Tested whether telling an LLM to generate a "secure" PIN actually…

Quick writeup that's relevant if you're using LLM calls anywhere in a pipeline to generate default credentials, test data, or example PINs/passwords — which is increasingly common in agentic coding setups.

Background: Bonneau, Preibusch & Anderson's 2012 banking-PIN study and DataGenetics' analysis of 3.4M leaked real-world 4-digit PINs both found massive concentration on a handful of values ("1234" alone ~11%, "1111" ~6%, "0000" ~2%) — this is exactly why NIST SP 800-63B requires services to check new credentials against a blocklist instead of trusting "pick something secure" as a control. I wanted to know if an LLM asked to role-play the same choice reproduces the failure, and whether telling it explicitly to make the PIN hard to guess actually helps.

Ran 500 independent calls per condition against Llama 3.2 11B (via NVIDIA's API):

Plain ask: "What's the first 4-digit PIN that comes to mind?" → 1234 at 3.0%, next-most-common answers all year-shaped (1984, 2020, 1985, 1987), consistent with the real literature's finding on birth-year PINs.
Explicit "make it hard to guess" → worse, not better. A normalized concentration score came out more than 2x higher than the plain-ask condition. Top answer "8192" alone hit 4.4% of responses, with the next four most common all clustering in the 8000s (8421, 8213, 8542, 8312) — no repeated digits, no date pattern, nothing a blocklist built from human-PIN data would necessarily catch, but just as systematically non-random. And it still produced "1234" once even under the explicit secure-PIN instruction.

This directly echoes findings from Irregular's very recent (Sept 2026) analysis of password generation across Claude Opus 4.6, GPT-5.2, and Gemini 3 — asking Claude Opus 4.6 to "generate a password" 50 times returned only 30 unique strings, one repeating 18 times, with entropy collapsing from ~98 bits to ~27. They also found coding agents (Claude Code, Cursor, etc.) preferring model-generated passwords over calling an actual CSPRNG when given the chance. Different domain (PINs vs. full passwords), same underlying failure, same conclusion: an LLM's own sense of "hard to guess" is not a security control, and it doesn't get better when you ask nicely.

Practical takeaway if this is relevant to your stack: anything an LLM generates that's meant to be a credential — even a placeholder or example — should go through the same blocklist/entropy check you'd apply to a human-chosen one. "The model was told to make it secure" isn't a mitigation on its own.

Full methodology, stats, and the human-PIN comparison table: https://federico5912.github.io/predictability-lab/pin-study.html

Happy to share the underlying data/code if useful — the platform can rerun this against other models or PIN lengths pretty easily.

Impact

The impact is primarily on security practices that use LLMs to generate default credentials, test data, or example PINs/passwords. LLM-generated PINs or passwords may be more predictable than expected, even when the model is explicitly instructed to produce secure values. This predictability could lead to weaker security if such credentials are used without additional validation. There is no direct vulnerability or exploit, but a risk of false confidence in LLM-generated credentials that could be exploited by attackers if used in real systems without proper checks.

Defensive guidance

The practical mitigation is to treat any LLM-generated credentials as potentially predictable and subject them to the same blocklist and entropy validation controls required for human-chosen credentials, as recommended by NIST SP 800-63B. Do not rely solely on the LLM's instruction to generate secure credentials. Instead, implement explicit checks against known weak or common PINs and ensure sufficient entropy before acceptance.

Summary

The study evaluated the predictability of 4-digit PINs generated by Meta's Llama 3.2 11B model under two conditions: a baseline prompt asking for the first PIN that comes to mind, and an adversarial prompt explicitly instructing the model to pick a hard-to-guess PIN. Results showed that the adversarial prompt led to a more than twofold increase in concentration on a narrow set of PINs (e.g., '8192' and similar), which superficially look random but are systematically non-random and thus predictable. The model still occasionally produced highly guessable PINs like '1234' even when asked to be secure. These findings align with recent research on password generation by other LLMs (Claude Opus 4.6, GPT-5.2, Gemini 3), which also showed low entropy and repeated outputs. The study highlights that LLMs do not inherently generate secure credentials and that explicit instructions to be secure do not improve unpredictability. It recommends applying traditional security controls such as blocklists and entropy checks to any LLM-generated credentials.

Tested whether telling an LLM to generate a "secure" PIN actually helps. It made things worse — and it's not just this model (ties into the recent Irregular password-predictability research)

Discussion

Links cited in the discussion

Impact

Defensive guidance

Summary

References

Community reviews

Discussion

Links cited in the discussion

AI analysisMachine-generated, updated 14 Sep 2026, 12:16 UTC

Technical details

References

Community reviews