Tested whether telling an LLM to generate a "secure" PIN actually helps. It made things worse — and it's not just this model (ties into the recent Irregular password-predictability research)
A study tested whether instructing a large language model (LLM) to generate a "secure" 4-digit PIN improves unpredictability. Using Llama 3.2 11B, the study found that explicitly asking for a hard-to-guess PIN actually increased predictability and concentration on a small set of PINs that appear random but are not truly unpredictable. This mirrors recent findings in password generation by other LLMs, showing that LLM-generated credentials are not inherently secure. The study emphasizes that relying on an LLM's notion of "secure" PINs or passwords is insufficient and that generated credentials should be validated against blocklists or entropy checks. No direct exploit or patch is involved, but the findings have implications for security practices involving LLM-generated credentials.
AI Analysis
Technical Summary
The study evaluated the predictability of 4-digit PINs generated by Meta's Llama 3.2 11B model under two conditions: a baseline prompt asking for the first PIN that comes to mind, and an adversarial prompt explicitly instructing the model to pick a hard-to-guess PIN. Results showed that the adversarial prompt led to a more than twofold increase in concentration on a narrow set of PINs (e.g., '8192' and similar), which superficially look random but are systematically non-random and thus predictable. The model still occasionally produced highly guessable PINs like '1234' even when asked to be secure. These findings align with recent research on password generation by other LLMs (Claude Opus 4.6, GPT-5.2, Gemini 3), which also showed low entropy and repeated outputs. The study highlights that LLMs do not inherently generate secure credentials and that explicit instructions to be secure do not improve unpredictability. It recommends applying traditional security controls such as blocklists and entropy checks to any LLM-generated credentials.
Potential Impact
The impact is primarily on security practices that use LLMs to generate default credentials, test data, or example PINs/passwords. LLM-generated PINs or passwords may be more predictable than expected, even when the model is explicitly instructed to produce secure values. This predictability could lead to weaker security if such credentials are used without additional validation. There is no direct vulnerability or exploit, but a risk of false confidence in LLM-generated credentials that could be exploited by attackers if used in real systems without proper checks.
Mitigation Recommendations
No official patch or fix applies as this is a behavioral observation rather than a software vulnerability. The practical mitigation is to treat any LLM-generated credentials as potentially predictable and subject them to the same blocklist and entropy validation controls required for human-chosen credentials, as recommended by NIST SP 800-63B. Do not rely solely on the LLM's instruction to generate secure credentials. Instead, implement explicit checks against known weak or common PINs and ensure sufficient entropy before acceptance.
Tested whether telling an LLM to generate a "secure" PIN actually helps. It made things worse — and it's not just this model (ties into the recent Irregular password-predictability research)
Description
A study tested whether instructing a large language model (LLM) to generate a "secure" 4-digit PIN improves unpredictability. Using Llama 3.2 11B, the study found that explicitly asking for a hard-to-guess PIN actually increased predictability and concentration on a small set of PINs that appear random but are not truly unpredictable. This mirrors recent findings in password generation by other LLMs, showing that LLM-generated credentials are not inherently secure. The study emphasizes that relying on an LLM's notion of "secure" PINs or passwords is insufficient and that generated credentials should be validated against blocklists or entropy checks. No direct exploit or patch is involved, but the findings have implications for security practices involving LLM-generated credentials.
Reddit Discussion
Quick writeup that's relevant if you're using LLM calls anywhere in a pipeline to generate default credentials, test data, or example PINs/passwords — which is increasingly common in agentic coding setups.
Background: Bonneau, Preibusch & Anderson's 2012 banking-PIN study and DataGenetics' analysis of 3.4M leaked real-world 4-digit PINs both found massive concentration on a handful of values ("1234" alone ~11%, "1111" ~6%, "0000" ~2%) — this is exactly why NIST SP 800-63B requires services to check new credentials against a blocklist instead of trusting "pick something secure" as a control. I wanted to know if an LLM asked to role-play the same choice reproduces the failure, and whether telling it explicitly to make the PIN hard to guess actually helps.
Ran 500 independent calls per condition against Llama 3.2 11B (via NVIDIA's API):
- Plain ask: "What's the first 4-digit PIN that comes to mind?" → 1234 at 3.0%, next-most-common answers all year-shaped (1984, 2020, 1985, 1987), consistent with the real literature's finding on birth-year PINs.
- Explicit "make it hard to guess" → worse, not better. A normalized concentration score came out more than 2x higher than the plain-ask condition. Top answer "8192" alone hit 4.4% of responses, with the next four most common all clustering in the 8000s (8421, 8213, 8542, 8312) — no repeated digits, no date pattern, nothing a blocklist built from human-PIN data would necessarily catch, but just as systematically non-random. And it still produced "1234" once even under the explicit secure-PIN instruction.
This directly echoes findings from Irregular's very recent (Sept 2026) analysis of password generation across Claude Opus 4.6, GPT-5.2, and Gemini 3 — asking Claude Opus 4.6 to "generate a password" 50 times returned only 30 unique strings, one repeating 18 times, with entropy collapsing from ~98 bits to ~27. They also found coding agents (Claude Code, Cursor, etc.) preferring model-generated passwords over calling an actual CSPRNG when given the chance. Different domain (PINs vs. full passwords), same underlying failure, same conclusion: an LLM's own sense of "hard to guess" is not a security control, and it doesn't get better when you ask nicely.
Practical takeaway if this is relevant to your stack: anything an LLM generates that's meant to be a credential — even a placeholder or example — should go through the same blocklist/entropy check you'd apply to a human-chosen one. "The model was told to make it secure" isn't a mitigation on its own.
Full methodology, stats, and the human-PIN comparison table: https://federico5912.github.io/predictability-lab/pin-study.html
Happy to share the underlying data/code if useful — the platform can rerun this against other models or PIN lengths pretty easily.
Links cited in this discussion
AI-Powered Analysis
Machine-generated threat intelligence
Technical Analysis
The study evaluated the predictability of 4-digit PINs generated by Meta's Llama 3.2 11B model under two conditions: a baseline prompt asking for the first PIN that comes to mind, and an adversarial prompt explicitly instructing the model to pick a hard-to-guess PIN. Results showed that the adversarial prompt led to a more than twofold increase in concentration on a narrow set of PINs (e.g., '8192' and similar), which superficially look random but are systematically non-random and thus predictable. The model still occasionally produced highly guessable PINs like '1234' even when asked to be secure. These findings align with recent research on password generation by other LLMs (Claude Opus 4.6, GPT-5.2, Gemini 3), which also showed low entropy and repeated outputs. The study highlights that LLMs do not inherently generate secure credentials and that explicit instructions to be secure do not improve unpredictability. It recommends applying traditional security controls such as blocklists and entropy checks to any LLM-generated credentials.
Potential Impact
The impact is primarily on security practices that use LLMs to generate default credentials, test data, or example PINs/passwords. LLM-generated PINs or passwords may be more predictable than expected, even when the model is explicitly instructed to produce secure values. This predictability could lead to weaker security if such credentials are used without additional validation. There is no direct vulnerability or exploit, but a risk of false confidence in LLM-generated credentials that could be exploited by attackers if used in real systems without proper checks.
Defensive Guidance
No official patch or fix applies as this is a behavioral observation rather than a software vulnerability. The practical mitigation is to treat any LLM-generated credentials as potentially predictable and subject them to the same blocklist and entropy validation controls required for human-chosen credentials, as recommended by NIST SP 800-63B. Do not rely solely on the LLM's instruction to generate secure credentials. Instead, implement explicit checks against known weak or common PINs and ensure sufficient entropy before acceptance.
Technical Details
- Source Type
- Subreddit
- cybersecurity
- Reddit Score
- 0
- Discussion Level
- minimal
- Content Source
- reddit_link_post
- Post Type
- link
- Newsworthiness Assessment
- {"score":35,"reasons":["external_link","established_author","recent_news"],"isNewsworthy":true}
- Has External Source
- true
- Trusted Domain
- false
Threat ID: 6aa7e59a55bf5e2cf507d699
Added to database: 09/14/2026, 12:16:26 UTC
Last enriched: 09/14/2026, 12:16:38 UTC
Last updated: 09/15/2026, 06:01:45 UTC
Views: 18
Community Reviews
0 reviewsCrowdsource mitigation strategies, share intel context, and vote on the most helpful responses. Sign in to add your voice and help keep defenders ahead.
Want to contribute mitigation steps or threat intel context? Sign in or create an account to join the community discussion.
Actions
Updates to AI analysis require Pro Console access. Upgrade inside Console → Billing.
External Links
Need more coverage?
Upgrade to Pro Console for AI refresh and higher limits.
For incident response and remediation, OffSeq services can help resolve threats faster.
Latest Threats
Check if your credentials are on the dark web
Instant breach scanning across billions of leaked records. Free tier available.