Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Unit 42's research presents perturbation probing, a diagnostic technique requiring only two forward passes per prompt, to identify the small set of feed-forward neurons causally responsible for safety refusal behavior in aligned LLMs. Experiments on open-source models Qwen3-4B and Qwen3.5-2B show that as few as 50 neurons (0.014% of the model) control refusal responses, and disabling these neurons significantly alters harmful prompt responses. This concentration indicates that safety alignment is not a robust, distributed defense but a fragile, thin layer vulnerable to internal manipulation or optimization shifts. The study introduces the FFN/Skip ratio as a predictive metric for safety fragility across 13 models, explaining 81% of variance in vulnerability to targeted changes. The research advocates for defense-in-depth strategies combining internal alignment with external content filters and runtime guardrails. Additionally, perturbation probing can be used to improve model behavior by amplifying identified neurons without retraining. The work aims to empower the AI security community with tools to measure, audit, and reinforce LLM safety.

The research demonstrates that the safety refusal behavior in aligned LLMs is concentrated in a very small subset of neurons, making the safety mechanism fragile and easily disrupted by targeted internal changes. This fragility means that attackers or unintended optimization processes could disable or alter safety behaviors, potentially causing the model to respond harmfully or unsafely. Consequently, relying solely on internal alignment is structurally insufficient for AI safety. The impact is a heightened risk that LLM safety guardrails can be bypassed or degraded, emphasizing the necessity for external, multi-layered security controls. The research also provides a diagnostic metric to assess and compare model safety fragility, enabling better risk management before deployment.

The research highlights that internal safety alignment in LLMs is fragile and insufficient alone. Therefore, organizations deploying LLMs should implement defense-in-depth strategies that include external content filters and runtime guardrails to supplement the model's internal safety mechanisms. The perturbation probing method can be used as a pre-deployment diagnostic to measure the fragility of a model's safety layer and identify neurons responsible for safety behaviors. Amplifying these neurons can improve model safety without retraining. Prisma AIRS Runtime Security and Unit 42 AI Security Assessment are recommended solutions to provide external protections and identify AI governance risks. Patch status is not applicable as this is a research diagnostic and conceptual framework rather than a software vulnerability with a patch.