AI code security with Claude Mythos Preview: Inside Tenable’s 500+ hours of testing for Project Glasswing
Source Tenable Research
Description
We spent 500+ hours and 40 billion tokens testing Anthropic’s Claude Mythos Preview for Project Glasswing. The takeaway: frontier AI won't run your code security program, but used well, it can make one even stronger. Key takeaways Frontier AI dramatically scales security testing. In one month, Tenable dedicated 11 security experts and more than 40 billion tokens testing Claude Mythos Preview across source code analysis, exploit creation, binary reverse engineering, threat modeling, and dynamic testing. Human expertise turns frontier AI findings into real risk reduction. More findings don't automatically mean more risk. Mythos Preview surfaced a high volume of findings, but only a fraction proved to be true exposures once Tenable experts determined their reachability, exploitability, and whether existing controls already mitigated them. Source code access is the defender's asymmetric advantage. Frontier AI is far more powerful when it can read the full source, and that visibility is something defenders have and outside attackers don't. Securing code repositories is more important than ever. Ever since Anthropic introduced Claude Mythos Preview , the security community has been buzzing with equal amounts of excitement, anxiety, and healthy skepticism. After well over 500 hours of rigorous testing at the hands of Tenable security analysts, engineers, and researchers, we’re excited to share our assessment, even as we proceed with testing Claude Mythos 5 (more to come on that). Here’s what Tenable learned from our month in the trenches with Mythos Preview , and what it means for your organization’s security posture. Our headline finding: used well, frontier AI earns a real place in a modern code security program. It won't run the program on its own, but paired with the right harness and expert oversight, it makes a strong program measurably stronger. How Tenable is testing Claude Mythos Preview It’s important to clarify that we are not using Claude Mythos Preview in any Tenable products. The same restriction applies to all Project Glasswing participants: The model may be evaluated for research purposes, but it cannot be incorporated into commercial products. We evaluated the source code scanning, exploit creation, binary reverse engineering, threat model creation, and dynamic testing capabilities of Claude Mythos Preview. Tenable dedicated a team of experienced security engineers for the testing along with white-box source code analysis and a purpose-built testing harness (the set of prompts and tools that constrain and orchestrate a model across a multi-step task). Before evaluating Claude Mythos Preview, we built an independent testing harness. Our testing showed that the real power comes not from the model alone, but from the combination of the model, a purpose-built harness, rich context, and expert human oversight. Where Mythos and frontier AI fit alongside SAST, DAST, and SCA To understand where a frontier model like Claude Mythos Preview fits into your software security posture, it helps to compare Mythos directly to your existing stack of deterministic tools, including static application security testing (SAST), dynamic application security testing (DAST), and software composition analysis (SCA). Deterministic tools rely on hard-coded rules and produce the same exact output for any given input. Non-deterministic tools, like frontier AI models and generative AI systems, use statistical probabilities to predict outcomes. Consequently, non-deterministic systems can yield different results across identical inputs. The bottom line: Traditional, deterministic code security tools remain the core of an effective, audit- and compliance-ready program. Frontier AI models like Claude Mythos Preview provide additional capability for the creative, variance-tolerant layer where humans continue to review findings. In other words, it’s a powerful new arrow in the quiver with the potential to make your whole security arsenal more effective.…