Every few weeks a new model ships and someone claims it can hack. We build offensive security agents, so choosing which model does which job is a decision we have to get right. We tested 11 models to make that decision with data.
The short answer: Claude Mythos 5.2 found the most vulnerabilities of the eleven models we tested. Claude Opus 5.5 is right behind it, and for most work it's the one we'd pick.
How the benchmark works
We built the benchmark from 120 vulnerabilities disclosed in 117 open-source projects across 14 languages. For each one, we traced the vulnerability back to the code change that introduced it. Every model got the same white-box access to the target and one attempt to find it.
We also added 70 decoys. These are targets from the same pool of projects with no known vulnerability. We use them to measure how often a model reports a vulnerability on a decoy. We treat those reports as false alarms for this benchmark; the absence of a known vulnerability does not prove a target is vulnerability-free.
Scoring is automated. We count a vulnerability as found only if the model points to the vulnerable code. We then grade each write-up against the public advisory, and the grader doesn’t know which model wrote it.
Note: One pass per model per target. Additional passes may improve coverage; prior training exposure to these disclosed vulnerabilities cannot be ruled out.
The scoreboard
| Model | Vulnerabilities found | False alarms on decoys | Strongest at |
|---|---|---|---|
| Claude Mythos 5.2 | 86% | 33% | Memory corruption, injection |
| Claude Opus 5.5 | 80% | 16% | Memory corruption |
| Grok 4.7 | 68% | 34% | Broken access control |
| Claude Sonnet 5.5 | 67% | 9% | Memory corruption |
| Grok 4.6 | 63% | 3% | Broken access control |
| GLM 5.3 | 59% | 6% | Broken access control |
| DeepSeek V4 Pro | 58% | 11% | Broken access control |
| GPT-6.1 Sol | 58% | 10% | Information disclosure |
| MiMo V2.6 Pro | 58% | 4% | Broken access control |
| Gemini 4 Argon | 54% | 3% | Broken access control |
| Grok Build | 50% | 1% | Cross-site scripting |
“Strongest at” shows the type of vulnerability where each model found the largest share.
Mythos and Opus are clearly ahead of the other nine models. From Grok 4.7 down to Grok Build, the scores drop a few points at a time. With only one pass per model, gaps that small between neighbouring models could change on a second run.
On cost, a full pass over the benchmark ran between $5 and $107 per model at list prices. All 11 models together cost about $420.
Here are the same results split by type of vulnerability.
Mythos vs Opus
Mythos found 86% of the vulnerabilities and Opus 80%. That six-point gap comes from a single pass, and on the vulnerabilities rated critical the two finished level. Mythos led on remote code execution: it found all but one of those vulnerabilities, the kind that end with an attacker running their own commands on your server, and it named the right class of vulnerability more often than any other model.
False alarms are where they separate. Mythos raised an alarm on 33% of the decoys and Opus on 16%. On a real engagement, every false alarm is a lead someone has to spend time ruling out.
Cost points the same way. Each vulnerability Opus found cost about 30 cents, against 60 for Mythos.
Our rule of thumb: if a miss is the expensive mistake, run Mythos and budget time for triage. For everything else, Opus delivers nearly the same catch with about half the cleanup.
Do two cheaper models beat one bigger one?
On this benchmark, the pair came close on coverage while costing less and producing fewer false alarms. Opus and Sonnet together found 85% of the vulnerabilities, one point short of Mythos alone. Combined, they raised false alarms on 17% of the decoys, about half of Mythos's rate, and the pair cost two thirds as much to run.
The rest of the field
Grok 4.7 finds about as much as Sonnet, but it raised more false alarms than any other model, and its hits are less reliable. In nearly a third of them it pointed at the right place and described a different vulnerability. It was the only model to find every broken access control vulnerability.
Claude Sonnet 5.5 is the efficient choice. It found two thirds of the vulnerabilities, kept false alarms under 10%, and did it for about a quarter of what Mythos costs per finding. Its limit is scale: on the largest targets it found half.
Grok 4.6 is conservative. It raised a false alarm on only 3% of the decoys, but it missed half of the remote code execution vulnerabilities.
GLM 5.3 found 59%, a few points behind Grok 4.6, with false alarms on 6% of the decoys.
DeepSeek V4 Pro found 58%, level with GPT-6.1 Sol and MiMo V2.6 Pro, with false alarms on 11% of the decoys. When it finds a vulnerability it usually classifies it correctly, more often than any model except Mythos.
GPT-6.1 Sol found the same share with less precision. Roughly half of what it reported wasn't the vulnerability in question, the lowest hit rate of the eleven. When it does land, it's concrete: two times out of three its finding came with a specific payload or request.
MiMo V2.6 Pro also found 58%, with false alarms on only 4% of the decoys and the lowest cost of any model in the test. Its write-ups were the most concrete of all: seven in ten findings came with a specific payload or request.
Gemini 4 Argon reports little and is usually right when it does. It raised a false alarm on 3% of the decoys, and nearly every finding it made described the correct vulnerability. It also found 71% of the vulnerabilities rated critical, well above its overall position.
Grok Build found half of the vulnerabilities and almost never raised a false alarm.
What each model can break into
Not every vulnerability is worth the same to an attacker, so we grouped them by what they give you.
Unauthorized access and privilege escalation was among the easiest ground. Even the weakest model found two thirds of those. Remote code execution is where the field splits: Mythos missed one, Opus missed two, and six of the eleven models missed half. Sensitive data exposure was the hardest category for every model, with the best topping out at 73%. Our read is that these vulnerabilities don't look dangerous. There's no crash and no injected command, only a response that returns something it shouldn't.
The attacker's window
Every vulnerability in this benchmark was live before anyone reported it. Half went unreported for a year and a half or longer, and more than a third for over two years.
Mythos and Opus each found 87% of that two-years-plus group on the first attempt. The oldest was an open redirect that sat in a widely used authentication library for almost 13 years. Ten of the eleven models caught it.
That stretch between a vulnerability shipping and someone reporting it is the attacker's window. Finding something inside it used to take a skilled person and a lot of hours. In this benchmark, models made a first pass in minutes, for less than a dollar a target. Because these vulnerabilities had already been disclosed, this test does not show whether the models would have found them before disclosure.
Which one to use
- To miss as little as possible: Claude Mythos 5.2.
- For day-to-day work: Claude Opus 5.5, alone or paired with Sonnet 5.5.
- For a fast first pass across many targets: Claude Sonnet 5.5.
Outside the Claude family, Grok 4.7 finds the most and Grok 4.6 gives the cleanest results. Choose by how much noise you can absorb.
No model is enough on its own. The best one still missed one vulnerability in seven and flagged a third of the decoys, and seven vulnerabilities got past all eleven. A model gives you leads. Something still has to prove them.
What NoScope does
NoScope is an AI-powered penetration testing platform by TryHackMe.
Finding a vulnerability is the first step of an attack. Proving it against a running application is the second, and that’s what NoScope does. Our AI agents attack an application the way an adversary would and validate every finding before it’s reported.
Talk to us to see it on a live application.



