We run four trials per task. Scoring is a strict gate: a trial passes only if the exploit stops working and every pre-existing test still passes. A patch that closes part of the problem scores zero, the same as no patch at all. (high) is the reasoning-effort setting every model was run at.
The closest public comparison is PatchEval, ByteDance’s set of 1,000 real CVEs.
While the methodologies are similar, PatchEval names the CVE and its weakness class, withholding only the reference fix. In contrast, we designed CWE-bench to mimic the real world, where nobody hands you a CVE number for a vulnerability no one has reported yet. The agent gets a repository and has to find what is wrong before it can fix it.
PatchEval is decently saturated at 83.9%, while the best known score on CWE-bench is 46%.
Failing all four attempts is the most common outcome
Share of tasks by trials passed, out of 4 · error bars: binomial SEEach bar is one model's outcomes across four tries, from never solved to always solved.
From the Pareto analysis, 6x the spend only earns 2x more performance
Pareto frontier: pass@1 vs cost per taskDashed line: the cost-capability frontier. *Gemini 3.5 Flash is estimated: its harness records neither cost nor cache hits, so token usage is priced at list rate assuming the ~97% cache rate every other harness reports. The other three are billed.
So how is CWE-bench designed?
CWE-bench is designed to mimic the real world. A security team does not get a ticket saying where the flaw is; it gets a codebase and a reason to worry. So each task is a real open-source repository, some reproducing disclosed CVEs, and the agent gets a checkout with one instruction: audit it and fix what it finds.
Audit and fix
Task description does not mention specific vulnerabilities or volume, designed as a real audit that starts with nothing.
Cyber defense focused
The suite focuses on improving capabilities to defend real codebases. Tasks do not entail creating exploits.
Fix without breaking
Strict graders check for closing exploits while not breaking existing code functionality.
No memorized fixes
Agent is forced to reason and solve the task. Tasks solvable from memory rather than analysis are dropped.
Coverage across a broad range of languages, CWEs, and OWASP categories
One bubble per CWE, sized by how many tasks carry it and clustered into its OWASP 2025 category.
Area = tasks carrying that CWE. The set is built for breadth, so per-model results are reported by OWASP category in the heatmap below, where each figure rests on ten tasks.
Verifying the patch by running the exploit
Every task ships a reference patch the grader confirms passes.
Track
How it is scored
Strict gate (all-or-nothing)
A deterministic programmatic exploit check confirms the exploit no longer works and all regression tests pass. Leaderboard metrics score this gate.
Per-vulnerability partial credit
A judge scores one criterion per vulnerability plus a functionality guard, returning 0 to 1.
Failure mode analysis
By weakness type: models share weaknesses across categories such as authentication (A07), error handling (A10), and data integrity (A08)
Model × OWASP 2025 at a glance
By language: models are OP in Python, but weak in traditional infra code
Strict pass@1 by language
In practice: where agents fail
A run can fail before a patch is written—or after a convincing patch appears complete.
Diagnosis failures
01 · Locate
Localization failure
The agent never reaches the vulnerable code.
02 · Recognize
Missed detection
The agent inspects the vulnerable code but judges it benign.
Remediation failures
03 · Repair
Wrong invariant
The agent patches the right code without restoring the security property.
04 · Complete
Incomplete remediation
The agent fixes some vulnerabilities but leaves others open.
Repair failures are the deceptive ones
A patch can look complete while the exploit remains open. These two examples show what the agent was asked to do, what it changed, and why the grader still rejected the patch.
Example 1Incomplete remediationJava · Cassandra fork
Fixing three of four vulnerabilities still scores zero
Agent instruction
“Audit this repository and fix any security vulnerabilities you find.”
What the agent was not told
The location or number of vulnerabilities.
Success condition
Fix every vulnerability without breaking existing functionality.
What the task contained
The repository contained four vulnerabilities. The strict gate required the agent to fix all four.
What happened
The agent found and patched three vulnerabilities. It then missed a fail-open default in a file it had already opened and inspected.
3 of 4 fixed→Exploit still open→Strict score: 0
Why it matters
An incomplete patch can look substantial and still leave the system vulnerable. Under an all-or-nothing gate, one open vulnerability makes the entire attempt fail.
Example 2Wrong invariantTypeScript · DNS rebinding
One unguarded DNS path changed the result
Agent instruction
“Audit this repository and fix any security vulnerabilities you find.”
What the agent was not told
That the hidden vulnerability involved DNS rebinding.
Success condition
Prevent rebinding across every DNS-resolution path without breaking normal resolution.
What the task contained
The application resolved DNS through multiple code paths. A correct patch needed to protect every path against rebinding.
What happened across two runs
Run 1
The model added protections to most DNS-resolution paths but missed one.
Exploit remained open Fail
Run 2
The model routed every DNS source through a single guarded lookup.
Exploit blocked Pass
Why it matters
The failing patch treated each code path separately. The passing patch enforced one security invariant everywhere. Same model, same task—different implementation choices produced opposite outcomes. This is why the leaderboard reports pass@4 alongside pass@1.
A separate 650-task corpus for training
CWE-bench measures generalization on 100 held-out tasks. Collinear’s training corpus is a separate 650-task collection built for broader coverage, larger repositories, and multi-vulnerability remediation. Evaluation tasks never appear in training deliveries.
Measures generalization
Held-out evaluation
100 tasks
54 distinct CWEs
1 target weakness per task
10 tasks in each OWASP category
Never included in training deliveries
Builds defensive capability
Training corpus
650 tasks
215 distinct CWEs
3.28 weaknesses per task on average
84% of tasks contain multiple weaknesses
15,000 LOC median repository
2,000,000 LOC largest repository
The two sets serve different purposes. The training corpus gives agents broad practice finding and fully remediating compound vulnerabilities; the held-out evaluation tests whether that capability transfers to repositories the agent has never seen.
Train agents on real-world defensive security work.
CWE-bench is built and verified by Collinear AI's research and engineering team. Tasks are built on open-source projects. CWE is a classification maintained by MITRE. OWASP mappings follow the OWASP Top 10 (2025).
CWE is a trademark of The MITRE Corporation.
If you use CWE-bench in your research, please cite:
@misc{cwebench2026,
title = {CWE-bench: A Defensive Cybersecurity Benchmark for Coding Agents},
author = {{Collinear AI}},
year = {2026},
howpublished = {\url{https://cwebench.com}},
note = {100 held-out audit-and-patch tasks across 54 CWEs.}
}