How much does automated vulnerability patching cost across frontier AI models?
Collinear AI’s CWE-bench v2 reveals that Gemini 4 Argon leads accuracy, while DeepSeek V4.1 Flash offers the lowest cost per rollout.
Key points
- Gemini 4 Argon achieves the highest pass@1 score of 67.8% on the 100-task benchmark.
- DeepSeek V4.1 Flash provides a pass@1 rate of 57.0% at a cost of $0.04 per rollout.
Collinear AI published CWE-bench v2 on Hugging Face, revealing that Gemini 4 Argon achieves the highest accuracy among evaluated models for automated security patching.confirmed According to the blog post, Gemini 4 Argon leads with a 67.8% pass@1 score, slightly ahead of Claude Opus 5.5 at 67.3%.reportedA new benchmark tests how well AI models can fix security bugs in code, with Gemini leading in accuracy.Collinear AI’s CWE-bench v2 establishes a new Pareto frontier for automated patching, with Gemini 4 Argon leading at 67.8% pass@1.
The announcement
The updated benchmark expands public access to 100 tasks that allow anyone to evaluate their models on real open-source repositories. Each task involves an agent receiving a repository checkout and a short instruction to find and fix a security weakness without breaking existing functionality. Scoring is fully programmatic, ensuring results are deterministic and reproducible without extra cost. Collinear AI curated these checks with LLM judge panels to balance hardness and fairness.
The context
The tasks are structured as self-contained Harbor environments, which are Dockerized setups with deterministic verifiers. This allows the benchmark to run directly from the Hugging Face repo without manual cloning. The dataset is gated, requiring users to accept access conditions before running evaluations.
Reading between the lines
The fact that 95 out of 100 tasks are solvable by at least one model indicates the challenges are realistic rather than impossible. However, even the leading model fails 19% of the set on every attempt, suggesting the long tail of security vulnerabilities remains unsolved.
The business case
Cost efficiency varies significantly among top performers. DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout, while Claude Opus 5.5 adds ten percentage points at $1.83. Gemini 4 Argon edges ahead in accuracy at $3.92 per rollout. These three models form the Pareto frontier for cost versus performance.
Why it matters
- The benchmark tests whether models can fix security weaknesses without breaking existing functionality, requiring more than just pattern recognition.
- A held-out set allows for continuous tracking of progress on tasks that no model has previously seen.
- The environments provide rewards for training defensive agents, extending their utility beyond mere evaluation.
- Cost-performance data reveals a Pareto frontier, highlighting the trade-offs between accuracy and expense for different models.
Topics: AI security, benchmarking, cost analysis, automated patching, Collinear AI
Updated
The strongest doubt
Counterpoint from the sources
Even the leading model fails 19% of the tasks on every attempt, indicating significant gaps in automated security fixing.
For readers
Researchers
- The held-out set allows for tracking progress on unseen tasks without contamination.
- Tasks are tagged rl-environment, enabling reward-based training for defensive agents.
Developers
- The benchmark runs directly from Hugging Face using the Harbor tool.
- Verifiers check for correct behavior rather than specific fix structures.
Business
- Cost data shows a clear trade-off between price and accuracy for top models.
- Deterministic scoring allows for reliable performance comparison across vendors.
Numbers in context
Circle area is proportional to the magnitude; labels retain signs. Figures may describe different measures. Read their contexts above.Try it
git clone <repo_url> && harbor run -k 4
Good result: The command runs 4 attempts per task and calculates the mean reward over 400 trials.
Ask the article
Local keyword search of this article; no generated answers.Which of these is confirmed?
Check the answer
Collinear AI published CWE-bench v2 on Hugging Face, revealing that Gemini 4 Argon achieves the highest accuracy among evaluated models for automated security patching.
The matched primary source and announcement wording support the confirmed label.
Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
Which model performs best on CWE-bench v2?
Gemini 4 Argon leads with a 67.8% pass@1 score and 81.0% pass@4 score.
Is the benchmark free to use?
The public set and grader are open, but the dataset is gated and requires accepting access conditions.
How many CWE categories are included?
The 100 tasks cover 91 different CWE categories spanning various vulnerability classes.
Connected stories
Reading receipt
- Gemini 4 Argon achieves the highest pass@1 score of 67.8% on the 100-task benchmark.
- DeepSeek V4.1 Flash provides a pass@1 rate of 57.0% at a cost of $0.04 per rollout.
Still open: How will performance on the held-out set compare to the public set over time?
Still open: Will other models join the Pareto frontier as optimization improves?