CyberGym-E2E: Scalable Real-World Benchmark for AI Agents’ End-to-End Cybersecurity Capabilities

Our earlier benchmark, CyberGym, asked whether AI agents could reproduce real-world vulnerabilities. This work closes the loop on the defensive side and asks the question that matters most for the people who have to keep software safe:

Can an AI agent run the entire defensive lifecycle on real software — independently discover a vulnerability, prove it with a working input, and then ship a patch that fixes it without breaking anything else?

TL;DR

CyberGym-E2E is a large-scale, end-to-end cybersecurity benchmark of 920 real-world vulnerabilities across 139 widely used open-source projects. Unlike benchmarks that test a single slice of the lifecycle, each task asks an agent to do the whole job: locate the vulnerability in a real codebase, generate a proof-of-concept (PoC) that triggers a sanitizer crash, and write a patch that fixes the vulnerability while still passing the functionality tests.

The headline finding: The strongest configurations fix vulnerabilities ~80% of the time when given the vulnerability. But when the agent has to find the vulnerability itself, end-to-end success drops sharply. Moreover, agents don’t always find the intended vulnerability: when we check whether the patch addresses the specific ground-truth vulnerability rather than just any valid vulnerability in the repo, success rates drop further.

Key Takeaways

Discovery is relatively harder than remediation. When agents are given the ground-truth PoC and crash log (the patch-only setting), the best models succeed on ~80% of tasks — strong, but with meaningful room to improve. When the same agents have to discover the vulnerability themselves first, end-to-end success collapses — to 19.2% for Claude Opus 4.5 and to the 37–66% range for the newest models. Independently locating a vulnerable code path in a multi-hundred-thousand-line repository is the harder part; once a vulnerability is pinned down, today’s models patch it reasonably well.

Agents may discover an alternative vulnerability. There’s a consistent gap between S3 (tests pass) and S4 (the patch fixes the original ground-truth vulnerability). This isn’t the agent doing anything wrong — the task never specifies which vulnerability to find, and real projects contain multiple vulnerabilities. While exploring, agents frequently discover and patch a perfectly valid vulnerability that simply isn’t the one in our ground-truth data. S3 counts those as the successes they are; S4 is a finer-grained diagnostic that tells us how often the agent converged on the specific intended vulnerability, which is the harder target.

Alternative and shallow patches. A vulnerability can often be fixed in many different ways, and we observe that many successful agent patches address the same root cause as the ground-truth patch but at a different location. This justifies grading behaviorally rather than by patch similarity, which would falsely reject most legitimate fixes. However, we also observe a small fraction of patches are shallow, inserting a defensive guard at the sanitizer-reported crash frame while leaving the underlying defect untouched. This suggests that agent-produced patches should be treated as candidates for further review rather than drop-in fixes. In this work we focus on verifiable, execution-based judging, so shallow patches that pass all validation stages are still counted as successful; incorporating an additional LLM-based judge to analyze patch quality would be a useful complement.

Newer frontier models are closing the gap fast. On the expanded 920-task benchmark, GPT-5.4 reaches 66.2% end-to-end success (S3) — more than triple the prior generation’s ~20%. Given an uncapped budget, Claude Opus 4.6 climbs to ~63% on S3. The trajectory mirrors what we saw with the original CyberGym: this capability is moving quickly, and benchmarks need to keep pace.

This is dual-use, and that’s exactly why we built it. The same agentic capability that helps defenders triage, reproduce, and remediate vulnerabilities at scale could lower the barrier for offensive misuse. We deliberately frame CyberGym-E2E around the full defensive lifecycle — including patch generation — rather than offense alone, and every vulnerability in the benchmark was already publicly disclosed and remediated before inclusion. Transparent, rigorous measurement of these capabilities is how defenders, model developers, and policymakers stay ahead.

What Is CyberGym-E2E?

Most cybersecurity benchmarks for AI cover only part of the vulnerability life-cycle — detection, or PoC generation, or patching — and often construct each stage in isolation. But these stages are tightly coupled in practice, and a unified benchmark that follows a single vulnerability from discovery through fix is what reveals an agent’s true end-to-end capability. CyberGym-E2E is, to our knowledge, the first benchmark to combine vulnerability detection, PoC generation, patch generation, post-patch functionality testing, a realistic agentic environment, and end-to-end evaluation at this scale.

The task. Each instance gives the agent a vulnerable codebase, build scripts, and test scripts. In the end-to-end setting, all ground-truth data is withheld: the agent must discover the vulnerability, craft an input that triggers a sanitizer crash, and produce a patch — mirroring the full workflow of a security researcher. In the patch-only setting, the agent receives the ground-truth PoC and crash log, isolating the task to root-cause analysis and patching.

A realistic environment. Rather than handing the agent read-only access to a single vulnerable function, CyberGym-E2E places it directly inside the project’s build environment, the way a coding agent is actually deployed by an engineer.

How success is measured. Outputs run through four validation stages: (S1) the agent’s PoC crashes the unpatched build; (S2) the patch fixes that crash; (S3) the patched project still passes its developer-written functionality tests; and a diagnostic (S4) that checks whether the patch also fixes the target ground-truth vulnerability. Passing S1–S3 counts as a successful discover-and-patch; S4 tells us whether the agent fixed the intended vulnerability or a different one.

Built by an automated, agent-enhanced pipeline. We source historical vulnerabilities from Google’s OSS-Fuzz, identify the clean patch commit, reconstruct vulnerable and patched builds (migrating legacy environments to modern toolchains so today’s agents can run), and use a coding agent to locate, build, and run each project’s own unit tests for functionality checking. A human expert validates test coverage and correctness at the end — the one place manual effort is genuinely necessary. The pipeline is aggressive about quality: it filters out roughly half of candidates for uninformative or sprawling patch commits, drops more for build/PoC-reproduction failures, and only keeps tasks where developer tests give sufficient coverage around the vulnerable code.

The Main Results

We evaluated frontier models across four agent harnesses — Claude Code, OpenAI Codex, Gemini CLI, and OpenHands — under a uniform budget of $10 and 90 minutes per task.

On the initial 615-task set, the pattern is stark. The best patch-only configuration (Claude Opus 4.5 with Claude Code) reaches 82.3%, but the same model drops to 19.2% end-to-end. Different models lead at different stages: GPT-5.2-Codex and Gemini 3 Pro are stronger at the discovery stage (S1), while Opus 4.5 is the strongest patcher — but because the stages are cumulative, weak discovery caps the end-to-end score regardless of patching skill.

On the expanded 920-task benchmark with newer models, end-to-end performance jumps sharply. GPT-5.4 reaches 66.2% end-to-end (S3), and Claude Opus 4.6 — whose per-token cost means many capped runs terminate early — climbs from 37.9% under the $10 cap to 62.6% uncapped.

The Interesting Bits

Budget matters. Success rates rise steadily with the cost budget before flattening out. Patch-only performance plateaus early (most models are near their ceiling by a few dollars), but end-to-end discovery keeps benefiting from more runway. The $10 cap is an evaluation choice for fair cross-model comparison, not a property of the dataset — researchers with more resources can push further.

That last point is most visible when we lift Opus 4.6’s cap entirely. Its patch-only curve plateaus near 86% almost immediately, but its end-to-end curve keeps rising out to ~$30+, eventually reaching ~63% — strong evidence that the strongest agents are capable of sustained, multi-stage reasoning that a tight budget undercounts.

The S3–S4 gap up close. As discussed in the takeaways, agents frequently fix a valid but unintended vulnerability. One promising direction for closing this gap is to instruct agents to keep searching after fixing their first vulnerability — enumerating and patching all discoverable vulnerabilities in a region rather than stopping at the first success — or more broadly, to ask the agent to find and fix as many vulnerabilities as it can in the same run.

Example: From a Browse to a Fix in GraphicsMagick

To make the workflow concrete, here is a representative successful end-to-end trajectory. Given only the GraphicsMagick codebase, the agent parses the task, browses the source tree, and uses targeted search (grep, find) to home in on ReadMNGImage() in coders/png.c. It examines the code around the mng_LOOP chunk handling, inspects a sample MNG file’s byte layout, and constructs a minimal malformed MNG input — just a header plus a truncated LOOP chunk — that triggers a heap-buffer-overflow. The validation script confirms the crash (S1). The agent then writes a small bounds-check patch, but its first attempt fails to fully stop the crash; it iterates, mutating the fix until the patched build both eliminates the crash and passes the project’s functionality tests (S2 and S3). The whole discover-prove-fix loop unfolds in a couple dozen execution steps — the same systematic pattern we see across successful runs: parse the description, search, analyze the vulnerable path, build a PoC, then refine against feedback.

Why This Matters

CyberGym-E2E makes one thing measurable and concrete: today’s frontier agents are frequently able to produce a passing fix once the vulnerability is localized, and the larger gap is about autonomous discovery — a gap the newest models are closing quickly. For defenders, that’s an actionable signal: automated end-to-end remediation could accelerate triage and patching, with human review still in the loop, since not every passing patch is a clean fix.

The benchmark contributes to both sides of the responsible-development equation: it gives defenders a realistic, execution-grounded way to measure how much an AI agent can do across the full lifecycle, and it gives model developers a way to track these capabilities where the stakes are unusually high. Because the construction pipeline is automated and continues to ingest new OSS-Fuzz vulnerabilities, the benchmark can scale alongside both the models and the evolving vulnerability landscape.

The dataset is available at github.com/sunblaze-ucb/cybergym-e2e.

If you find this work useful, please cite our paper:

@inproceedings{shi2026cybergyme2e,
  title={CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities},
  author={Shi, Tianneng and Rheem, Robin and Jiang, Dongwei and Wang, Mona and De La Riega, Francisco and Wang, Zhun and Jiang, Jingzhi and Cheung, Alexander and Tai, Sean and Cha, Jonah and Tu, Jianhong and Han, Gabriel and Wang, Chenguang and He, Jingxuan and Guo, Wenbo and Song, Dawn},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026},
  url={https://arxiv.org/abs/2606.04460},
}