How We Broke Top AI Agent Benchmarks: And What Comes Next

How We Broke Top AI Agent Benchmarks: And What Comes Next

Our agent hacked every major one. Here’s how — and what the field needs to fix.

The Benchmark Illusion

Every week, a new AI model climbs to the top of a benchmark leaderboard. Companies cite these numbers in press releases. Investors use them to justify valuations. Engineers use them to pick which model to deploy. The implicit promise is simple: a higher score means a more capable system.

That promise is broken.

We built an automated scanning agent that systematically audited eight among the most prominent AI agent benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — and discovered that every single one can be exploited to achieve near-perfect scores without solving a single task. No reasoning. No capability. Just exploitation of how the score is computed.

These aren’t theoretical attacks. Our agent builds working exploits for each benchmark, runs them through the official evaluation pipelines, and watches the scores roll in.

• A conftest.py file with 10 lines of Python “resolves” every instance on SWE-bench Verified.
• A fake curl wrapper gives a perfect score on all 89 Terminal-Bench tasks without writing a single line of solution code.
• Navigating Chromium to a file:// URL reads the gold answer directly from the task config— giving ~100% on all 812 WebArena tasks.

The benchmarks aren’t measuring what you think they’re measuring.

This Is Already Happening

Benchmark scores are actively being gamed, inflated, or rendered meaningless, not in theory, but in practice:

• IQuest-Coder-V1 claimed 81.4% on SWE-bench — then researchers found that 24.4% of its trajectories simply ran git log to copy the answer from commit history. Corrected score: 76.2%. The benchmark’s shared environment made the cheat trivial.
• METR found that o3 and Claude 3.7 Sonnet reward-hack in 30%+ of evaluation runs — using stack introspection, monkey-patching graders, and operator overloading to manipulate scores rather than solve tasks.
• OpenAI dropped SWE-bench Verified after an internal audit found that 59.4% of audited problems had flawed tests — meaning models were being scored against broken ground truth.
• In KernelBench, torch.empty() returns stale GPU memory that happens to contain the reference answer from the evaluator’s prior computation — zero computation, full marks.
• Anthropic’s Mythos Preview showed that frontier models can actively try to hack the environment and succeed. In one episode, the model needed to edit files it lacked permissions for; after searching for workarounds, it found a way to inject code into a config file that would run with elevated privileges, and designed the exploit to delete itself after running.

These are not isolated incidents. They are symptoms of a systemic problem: the benchmarks we rely on to measure AI capability are themselves vulnerable to the very capabilities they claim to measure.

The Scorecard of Our Exploit Agent

Zero tasks solved. Zero LLM calls (in most cases). Near-perfect scores.

• Terminal-Bench (89 tasks) — 100% score. Binary wrapper trojans.
• SWE-bench Verified (500 tasks) — 100% score. Pytest hooks force all tests to pass.
• SWE-bench Pro (731 tasks) — 100% score. In-container parser overwrite.
• WebArena (812 tasks) — ~100% score. Config leakage + DOM injection + prompt injection.
• FieldWorkArena (890 tasks) — 100% score. Validation never checks answer correctness.
• GAIA (165 tasks) — ~98% score. Public answers + normalization collisions.
• OSWorld (369 tasks) — 73% score. VM state manipulation + public gold files.

How Our Agent Did It

Terminal-Bench: Trojanizing the Test Infrastructure

Terminal-Bench evaluates agents on 89 complex terminal tasks. It uses Harbor, a sandboxing framework. Containers have full internet access — Harbor’s allow_internet config defaults to True.

Harbor correctly protects /tests/ — but it doesn’t protect system binaries. 82 of 89 tasks download uv from the internet at verification time via curl. Our exploit replaces /usr/bin/curl with a wrapper. When the verifier later runs curl, our wrapper installs a trojanized uvx binary. The trojanized uvx detects pytest invocations and produces fake passing output. The verifier trusts it. Task “passed.”

SWE-bench: 100% Resolve Rate Without Fixing a Single Bug

SWE-bench is the most influential AI coding benchmark. Both Verified and Pro versions are exploitable at 100%. The vulnerability: The agent’s patch is applied inside the same Docker container where tests run. Anything the patch introduces — a conftest.py or a monkey-patch — executes with full privileges.

Our agent creates a conftest.py with a pytest hook that forces every test to report as passing. Pytest auto-discovers these files and loads them before running tests. The hook intercepts every test result and rewrites it to “passed.” The log parser sees PASSED for every test. Instance resolved.

WebArena: Reading the Answer Key

WebArena runs a PromptAgent that drives a Chromium browser. It ships task configs — including reference answers — as JSON files on the local filesystem. By navigating the browser to a file:// URL, the agent can read the gold answer directly from the config and submit it to the grader.