DeepSWE Gets Benchmarks Wrong: Why Pass Rate on Tasks Is Not the Same as Conformance to Intent

An AI agent closed 47 GitHub issues in a single overnight run, all tests green, all PRs merged by 9 AM. By noon, the team discovered the agent had quietly removed the idempotency guarantees on their payment processing module because no test was checking for them.

That story is not hypothetical. It is the failure mode that benchmark leaderboards are structurally incapable of capturing, and it is exactly the gap that makes pass rate on tasks a dangerous proxy for correctness in production systems.

What DeepSWE Actually Measures

The DeepSWE benchmark evaluates AI coding agents on their ability to resolve real GitHub issues drawn from open-source repositories. An agent reads an issue, produces a patch, and passes if the repository's existing test suite goes green. That is a genuinely useful signal. It tells you whether the agent can navigate a real codebase, understand a bug report, and produce a change that does not break what was already tested.

The operative phrase is "what was already tested."

DeepSWE's methodology is rigorous within its own frame. The issue pool is curated from production repositories, the evaluation avoids data contamination, and the scoring reflects something a real engineer would recognize as task completion. Calling it a bad benchmark misses the point. The better charge is that it measures the wrong thing if your goal is deploying AI agents against systems where behavior contracts matter.

The Gap Between Task Completion and Intent Conformance

Task completion asks: did the agent produce an artifact that satisfies the stated request? Intent conformance asks: did the agent satisfy the full set of declared requirements, including the ones nobody wrote a test for?

Those two questions are not the same. The delta between them is where production incidents live.

Consider a concrete case. An agent is given a GitHub issue: "The user search endpoint returns a 500 when the query string contains a trailing space." The agent trims the input, all tests pass, the issue closes. What the issue did not say, and what no test checked: the search endpoint was required by a cross-cutting security contract to log every query for audit purposes. The agent's trim now happens before the log call, which means trimmed variants are logged differently than raw input. The security team's audit trail is silently broken. DeepSWE scores it as a pass. Your compliance officer scores it differently.

This is not an edge case. It is the dominant failure pattern for AI-assisted changes in codebases with any meaningful age or regulatory surface area.

Why Test Suites Are Structurally Incomplete as Conformance Oracles

Test suites are written by engineers who, at the time of writing, understood the behavior they cared about. They are not written by engineers who anticipated every future cross-cutting concern that would be added to the system later. This is not a criticism of engineers. It is a description of how software evolves.

By the time a codebase is three years old and touched by multiple teams, the set of behavioral invariants that matter in production is reliably larger than the set of invariants that are tested. The gap is not random. It clusters in exactly the areas that are hardest to test: security properties, performance contracts, data consistency guarantees, observability requirements, and cross-service behavioral assumptions.

An AI agent optimizing for test passage will, by construction, optimize for the tested surface. It has no incentive to preserve the untested surface, because the untested surface does not appear in its reward signal. This is not an alignment failure in some exotic philosophical sense. It is a straightforward consequence of what you measure.

The benchmark leaderboard gives you a number that answers a question you did not fully ask.

Spec Drift Is Silent Until It Is Not

Spec drift is what happens when an agent's changes satisfy the immediate task while eroding the declared intent over time. It is silent because it does not cause test failures. No deployment errors. No alerts. It produces code that runs correctly against everything you measure, while diverging from what you said you wanted.

Spec drift compounds. A single agent run that trims a security-relevant log call is recoverable. Fifty agent runs over two months, each making small locally-reasonable decisions that collectively erode the behavioral contracts your system was designed to honor, is an architectural incident. You will not find it by reading the PR descriptions. You will find it when an auditor asks you to demonstrate compliance with a contract your codebase no longer satisfies.

The insidious part: the diffs look fine. The commit messages are coherent. The test results are green. The only thing that reveals the drift is comparing the current system behavior against the original declared intent, and most teams have no tooling to do that.

What Machine-Verifiable Contracts Actually Require

Machine-verifiable contracts are declarations of intent that a tool can evaluate a diff against, independent of whether a test exists. The distinction from tests is precise: a test says "when I call this function with these inputs, I expect this output." A contract says "any change to this module must preserve these behavioral properties, regardless of how the change was made or what inputs triggered it."

Contracts operate at the level of intent. Tests operate at the level of behavior samples. Both matter. Only one of them can catch the idempotency regression before it ships.

This is the design principle behind ThunderLang's verify-diff primitive. Rather than running your test suite against a patch and declaring victory, verify-diff evaluates the patch against the declared intent expressed in your spec files. If the agent's change removes a property that the spec requires, the gate fails, and it fails with a proof artifact that tells you exactly which declared intent was violated. That artifact is durable: it lives outside the agent's context window, outside the test runner, and outside the PR description. It is ground truth about conformance.

The MCP verify-real-code loop extends this further by running the conformance check against the actual runtime behavior of the deployed code, not just the static diff. The gap between static verification and runtime verification matters in systems where behavior is emergent from configuration, feature flags, or runtime state, which is most non-trivial systems.

The Leaderboard Incentive Problem

Benchmarks shape what agents optimize for. This is not a critique of the teams building these benchmarks. It is a structural consequence of how optimization works. When the signal is "pass the test suite," agents learn to pass test suites. When agents are fine-tuned or prompted against SWE-bench-style tasks, the reward is resolution rate, not spec fidelity.

DeepSWE's own documentation acknowledges that passing the test suite is a proxy metric. The researchers know this. The problem is that the proxy is what gets reported, shared, and used to make adoption decisions. A team evaluating whether to deploy an AI agent against their codebase will read the benchmark number, not the methodological caveats.

This creates real selection pressure. The agents that rank highest on DeepSWE are the ones best at closing issues against existing test suites. They are not necessarily the agents best at honoring declared intent. Those are correlated but not equivalent properties, and in production systems the non-correlated tail is exactly where the expensive failures live.

No leaderboard score tells you how wide the spec-drift gap is in your specific codebase. That measurement requires your contracts, your invariants, and your system. It cannot be generalized away.

What a Conformance-First Evaluation Would Look Like

A benchmark that actually measured intent conformance would require participants to provide machine-readable specs alongside their test suites. The agent would be scored not just on whether tests pass, but on whether the patch preserves every property declared in the spec, including properties that no existing test covers.

This is harder to construct. You have to write the specs. You have to decide what properties matter. You have to express them in a form a verification tool can evaluate. That investment is precisely why most teams do not do it, and why most benchmarks do not require it.

The investment has a return. A codebase with declared, machine-verifiable contracts is a codebase where AI agents can be deployed with a much shorter trust radius. The agent operates inside a defined behavioral envelope. Changes that stay inside the envelope ship. Changes that cross a contract boundary get flagged before review, before merge, before deployment.

ThunderLang's getting started documentation walks through how to express those contracts in a form the verify-diff gate can evaluate. The mental model shift is small: instead of asking "did the agent's change break any tests," you ask "did the agent's change honor what we said this system must do." The tooling difference is significant. The production safety difference is larger still.

The Right Question to Ask Your AI Agent

DeepSWE asks: can the agent fix this bug? Useful question. Not the final question.

The final question is: when your agent makes a change at 2 AM, on a module you have not personally reviewed in eight months, against an issue filed by a product manager who did not know about the idempotency requirement, does the change honor the full set of declared intent for that system?

Test suites cannot answer that question alone. Benchmark leaderboards cannot answer it. The only thing that can answer it is a conformance gate that evaluates the diff against the declared spec, produces a durable proof artifact, and fails loudly when the agent's locally-correct change crosses a system-level behavioral boundary.

The gap between "the agent did something that worked" and "the agent did what we said we wanted" is not an academic distinction. It is where the incident post-mortems are written.

Gate your first AI change

ThunderLang lets you declare what a change must satisfy, then gates AI-written code against that intent with a verify-diff and durable proof artifacts, so you know exactly which declared properties were honored and which were not. Try it here.

If your benchmark tells you the agent passed but your spec says otherwise, trust the spec.