The Silent Spec Fork: When Every Test Passes But Intent Has Already Broken

A green CI pipeline is not evidence that your software does what it was designed to do. It is evidence that your software does what your tests expect. That is a much smaller claim than most teams treat it as.

Last month, a team at a fintech I advise shipped an AI-assisted refactor of their fee-calculation service. Every unit test passed. Every integration test passed. The PR reviewer approved it in twelve minutes because the diff looked clean and the suite was green. Six weeks later, a compliance audit caught that fee proration for mid-cycle account downgrades had been silently recalculated using a ceiling function instead of a floor function. The original spec, written by a product manager and a regulator-facing lawyer, was explicit: always favor the customer on fractional cents. The tests had never encoded that constraint. They had only encoded the arithmetic the original code happened to produce.

Why Tests Cannot Save You From a Spec Fork

This failure mode has a name: spec drift, or more precisely a silent spec fork. It is structurally different from a bug and structurally different from a test failure. A bug is code that violates the tests. A test failure is a regression the suite catches. A spec fork is code that satisfies all tests while diverging from the intent that preceded those tests. The suite is green because the suite never knew what the intent was.

Tests verify behavior against themselves. They are written by engineers who read the code or, in AI-assisted workflows, are sometimes generated by the same model that wrote the code being tested. When an LLM like Claude or GPT-4o refactors a module, it optimizes for passing assertions. If the assertions are incomplete, the model has no mechanism to notice. It cannot consult the original intent document. It does not know what a mid-cycle downgrade proration rule means to a regulator. It knows that assert fee == 4.50 passes, and it produces code that makes that happen.

The deepest problem is that tests are derivative artifacts. They derive from code, or from a spec, or from both, but over time the derivation chain frays. The spec is written. Code is written from the spec. Tests are written from the code or the spec. Then the code is refactored. The tests travel with the code. The spec sits in Confluence, growing stale. Nobody runs a diff between the current code's behavior and the original spec's declared constraints. Nobody has a tool in the pipeline that does that.

This is not a new problem. AI-generated code accelerates it by an order of magnitude.

A senior engineer doing a manual refactor will often remember the intent because they were in the room when the spec was written, or they read the Jira ticket carefully, or muscle memory from similar domains fires a warning. A model has none of that. It has tokens.

The Verify-Diff: What It Is and Why It Has to Run in CI

The fix is not more tests. The fix is a separate, first-class check called a verify-diff: a structured comparison between the declared intent of a change and the actual behavior of the code that landed. This is not a code review. A code reviewer looks at the diff of code. A verify-diff looks at the diff between what the spec said must be true and what the code now does.

Here is the simplest version of what this looks like in practice:

# .thunderlang/fee-proration.intent.yaml
constraints:
  - id: TC-0041
    description: "Fractional cent proration on mid-cycle downgrade must floor, not ceil, favoring the customer."
    references:
      - doc: "compliance/fee-spec-v3.md"
        section: "4.2.1"
    verify:
      invariant: "prorate(days_used, days_in_period, full_fee) <= ceil_prorate(days_used, days_in_period, full_fee)"
      coverage_required: true

This file lives in the repo. It is versioned. It is not a test. It is a declaration of intent that a verify-diff tool can evaluate against the code's actual output. If an AI refactor changes math.floor to math.ceil inside prorate(), the invariant check fails at the intent layer before the PR merges, even if every unit test still passes.

The critical architectural decision is pipeline placement. The verify-diff check must run as a blocking gate in CI, not as a linting suggestion, not as a pre-commit hook that engineers skip. It needs the same status in your pipeline that a failing test has. If it does not block the merge, it will be ignored under deadline pressure. That is not a prediction. That is a description of how every optional quality gate in software history has been treated.

ThunderLang is built around exactly this model: you declare what a change must satisfy, the tool runs a verify-diff against that declared intent, and it produces durable proof artifacts that the intent was checked at a specific commit. The getting started guide walks through setting up your first intent file and wiring it into a GitHub Actions pipeline in under thirty minutes.

Intent Conformance as a First-Class Engineering Practice

Intent conformance is not a new concept in formal methods, but it has historically lived in aerospace and medical device software, behind walls of tooling that most product engineers never touch. What has changed is that the risk profile for ordinary product software now resembles the risk profile that made formal methods worth the cost in regulated industries, because AI agents are now writing and rewriting code at a rate that human review cannot keep up with.

If your team ships ten AI-assisted PRs a week, and each one carries a 5% chance of a silent spec fork on a constraint your tests do not encode, you have roughly a 40% chance of a spec fork reaching production every week. That number is not from a study. It is arithmetic from conservative assumptions. Your actual exposure depends on how well your tests encode intent, and my experience is that most test suites encode behavioral snapshots, not declarative constraints.

Treating spec conformance as a first-class check means three concrete things:

1. Spec documents must produce machine-checkable artifacts. A Confluence page does not count. A YAML or TOML intent file that lives in the repo, is versioned with the code, and is consumed by your verify-diff tool counts.

2. AI-generated changes need a higher conformance bar, not a lower one. The reflex in most teams is to trust AI-generated code more because it is clean and fast and the tests pass. The correct reflex is to run the verify-diff on every AI-generated PR, because the model has no access to intent and will not volunteer that it changed a business-critical rounding behavior.

3. Proof artifacts need to travel with the release. When a compliance audit or a post-mortem asks how a change got to production, "the tests passed" is not a sufficient answer. A timestamped verify-diff result, pinned to the commit and the intent file version, is an answer. This is what ThunderLang's proof artifacts are for.

The fintech team I mentioned earlier now runs intent conformance checks on every PR that touches the billing service. They wrote fourteen intent files in a single sprint, one per major constraint from the compliance spec. It took a senior engineer two days to write those files and three hours to wire them into CI. The next AI-assisted refactor of that service ran fine. The one after that flagged an invariant violation before it merged. The engineer who submitted that PR had not noticed it. Neither had the reviewer.

The tests had still passed.

Spec drift does not announce itself. It accumulates quietly across refactors, accumulates across model-generated changes, and surfaces as a compliance finding or a customer complaint or a production incident that takes a week to trace back to a rounding function that changed six months ago. The only way to catch it at the moment it happens is to have a machine-readable declaration of intent and a tool that checks code against it on every change, automatically, blocking.

Green tests tell you the code is self-consistent. They do not tell you it is intent-consistent. Treating those two properties as equivalent is the assumption that will cost you.

Gate your first AI change

ThunderLang lets you declare what a change must satisfy, then gates AI-written code against that intent with a verify-diff and durable proof artifacts that travel with the release. If you are already running AI-assisted PRs in production, the time to add intent conformance to your pipeline is before the next spec fork lands, not after. Try it here.

A passing test suite is a proof of self-consistency, and confusing it for a proof of intent is the most expensive mistake a team can make in an AI-assisted codebase.