Spec Time vs. Runtime: Where Intent Verification Actually Breaks Down

Roughly 68% of production incidents traced to configuration drift originate in code that passed every pre-merge check, including static analysis, type checking, and unit tests. The failure surface is not the code itself. It is the gap between what the spec declared and what the live environment actually delivered at runtime.

The Snapshot Problem With Spec-Time Verification

When an AI agent generates code from a spec, the moment of verification is the generation moment. The spec says "this endpoint must return HTTP 200 when the feature flag payments_v2 is enabled." The agent produces code that satisfies that clause. A diff tool confirms the generated code matches the intent. Everyone ships it.

The spec is a snapshot. The feature flag is a runtime fact that changes without a deploy.

This is the foundational crack in most spec-driven development workflows today. Specs are written against an assumed world state, and that world state drifts constantly. Feature flags toggle. Environment variables get rotated. Third-party services deprecate endpoints. Downstream Kafka topics shift their schema. The spec is never updated to reflect any of this, because from the spec author's perspective, nothing changed in the codebase.

What Static Gates Actually Catch

Let me be precise about what static verification does well, because the argument is not that it is useless.

Static gates catch:

• Type mismatches between the spec's declared interface and the generated code's signature
• Missing branches that the spec required ("must handle 404 from the upstream service")
• Obvious contradictions between spec clauses ("must be idempotent" + "must increment a counter on each call")
• Structural conformance: does the code even address the surface the spec described

Static gates do not catch:

• Whether PAYMENT_PROVIDER_URL points to staging or production in the environment where this runs
• Whether a third-party API the spec assumed is still returning the schema the spec was written against
• Whether the feature flag logic in the spec was written before ops disabled that flag in the production config store six weeks ago
• Whether a Postgres view the spec cited exists in the current migration state of the target database

None of those failures are code bugs. They are intent drift: the declared intent was correct at spec-authoring time and is now false in the environment where the code executes.

The Class of Bugs That Pass Every Gate

Here is a concrete failure mode I have seen in production twice. A spec requires that a payment retry handler respect a max_retries configuration value pulled from the environment. The AI generates the following:

import os

MAX_RETRIES = int(os.getenv("MAX_RETRIES", "3"))

def retry_payment(payment_id: str, attempt: int) -> bool:
    if attempt >= MAX_RETRIES:
        raise MaxRetriesExceeded(f"Payment {payment_id} failed after {MAX_RETRIES} attempts")
    return _attempt_charge(payment_id)

This code is correct relative to the spec. The diff check passes. The unit tests pass with MAX_RETRIES=3 in the test harness. The type signature is correct. The spec clause is satisfied.

In production, MAX_RETRIES is unset on the payment worker pods because someone renamed it to PAYMENT_MAX_RETRIES during a config consolidation two months ago and nobody caught that the old name was still referenced in specs. The environment silently falls back to "3". Every payment that should have retried five times before escalating to the fraud queue now retries three times. Fraud catches fire. Nobody connects it to a config rename.

That bug has a perfect static verification record. It is invisible to every pre-merge gate. It shipped because runtime conformance was never checked against declared intent, only code structure was.

Why Intent Drift Is Structurally Invisible at Commit Time

Commit-time verification tools operate on the repository. The repository contains code and, sometimes, spec files. It does not contain the live state of the environment the code will execute in.

You cannot diff os.getenv("MAX_RETRIES") against the live Kubernetes ConfigMap at commit time without instrumenting your CI pipeline to interrogate the production config store. Almost no teams do this. The operational cost is high, the security surface is uncomfortable (CI touching prod secrets), and the tooling barely exists.

So the gap persists by default. Specs accumulate assertions about environment shape that nobody verifies after the spec is written. The code honors those assertions faithfully. The environment has moved on.

This is not a discipline problem. It is a tooling gap. The tooling assumes spec verification ends at generation or commit. Runtime is treated as a separate domain owned by observability teams, not the intent-verification system.

What Durable Conformance Actually Requires

Durable conformance means the link between declared intent and observable behavior stays live after the code ships, not just at the moment it was generated.

This requires three things that most current workflows omit:

1. Spec assertions that are executable at runtime, not just readable at commit time. A spec clause like "the service must read MAX_RETRIES from the environment and default to 5" should generate a startup assertion that verifies the variable exists and is a valid integer before the process begins serving traffic. Not a unit test. A production gate.

2. Intent signal in observability. When the code does something the spec declared it would do, emit a structured trace attribute that identifies the spec clause. When the code takes a path the spec did not anticipate, that is a signal, not just a log line. Datadog custom metrics and OpenTelemetry span attributes are both viable here. Intent is an observable dimension, not just a code property.

3. A feedback loop from runtime observations back to the spec. If an environment variable the spec cited is absent at startup, the spec is now violated. That violation should surface as a spec-level alert, not just a generic config error. The owning team needs to know the spec clause that is broken, not just that a variable is missing.

ThunderLang's verify-diff model gets at point one by generating proof artifacts at change time, but durable conformance extends that into runtime instrumentation. The spec is not done proving itself when the PR merges.

Feature Flags Are the Worst Offender

Feature flags deserve their own treatment because they are the most common source of spec-runtime divergence in modern systems.

A spec written against a feature flag condition is a conditional spec. "When dark_mode_billing is enabled, the invoice PDF must use the dark color palette." The AI generates code that branches on the flag. The branch is verified against the spec. The test suite runs both branches. Everything is green.

Six months later, dark_mode_billing is permanently enabled in production and the old flag-off path is dead code. The spec still describes both branches as if they are live. A new engineer writes a new spec that contradicts the old flag-off behavior, not knowing the branch is unreachable. The AI agent generates code that satisfies the new spec in the live branch but reintroduces a subtle regression in the dead branch, a branch that a future experiment might re-enable.

The spec had no runtime signal telling it that one of its conditional clauses had become permanently false. The spec aged silently. The AI had no way to know.

This is one of the places where connecting declared intent to a feature flag service (LaunchDarkly, Statsig, or a homegrown config store) changes the calculus. If the spec-tracking system knows which flags are permanently enabled, it can prune dead branches from the intent surface automatically and stop generating verification artifacts for unreachable code paths.

The Engineering Investment Required

I want to be honest about the cost here. This is not a free lunch.

Closing the loop between spec-time intent and runtime conformance requires:

• Startup assertions or readiness probes that interrogate live configuration against spec-declared expectations
• A convention for tagging observability signals with spec-clause identifiers (this is a cultural investment as much as a technical one)
• Integration between your spec tooling and your feature flag service so that spec clauses can be marked as conditionally active
• A process for alerting on spec violations at runtime, separate from general incident alerting, because the owning team for a spec clause may not be the on-call team

The teams where I have seen this work well are the ones that treated spec clauses as first-class runtime objects, not just documents. The spec is not a Word doc that lives in Confluence. It is a versioned artifact with an identity, and that identity travels with the code all the way into production.

ThunderLang approaches this by making intent machine-verifiable at the point of change, which is the prerequisite step. Without machine-readable specs, you cannot generate runtime assertions from them. The machine-readable spec is the foundation; runtime conformance is the next layer.

Where To Draw the Verification Boundary

My opinionated position: spec-time verification is necessary but not sufficient, and teams that treat it as the full solution are carrying hidden runtime risk they cannot see.

If your spec contains any of the following, you need runtime conformance checking, not just commit-time diff verification:

• References to environment variables or secrets
• Conditions on feature flags or configuration values
• Assertions about the schema or availability of third-party APIs
• Assumptions about the state of a database (views, indexes, schemas) that are not managed in the same migration pipeline as the application code

That covers the majority of non-trivial specs in production systems. Which means the majority of teams running AI-generated code from specs are carrying unverified runtime assumptions today.

The fix is not to distrust AI code generation. The fix is to extend the verification contract forward in time. A spec verified only at generation is a snapshot. A spec that generates runtime assertions, instruments observability with intent signals, and triggers alerts when the live environment violates its clauses is a contract. The engineering investment is real, but the alternative is a growing inventory of silently wrong behaviors that passed every gate you had.

Gate your first AI change

ThunderLang lets you declare what a change must satisfy, then gates AI-written code against that intent with a verify-diff and durable proof artifacts. If you want to see what machine-readable intent looks like before you build the runtime layer on top of it, start with the getting-started guide and define your first spec clause against a real change in your codebase.

The gap between what a spec declares and what a running system does is not a testing problem; it is a verification boundary problem, and moving that boundary to runtime is the only way to close it durably.