Test AI-Generated Cleanup Against an Event-Order Contract

Summary slide. Before accepting the fix, test who owns cleanup, when it happens, and which error reaches the caller. Save the slide (PNG)

This helper returns the right result on success but skips cleanup when execute() raises. An event trace makes that ownership violation visible before you accept the diff.

def use_connection(acquire):
    connection = acquire()
    result = connection.execute()
    connection.close()
    return result

In this illustrative Python example, an exception bypasses close(). A test that checks only the returned value misses the failure-path defect.

A correct result does not establish correct cleanup.

Write the ownership contract before changing the code

For this example, declare the requirement as follows:

After successfully acquiring a connection, the helper owns its cleanup. It must attempt to close that connection exactly once before returning or propagating an operation error. If acquisition fails, the helper must not attempt cleanup.

Also declare the assumptions:

• Execution is synchronous.
• Acquisition either returns a connection or raises without transferring a connection to the helper.
• The connection’s close() operation succeeds.
• On success, the helper returns the operation’s result; on operation failure, it propagates the original exception.

The second assumption matters. An acquisition function that allocates a resource internally and then raises must manage that partial acquisition itself. This helper cannot close a connection it never received.

Ownership begins when acquisition returns, not when acquisition starts.

These boundaries make the requirement checkable without pretending it covers every possible resource failure.

Use the failure trace as the counterexample

The acceptance contract is observable behavior: ownership, event ordering, and exception identity. It is not a requirement to use a particular implementation shape.

Give the fake connection an operation that raises. For the original implementation, the observable trace would be:

acquire → operate → exception reaches caller

The required trace is:

acquire → operate → close → exception reaches caller

That difference gives a coding agent a concrete acceptance condition, rather than the vague instruction to “handle errors better.”

For this scoped example, try/finally is the repair:

def use_connection(acquire):
    connection = acquire()
    try:
        return connection.execute()
    finally:
        connection.close()

Acquisition stays outside the try block deliberately. If it raises, the helper has no acquired connection to close. Once acquisition succeeds, finally runs before the helper returns or lets the operation exception reach its caller.

This implementation relies on the stated assumption that close() succeeds. If closing can raise, you need an additional policy for competing operation and cleanup errors.

Check three paths with an event-recording fake

The following acceptance tests use Python’s standard unittest module. Keep the corrected helper above in the same file as this example.

The fake records lifecycle events directly. There is no garbage-collection timing, real database, network connection, or live customer data involved.

import unittest


class FakeConnection:
    def __init__(self, events, operation_error=None):
        self.events = events
        self.operation_error = operation_error

    def execute(self):
        self.events.append("operate")
        if self.operation_error is not None:
            raise self.operation_error
        return "result"

    def close(self):
        self.events.append("close")


def make_acquirer(events, operation_error=None, acquisition_error=None):
    def acquire():
        events.append("acquire")
        if acquisition_error is not None:
            raise acquisition_error
        return FakeConnection(events, operation_error)

    return acquire


class ConnectionContractTests(unittest.TestCase):
    def test_success_closes_before_return(self):
        events = []
        result = use_connection(make_acquirer(events))
        events.append("returned")

        self.assertEqual(result, "result")
        self.assertEqual(
            events, ["acquire", "operate", "close", "returned"]
        )

    def test_operation_failure_closes_before_propagation(self):
        events = []
        error = RuntimeError("operation failed")

        with self.assertRaises(RuntimeError) as caught:
            use_connection(make_acquirer(events, operation_error=error))
        events.append("propagated")

        self.assertIs(caught.exception, error)
        self.assertEqual(
            events, ["acquire", "operate", "close", "propagated"]
        )

    def test_acquisition_failure_does_not_attempt_cleanup(self):
        events = []
        error = RuntimeError("acquisition failed")

        with self.assertRaises(RuntimeError) as caught:
            use_connection(make_acquirer(events, acquisition_error=error))
        events.append("propagated")

        self.assertIs(caught.exception, error)
        self.assertEqual(events, ["acquire", "propagated"])


if __name__ == "__main__":
    unittest.main()

The exact event-list assertions serve several purposes:

• A missing close exposes skipped cleanup.
• A repeated close exposes duplicate cleanup attempts.
• A close before operate exposes incorrect ordering.
• The caller-side markers establish that cleanup happened before the caller observed completion.

The exception identity assertions check that the helper did not silently replace the original error with another exception of the same type. Keep those assertions even after the implementation becomes a short try/finally block: the implementation’s shape alone is not acceptance evidence.

Save the four-part cleanup contract

Use this checklist when reviewing an agent’s diff or writing its task instructions. The first three rows describe behavior to verify. Scope is a declared boundary, not evidence that behavior was verified.

Signal	Declared intent	Acceptance evidence or boundary
Ownership	The helper owns a connection only after acquisition returns.	Evidence: acquisition failure produces no operation or close event.
Cleanup timing	Attempt cleanup exactly once before completion.	Evidence: the exact trace contains one close event before return or propagation.
Error propagation	Preserve the original operation error when cleanup succeeds.	Evidence: the caller receives the same exception object.
Scope	Synchronous execution with successful close.	Boundary: name excluded behavior without claiming it has been checked.

I would require these checks before accepting a change to this owning helper. I would not impose the same cleanup rule on a helper that merely borrows a caller-owned connection. Closing a borrowed connection could itself violate the requirement.

The test must follow ownership, not just the presence of a connection variable.

For an AI-assisted change, keep the contract and acceptance checks alongside the code. Ask the agent to preserve them when editing the helper. Review changes to the tests as carefully as changes to the implementation: deleting the failure-path assertion does not resolve the failure-path defect.

This makes specification drift concrete. If the declared requirement still says “close on operation failure” but the diff removes that behavior, the acceptance check provides an observable disagreement between intent and implementation. For getting started with ThunderLang, see ThunderLang’s getting-started guide.

Background reading, not evidence for this test design: Stack Overflow’s “Elevators, toasters, and the birth of Java”.

A passing fake test does not prove leak freedom

These checks cover the modeled lifecycle paths. They establish neither real-driver cleanup nor all possible resource behavior.

They do not establish:

• That a real driver releases its underlying socket when close() is called.
• Correct behavior when cleanup itself raises.
• Correct behavior under cancellation, asynchronous execution, or concurrent use.
• Correct cleanup inside an acquisition function that partially allocates and then fails.

A close event records an attempted method call, not independently confirmed resource release. Real-driver integration checks and additional contracts are needed for those broader questions.

Try the counterexample before accepting the fix

For a five-minute exercise, save the success-only helper and the tests as test_connection_contract.py. Run:

python -m unittest test_connection_contract.py

Inspect the operation-failure test’s missing close event. Then replace the helper with the try/finally version and rerun the same command.

This exercise checks whether your test distinguishes the specific defect from the intended behavior. It does not establish that every possible leak has been eliminated.

Save the field guide

A field guide distilled from this article. The text version follows below.

Save the infographic (PNG)

Text version

• Establish who owns cleanup: The helper receives a connection only when acquisition returns successfully. Declare ownership at that boundary, and require no operation or close event if acquisition fails.
• Check successful completion: The helper returns the expected result, but that alone says nothing about cleanup. Assert the exact acquire, operate, close, returned sequence to verify one cleanup attempt before return.
• Force an operation failure: The operation raises after the helper has acquired its connection. Assert one close before propagation and the same exception object, so skipped cleanup and replaced errors are visible.
• Review the acceptance checks: An agent's diff changes the helper or its failure-path assertions. Review test edits alongside code edits, because removing an assertion does not repair the defect.

Illustrative Python failure path: When execute() raises, the original helper skips close(). Keeping acquisition outside try and placing close() in finally attempts cleanup before the operation error reaches the caller.

These fake tests assume synchronous execution and successful close. A close event proves an attempted call, not real resource release.