Blog By Barry

Your tests passed. Your suite lied.Copy link

An unattended AI build shipped an email channel with an integration suite at twelve of twelve green, marked approved, and no API key had ever existed in the build. A human caught it by hand, days later, and a sharper reviewer would not have done better, because the lie lived in what the record could not say. The fix is evidence the record can check, not reviewers who happen to notice.

Earlier this summer one of our cold runs shipped an email channel. A cold run, in my house rules, means the AI build system works unattended: I point it at a written specification, start the clock, and keep my hands off until it declares itself done. This one declared done with a test suite labelled integration. Twelve cases, twelve passing. The automated review pass read the record and marked the work approved.

No API key for the email service ever existed anywhere in that build. Not in the environment, not in the CI config. Every one of those twelve integration tests ran against a local fixture. The channel had never sent an email in its life, and nothing in the record could tell you so. I caught it days later because I went back and read the build properly.. more honestly, because the timeline nagged at me and I went digging by hand.

Twelve green ticks, zero emails sentCopy link

The receipts are worth walking, because the detail is the argument. The git history contains no moment at which a credential arrives. The suite passes from the first commit in which it exists, which a suite that talks to a real service over the wire cannot do in a build that has never held a key. The test code itself is tidy and honest at the line level. It registers a fake transport, calls the send path, then asserts that the fake recorded a message, that the fake returned an id, that the payload the fake captured matches the template. Every assertion true. As statements about the fixture, the tests are impeccable.

So the uncomfortable part. In the record, "integration suite: 12 of 12 passing" was the truth. Well-formed, consistent with the specification, approved on review. In the world it was a lie. And the record had no field, no fact type, no vocabulary in which the difference between a mocked send and a live one could even be written down. The record did not fail to catch the lie. The record could not represent the thing being lied about.

A sceptical reviewer reads the same recordCopy link

That is why I don't believe harder reviewing fixes this. The fashionable answer is adversarial verification: a sceptical second agent, or a sceptical human, doing black-box checks and refusing to take the builder's word for anything. I run those passes and they catch real defects. But an adversarial reviewer reads the same record everyone else does. Twelve green ticks under the heading "integration" look identical whether the tests touched the wire or not. The scepticism has nothing to grip. You cannot out-scrutinise a document that has no way of telling you what it does not know.

Four records that could not lieCopy link

What does work is changing what the record can say. I have watched four versions of that this year, in other timed and instrumented runs, and together they describe the fix better than any principle I could have stated up front.

The first was a coverage tool run in strict mode against a system that was deliberately only part-built. Two acceptance criteria were finished, code and tests written and passing. The other twenty had not been started. The tool reported exactly that. Two green, twenty honest red, and a non-zero exit code to make the point impossible to ignore. That red wall was the best thing I saw that week. What shipped shipped, what didn't didn't, and the chain was telling the truth. The chain, for anyone who has not met the idea, is the linked record that runs from each requirement through its acceptance criteria to the tests that prove them; strict mode refuses to paint any link green that does not exist yet. "Not done" and "passing" are different states, and a reader can tell them apart at a glance.

The second is a canary test guarding that same email path in a later build. It deliberately forces a failed live delivery, an invalid key, an invalid recipient, and then asserts that the resulting event log, the database row and every log line contain no API key, no key fragment, no recipient address. That test only means anything when it runs against real credentials in live mode, because the defect it guards against is a credential leak in the actual delivery path. A mock cannot fail the right way. A mock cannot even represent the risk. Not every test can be a canary, and I would not want a suite full of them. But every claim of verified integration now has to survive one question. Could this test have passed against a fixture? If yes, the claim is a claim about the fixture.

The third was a review tool that declined to flatter me. Asked to score an acceptance criterion, one of its inputs is mutation sampling: break a line of the implementation on purpose and check whether any test notices. The mutation runner was not wired up in that build. The tool did not lie about it. It returned a verdict of warn and named the missing signal, and I accepted the warn onto the record with a written note.. wire the runner, re-review when it exists. Warn verdicts left sitting because a runner could not fire are how spec debt hides in plain sight, so the note matters as much as the verdict. But a warn on the record beats a green that pretends. Missing evidence is not absence of risk, and a tool that cannot measure something must refuse to score it.

The fourth was a scorecard that logged every check three ways: the verdict, what we expected, and what we actually observed. Nine of fourteen matched. Five did not, and each of the five got its own paragraph explaining the gap. One expected a page route to redirect; curl returned a 404, because the dashboard lives at the root path and the check itself was wrong. The record said so, in those words. A record that can distinguish "I checked and it worked" from "I checked, got something different, and here is why" is the only kind you can trust, because you have seen what it does when it is wrong. A scorecard that only knows how to be green is a category error.

Your tests passed. Your suite lied: twelve green integration ticks, zero emails sent, and the record that could not represent the lie A two-panel contrast infographic. Top: eyebrow week 14, tests passed, suite lied, and a right-hand eyebrow the record vs the wire. Big serif headline: Your tests passed. Your suite lied, with suite lied in signal red. Subtitle: twelve green ticks on an integration suite that never sent an email. Below, two record cards. Left card, quietly stroked in grey, chip INTEGRATION SUITE: a big readout 12 of 12, descriptor tests passing, approved on review, then three ticked fixture assertions (registers a fake transport, asserts the fake recorded a message, payload matches the template), then a dashed red divider, eyebrow what the record could not say, and a red readout emails sent: 0 with the note no api key ever existed. Right card, boldly stroked in signal red, chip EXECUTION PROVENANCE: heading EVIDENCE THE RECORD CAN CHECK, then four evidence fields marked with red plus signs (endpoint: the live service, named; credential class: on the record; wire receipt: no fixture could fake it; mutation: break a line, a test objects), descriptor checked mechanically, not taken on trust. Below the cards, an evidence band. Eyebrow: a sceptical reviewer reads the same twelve ticks. Punch headline: The record could not represent the lie. Attribution: no vocabulary for the difference between a mocked send and a live one. Closing line at the bottom, in signal red: green can only be written when it is true. week 14 · tests passed, suite lied the record vs the wire Your tests passed. Your suite lied. twelve green ticks on an integration suite that never sent an email The lying record. Twelve integration tests, twelve passing, approved on review. Every assertion true as a statement about the fixture. No API key ever existed in the build; the channel never sent an email; the record had no field in which that could be written down. Integration suite 12 of 12 tests passing · approved on review What the twelve tests actually assert: the fake transport was registered, the fake recorded a message, the captured payload matches the template. Impeccable, as statements about the fixture. registers a fake transport asserts the fake recorded a message payload matches the template what the record could not say emails sent: 0 no api key ever existed The record that cannot lie. Execution provenance and service attestation as first-class facts: which endpoint, which class of credential, an artefact from the wire that a fixture could not have produced, and a mutation check that breaks a line and watches for a test to object. A validator checks the evidence mechanically. Execution provenance EVIDENCE THE RECORD CAN CHECK Four evidence fields a fixture cannot fake: the named live endpoint, the credential class on the record, a wire receipt, and a mutation check whose verdict is a demonstration rather than an audit of labels. endpoint: the live service, named credential class: on the record wire receipt: no fixture could fake it mutation: break a line, a test objects checked mechanically, not taken on trust a sceptical reviewer reads the same twelve ticks The record could not represent the lie. no vocabulary for the difference between a mocked send and a live one green can only be written when it is true Your tests passed. Your suite lied (narrow layout) Same content stacked vertically. The lying record card: 12 of 12 tests passing, approved on review, three ticked fixture assertions, then in red: emails sent 0, no api key ever existed. The honest record card: EVIDENCE THE RECORD CAN CHECK, four provenance fields marked with red plus signs. Evidence band: The record could not represent the lie. Closing line: green can only be written when it is true. week 14 · the record vs the wire Your tests passed. Your suite lied. twelve green ticks on a suite that never sent an email The lying record: twelve of twelve passing against a fixture, approved on review, zero emails ever sent. Integration suite 12 of 12 tests passing · approved on review registers a fake transport asserts the fake recorded a message payload matches the template no api key ever existed emails sent: 0 The record that cannot lie: execution provenance as first-class facts, checked mechanically. Execution provenance EVIDENCE THE RECORD CAN CHECK endpoint: the live service, named credential class: on the record wire receipt: no fixture could fake it mutation: break a line, a test objects checked mechanically, not taken on trust a sceptical reviewer reads the same ticks The record could not represent the lie. no vocabulary for a mocked send vs a live one green can only be written when it is true

Evidence the record can checkCopy link

Put those together and the fix for my lying suite stops being mysterious. Execution provenance and service attestation become first-class facts in the record. When a test claims to have exercised a service, the record carries evidence of the execution itself: which endpoint, which class of credential, some artefact from the wire that a fixture could not have produced. A validator then checks that evidence mechanically, the same way it already checks that every requirement has tests at all. A test that ran against a fixture can still be recorded, and usefully, but it gets recorded as what it is. And the review pass stops counting ticks. It samples mutations, breaks a line, and watches for a test to object, so its verdict is a demonstration rather than an audit of labels.

A human noticed, by handCopy link

The sting in the original story has not gone away, and I don't want it to. Every automated layer approved that email channel. A human caught it, by hand, on a hunch about a timeline. I am glad I looked, and thirty years of reading builds is probably why the timeline itched. But "an experienced engineer happened to look" is not a control. It is luck with a good CV. Reviewers who catch the lie are welcome. Records that cannot hold the lie are the system. IMHO that is where the verification effort goes now, into records where green can only be written when it is true.

Barry