Blog By Barry

Harness without a spec is a loop without a target.Copy link

Harness engineering has the loop close to solved. Plan, act, verify, repeat, with humans steering rather than typing. But almost every harness points the verify step at proxies, tests the agent wrote, lint, a second model's opinion of the diff, and a loop aimed at a proxy converges on the proxy. A guard job in my own CI sat green for five weeks while blind to the one thing it existed to check. The missing half of the discipline is the target, a spec layer the loop can verify against.

A few weeks back I wrote about the harness wave, and the discipline that has crystallised out of it. Harness engineering, everything around the model that isn't the model. The instructions, the custom linters, the sensors, the feedback loops that let an agent catch its own mistakes before a human ever looks. It has gone from informal practice to named discipline inside a year, and credit where due, the loop mechanics are close to solved. Agents plan, act, verify and go round again, with humans steering rather than typing.

Which is exactly why I want to pick at it. The better the loop gets, the more the whole argument concentrates in one word. Verify. Most harness write-ups treat the verify step as a slot you plug sensors into. It isn't a slot. It is the step that decides what the other three converge on. A loop is an optimisation process, and an optimisation process needs a target. So, verify against what?

The proxy targetCopy link

Ask that question of most harnesses and the honest answer is a list of proxies. Tests green. Build green. Lint clean. Types check. A second model read the diff and thought it looked fine. Every one of those is a real signal, and my own loops ran on exactly that stack before the spec layer went in. But look at what each one is a signal of. In that setup the tests were written by the same agent, minutes after the code they test. The lint checks style. The types check internal consistency. The reviewing model reads the same repo the building model wrote. Each sensor compares the code with the code, or with an artefact derived from the code. Nothing in that list has ever seen what the business asked for.

Last Monday I walked through a suite that passed twelve of twelve while the email channel it claimed to verify had never sent an email. That suite wasn't badly written. It was pointed at a fixture and asked questions the fixture could answer, and it answered them truthfully. The loop optimises what you measured, not what you meant, and it does so with complete sincerity.

What converging on a proxy looks likeCopy link

You'll have seen the small versions if you run agents in anger. The agent that can't make a failing test pass, so it edits the assertion instead of the code. The fix that satisfies the check and breaks the behaviour the check was standing in for. The ones you catch are the clumsy ones.

The version that got me this month was better, because nothing edited anything. One of my repos carries a guard job in CI whose entire purpose is to catch a breaking release of a package. Install the latest release, exercise it, fail loudly if a release ships broken. It ran on every push. Green for five straight weeks.

The package had been renamed. The guard was still installing the deprecated name, which resolved happily to the last version ever published under it, months stale. So the job was faithfully installing a frozen artefact, exercising it, and reporting success, while release after release went out unchecked. The one job whose reason to exist was watching releases was blind to every release it existed to check. Green wasn't health. Green was staleness.

I repointed it at the current name and it went red on the next run. That red was not a regression. It was the guard doing its job for the first time in five weeks, and it was the most trustworthy signal that pipeline had produced all month.

The spec layer is the targetCopy link

None of this gets fixed by a better sensor, and I made that argument at length in the harness piece. A sensor compares the system with a reference it can reach. If the only reference in the repo is the code and artefacts derived from the code, the loop closes on the code, and verified means self-consistent. The missing component is the reference itself. The requirement, written down where a check can see it.

That is what I mean by a spec layer, and I mean something far more mechanical than a requirements document humans nod at and ignore. Product requirements broken into atomic requirements. Requirements into user stories. Each story carrying acceptance criteria precise enough that a test either passes against them or doesn't. Every artefact carries an ID, and the links between them are explicit, so tooling can walk the chain in both directions.. requirement down to the tests that prove it, failing test back up to the requirement it just falsified. A criterion with no test is a reported failure, the same class of failure as a broken build.

Aim the loop at that and the verify step changes meaning. A green run stops asserting that the code agrees with itself and starts asserting that the code does what somebody asked for. The tests sit downstream of the criteria instead of downstream of the implementation, so the agent can't mark its own homework, because it didn't set the questions.

Harness without a spec is a loop without a target. Same loop mechanics twice, two different targets, two very different destinations. A two-panel contrast infographic. Top: eyebrow week 15, half a discipline, and a right-hand eyebrow loop vs target. Big serif headline: Harness without a spec is a loop without a target, with the word target in signal red. Subtitle: the loop optimises what verify points at, so verify against what. Below, two loop cards. Left card, quietly stroked in grey, chip PROXY TARGET, heading LOOP AIMED AT PROXIES: a plan, act, verify cycle drawn as three linked nodes with a dashed grey arrow leaving verify and pointing right at a list of proxies with grey ticks, tests green, build green, lint clean, model says fine. Below the list, the label target followed by self-consistent in red, and the descriptor code checked against code. Right card, boldly stroked in signal red, chip SPEC LAYER, heading LOOP AIMED AT INTENT: the same plan, act, verify cycle with a dashed red arrow leaving verify and pointing right at a chain of four connected red pills, requirement, story, criterion, test. Below the chain, the label target followed by intent in red, and the descriptor code checked against what somebody asked for. Below the cards, an evidence band. Eyebrow: autonomy scales with the gate, not the model. Punch headline: A loop with no target still gets somewhere. Attribution: just not anywhere you chose. Closing line at the bottom, in signal red: build the target before you tune the loop. week 15 · half a discipline loop vs target Harness without a spec is a loop without a target. the loop optimises what verify points at. verify against what? The proxy loop. Plan, act, verify running on well-oiled harness mechanics. The verify step points at a list of proxies: tests green, build green, lint clean, model says fine. Every one of them is a signal about the code, not about intent. The loop converges on self-consistent. Proxy target LOOP AIMED AT PROXIES Plan, act, verify. Three nodes in a triangle joined by curved arrows going clockwise, the harness loop that is close to solved. plan act verify verifyagainst The proxies the verify step actually points at. Each one a real signal, and every one of them describes the code, or an artefact derived from the code. Nothing in the list has ever seen what the business asked for. tests green build green lint clean model says fine target self-consistent code checked against code The spec loop. Same plan, act, verify mechanics, aimed through a chain of small linked documents: requirement, story, criterion, test. The verify step compares the system with something a person wrote down as intent, so the loop converges on intent instead of on itself. Spec layer LOOP AIMED AT INTENT Plan, act, verify. Identical mechanics to the proxy loop on the left. The loop is not the problem. The target is. plan act verify verifyagainst The spec layer. Product requirements broken into atomic requirements, requirements into user stories, stories carrying acceptance criteria precise enough that a test either passes or does not. Explicit IDs, walkable in both directions. requirement story criterion test target intent code checked against what somebody asked for autonomy scales with the gate, not the model A loop with no target still gets somewhere. just not anywhere you chose build the target before you tune the loop Harness without a spec is a loop without a target (narrow layout) Same content stacked. Proxy card: chip PROXY TARGET, heading LOOP AIMED AT PROXIES, a plan act verify cycle, dashed grey arrow to a list, tests green, build green, lint clean, model says fine, target self-consistent, code checked against code. Spec card: chip SPEC LAYER, heading LOOP AIMED AT INTENT, same cycle stroked in red, dashed red arrow to a chain of red pills, requirement, story, criterion, test, target intent, code checked against what somebody asked for. Punch: A loop with no target still gets somewhere. Attribution: just not anywhere you chose. Closing: build the target before you tune the loop. week 15 · half a discipline Harness without a spec is a loop without a target. the loop optimises what verify points at. verify against what? Proxy loop: same plan act verify, aimed at proxies, converges on self-consistent. Proxy target LOOP AIMED AT PROXIES plan act verify verifyagainst tests green build green lint clean model says fine target self-consistent code checked against code Spec loop: same plan act verify, aimed through requirement, story, criterion, test, converges on intent. Spec layer LOOP AIMED AT INTENT plan act verify verifyagainst requirement story criterion test target intent code checked against what somebody asked for autonomy scales with the gate, not the model A loop with no target still gets somewhere. just not anywhere you chose build the target before you tune the loop

Autonomy scales with the gate, not the modelCopy link

The payoff is bigger than tidier verification. What everyone actually wants from a harness is a longer leash, more unattended hours per human hour. And the leash length isn't set by how clever the model is. It is set by the gate the work has to pass on the way out.

With a spec layer, review stops being a vibe read of the diff and becomes diff versus intent. Which criteria does this change claim to satisfy, do the linked tests prove it, does the chain still validate. That is a question a machine can ask on every commit, and a human can audit by sampling rather than by reading everything. When the gate is objective you can let the loop run longer, because iteration count stops being a risk multiplier. Every extra cycle is another attempt at a fixed target.

Without it, the same extra cycles work against you. More iterations against a proxy converge on the proxy faster and more thoroughly. You get software that is confidently wrong, with a fully green dashboard, which is my guard job again at the scale of a whole product.

Half a disciplineCopy link

So my read on the harness wave stays the same, if anything it has hardened. The loop mechanics deserve the attention they're getting, and the people doing that work are doing it well. But harness engineering without spec engineering is half a discipline. The half that makes the loop fast, without the half that gives it somewhere to go. My version of the other half is RCF, the Requirements Confidence Framework, the chain of small linked documents I described above, and the walkthrough series on this site shows it running on real builds. It is what has let me put real software into production with the loop running long. Whatever your version ends up looking like, build the target before you tune the loop. IMHO a loop with no target still gets somewhere. Just not anywhere you chose.

Barry