Harness Engineering Has a Hole Where the Requirements Should BeCopy link

If you run coding agents in anger you'll have met the word harness by now. It means everything around the model that isn't the model: the instructions, the custom linters, the sensors, the feedback loops that let an agent catch its own mistakes before a human ever looks. Over the last year harness engineering has gone from informal practice to named discipline, and three write-ups define the current state of it.

OpenAI built an internal product with no manually written code at all: a million lines, roughly 1,500 pull requests, three engineers driving Codex, and throughput that went up as the team grew. Anthropic worked out how to keep an agent productive across context windows, where every session starts with no memory of the last. And Birgitta Böckeler at Thoughtworks pulled the whole area into a proper taxonomy on martinfowler.com: guides that steer an agent before it acts, sensors that catch it after, and a steering loop where humans tune both. All three are worth your time, and there's plenty in each I intend to steal.

The loop mechanics, the part everyone was nervous about two years ago, look close to solved. Agents write, review, fix and merge, with humans steering rather than typing. What I want to look at is a question all three pieces raise and none of them closes: what does the loop verify against?

Every sensor points at the codeCopy link

Follow the checks. OpenAI's harness enforces layer boundaries, naming conventions, structured logging and file size limits with custom linters, and validates dependency directions with structural tests. Böckeler's catalogue of sensors runs from type checkers and lint through test suites, coverage and mutation testing, up to AI code review. Anthropic's agents run the tests and then drive the app through browser automation, clicking round it the way a user would.

Each of those compares the code against something. And in every case the something is either the code itself, its types, its structure, its style, or an artefact derived from the code, usually tests the same agent wrote moments after writing the thing they test. A green run tells you the code agrees with itself. Nothing in the loop knows what the business asked for, because nothing in the loop has ever seen it.

The person who mapped the field says soCopy link

Böckeler is straight about this, to her credit. She splits harnesses into three regulation categories. The maintainability harness, structural code quality, is well developed. The architecture fitness harness is emerging. And the behaviour harness, whether the application functionally does what it should, is the one she calls "the elephant in the room". Current practice, as she describes it, is to feed a functional specification in at the front, of wildly varying detail, and check whether the AI-generated test suite comes out green at the back. Her verdict: "This approach puts a lot of faith into the AI-generated tests, that's not good enough yet." Her summary of the whole area is that "we still have a lot to do to figure out good harnesses for functional behaviour".

So the practitioners' own account of the discipline has maintainability solved, architecture in progress, and the question of whether the software does its job marked unsolved. That ordering should bother anyone signing off an agent-driven delivery.

An agent marking its own homeworkCopy link

The closest any of the three gets is Anthropic's feature list, and it's the detail I found most interesting in the whole wave. Their initialiser agent expands the user's prompt into a JSON file of feature requirements, over 200 of them for a claude.ai clone, every one marked failing on day one. Later sessions are only allowed to flip a feature to passing, policed by firm instructions that editing or removing entries is unacceptable. That is a requirements artefact, in the repo, that the loop closes on. Right shape, right place.

Now look at who wrote it. The agent did, by expanding a one-line prompt. It's marking its own homework against a mark scheme it also wrote. For a demo, where the prompt really is the whole requirement, that works. Point the same trick at commercial software, where the requirement is spread across a discovery workshop and the memory of a product manager who has since left, and the file the agent invents has no connection to any of it.

OpenAI's arrows point backwardsCopy link

OpenAI deserve credit here too, because they've had the right instinct and taken it further than most. Their repository carries product specs, design docs and execution plans, all versioned, on the stated principle that anything the agent can't reach in the repo effectively doesn't exist. That principle is the key to this whole subject, and I'll come back to it.

But watch the direction of enforcement. The mechanical checks guard architecture and taste. The documentation is kept honest by a doc-gardening agent that hunts for docs no longer reflecting the real behaviour of the code, and opens fix-up pull requests. When the description and the code disagree, the description is corrected to match the code. The spec is downstream of the implementation. And the verification that matters most is named plainly in their own write-up: humans translate user feedback into acceptance criteria by hand, humans validate outcomes, and the bottleneck on the whole operation is human QA capacity. The one check the harness can't run is the one that decides whether the product is any good.

A sensor needs something to point atCopy link

None of this is a complaint about the sensors. It's what sensors are. A check compares the system to a reference, and it can only use a reference it can reach. Requirements, in most organisations, live nowhere a loop can reach: closed tickets, chat threads that scrolled away, and whatever the people in the room still remember. The repo holds the code and the tests, so the loop closes on the code and the tests, and everyone is then puzzled that functional behaviour is the unsolved harness. The checking isn't the hard part. The reference was never written down where a checker could see it.

So the missing harness component isn't a better sensor. It's an artefact. The requirement has to make it into the repository in a form a loop can use: versioned with the code it justifies, atomic enough to carry an ID, phrased so a check can pass or fail against it, and linked to the tests that prove it and the code that implements it, tightly enough that a validator can walk the links and fail loudly when one is missing. Prose in Confluence meets none of those. A well-meaning requirements.md meets one.

What I do about itCopy link

This is the problem I've spent the last year building against, so I'll declare the interest and keep it brief; the hole is the point of this piece, not my way of filling it. My answer is RCF, the Requirements Confidence Framework. It keeps a chain of small documents in the repository: product requirements broken into atomic requirements, those into user stories, each story carrying acceptance criteria written as Given/When/Then. Everything has an ID. Test suites are generated from the acceptance criteria, one suite per criterion, before implementation, and the code that implements a criterion links back to it. Tooling validates the chain the way a linter validates style: a criterion with no test suite is a reported failure, and so is a link that no longer resolves.

The agent's loop still runs tests and lint. The difference is what the tests are downstream of. When they descend from the criterion rather than from the code, a green run finally asserts something about the requirement, and the trace runs both ways: failing test to criterion to requirement, or requirement to everything claiming to satisfy it. It's put real software into production for me over the past year. RCF is one way to build the artefact, and I'd be surprised if it ends up the only one.

The harness wave has done the industry a favour, and the honest ones say where the edge is. Böckeler's unsolved behaviour harness isn't waiting for a smarter sensor. It's waiting for the requirement to turn up: in the repo, versioned, checkable, with the same standing as a failing test. Until then the loops will keep getting faster at proving the code agrees with itself. IMHO that's the next job in agent-driven delivery, and it belongs to whoever owns the requirements. Assuming anyone still does.