Vibe coding at enterprise scale: the review that never happens.Copy link

Vibe coding has stopped being a joke. The search interest keeps climbing year on year, the posts are everywhere, and the enterprise governance write-ups have caught up with their own version of the story. Read a few and the same sentence turns up in all of them, worded slightly differently. All AI-generated code is reviewed by a senior engineer before merge. Then, a paragraph or two later, usually in the risks section, the admission. Review capacity is the constraint. Senior engineers do not have the hours. The teams producing the most code are the teams with the least time to read it.

So the review is written into the policy and skipped in practice. Not maliciously. The agents produce code faster than a human can read it line by line, and the human line-by-line read was the control the policy relied on. The mature end of the industry knows this already; the orgs with proper QA functions and automation have been living with the gap for a year. What I have not seen anyone do is say plainly what happens when you stop skipping it. So, twice last week, on my own code, I did.

Some context for anyone landing here cold. I build software with AI coding agents full time and ship it into production. One of the internal tools I have been building runs a structured argument between two coding agents from two different vendors. One is briefed to attack a codebase and find defects. The other is briefed to defend it. A referee sits between them and only accepts a claim that arrives with evidence, a file and a line, a quoted clause, a reproduction. Where the two sides cannot settle a finding, a human adjudicates. It is rough, it is internal, and I am not selling it. What it gave me was a way to force the review that never happens onto my own work and watch what came out.

Six findings in four filesCopy link

The first target was a small tool of mine: four files that pass messages between two machines over a shared network folder. Send a message, acknowledge a message, check the other side is reachable, write the file without leaving a half-written one behind. It had been through my normal loop. An agent wrote it, the tests were green, and it had been in use between my machines without anything visibly going wrong.

The attacking side came back with six findings, and the defending side conceded all six. Two were high severity. The write path had no retry when two messages landed with the same name, so the second silently replaced the first and a message was lost. And the directory read that was supposed to spot those collisions swallowed every error into an empty list, which meant the collision check went blind the moment the mount misbehaved. The other four were the kind of thing a careful reviewer finds on a good day: an audit folder that overwrote its own history, a race between finding a message and moving it, a timeout guard that only covered one of the file operations, and temporary files created inside the inbox itself and left there on a crash.

Every one of those was fixed the same day. And tbh, I probably would have let all six through. A tool that is tests-green and works is not one I generally go back and read.

The review of the fix found defects in the fixCopy link

The second run, on the fixed code, is the part that changed how I think about this. Seven findings. Six conceded, and every one of those six lived inside the fixes for the first six. Two of the six were high severity. An error-cleanup path, added to tidy up after a failed rename, could not tell an ambiguous timeout from a real failure, and in the ambiguous case it deleted a message that had already been delivered. And the new collision-safe write reserved its filename with a zero-byte placeholder that readers could see in the window before the real content arrived, and that stayed there forever if the process died in between.

So the first review found six defects, the fix for those six introduced at least six more, and nothing in my loop would have caught the second set. Tests were green both times. That is what the policy line misses. It is not one review. Every fix is new code, new code needs the review again. Re-requesting it is a repo setting; the hours are not.

The seventh finding I overruled, after reading the code myself. The attacker claimed a file handle was being leaked, but it could only see the four files under review and not the helper module that closes the handle. It guessed, and it guessed wrong. That cuts both ways. A reviewer that cannot see the whole picture makes confident mistakes. So does a human with forty minutes and a policy to satisfy.

A concession without an argument is worth lessCopy link

Now the honest bit about the tool. In that second run the defending agent conceded six findings without producing a single piece of evidence of its own. It read the attack, agreed, and folded. That is a weakness in the setup, not a strength. A finding that survives a proper defence, where the other side went looking for a counter-example and failed to find one, is worth a lot. A finding the defender waved through is worth exactly as much as the attacker's evidence and nothing more. In this case the attacker's evidence was good, and I checked the two high ones by hand before I believed them. But I would not want anyone to read "six conceded" as "six proven". The referee demands evidence from the side making a claim. It does not yet demand a fight from the side giving way, and it should.

The same trick on the specCopy link

I ran the same shape against the tool's own design spec, one side interrogating it for anything an implementer would trip on, the other defending the document. Eight defects. Sections that contradicted each other on the same behaviour. A requirement no test could ever pass or fail. A rule ambiguous enough that two builders would ship two different things. Gaps where a decision had never been written down. Seven conceded, and one I upheld myself after checking it against a live run.

That run matters for the argument more than the code runs do. The code reviews worked because there was something concrete to review against: the files, the behaviour the tool was meant to have, and the second time round, the list of findings the fixes were meant to close. The spec review worked because the spec was written down where an agent could read it and quote it back. Take the written target away and two agents arguing about a codebase turn into two agents arguing about taste.

What I would ask before buying vibe coding for an enterpriseCopy link

Speed is not the problem. The tools produce code faster than any team can read it, and that only goes one way. The question for whoever signs the procurement is what the review step looks like at the volume the tool is about to produce, and what that review checks against.

I would go so far as to say the review you cannot staff has to be a review a machine can run, and a machine can only review against something written down. That is the spec-first position, and it is the same one I made last Monday about the verify step in an agent loop. My version of the written target, for product builds, is the RCF methodology: requirements broken into acceptance criteria with the tests hanging off them. Adversarial review with a referee is one shape the machine-run review can take. It is not the only one, and mine is rough and still being developed. But it found twelve real defects in four small files that had been through a green pipeline, and it found six of them in my fixes.

The review that never happens took two machines and a referee. Nobody had to find a senior engineer with the hours. I would rather a rough version of that review ran on every fix than a well-written policy sat in a folder describing a review that never runs.