Writing By Barry

How much do you trust human output?Copy link

Every AI failure mode has a human equivalent that predates large language models by decades. The common root cause is the one nobody wants to examine.

Can you trust AI output?

Serious question. Entire conferences are being organised around it. Governance frameworks. Audit trails. Trust scores. The industry has assembled a detailed taxonomy of AI failure modes and is busy building guardrails for every one of them. Hallucination. Drift. Confident fabrication. Context loss. Scope creep. The vocabulary is new. The behaviour isn't.

Because here's a better question. How much do you trust human output?

The failures you already tolerateCopy link

AI hallucinates. It produces confident, plausible answers that are completely wrong. This is treated as a damning indictment of the technology.

Meanwhile, every team I've worked on in thirty years has had at least one engineer who does the same thing. They don't know the answer, but pride or politics prevents them from saying so. They use the right words. They nod at the right moments. They deliver work that compiles and passes a cursory review and is quietly, structurally wrong. We don't call it hallucination when a human does it. We call it "getting away with it." And most of them do, for months or years, because nobody checks closely enough to notice.

AI drifts. The codebase grows, assumptions compound, and the model's understanding quietly diverges from the operator's intent. Again, technology problem. Except I've watched human teams do this on every project longer than three months. Six developers, six months in, three different mental models of the same system. Nobody notices because everyone is busy, and the divergence only surfaces when something breaks in production. The shared understanding wasn't maintained well enough. It rarely is.

AI is sensitive to prompts. Small differences in phrasing can produce wildly different outputs. We call that fragile. Unreliable. But this is just briefing. A team lead writes a requirement in twenty minutes between meetings. The developer reads it, interprets it, builds something. The team lead looks at the result and says "that's not what I meant." The words were the same. The readings were different. We've been living with this failure mode since the invention of the written specification, and we still haven't fixed it. We just blame the developer.

AI loses context. The session grows, earlier instructions drop out of the window, and the model starts making decisions that contradict what you told it an hour ago. Sound familiar? It should. It's every project that was crystal clear in the kickoff meeting and incomprehensible by sprint four. Nobody maintained the shared understanding. Nobody went back and checked the model (the mental one, this time) still matched reality. The context fell on the floor between meetings and nobody picked it up.

The operator problemCopy link

There's a pattern in all of this, and it's not the technology.

A human team falls apart when operated badly. Everybody accepts this. You recruit an engineer who turns out to be less capable than they seemed at interview. You communicate a requirement ambiguously. You fail to notice the team's shared understanding has forked into three incompatible versions. You don't check the work at the stages where checking would have caught the drift. Nobody blames the programming language when a team ships a bug.

With AI, we do the equivalent of all of this and then blame the model, or the coding agent. We stuff loosely correlated thoughts into a prompt and call it a requirement. We don't manage the context as the session builds. We don't verify at intermediate stages. We don't notice when the model's understanding has diverged from ours. And when the output is wrong, we write a LinkedIn post about how AI isn't ready.

I've watched a developer "improve" a design during implementation because they thought they knew better than the spec. Unrequested features, unrequested architectural decisions, baked in silently. When AI does this we call it scope creep and shout about it. When a human does it we call it initiative, right up until the production incident.

We've been deploying unreliable entities into production environments for decades. We used to call them under-managed teams. Or under-processed ones. Same failure mode.

The uncomfortable bit is this: we have collectively decided that an intelligence we've been working with for a couple of years should be held to a higher standard than the teams we've been failing to manage for thirty. AI is expected to infer intent from vague instructions, maintain perfect context across long sessions, never fabricate when uncertain, and never overstep its brief. We don't expect that from humans. We just accept that humans need management, structure, verification, and clear communication to produce reliable output. Then we hand an AI agent a half-formed prompt and wonder why the result is wrong.

Same failure modes. Different vocabulary. A two column comparison of five AI failure modes paired with their long standing human team equivalents. Each row matches one AI label on the left, signal outlined, to the equivalent human pattern on the right, neutral outlined, separated by an equals glyph. Hallucination pairs with getting away with it, the engineer who delivers confident structurally wrong work. Drift pairs with forking mental models, six developers and three incompatible understandings of the same system. Fragile to briefing pairs with the misread requirement spec, twenty minutes of writing read three different ways. Lost context pairs with sprint four amnesia, a project crystal clear at kickoff and incomprehensible by month four. Scope creep pairs with initiative, the developer who improves the design without asking. The footer states the load bearing point: both columns fail the same way under bad operation. The article framing is that every AI failure mode predates large language models by decades, the common root cause is the operator. AI failure: hallucination. Confident, plausible answers that are structurally wrong. HALLUCINATION confident, structurally wrong Human equivalent: getting away with it. The engineer who confidently delivers structurally wrong work because nobody checks closely enough. GETTING AWAY WITH IT confident, nobody checks AI failure: drift. The model's understanding quietly diverges from the operator's intent as a session or codebase grows. DRIFT understanding silently diverges Human equivalent: forking mental models. Six developers, six months in, three incompatible mental models of the same system. Surfaces in production. FORKING MENTAL MODELS six devs, three understandings AI failure: fragile to briefing. Small changes in prompt phrasing produce wildly different outputs. Called unreliable, but it is just briefing. FRAGILE TO BRIEFING small phrasing, big swing Human equivalent: the misread spec. Team lead writes a requirement in twenty minutes. Developer reads it, builds the wrong thing. Same words, different readings. THE MISREAD SPEC same words, different readings AI failure: lost context. Long sessions drop earlier instructions; the model starts contradicting what it was told an hour ago. LOST CONTEXT earlier instructions drop out Human equivalent: sprint four amnesia. Project crystal clear at kickoff, incomprehensible by sprint four. Nobody maintained the shared understanding. SPRINT-FOUR AMNESIA clear at kickoff, lost by month four AI failure: scope creep. Unrequested features, unrequested architectural decisions, baked in silently. SCOPE CREEP improves the brief uninvited Human equivalent: initiative. The developer who improves the design during implementation because they thought they knew better. Praised until it causes the production incident. INITIATIVE improves the design unasked
same failure · different vocabulary · same root cause

The one holding the reinsCopy link

Teaching is a useful lens here.

If a student consistently fails to learn, the first question a good teacher asks is not "what's wrong with this student?" The first question is "what's wrong with my teaching?" Am I being clear enough? Am I checking comprehension at the right intervals? Am I actually telling them what to fix, or just grading the result? The assumption, in any serious teaching practice, is that the teacher bears the primary responsibility for the student's understanding.

If an AI agent hallucinates, the first question shouldn't be "what's wrong with the model, or the agentic tooling?" It should be "what context did I give it, and was that context actually good enough?" Did I state the requirement clearly? Did I verify the model's understanding before it built the thing? Did I check the intermediate output, or did I wait until the end and then complain about the result?

The answer, in almost every real failure case I've observed over the last two years, is the same. The problem is the one holding the reins. The operator. The orchestrator. The person who set the task, defined (or failed to define) the constraints, and then walked away until it was time to judge the output.

I include myself in this, by the way. The first year I spent working with AI agents, I blamed the models constantly. Too flaky. Too confident. Too prone to wandering off brief. It took longer than I'd like to admit before I turned the lens around and asked whether my briefs were actually any good. They weren't. The quality of the output tracked the quality of the operation, not the capability of the model.

This applies to human teams and AI agents equally. It always has.

There are ways to operate AI well. They look suspiciously like the ways we were supposed to operate human teams all along.

Blurted out by Barry, refined by Dave.