On this page

Related reading

Blog By Barry

One agent, three powers, no gate.

I approved an overnight agent design in principle that put three powers in one session without a human gate. An adversarial review caught the problem before we built any of it. Meta's Agents Rule of Two helped us redraw the boundary.

We designed an agent that would wake up at 04:00, read what other agents had filed, and decide what to do about it without asking anyone.

It was one scheduled job for a shared repository. Colleagues' own agents would file issues, and our agent would work through them overnight. It had code hosts and an issue tracker in reach, and permission to write wherever it decided it should. There was no human anywhere in that loop.

The three powers were untrusted input, sensitive access, and state change or external communication. All three sat in one session. An issue written by a colleague's agent is untrusted input by definition. Anything that agent writes, an attacker could influence. Knowing the colleague doesn't establish where every sentence in the issue came from.

I approved this in principle for a closed beta with a handful of colleagues. After thirty years in engineering, I still let the small audience make the permission problem feel smaller. The convenience was easy to picture. I hadn't given the combination of permissions the same attention.

An adversarial review stage caught it before a line of it was built. That's the evidence we have. The usual defence, "it has never done any damage", is a bet, not a control. Our plan hadn't run once, so it couldn't even offer that much.

Meta's rule made the split explicit

Meta's Agents Rule of Two: A Practical Approach to AI Agent Security gave me the words for the split we should have drawn at the start.

The rule says an agent should combine at most two properties in a single session, choosing among processing untrustworthy inputs, accessing sensitive systems or private data, and changing state or communicating externally. If the task needs all three without a fresh session, Meta says autonomy must give way to supervision, with human-in-the-loop approval or another reliable form of validation as the minimum.

In our nightly design, the issues supplied the untrustworthy input, the code hosts and issue tracker supplied the sensitive access, and the permission to write wherever the agent chose supplied the state change and external communication. All three were in the one proposed job, and I could point to each of them.

This is Meta's heuristic for prompt injection, an unsolved problem. Applying it doesn't prove that an agent will behave correctly. It does give me a concrete way to examine which permissions I'm combining. Our design needed that examination before anyone started implementing it.

What we changed the design to

We changed the design to an hourly read-and-recommend loop, a human approval for each item, and a separate lane for the approved work. The hourly loop would read new issues, classify them, do cheap first-party verification and maintain a daily triage list. Its only write would mirror a status onto issues already linked upstream. That leaves two powers in the autonomous loop. It reads untrusted input, and the only thing it can write is one status word picked from a fixed list of five, so nothing it writes can smuggle text or a command anywhere. The broad access to sensitive systems would stay out of that loop. Those limits need to be enforced by the permissions and the write interface when we build it.

Then the operator would read a page and approve each item as one specific action: file this upstream with a link back, create this ticket, close that one with this reason. A separate, narrowly scoped runner would carry out only the approved action. It would never see the raw issue text, only the instruction I approved. Meta says that when a task needs all three properties, autonomy gives way to supervision, and here the approval is that supervision. It does more than sign off on the job. It replaces free text an attacker could have influenced with one bounded instruction on a named target. The runner holds sensitive access and can write out, but the untrusted input never reaches it. Two powers, not three, because the approval turned the third into an instruction. No session in the workflow holds all three. In this design, approval is the gate, not the log afterwards. Today this is a design. The gate is in it by construction, and whether it holds will be proven when it is built.

No session holds all three powers Three named boxes. Across the top, a box named Overnight session holds all three powers side by side, untrusted input, sensitive access and external writes, and is marked no gate. Below it, two equal boxes face each other. On the left, the Hourly loop, read and recommend, holds untrusted input and a fixed status write from a closed set of five values. On the right, the Post-approval lane, one approved action, holds two powers, sensitive access and external writes, and one approved instruction, a card with a tick; the untrusted input never reaches it. Between the two boxes a gate named Human approval stands on the only path from the loop into the lane. The drawing is a design, not a measure of scale or time. No session holds all three powers Overnight session No gate ? Untrusted input Sensitive access External writes Hourly loop Read and recommend ? Untrusted input Fixed status write Closed set of five values Post-approval lane One approved action Approved instruction Sensitive access External writes Humanapproval

The same gate applies to code

A reader of the trust piece back in June said the same argument applied to agents that send things on your behalf; he was right, and it took an agent of our own to show me where the line actually goes.

The send log and the build log are both the wrong instrument for deciding whether an action should proceed. They tell me what happened, and by then the thing that matters has already been touched. I still want the records. I just can't ask a record of an action to supply the permission that should have preceded it.

I need to decide what good looks like before the run and put the gate before the thing that matters, whether that's a merge, a customer, or an issue tracker my colleagues rely on. For this design, that means being able to read the proposed action and judge it before granting the agent the authority to carry it out. A general willingness to try an overnight job doesn't do that work.

Last month I wrote about what the record can say; this is about what the agent is allowed to do.

My scepticism about AI code and my scepticism about AI agents acting are the same scepticism, and the fix is the same gate.

This general scepticism is why I created the RCF methodology, where putting the gate before the thing that matters is written down.

Barry