Blog By Barry

The knowledge library behind a digital operatorCopy link

My digital operator runs across a dozen surfaces and has to remember what it learns. Not in a chat log, not in a vector blob. Here is the library I built for it, and where I stole the shape from.

I've spent the last year building a digital operator: an AI system that runs my estate, drafts under my name, dispatches work and generally acts on my behalf across a dozen surfaces. The thing nobody warns you about when you start is that the model is the easy part. The hard part is memory. An operator that can't remember what it learned yesterday is just an expensive autocomplete you have to re-brief every morning.

So the build that's taken the most care isn't the agent. It's the library behind it. This is how it actually works, and where I borrowed the shape from, because I borrowed most of it.

Markdown is the truthCopy link

Start with the decision everything else hangs off. The knowledge is markdown files, and the markdown is the source of truth. Not a database, not an index, not embeddings. Plain .md files in a git repo, sorted into domains: engineering, systems, writing, the operator's own interface preferences, and so on. One idea per file where I can keep it that tidy. An H1 for the title. Human-readable, git-diffable, and reviewable by me in an editor when I don't trust what the machine wrote.

I didn't invent this. The first push is the pattern Andrej Karpathy popularised as the "LLM wiki": a knowledge base written for a language model to read and maintain, small cross-linked pages with an index the agent reads first, kept current by the agent itself instead of rotting in a drive nobody opens. The second is Google Cloud's Open Knowledge Format, OKF, which landed in June and formalises that same pattern into a portable spec: a directory of markdown files with YAML frontmatter, deliberately minimal, vendor-neutral, "format not platform". When Google ships the thing you already built, you feel slightly less mad for having built it.

The older lineage is just the wiki. Cross-links between pages, the [[double-bracket]] style Obsidian and Roam readers will recognise, and a backlinks index so every page knows what points at it. None of that is new. What's new is who's reading it. The pages aren't for a human who'll click around at leisure. They're for an agent that has to land on the right one, fast, mid-task.

The graph is derived, not authoredCopy link

Here's the part I'm happiest with. I don't hand-maintain a knowledge graph. I generate it.

A nightly pass reads every markdown file, pulls the links out (the [[wiki]] ones and the ordinary [text](path.md) ones), and rebuilds three derived indexes: a domains list, a backlinks map, and the graph itself as plain nodes and edges in JSON. The markdown is the truth; the graph is a projection of it. Delete the graph and I've lost nothing, the next rebuild reconstructs it whole. That's the trick to keeping it honest. There's nothing I have to remember to update by hand, because the thing that rots is always the hand-maintained copy.

Retrieval runs over that. There's a small CLI, pa knowledge, and the operator uses it the way I'd use a good index. search for a term. related for the one-hop neighbourhood of a doc. path for the shortest hop between two. orphans to find pages nothing links to. domain to pull everything in an area.

I'll be straight about what that retrieval is, because it's fashionable to overclaim here. It's keyword search and graph navigation. It is not vector RAG, and there are no embeddings anywhere in it. That's a deliberate call, not a corner I haven't got to yet. Explicit links and an index the model follows like a human would are legible in a way semantic-similarity chunks are not. I can look at why a search hit or missed. I can't easily look at why a cosine distance did. For an operator I actually have to trust, legible wins. RAG can compose on top later if I want it; it doesn't get to be the foundation.

Staging, then distillationCopy link

New knowledge doesn't get written straight into the library. That would let any half-formed observation corrupt the canonical set. Instead, agents drop raw observations into a staging area as they work. Staging is a holding pen, not the library.

Then the librarian sweeps it. On every sweep, each staged observation gets two questions asked of it, and this is the real judgement work. What is it: an operator preference, a piece of domain knowledge, an operational rule, a one-off worth nothing? And who should see it: private to me, shared with a team, or general enough to promote across the company? The first question decides where it lands. The second decides how far it travels.

The rule I care most about is this: synthesise, don't append. The lazy move, the one every system drifts towards, is to bolt the new line onto the bottom of an existing page. Banned. If an observation refines what a page says, the librarian rewrites the relevant section so the page still reads as one coherent thing, and references the observation's ID in the commit so I can trace where it came from later. A knowledge base that only ever grows by accretion isn't a library. It's a landfill with good search.

And no change reaches the library unchecked. Every one, from every agent, arrives as a pull request first, the same review-before-merge you'd run on code, and only clears the merge gate once it's been scanned and tested. Each agent writes with its own scoped deploy key, one key each, so a write can only touch the area that agent owns, and every change carries a name. The library maintains itself, but under the review-and-audit discipline I'd demand of any code.

The operator knowledge library: the record, the librarian, and the gates A record-centred diagram of the knowledge library behind a digital operator. The hero is a large central card, THE LIBRARY: canonical markdown, the single source of truth. Inside it sit three shelves, one per canonical repository (team A's team library, team B's team library, and a company-wide shared commons), plus the operator's own profile and private vault. Below the shelves, inside the same card, is THE LIBRARIAN: one tireless agent that is the only writer to the shelves, drawn as the five roles it performs: sweep and distil (read each team's new notes and rewrite the pages, synthesising rather than appending), fast-lane corrections (your fixes jump the queue and land in minutes), promote across teams (what everyone should know moves up to the shared shelf), admit and merge (the gate that checks incoming notes and lets the good ones in), and nightly housekeeping (rebuild the index and graph, find orphans and broken links). On the left, agents drop raw observations into a dashed staging pen, a holding area where nothing is canonical yet; each agent writes only its own shelf, with its own scoped key. Those notes flow into the library through a merge gate: they arrive as a pull request, are secret-scanned and tested, and only passing notes are admitted, and failures bounce back. On the right, the canonical shelves flow out and are mounted onto every device, always the latest canonical copy, and only the shelves a device is entitled to; from there the operator reads the library through a small tool, pa knowledge, with search, related and path, over a graph that is derived and rebuilt nightly rather than hand-authored. Supporting cast along the bottom: the librarian runs on a schedule, ticking every few minutes, with housekeeping nightly, and one agent is deployed per device. Operator knowledge library markdown = the record Raw in · the holding pen Staging. Agents drop raw observations here as they work: a holding pen, not the library. Nothing here is canonical until the librarian has swept it. STAGING agents’ holding pen · raw notes one agent per team · own shelf only nothing here is canonical yet The merge gate. Raw notes arrive as a pull request; they are secret-scanned and tested, and only passing notes are admitted to the library. Failures bounce back for a human. gate ① The merge gate notes arrive as a pull request secret-scanned & tested on the way in only passing notes admitted · the rest bounce back ② One key each scoped write keys every writer holds one key, for one shelf only each agent → its own team’s shelf the librarian → the canonical shelves The library. Plain markdown files in canonical git repositories: the single source of truth. Every mount, index and graph downstream is derived from it. THE LIBRARY canonical markdown · the single source of truth The shelves: the record The shelves. Three canonical library repositories: one per team plus a company-wide commons. One team maps to one shelf; a team cannot read another team's shelf. library-team-a team A’s shelf library-team-b team B’s shelf library-company shared commons + operator profile & private vault The librarian. One tireless agent, the only writer to the canonical shelves, via its own per-repository keys. It works inside the library, performing five roles. THE LIBRARIAN the only writer to the shelves · works inside, role by role SWEEP & DISTIL reads each team’s new notes, rewrites the pages: synthesise, not append FAST-LANE CORRECTIONS your fixes jump the queue: live in minutes, not hours PROMOTE ACROSS TEAMS what everyone should know moves up to the shared shelf ADMIT & MERGE the gate: checks incoming notes and lets the good ones in NIGHTLY HOUSEKEEPING rebuild the index & graph, find orphans & broken links one agent · never bored, never off Legible out · onto devices Mounts. The canonical shelves are materialised onto every device: always the latest canonical copy, read-only, and only the shelves that device is entitled to. MOUNTS ONTO EACH DEVICE always the latest canonical copy only the shelves it’s entitled to read-only · never writes back What the operator reads. A small CLI, pa knowledge, with search, related and path, over a graph that is derived from the markdown and rebuilt nightly, not hand-authored. Keyword search and graph navigation, not vector RAG. WHAT THE OPERATOR READS pa knowledge: search · related · path the graph is derived, rebuilt nightly markdown stays the truth The operator knowledge library: the record, the librarian, and the gates (narrow layout) A vertical rendering. Agents drop raw notes into a dashed staging pen. The notes pass down through a merge gate (arriving as a pull request, secret-scanned and tested, only the passing ones admitted) into the hero card, THE LIBRARY: canonical markdown, the record, holding three shelves (team A's team library, team B's team library, and a company-wide shared commons) with THE LIBRARIAN working inside it, decomposed into five roles: sweep and distil, fast-lane corrections, promote across teams, admit and merge, and nightly housekeeping. Each writer holds one scoped key for one shelf only. Below, the canonical shelves are mounted onto every device, always the latest canonical copy and only the shelves it is entitled to; the operator reads them through pa knowledge over a graph that is derived and rebuilt nightly. Operator knowledge library md = record Staging: raw observations, a holding pen. Each agent writes only its own shelf. STAGING agents’ holding pen · raw notes one agent per team · own shelf only nothing canonical yet ① merge gate arrives as a PR · scanned + tested · only passing admitted The library: canonical markdown, the record. Shelves are canonical repos; the librarian works inside. THE LIBRARY canonical markdown · the record The shelves library-team-a team A’s shelf library-team-b team B’s shelf library-company shared commons + operator profile & private vault THE LIBRARIAN the only writer to the shelves · works inside SWEEP & DISTIL new notes → rewritten pages FAST-LANE FIXES your fixes, live in minutes PROMOTE shared knowledge moves up ADMIT & MERGE the gate on the way in HOUSEKEEPING nightly: index, graph, orphans never bored, never off ② one key each each agent → its own shelf · librarian → canonical Mounts: the canonical shelves materialise onto each device, always the latest canonical copy, entitled shelves only. MOUNTS ONTO EACH DEVICE latest canonical copy · entitled only read-only · never writes back What the operator reads: pa knowledge over a graph derived from the markdown, rebuilt nightly. WHAT THE OPERATOR READS pa knowledge: search · related · path graph derived, rebuilt nightly
raw notes → the merge gate → the library the librarian tends → mounted onto every device

The librarian never gets boredCopy link

The thing doing all this is its own agent. I call it the Librarian, and it runs on a schedule rather than on demand, in scoped passes. Hourly sweeps over the active agents. A fast lane every fifteen minutes for corrections I've just made, because feedback I gave five minutes ago shouldn't have to wait four hours to take effect. A slower cross-team pass that judges what generalises. And a nightly housekeeping pass that does the un-clever mechanical work: rebuild the indexes, rebuild the graph, hunt for orphans and broken links.

That last part matters more than it sounds. The reason to hand this to an agent rather than a plain script isn't cleverness, it's stamina. As the OKF write-up puts it, LLMs don't get bored, don't forget to update a cross-reference, and can touch fifteen files in one pass. That's the whole case. The drudgery of keeping a cross-linked library consistent is exactly the drudgery a human stops doing by week three.

And because I don't trust a system I can't measure, there's a retrieval eval that runs nightly too. A fixed set of queries phrased the way the operator actually phrases them, scored for whether the right document comes back first, or at all in the top five. Last night it had hit-at-five sitting around 85%, and it was also shouting at me that orphan pages are climbing and a handful of domains have drifted out of the graph entirely. Which is the point. The library is not pristine and I won't pretend it is. What I've got instead is a number, every morning, telling me where it's slipping, so I hear about the drift from the eval rather than from an operator that confidently retrieved the wrong thing.

Where this leaves itCopy link

Markdown as the truth. A graph derived from it, never hand-authored. Retrieval that earns trust by being legible before it's clever. Raw observations staged, then distilled by an agent that synthesises instead of piling up. And an eval that keeps me honest about how well any of it is working on a given day.

None of the individual pieces are novel, and I'd rather say so than dress it up. The LLM-wiki crowd, the OKF spec, thirty years of wikis all got there first. The point worth making is that the boring version works. You don't need a vector database and a reranker to give an operator a memory. You need plain files, a librarian with no ego and no fatigue, and the discipline to measure whether the thing can still find what it knows.

Blurted out by Barry, refined by Dave.