Engineering · Architecture · Postgres
Event sourcing for AI agent runs: Postgres is the truth
Event sourcing for AI agent runs: an append-only Postgres log, one translation for every engine, a one-transaction finalize, and a UI that only reads.
On this page
An agent run in UseAgent can last a few seconds or a few hours. In that time the browser tab reloads, the backend gets redeployed, and once in a while the sandbox dies under the agent. If the screen shows a live process, every one of those events shows the user something different, or nothing. Event sourcing for AI agent runs was our answer: the run is an append-only log in Postgres, and everything else, the web timeline, the Slack reply, the bill, reads from it.
We got there the hard way. Our first version rendered the engine's own session UI, and the control plane post covers why that broke. This one is about what replaced it.
Event sourcing for AI agent runs means persist before publish
Four kinds of rows carry a run:
| Table | What it holds |
|---|---|
| runs | one row per turn: status, engine, model, thread, summary, timings |
| commands | the durable per-thread mailbox: start, reply, steer, stop |
| steps | the durable tool and message timeline of a run |
| provider_events | the engine's raw native frames, with stable ids and per-run sequences |
Plus canonical_events, the translated, engine-neutral version, which gets its own section below.
The rule for all of them is persist before publish. A frame is written to Postgres first, then pushed to the live bus. A browser that connects late, or reconnects after a network blip, replays from the database and then follows the bus. Live and replay are the same data, deduped by id. We wrote up the streaming side in one SSE stream per thread.
Commands are the other half. Every channel, the web composer, Slack, schedules, skills, enters through one function that accepts a run command. That gives each thread a FIFO lane with one dispatched command at a time, idempotency keys derived from the channel's own stable event id, cancellation, and recovery at boot. When the backend restarts, boot recovery reconciles in-flight runs from these rows. That's also why we never point a second backend at a shared database: its boot would reconcile the first one's runs.
One grammar for every engine
Claude Code, Codex and OpenCode each speak their own event dialect. OpenCode has sessions, messages and parts; Claude Code and Codex each have their own shapes. If the UI understood all three, every engine release would be a UI change.
So the backend translates each engine's native events into one canonical vocabulary. Every canonical event carries the same envelope: a schema version, a stable event id per run and native event, our own sequence, the run and thread, a timestamp we assign rather than trust from the engine, and the native identity (session, message and part ids) for correlation. Large payloads like tool output and diffs are stored out of band with a hash and a byte count, and the row keeps a bounded preview.
Two rules made the translation trustworthy.
It's lossless and fully accounted. Every source frame gets a disposition: mapped, or suppressed with a named reason. No silent continue. On the first big OpenCode fixture that meant 1,188 frames in, 911 frame events out, zero unmapped.
It's proven against the old renderer, not against itself. We built the canonical timeline next to the existing one and required them to match on a golden fixture from a real run: 281 tool nodes from the old path, 281 from the canonical path, the same kind and key sequence, then full deep equality of every timeline node. The fixture is sanitized before it's committed, and the sanitizer has a negative control: smuggle the prompt into a frame and it fails with SANITIZATION FAILED. A guard in CI checks every committed fixture string against a controlled vocabulary, so customer data can't sneak into test data.
Delivery order is not run order
The canonical table separates two sequences. Each run has its own seq. The thread has a deliverySeq, an append-only bigserial that is never reused. The browser resumes from deliverySeq.
An early version used one sequence for both and mutated rows when a revision came in. A browser that had already seen a cursor would never see the revision. The fix was to make revisions append (revision plus one, new delivery cursor) and to publish on the thread channel instead of the run channel. A test proves it: reconnect from run one's final cursor and you get every event of run two, a run created live after the first one.
Finalize is one transaction
Settling a run touches a lot: status, the sandbox lease, memory capture, the Slack reply, automation delivery, the canonical translation, learning, the member's spend. If those happen one by one, a crash between any two leaves a run that says completed while its Slack reply never goes out, or its spend never gets charged.
So finalize is one transaction:
drain in-flight provider event captures (usage lands before the charge is read)
BEGIN
lock the run row
if a Stop was recorded: the run fails as stopped
completeRun(...) where status in (queued, running) -> 0 rows: someone else won, do nothing
release the sandbox lease, accrue sandbox minutes
enqueue memory capture (completed, non-internal runs only)
enqueue the Slack reply and any automation delivery
enqueue the canonical translation intent
enqueue the learning intent
accrue the run's spend (last: it takes the member's row lock)
COMMIT
kick the outbox workersStop takes the same row lock, so Stop and finalize are linearized. The first finalizer wins: a zombie cancel racing the reconcile loop updates zero rows and skips every side effect, so it can't flip a status or double-enqueue anything. A transient database error retries the whole transaction with backoff, which is safe because every step is idempotent and the transaction commits whole or not at all.
The side effects are outbox rows, not calls. Workers pick them up after commit. Slack delivery retries with jittered backoff and dead-letters. If the process dies a millisecond after commit, the rows are there when it comes back.
Never publish provisional rows
The canonical translation runs off an outbox too, and this was the part with the most bugs.
The worker claims an intent with FOR UPDATE SKIP LOCKED, reads a watermark of the source (the highest native frame plus a signature of the steps), translates, and reads the watermark again. If anything moved, it retries instead of freezing a partial translation. The step signature started as count(*) and became an md5 over each step's id, index, kind and content, because tools rewrite a step in place when the result arrives, and a count doesn't change when a tool call becomes a tool result.
The bigger fix was about what clients can see. Our first version wrote provisional canonical rows and marked them complete later. A crash and retry could leave stale rows that a browser had already trusted. Now the rows and the outbox's complete mark commit in one transaction, and the live publish happens after commit. The canonical table never holds a provisional row. The browser, for its part, trusts a run's canonical events only when the durable completion record says so, never because the list is non-empty.
That lane fixed a real gap. Claude Code threads used to reload to an empty timeline because the old renderer understood only OpenCode's events. A real Claude Code run through the canonical lane rendered correctly and stayed correct on reload. In the React app the canonical timeline still sits behind a flag, and the native lane remains the default render path while we finish moving over.
What we capture per run
The log isn't only for rendering. On the hosted app we keep everything we need to answer "what did the model actually see and say":
- the composed prompt
- the engine's own session store, harvested after every turn, thinking included
- the repo diff per turn
- every model call through our gateway, raw: the request, headers minus credentials, every SSE byte
- the backend's own model calls (chat, summaries, embeddings)
- runs settled by recovery after a deploy
All of it goes through one writer. It scrubs known secret values from the run's secret set before anything is stored, and when it can't load the full set, it stores the content anyway and marks the scrub partial instead of dropping data. There's a per-run ceiling of 5 GB and a free-disk floor checked per chunk. The gateway's database role can only insert into those tables, and there's no read API. Accounting never depends on capture: a capture failure can't change what a run costs.
Files work the same way. A PDF the agent wrote used to go missing when the model forgot to call the publish tool. Now the control plane lists the workspace after every turn and publishes what changed. It finds changes by inode change time, because cp -p keeps the old mtime and nothing in user space can set ctime, and it reads the listing as NUL-delimited records so a file name can't forge a record.
The cost
One backend per database. The live fan-out and the drain barrier before finalize are in-process, and we enforce it with a Postgres advisory lock at boot. It's a real ceiling and we chose it on purpose: the log is distributed-safe, the realtime layer on top of it isn't yet.
The run model, the outboxes and the canonical translation are in the open-source repo; backend/src/runs/finalize.ts is the place to start.
Frequently asked questions
Why use event sourcing for AI agent runs?
Agent runs outlive browser tabs, deploys and sometimes the sandbox itself. If the durable log is the source of truth, a reload, a reconnect or a restarted backend rebuilds exactly what happened instead of guessing from a process that may be gone.
How do you show a live agent run if the UI only reads the log?
Events are persisted first and then published on a live bus, and the browser holds one stream that replays from the database and then follows live frames, deduped by id. Live and replayed views are the same data.
What happens if two processes try to finalize the same agent run?
The first one wins. The terminal write is guarded on the run still being queued or running, so a second finalizer updates zero rows and skips every side effect, which prevents double delivery and double charges.