Skip to content
UseAgent

Engineering · Realtime · Architecture

Streaming AI agent responses with SSE: one stream per thread

Streaming AI agent responses with SSE broke for us on every reply. The fix: one thread-scoped stream instead of one per run, with Postgres as the truth.

Abhishek6 min read
On this page

Send a second message in a thread and, for about a second, the tool rows from the previous turn went blank. Then they came back. Separately, a reply started from Slack showed up in Slack instantly but took up to five seconds to appear in the web app. Both bugs came from the same decision about streaming AI agent responses with SSE: we had one stream per run, and a conversation is many runs.

This is how we replaced it with one stream per thread, and why the stream is only a window onto the event log in Postgres, never the truth itself.

One run per turn, one stream per run

In UseAgent a thread is a conversation and every turn in it is a run. Runs are the durable unit: they're queued, admitted, executed by an engine in a sandbox, and settled. Their steps and the engine's native frames are written to Postgres before anything is published live.

The session page used to subscribe to whichever run was active. It looked reasonable. Here's the chain it actually produced, as we wrote it down in the design doc for the fix:

text
thread gains another run
  -> active/newest run selection changes
  -> the stream hook receives a different run id
  -> the native store resets and the EventSource reconnects
  -> the page tries to keep prior state in a projection cache
  -> polling and refetch race the live state
  -> tools, text or status disappear, freeze, or target the wrong turn

Every link in that chain had a stopgap. Active-run selection logic. A projection cache to hold the old run's state while the new stream connected. A refetch when a run reached a terminal state. A five-second poll to discover turns started outside the browser, which is why Slack-started replies lagged. Each stopgap fixed one symptom and added a race with the others.

The flash on reply was the cache losing that race. The new run arrived, the subscription switched, the store reset, and the previous turn rendered empty until the cache caught up.

Streaming AI agent responses with SSE, one connection per thread

The fix was to stop switching. The page opens one EventSource for the root run of the thread, for the whole page lifetime:

text
GET /api/runs/:rootRunId/thread-events

That endpoint multiplexes every run in the thread onto one connection. Every frame names its run:

Frame Carries
snapshot the full thread from durable state
run one run upserted: new, queued, running or settled
step a durable step, same index may arrive again enriched
delta transient live narration
native a versioned engine frame, deduped by event id and sequence
done settles one run and does not close the stream

There are also canonical frames for the provider-neutral event lane, covered in event sourcing for agent runs.

The important design call: this route is not a second execution system and not a new source of truth. It reads the existing durable sources (runs, steps, stored native frames) and relays the existing live buses (the worker's step bus, the delta stream, the native bus). Postgres stays authoritative. The stream is an aggregation over it.

Subscribe first, then read the snapshot

There's a window between "read the thread from the database" and "start listening for changes." A run accepted inside that window is invisible. So the order is fixed:

ts
// 1. Listen for thread changes BEFORE loading anything.
const unsubscribe = subscribeThread(threadId, (change) =>
  push({ type: "signal", runId: change.runId }),
);

// 2. Load the authoritative thread and attach each run's live sources,
//    so frames produced during the replay queue up instead of vanishing.
const thread = await getThreadForRun(orgId, rootRunId);
for (const run of thread) attachRun(run.id);
sendFrame("snapshot", { threadId, runs: thread });

// 3. Drain the queue. A signal re-reads durable state for that run.

The signal handler only enqueues. It never trusts the payload. When the drain loop sees a signal, it re-reads that run from Postgres, checks the run belongs to this thread, and only then attaches its live listeners.

That ordering inside the signal handling came from review. An early version attached the live listeners first and checked thread membership after. A signal carrying a run id from another thread would withhold the durable frame correctly but leave step, delta and native listeners bound to the wrong run. Authorize, then attach.

The thread-change signal itself comes from one place. Every channel that starts work, the web composer, Slack mentions, schedules, skills, goes through the same command acceptance function, and that function publishes the signal after the transaction commits. A turn started from Slack shows up in an open browser tab right away, and no channel needs its own UI wiring. The Slack post has the other half of that.

Dedupe everything, reset nothing

Replay and live overlap by design, so every lane has a dedupe key:

  • steps by run, index and a fingerprint of the step's content. A step whose tool output gets attached in place passes again; a true duplicate doesn't.
  • native frames by event id, keeping only a higher sequence
  • canonical events by a delivery cursor, an immutable thread-wide sequence that never gets reused

On the browser side, the thread store is a pure reducer. Applying a snapshot merges into what's there and never resets it. Adding, queueing, starting or settling a run touches only that run's slice. No run switch can blank another run's tools, because there is no run switch anymore.

We deleted the stopgaps in the same change: active-run selection, the projection cache, the terminal-state refetch and the five-second poll.

One performance trap showed up later. Opening a long settled run replays hundreds of native frames at once. Notifying React per frame meant a full timeline rebuild per frame, and a 600-frame run froze the tab for minutes. The store now takes a batch: apply the burst, notify once, paint once.

Reloads still double-render for a moment, the server-rendered thread and then the SSE replay of the same turns, and it settles by merging on id. Any test that counts turns has to wait for the settled, distinct count.

Backpressure and the boring headers

A browser tab that stops reading would otherwise make the server buffer forever. Each connection has a queue capped at 20,000 frames. On overflow the server closes the connection, and the browser reconnects to a fresh authoritative snapshot. Closing is cheaper than guessing which frames to drop, because the snapshot is always correct.

The rest is plumbing every SSE endpoint needs:

http
Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

Plus a comment line every 25 seconds as a heartbeat, so proxies don't decide the connection is idle. The route resolves and authorizes the root run before opening the stream. A run from another org is a 404, the same as a run that doesn't exist, and the thread id is derived on the server, never taken from the browser.

What we proved, and what we didn't

The cutover shipped with 12 route tests, 6 reducer tests and a real-Chrome proof that passed 15 of 15 checks against an isolated throwaway database. That proof used a mock engine. The real-OpenCode leg wasn't run in that change, and we said so at the time. The path is engine-agnostic, and a later end-to-end run with a real OpenCode sandbox on the canonical lane passed 18 of 18, including reload and reconnect.

We kept the old per-run endpoint for a while, so rolling back was one import.

The real limit is deployment shape. The live buses are in-process event emitters. A second backend wouldn't see the first one's frames, so production runs exactly one backend per database, and boot takes a Postgres advisory lock to refuse a second. If we ever need two, the buses move to something shared. We haven't needed it.

The stream code lives in the open-source repo, under the runs routes and the chat thread store.

Frequently asked questions

Should I open one SSE connection per agent run or per conversation?

Per conversation. A per-run connection has to be torn down and reopened whenever a new run starts, and anything rendered from the old one is at risk during the switch. One thread-scoped stream that tags every frame with its run id avoids the switch entirely.

How do you avoid losing events between loading a snapshot and opening the live stream?

Subscribe to the live signal before you read the snapshot, queue anything that arrives, and dedupe by id when you drain the queue. A run accepted mid-load is then discovered instead of missed.

What happens when an SSE client falls behind?

We cap the per-connection queue at 20,000 frames. On overflow the server closes the connection and the browser reconnects to a fresh authoritative snapshot, rather than the server buffering without limit.

All posts →

Hand your next task
to a coding agent.

UseAgent runs Claude Code, Codex and OpenCode in cloud sandboxes. The core is open source, so you can read the code or host it yourself.