Skip to content
UseAgent

Engineering · Deployment · Reliability

Deploy long-running AI agents without downtime: what broke

How we deploy long-running AI agents without downtime: admission drain, runs that survive a restart, a 3.5-minute outage, Caddy POST retries, deploy batching.

Abhishek8 min read
On this page

A coding agent run on UseAgent can go for 25 minutes or more, and a person can leave a question card open for as long as they like. Our first blue/green promote drained for ten seconds and swapped, which kills both. This is how we now deploy long-running AI agents without downtime, what we tried first, and the 3.5-minute outage we caused on the way.

Some context. Production is one host with two colors, blue and green, behind Caddy. Exactly one backend may run per database because realtime fan-out and boot recovery are process-local (the control plane post explains why), so "two backends side by side, old one finishes its runs" isn't available to us. The new backend can only start after the old one lets go.

The release gate took hours, the promote takes two minutes

In September the deploy was a 17-step release gate that ran parity canaries on the host before it switched anything. One clean attempt took 15 to 20 minutes. The bad day was nine attempts in four and a half hours, each aborted by a probe bug, rolled back, fixed, and rerun from zero. Worse, the gate closed run admission before step 1 and kept it closed until step 15, so production refused new runs for the whole matrix on every attempt.

We replaced it with images built in CI and a promote workflow that takes about two minutes. The release controller owns a host lock, stages the release, runs its migration (only if the SQL file starts with -- fast-deploy: expansion-safe, otherwise it refuses with migration transition is not promote-safe), warms the inactive color's frontend and gateway, closes admission, waits for in-flight runs, swaps the backend, switches Caddy, verifies the public origins and reopens admission.

Then we made deploys continuous. Every green head of main shipped within about a minute. One day in October that meant 34 promotes, and every promote restarts the backend, reconnects every SSE client and makes a few requests fail. The host-check mail kept reporting short 502s. Earlier, seven promotes in ninety minutes meant an in-process 15-minute timer never reached its first tick all morning. The deployer now ships at most once every 30 minutes and only when nothing is in flight. Urgent fixes go out by hand.

A ten-second drain cuts runs, a two-hour drain blocks everyone

Our first drain waited ten seconds. Any real run turned that into backend did not drain within 10000ms and a compensated promote. So we raised the wait to a default of 120 minutes, capped at 300.

That fixed the cut runs and created a new problem. While a promote waits, admission is closed, and new tasks got a 503: "A release is being installed. Send your task again in a moment." One 25-minute run held admission closed for everyone else. Another time a run parked on an approval card held the promote lock for over 20 minutes. Deploys were waiting on a person.

How we deploy long-running AI agents without downtime now

Three changes, each narrow.

Held sends. A web send that arrives during a drain is accepted and stored as held (HTTP 201), and runs on the next release instead of bouncing with a 503. A queued turn behind a held send waits with it.

Survivors skip the drain. The drain only waits for runs whose turn depends on the backend process. A run survives a restart when all of these are true:

  • its engine is OpenCode or Claude Code, the two we proved by recreating the production backend mid-turn
  • its prompt has been delivered
  • it is bound to a native session in a cloud sandbox, with the sandbox id matching
  • its driver identity (provider, protocol, generation) matches what the new release will recover it with
  • the target release reports the same survivor rule version, so a release that changes recovery rules waits for everything

Codex on a ChatGPT plan still drains, because its relay lives in the backend process. A Codex turn cut by a restart gets one automatic continuation.

Parked runs skip the drain. A run waiting on a question or approval card is held by the sandbox runtime, not by us. The next backend follows it. It took a few tries to get right, because a kept run needs everything a live worker used to do for it. The fleet lease heartbeat has a 20-minute TTL; miss it and the reconciler reclaims the lease and deletes the sandbox. The sandbox needs keep-alives, and on Daytona only SDK calls count as activity, not traffic through the preview. And recovery keeps a parked run only if the runtime still holds every open card under the same request id. If a card is lost, the run fails with "Lost during deploy, please retry." A recovery check that throws keeps nothing alive.

The rule we wrote down: any per-run resource a worker refreshes must also be refreshed by recovery for parked runs.

The 3.5-minute outage

On 2026-10-11, production was down from 20:56:50 to 21:00:11 UTC during a promote. Three things lined up.

Boot recovery probes every run that survives a restart. Each probe had an 11-second backstop, and seven survivor probes held boot for 11.3 seconds, so the new backend took 14 seconds to come up. The controller waited 10.

The controller then compensated: it started the old color again. It did not stop the new one first. The new backend, still booting, already held the single-backend advisory lock, so the old one crash-looped on "Refusing to boot."

Caddy was 2.6.2 with lb_policy first and no health config. It kept retrying the first upstream for the whole lb_try_duration and never tried the second.

The fixes went live about 80 minutes later. Compensation always stops the target first and waits for it to exit. The target health wait is 30 seconds and the swap window 45. Boot recovery waits at most 12 seconds before the server listens; runs still resolving after that keep going in the background and fail if they haven't resolved within another 60 seconds. If the controller dies after committing a swap, its recovery finishes forward, or leaves the target serving and fails closed. Caddy got passive health (fail_duration 2s, max_fails 2).

Before that day, a failed promote was silent. Now the deployer pauses itself and sends a plain-language mail.

Caddy 2.6 could not replay a POST body

Even without an outage, every swap cost us failed POSTs. Caddy 2.6.2's buffer_requests can't replay a body under concurrent traffic. The retry goes out empty and the log says ContentLength=N with Body length M. An in-place backend restart lost 7 of 358 sign-out POSTs. On the host we reproduced 43 to 49 failures out of 239.

Setup POST failures during one promote
Caddy 2.6.2, no health config 75 slow 502s
2.6.2 with passive health 25 fast failures (~175 ms, "no upstreams available") in a 6.8 s window
2.11.7, request_buffers 1MB, active health, backend drain 1 connection error

On 2.11.7 the active checks hit /api/health/live: one pass marks an upstream up, two failures mark it down. Not /api/health, which pings the database and peaked at 1.9 seconds during a load test, while /live stayed at a p99 of 348 ms with 40 runs going. During the first promote on 2.11, requests sent while neither color was up waited and succeeded; the slowest took 8.7 seconds. GETs had zero failures.

The one remaining error is a POST sent on a keep-alive connection the old server is closing during a config reload. We accept it. A reload grace_period would fix it and also cut every open SSE stream, which is worse. Related trap: a Caddy config reload cancels the old handlers' retry loops, so any request still waiting in lb_try_duration at the switch gets a 502.

Two smaller Caddy facts from the switch: two versions can only share :443 through SO_REUSEPORT when both run as the same user, and HTTP/3 has to be off during that overlap.

Env changes only load on a new release

Our containers read their env file when a release starts its new color, and only then. Re-dispatching the promote with the live sha is refused with "release manifest is already active", which doesn't touch production but did pause the auto-deployer and mail us "deploy failed". An env-only fix looks deployed and isn't, until the next real commit ships. We now check the variable inside the live container before calling an env change done.

The gateway has its own env file, generated from the backend's through a whitelist when the host was provisioned. Keys added to the backend later never reached it. On 2026-10-13 that broke integration actions: the gateway was missing the connector settings and disagreed with the backend about the approval capability format, so actions failed with "approval capability unavailable". We now compare which whitelisted keys exist in each file, and hashes of their values, never the values.

Never swap half a release

The frontend bundle carries the release it was built for, and the backend answers every write with 409 frontend_release_mismatch when the two disagree. Once, a hand-swapped backend under an old frontend meant every send in every browser said "Frontend was updated. Reload before retrying this action." Reloading couldn't help; the served bundle was still the old one. The promote always moves both images together. Hand fixes have to as well.

What's still open: a promote still restarts the backend, so in-process timers reset and every SSE client reconnects. Batching deploys hides that rather than fixing it.

Boot recovery and the Compose release controller are in the open-source repo, in backend/src/runs/recovery.ts and deploy/compose.

Frequently asked questions

How do you deploy without killing in-flight AI agent runs?

Close admission, hold new sends instead of rejecting them, and only wait for runs whose turn depends on the backend process. Runs whose turn lives in the sandbox runtime, and runs waiting on a person, are recovered by the next backend instead of drained.

Why does a blue/green swap return 502 for POST requests?

If the proxy cannot replay a request body, a retried POST either goes out empty or fails. Caddy 2.6.2's buffer_requests sent empty retries under concurrent load for us; Caddy 2.11's request_buffers plus active health checks fixed it.

Why didn't my environment variable change take effect after a deploy?

Our containers read their env file only when a release starts its new color. Re-promoting the live release is refused as already active, so an env change goes live with the next real release.

All posts →

Hand your next task
to a coding agent.

UseAgent runs Claude Code, Codex and OpenCode in cloud sandboxes. The core is open source, so you can read the code or host it yourself.