Skip to content
UseAgent

Engineering · Computer Use · Sandboxes

Building a computer use agent sandbox with a real desktop

What it took to make our computer use agent sandbox usable: clicks that land, typing through the clipboard, a desktop started at boot and a real Xorg session.

Abhishek6 min read
On this page

On September 9 we watched our own agent use its desktop and it was bad. We closed Chrome and Chrome came back. It made typing mistakes in the address bar. One run took 4 minutes 50 seconds to open a single website. And every time someone opened the desktop tab there was a 15 to 20 second spinner first. A computer use agent sandbox that works in a demo and fails on the third click is worse than none, so we went through it piece by piece.

Every UseAgent sandbox has a desktop the agent can drive and you can watch. Engines get a computer tool through our gateway: screenshot, click, type, key, scroll, drag, navigate, batched into sequences. Here's what made it usable.

A computer use agent sandbox needs clicks scaled at run time

The model never sees the real screen. It sees a screenshot resized to 1280 pixels wide, and it answers in those pixels. The display is 1920x1080. Send the model's x straight to xdotool and every click lands in the wrong place, by more the further right it is.

So every coordinate is scaled inside the sandbox, at the moment the batch runs:

bash
sw=1280                                   # width of the last screenshot the model saw
dw=$(xdpyinfo -display :1 | awk '/dimensions:/{split($2,a,"x"); print a[1]; exit}')
[ -n "$dw" ] || dw=$sw
xdotool mousemove $((640*dw/sw)) $((360*dw/sw)) click 1

Reading the display width live means the same command works whatever resolution the image ships with. The gateway remembers the width of the last full screenshot per sandbox and falls back to 1280 after a restart. A zoom, a screenshot of a region, never updates it. If it did, the next click would be scaled against a crop.

A batch runs as one shell. After each action we print a marker, USEAGENT_STEP_OK_1, USEAGENT_STEP_OK_2 and so on. When a batch fails, the markers in stdout say exactly which action broke, and the error the model gets names that action instead of saying something failed. Our sandbox provider returns stdout and stderr together, so the markers survive a non-zero exit.

Typing goes through the clipboard

xdotool types one keystroke per character. It's slow and GUI toolkits drop keys, which is where the address bar typos came from. Text up to 200 characters is still typed. Anything longer goes through the clipboard:

bash
printf '%s' "$B64" | base64 -d | xclip -selection clipboard -in >/dev/null 2>&1 \
  && xdotool key --clearmodifiers ctrl+v

The >/dev/null 2>&1 is the whole bug. xclip forks a process that stays alive to serve the selection, and that child keeps the batch's stdout open. Whatever runs the batch waits for the pipe to close, which doesn't happen, so every long paste hung until the 60-second command limit. Detach xclip's output and the batch exits normally. Text is passed base64 encoded so quotes and newlines in it can't break the shell. We had to rebake the sandbox image to include xclip at all.

Key presses are explicit: keydown, sleep 0.1, keyup, sleep 0.2. Chords release every modifier first, press the modifiers, press the key, and release them in reverse. One known gap: a plain key press still exits 0 when the keydown itself failed, so the model can believe it pressed Enter when it didn't.

The browser starts when the agent needs it

Chrome coming back after we closed it was our own launcher, which started Chrome along with the desktop. Now it doesn't. A small relay inside the sandbox speaks the Chrome DevTools protocol, answers a health check, and starts Chrome on first use. Playwright's MCP server connects to it over loopback. Desktop readiness waits for the relay, not for Chrome, so a closed browser stays closed until something needs a page.

Starting Chrome through the relay can take up to 20 seconds, so the request for the list of browser targets allows 30.

Some providers ship their own desktop with no relay. Box does, on display :0. There, navigate goes through the address bar like a person would: activate the visible Chrome window, ctrl+L, type the URL, press Delete, then Return. The Delete matters, since the address bar autocompletes a typed prefix from history and Return would accept the suggestion. If no browser window exists, it starts one with the URL, and if no browser is installed it fails loudly instead of pretending.

The 15 to 20 second spinner was a desktop started too late

We timed a cold desktop inside a live sandbox:

Component ready Seconds from start
Xorg 3.4
Budgie session 9.3
VNC server (RFB) 12
Chrome and relay 13.5
noVNC 14

None of that is slow for a desktop. The problem was when it started. The launcher ran when someone opened the desktop tab, never at sandbox boot, so our warm pool of pre-booted sandboxes did nothing for it. Every viewer paid the full cold start.

The fix moves the desktop into the image's boot entrypoint, after the runtime is warm. The launcher and relay are baked into the image, the relay token is written first, and a marker file sits in the home directory while the desktop boots. When the backend needs the desktop, it waits up to 60 seconds on the marker instead of launching a second one. On a live E2B sandbox the marker cleared at 13.4 seconds, before anyone had asked. A sandbox from the warm pool hands over a desktop that's already up.

Two review rounds tightened the edges. A relaunch now stops any earlier desktop first, by the recorded process group plus the named services. Boot records the launcher's pid immediately, and a marker without a pid counts as still booting. A stale pid from a previous boot is cleared.

Each tool call paid for a readiness check

With the desktop warm, single computer actions still took 4 to 10 seconds. The sandbox work was small:

  • screenshot capture: 0.4 s warm, 1.35 s cold
  • PNG download: 0.3 s
  • a click, type and Return batch: 0.8 s
  • one command round trip to an E2B sandbox: 0.27 s

The time was ours. The gateway ran the full desktop readiness pass on every call: probe the display, check the relay files (with uploads), chmod. Now readiness is cached per sandbox for 60 seconds, and any failed command clears the cache. The gaps between tool calls that remained, 3 to 5 seconds, were the model thinking, and the first tool call landed 5 to 13 seconds after dispatch. That part isn't ours to fix.

A real Xorg session, not Xvfb

The old image ran XFCE on Xvfb at 1440x900. It worked, and it looked like a test fixture. We rebuilt it as a real desktop session on our Debian bookworm base:

  • Xorg with the dummy video driver, a 1920x1080 modeline at 60 Hz, 96 DPI, run as root inside the container
  • a Budgie desktop with one bottom panel, dark Adwaita, icons on the desktop
  • x11vnc and noVNC to stream it, since we don't have a WebRTC path yet

Getting Budgie up in a container had its own traps. Debian's Budgie launches through gnome-session, which wants a system bus, logind and polkit, and shows the "oh no" failure screen in a container. So we skip the session manager and start the parts directly under dbus-launch: the xsettings daemon, budgie-daemon, budgie-wm (which needs XDG_SESSION_TYPE=x11), then budgie-panel. Desktop icons come from pcmanfm --desktop, because nemo-desktop refuses to run as root. Symbolic icons need librsvg2-common, the wallpapers are .webp and need webp-pixbuf-loader, and gnome-terminal won't start without LANG=C.UTF-8 and a session bus.

What's still missing

Research after these fixes ranked the gaps we haven't closed: browser tools like tabs, file upload and find-text on every engine; path drags and separate mouse down and up; a confirmation step before consequential actions; downloads and stored logins; and the cursor position.

The desktop recipe and the computer tool are in the open-source repo; the hosted version is one sign-up away.

Frequently asked questions

Why do computer use agent clicks miss their target?

Usually a coordinate space mismatch. The model sees a resized screenshot and answers in its pixels, while the display is larger. Scale every coordinate by display width over screenshot width, and don't let a zoomed screenshot change the width you scale by.

How do you make a computer use agent type long text reliably?

Paste it. Per-character keystrokes are slow and GUI toolkits drop some. We put text over 200 characters on the clipboard with xclip and send ctrl+v, with xclip's output detached so the command can exit.

Why use Xorg with the dummy driver instead of Xvfb for an agent desktop?

It behaves like a real display, which the desktop components expect. We run Xorg on the dummy driver at 1920x1080 and 60 Hz with a Budgie session, where we used to run XFCE on Xvfb at 1440x900.

All posts →

Hand your next task
to a coding agent.

UseAgent runs Claude Code, Codex and OpenCode in cloud sandboxes. The core is open source, so you can read the code or host it yourself.