Most writing about AI agents is about the model: prompts, tools, reasoning. Very little is about the machinery underneath — who owns a run, what happens when the process executing it dies, and whether anything can be safely repeated afterwards. That machinery decides whether an agent is trustworthy far more than the prompt does.
I think about this mostly through one kind of system: a self-hosted agent platform that runs on the customer’s own server. No shared cloud tenant, no managed queue, no platform team. The whole stack — API, workers, Redis, a code sandbox, a browser, and an agent that can use all of them — runs under Docker Compose on one machine. When something breaks at 2 a.m., the system has to explain itself, because nobody else will.
That constraint makes the trade-offs unusually clear, so it’s a good lens for the general problem.
Keep infrastructure decisions away from the model
The first rule I’d write down: the model never decides whether a container should be restarted, waited on, or replaced.
It sounds obvious until you give an agent tools like “run Python” and “open a browser”. The tempting design is to let the agent see a failure and try again. Then a sandbox service that was briefly unreachable turns into three identical code runs, or a browser that was still starting gets a second one stacked on the same profile.
Instead, each execution backend reports a small, uniform health record — available or not, which runtime, why — and recovery is plain code with fixed numbers. Something like: three attempts, a short jittered backoff, a readiness timeout, and a circuit breaker that opens after a few consecutive failures. When a backend is down, the tool fails closed and says so in words a person can act on. The agent gets “browser unavailable: no isolating runtime on this host”, not a stack trace it will try to reason its way around.
Draw the topology around trust
With Compose, profiles are a cheap way to make the default install small and opt into risk. The core profile has the API, the frontend, Redis, and a bootstrap step that generates secrets on first boot. Everything with a bigger blast radius — the code sandbox, the browser, the worker pools — is a separate profile someone has to turn on.
Two details carry more weight than they look. Only the services that launch containers get the Docker socket; the API never does. And the browser’s network is marked internal: true, so the only way a browser session reaches the internet is through an egress proxy.
For a single-customer deployment I’d pick this over Kubernetes without much hesitation. Kubernetes brings scheduling you don’t need on one machine, and it moves failure modes somewhere the customer’s IT team can’t see them.
Every run has exactly one owner
The primitive I keep coming back to is a small, immutable ownership record:
@dataclass(frozen=True)
class Owner:
run_id: str
attempt: str
epoch: int
worker: strA worker that picks up a run holds a lease on it and renews it in the background — say every 5 seconds against a 20-second TTL. If renewal keeps failing past that grace period, the worker stops treating the run as its own. It doesn’t try to win it back. An old worker must never adopt an attempt that a newer one has already claimed.
Renewal has to be atomic, because the obvious version is subtly wrong:
-- KEYS[1] = lease key, ARGV[1] = my token, ARGV[2] = ttl in ms
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("PEXPIRE", KEYS[1], ARGV[2])
end
return 0With a GET followed by an EXPIRE from application code, there’s a window where your lease expires, the next owner takes it, and your EXPIRE lands on their key. You’ve just extended someone else’s ownership based on a stale read. Release has the same problem, so it’s a compare-and-delete.
When a supervisor actually observes a worker process exit, it should expire that worker’s leases immediately rather than make the next worker wait out the TTL. The important word is observes. Missing metadata isn’t death. A Redis restart can make every healthy worker look dead for a few seconds, and treating that as proof is how you get two owners.
Own your retries
I’d call the model provider’s SDK with its built-in retries switched off and do retries in the application.
Here’s why. Provider SDKs honour Retry-After, which is correct from their point of view. But a quota error can come back with a Retry-After measured in hours. If that happens inside a job with a ten-minute timeout, the run sits silently until the job is killed, and the user sees a spinner and then a generic error.
An application-level policy can classify the failure first — transient network error, rate limit, exhausted quota, unknown tool, repeated identical tool call, a tool result without a matching call — and treat each class differently. Rate limits get backoff with a cap on any server-requested delay. Exhausted quota isn’t retried at all; it becomes a clear message. An idle stream gets a hard limit rather than waiting forever for the next token.
Turn uncertainty into a receipt
This is the idea I’d keep if I had to throw away everything else.
When a worker dies, some operations are in a state nobody can resolve from the inside. A model call was sent but no usage came back. A browser was asked to start and the client got an HTTP 499, but the container may well be running. In each case the tempting move is to retry, and in each case a retry can double the effect.
So don’t. Record an interrupted call as a visible “outcome unknown” entry in the run’s history. Carry the accepted results — and the exact identities of resources and calls — across the crash in a small recovery ledger, so the next attempt continues from facts rather than from a reconstructed guess. And be explicit about what that ledger can’t do: it can’t make an interrupted mutation safe to repeat.
Reads are different. Within a run, a read can be replayed, with at most one in-flight call per read. A successful write should bump a per-domain version, so a read → edit → verify sequence isn’t mistaken for the agent going round in circles.
Two bugs that are easy to write
Holding a database transaction across a slow call. An agent runner reads some state, then calls a health endpoint, a tool server, or the model, and the read transaction stays open the whole time. Under load the connection pool empties. The fix sounds trivial — commit or roll back before any network wait — but SQLAlchemy has a trap: rolling back expires loaded objects, so the next attribute access quietly opens a new transaction, possibly from a different task. Committing clean reads with expire_on_commit=False, and rolling back only when the session is actually dirty, avoids it. To test it, run against a pool of exactly one connection with no overflow. Any leaked transaction then deadlocks immediately, which is a far better test than hoping a bigger pool hides it.
Two things driving Compose at once. A deploy script and an automatic scaler both run docker compose. Sooner or later they race: one removes a container the other expects, and the deploy fails with a “missing container” error. The natural reaction is to run a recovery command straight away — while the failing command is still running — which makes name conflicts worse. The fix is dull: every tool that touches Compose takes the same exclusive flock, and you always wait for a command’s process to exit before starting the one meant to recover from it.
What “deterministic” means
It doesn’t mean nothing fails. Workers die, providers rate-limit, Docker does strange things with volumes. It means that for any run you can answer three questions from stored state, not from logs and guesswork: who owned it, which effects were accepted, and which outcomes are still unknown.
If the system can’t answer the third, it should say “unknown” rather than pick whichever answer looks better. I’d trust that rule more than any amount of clever retry logic. The rest of this series — the sandbox, the browser, worker capacity — keeps coming back to it.