Most autoscaling advice assumes a cluster: add nodes, point a horizontal autoscaler at a metric, done. A self-hosted agent platform often runs on one machine instead — the customer’s server. There is no next node. Every worker you add competes for the same RAM as browser sessions, the code sandbox, the database, and whatever else the customer runs.
So the real question isn’t “how many replicas?”. It’s “how much work can this box take right now without hurting something else?” — answered every second, by code that has to behave sensibly when the numbers it reads are wrong.
I used to think queue age was the signal to scale on. On a single machine I no longer do, and the reasons are most of this post.
Running processes aren’t ready capacity
Assume agent runs execute on RQ workers. A parent process in each worker container spawns children, each a fresh python -m process rather than a fork after imports, and each child runs one SimpleWorker.
The naive scaler compares queue depth with the number of child processes. It over-counts constantly. A process that’s spawned but still importing the model SDK and tool definitions — several seconds and a lot of memory — isn’t capacity. Neither is one that’s alive but wedged.
A better scaler asks RQ which workers are registered and idle, matching on hostname and PID (PIDs collide across containers), and counts three groups: ready and idle, busy, and starting.
pending = depth - ready_idle - starting
target = max(floor, active + pending)
target = min(target, cpu_cap, active + memory_slots)Have children do their heavy imports before registering with RQ, so that “ready” really means a run won’t pay the cold start. Wake the parent with a blocking pop on a small Redis list when work arrives, rather than polling, and limit how many processes it spawns per tick. Imports allocate memory before your next memory sample shows it.
Read limits from the cgroup, not the host
Inside a container, os.cpu_count() tells you about the host, which is the wrong answer. A CPU cap should come from the cgroup’s cpu.max, multiplied by an oversubscription factor. Agent runs spend most of their time waiting on the model, so one core can carry several of them. Memory slots should come from memory.max − memory.current − reserve, divided by a measured per-worker footprint.
The rule I’d care most about: if queue telemetry fails, the answer is unknown, not empty. A Redis timeout that reads as a quiet queue starts the idle timer, and a few minutes later you scale down in the middle of a busy period. When you can’t read the queue, hold current capacity and don’t start the idle clock.
Scale down by draining, never by killing
Scale-down should send one SIGTERM. In RQ that means “finish the current job, then exit”. Never send a second signal. An agent run can take minutes, and killing it halfway turns a capacity decision into a user-visible failure with uncertain side effects. (On Windows, SIGTERM is effectively a kill, so there’s no graceful drain in a Windows dev environment. It’s worth knowing that before you debug “lost” runs on a laptop.)
A drain file is a handy escape hatch: if it exists, the target is zero, which gives deploys a polite way to empty a container.
Somebody has to manage the containers
Above the in-container pools you need something that decides how many worker containers exist. On one machine, a small host-level controller running as a systemd service is enough, provided it’s allowed to touch only a short, explicit list of services.
Each tick it reads demand — counting only workers with a recent heartbeat — and computes slots from the host’s available memory, after leaving room for the full sandbox and browser reservations plus headroom. Share slots across worker roles by weighted fair rounds, never take busy work away because an estimate dropped, and require a lower target to hold for several minutes before applying it.
Removing a container has to be more careful than Compose allows on its own. docker compose up --scale won’t let you choose which replica goes. So check that the replica’s workers aren’t busy, touch its drain file, wait for its workers to deregister and for any preloading children to finish, and only then stop it with a generous grace period. Refuse to act at all if the image isn’t pinned by digest or a release is half-applied, and take the same lock as anything else that runs Compose.
A deadlock you can build in an afternoon
Say your agent can delegate to child runs, and parents and children use separate queues. Idle capacity is wasteful, so you let idle workers steal jobs from other queues.
Now saturate it. A parent waits for its children while holding its worker. Child workers, allowed to steal, pick up more parents, which delegate and wait too. Eventually every slot holds a parent waiting for a child that has nowhere to run.
The fix is a single rule: child workers serve only the child queue. Keep work stealing if you like, off by default, but never let it lend child capacity to parents. I’d put a load test for this in CI before anyone writes the stealing code.
An admission bug that rejects work you have room for
Browser sessions are expensive — a gigabyte or two per container — so the browser service should do its own admission. A reasonable ceiling is the minimum of three numbers: a configured maximum, a share of total host memory, and available memory plus what current sessions are actually using.
That third number is easy to get wrong. MemAvailable already excludes what running sessions use. If you then charge each session its full memory limit again, every resident session is counted twice, and users see “no capacity” with gigabytes free.
The fix is to credit back only measured anonymous memory from each session’s cgroup — never file cache, which the kernel can reclaim — capped per session, and only for containers your deployment owns. Give the sampler a short time budget and short Docker socket timeouts. If anything fails, credit nothing: fall back to the conservative answer, not the optimistic one.
Where queue age fits
Queue age is still worth exposing. Every queue should report its oldest waiting job in its health endpoint, and it’s the first thing I’d look at when someone says “the agent is slow”. But on one machine it can’t drive scaling. Queue age tells you you’re late. Whether you can do anything about it depends on memory, and memory is a hard ceiling, not a dial.
The other habit I’d keep is writing down the gates you haven’t passed. Something like “a burst of a few hundred runs drains in under two minutes” or “N concurrent runs plus M sandboxes plus K browsers on this box” should be a test with a pass or fail, and a failure should be recorded as a failure. That list has been more useful to me than any dashboard, and it’s the same honesty the deterministic infrastructure post argues for, applied to capacity.