Python RQ for Long-Running Work — Keep the Queue Dumb, Put the Guarantees in SQL

RQ is easy to underestimate. It’s Redis lists, a job hash, and a worker loop. There’s no workflow engine, no durable timers, no exactly-once anything. For a lot of teams that reads as “fine for sending emails, not for real work”.

I think it’s the other way round for one class of system: self-hosted software that a small team has to operate, where Redis is already in the stack and the person on call might not be a distributed-systems specialist. A queue you can explain on a whiteboard in two minutes is a real operational advantage. The catch is that RQ gives you very few guarantees, so the ones you need have to live somewhere else. In my experience the right place is your SQL database.

This post is about long-running jobs — agent runs that take minutes, call paid APIs, and sometimes have side effects — and the patterns I’d use to run them on RQ without lying to anyone about what happened.

The job should be a pointer

Keep the job payload tiny. Ideally it’s one id:

queue.enqueue(
    "runs.execute",
    run_id,
    job_id=f"run-{run_id}",   # deterministic
    result_ttl=86_400,
    failure_ttl=86_400,
)

Everything the job needs — the user’s request, the configuration, the permissions it runs under — lives in a database row. Two things follow from that. You never have to worry about pickling an object graph or about a stale snapshot of the user’s settings riding along in Redis. And the database becomes the place where you can answer “what was this run supposed to do?” without trusting Redis to still have it.

Order matters here. Write the intent first, commit, close the session, then enqueue:

with session_scope() as db:
    run = Run(id=run_id, status="admitted", lane=lane_for(request))
    db.add(run)
# transaction committed and connection returned before any Redis I/O
queue_for(run.lane).enqueue("runs.execute", run_id, job_id=f"run-{run_id}")

If Redis is down at that moment, the run is still admitted. Something can enqueue it later. If you do it the other way round — enqueue, then write the row — you can end up with a job pointing at a run that doesn’t exist. And if you hold a database transaction open while talking to Redis, a slow Redis turns into a slow database.

Reconcile instead of retrying

I’d leave RQ’s Retry off for this kind of job, and probably its timeout too.

RQ’s retry knows the job failed. It doesn’t know why, and for agent runs the why decides everything. A provider rate limit wants a short backoff. A quota error wants no retry at all and a clear message. A malformed tool call wants a corrected prompt, not a replay. That logic belongs in the code that makes the model calls, which can classify failures properly.

Timeouts are similar. RQ’s death penalty kills the job wherever it is, including halfway through a paid call or a write. I’d rather enforce an absolute deadline inside the run, at points where stopping is safe, and make sure that deadline never slides forward when someone extends a job.

What replaces RQ’s retry is a reconciler: a loop that compares what the database says should be running with what Redis actually has.

def ensure_queued(run_id: str) -> None:
    job = Job.fetch_or_none(f"run-{run_id}")
    if job is None:
        enqueue(run_id)
    elif job.get_status() in {"failed", "finished", "stopped"}:
        job.delete()
        enqueue(run_id)
    elif job.get_status() == "queued" and redis.lpos(queue.key, job.id) is None:
        queue.enqueue_job(job)   # the hash says queued, the list disagrees

That last branch handles a case that sounds impossible until you see it: the job hash says queued, but the id isn’t in the list any more — a crash between two Redis operations, or a manual cleanup that removed one and not the other. Because the job id is deterministic, running the reconciler twice does no harm.

Add fencing to the run itself. Give each attempt an id and an epoch, and have the worker claim the run with a compare-and-swap in SQL. A worker that lost its claim can still be running — paused by GC, stuck on a slow network call — and it must not be able to write a result for an attempt that has moved on.

SimpleWorker, one process per worker

RQ’s default Worker forks a child per job. For short jobs that’s a nice isolation story. For long jobs with heavy imports, it’s expensive, and forking after you’ve opened database pools is a well-known way to get two processes sharing one connection.

The arrangement I like is SimpleWorker (no fork per job) inside processes you spawn yourself, from a small parent that decides how many to run. Each process owns its pools for its entire life. Have each one import its heavy dependencies — model SDKs, tool registries — before it registers with RQ. A worker that’s registered should be ready to work, not about to spend five seconds importing.

Two heartbeats are worth separating: one that says “this worker is alive and listening”, and one that says “this job is still making progress”. A worker can be alive with a wedged job, and the two signals lead to different actions.

Don’t let children starve their parents

If runs can delegate — a parent run that enqueues child runs and waits for them — you have a deadlock waiting for load.

Here’s how it happens. Parents and children have separate queues, which is sensible. Then someone enables work stealing so idle workers can help with other queues, also sensible. Under load, child workers pick up parent jobs. Those parents enqueue children and wait, holding their worker. Eventually every worker holds a parent waiting on a child, and no worker is free to run a child.

The fix is a one-line rule: workers that serve the child queue only ever serve the child queue. Steal in other directions if you want, but never lend child capacity to parents.

Fairness is the other queue-level problem. RQ is FIFO, so one user submitting thirty runs puts everyone else behind them. You can reorder pending jobs round-robin by user inside a WATCH/MULTI transaction: read the list, compute the new order, and write it back only if nobody touched the list in between. If the transaction aborts, try again on the next pass. Never overwrite a job someone else just accepted.

Pin work to the code that accepted it

One more subtle problem shows up during deploys. A run is admitted under version N of your code. Halfway through a rolling deploy, a version N+1 worker picks it up and reads its stored state with slightly different assumptions.

You can avoid that by putting a build identifier in the queue name, so version N’s runs are only consumed by version N’s workers. It works, but it creates a new failure: a run admitted just before a deploy, on a queue that no worker serves any more, waiting forever. And if turns within a conversation run in order, it blocks everything behind it too.

So the reconciler needs one more job. For runs that never started, if no worker with a recent heartbeat is listening on their queue, move them to the current build’s queue and stamp them with the current build. Never move a run that has started. That one should stop and wait for a person to decide.

Cron isn’t exactly-once, and it should say so

For scheduled work I’d skip rq-scheduler and use the database. A short Redis lease can decide which process bothers to look for due work, but the database decides who owns a row:

UPDATE schedules
   SET next_fire_at = :next, owner = :me, lease_until = :lease
 WHERE id = :id AND next_fire_at = :due AND owner IS NOT DISTINCT FROM :prev_owner;

Advance the schedule and insert an occurrence row in the same commit. Then move the occurrence through states: claimed, dispatching, then ok, error, or skipped. If one sits in dispatching longer than it plausibly could, mark it unknown. Don’t replay it. You don’t know whether the side effect happened, and replaying something like “send the alert” twice is often worse than a visible gap.

Write that limit down in the code: this isn’t exactly-once delivery. Then if someone edits a schedule after an occurrence was claimed, skip the stale occurrence instead of running yesterday’s claim with today’s configuration.

When to stop using RQ

None of this is RQ’s fault, and it isn’t really a criticism of RQ. It does what it promises, which isn’t much. Every pattern above is about writing your real assumptions into SQL where you can check them.

There’s a point where that becomes a workflow engine in disguise. If you need durable timers measured in days, sagas with compensation steps, or many-step workflows that resume halfway through, use something built for that. But for “run this long job once, tell me honestly if you couldn’t”, RQ plus a database and a reconciler is plenty, and it’s much easier to debug at 2 a.m. The RQ docs on workers are short enough to read in full, which is part of the appeal.