One Redis, Too Many Jobs — Split It by What You Can Afford to Lose

Redis tends to accumulate responsibilities. It starts as a cache. Then it’s the job queue, because it’s already there. Then live price streams, a few distributed locks, a write-behind buffer, and wake-up signals for server-sent events. Each addition is reasonable on its own. Together they end up sharing one maxmemory-policy and one persistence setting, and that single configuration is wrong for at least half of them.

The question I find most useful for any key pattern in Redis is: what happens if this data disappears right now? The answers vary a lot, and they should drive the configuration.

Two instances with opposite rules

Look at a typical mixed workload through that question:

The first two can be evicted at any time. The last three must never be evicted, and should survive a restart. You can’t have both under one maxmemory-policy. With allkeys-lfu, Redis will cheerfully evict a queued job to make room for a cached search. With noeviction, a cache that grows past its budget starts failing writes to your queue.

So split them. One instance for disposable data: allkeys-lfu, no persistence, a modest memory cap. Another for things that matter: noeviction, AOF with appendfsync everysec, and an alert when it’s close to full, because for this instance “full” means “refusing work”. Running two small Redis processes costs almost nothing compared to debugging why a job evaporated.

Cache keys are part of your permission model

If cached data was fetched with a particular user’s credentials or entitlements, the key needs to say whose it is. A price fetched through one user’s market-data subscription shouldn’t become a cache hit for another user who doesn’t have that subscription.

A key shape I like:

app:price:v2:{deployment}:{user}:{account}:{source}:{exchange}:{symbol}

Each segment does a job. The version lets you change the value format without a migration. The deployment fingerprint stops two environments that accidentally share a Redis from reading each other’s data. User and account scope the entitlement. URL-quote every segment, so a strange symbol can’t break out of its scope during a SCAN. And keep credentials out of both the key and the value — it’s surprisingly easy to cache a whole upstream response that happens to contain a token.

Refuse older observations

Prices arrive from several places — a streaming feed, a polling fallback, a batch fetch triggered by a scheduled job — and not in order. A plain SET lets a slow poller overwrite a newer tick with an older one, and nobody notices until an alert fires on a price that was already stale.

Make the write conditional on time, and make it atomic with a small Lua script:

-- KEYS: value, clock, pending, dirty    ARGV: payload, ts
local last = tonumber(redis.call("GET", KEYS[2]) or "0")
if last > tonumber(ARGV[2]) then return 0 end
redis.call("SET", KEYS[1], ARGV[1], "EX", 30)
redis.call("SET", KEYS[2], ARGV[2], "EX", 604800)
redis.call("SET", KEYS[3], ARGV[1], "EX", 604800)
redis.call("ZADD", KEYS[4], ARGV[2], KEYS[3])
return 1

The last two lines turn the cache into a write-behind buffer as well: a pending copy plus a sorted “dirty” set that a separate process drains into SQL. When that process acknowledges a flush, it should delete the pending copy only if its bytes haven’t changed since it read them — another short Lua script. Otherwise a flush can erase an update that landed while it was writing.

Also reject values that aren’t observations. A zero or missing price is an error from upstream, not a market event, and caching it is how you get “price fell to 0” alerts.

On the client side, I’d make cache reads impatient: sub-second connect and socket timeouts, no retries, and a short local circuit breaker after a failure. If Redis is struggling, fall through to the source of truth. Retrying a cache read only adds load to the thing that’s already behind.

One consumer, a fenced checkpoint

For an ordered computation over a stream — say, evaluating alert rules tick by tick, where “crossed above” depends on the previous value — I usually prefer one owner over consumer groups. Consumer groups spread entries across consumers, which is great for independent jobs and awkward when every entry depends on the one before.

The single-owner version looks like this. One process holds a lease (SET key token NX PX 15000). It reads with plain XREAD. Its progress is a checkpoint key per stream, written by a Lua script that refuses unless the caller still holds the lease and the new offset is ahead of the old one. Any state it keeps — rolling windows, previous values — is namespaced by its lease token, so a paused process that wakes up after losing its lease can’t corrupt its successor’s state.

Then handle the gap. Streams are usually capped with MAXLEN ~ N. If the oldest entry still in the stream is newer than your checkpoint, entries were trimmed before you read them. Don’t pretend otherwise: start a new epoch and reset any computation that depended on continuity. A “crossed above” across a gap you can’t see is a guess.

Leases are an optimisation

Every lease should be SET NX with a random token, renewed and released only through compare-and-set scripts:

if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("PEXPIRE", KEYS[1], ARGV[2])
end
return 0

The tempting Python version — GET, compare, then EXPIRE — has a gap. If your lease expires between the two calls and someone else acquires it, your EXPIRE extends their lease. Renewal should never re-acquire a lease you’ve lost, either.

Even done right, a Redis lease isn’t a lock you should bet correctness on. A process can pause for longer than its TTL — a GC pause, a VM hiccup — and wake up still believing it owns something. Martin Kleppmann’s note on distributed locking covers why. So where correctness matters, let the database fence: an epoch or version column checked in the same UPDATE that does the work. The Redis lease then just decides who bothers to try, which saves wasted effort without carrying any guarantee.

One small, very real bug: if the code between acquiring a lease and the try that releases it can raise — a missing configuration check, say — the lease stays held until its TTL. With a long TTL and a periodic job, that looks like a job that silently stopped. Acquire as the last step before the guarded block.

Streams as hints, SQL as truth

For server-sent events on top of a database-backed event log, I like using Redis Streams as a doorbell rather than a mailbox. The event goes into SQL. Redis gets a tiny XADD with just the sequence number — enough to wake up whoever is streaming that log to a browser.

The reader then does this:

1. Read the tail ID of the stream.
2. Read new events from SQL.
3. XREAD BLOCK from the tail ID you captured in step 1.
4. When woken, go back to step 2.

Capturing the tail before the database read closes the race: anything committed in between either shows up in that read or wakes the next XREAD. If the XADD fails, readers fall back to polling SQL on a timer, rate-limited per viewer. Cap connection lifetime so reconnects re-check authorisation. And let a terminal status in SQL end the stream, even if its wake-up hint was lost.

Use an async client for this. Blocking XREAD calls run in a thread pool hold one thread each for the whole block timeout, and with a few dozen viewers that starves everything else using the pool.

The sentence test

Before adding a new key pattern, try writing one sentence: “If this data disappears, then…”. If the sentence ends with “we recompute it”, it’s a cache, so put it on the evicting instance. If it ends with “the database still has it”, Redis is an accelerator, so design the fallback. If you can’t finish the sentence, the data probably shouldn’t live only in Redis.