Redis streams: find and fix a dead consumer

Redis consumer groups in SQLly, hands-on: spot a dead consumer by lag and idle time, then XAUTOCLAIM, XACK, XCLAIM and trim, exact command shown first.

By Updated

The short version

A dead consumer leaves entries pending forever. Here's how to find them and hand them to a live one.

A stream key in SQLly opens with four tabs: Entries, Groups, Consumers and Pending. Together they show you who reads the stream, how far behind each group is, and which entries were delivered to a consumer that never acknowledged them. From the same view you can auto-claim those entries for a live consumer, acknowledge, claim, create or destroy groups, and trim. Each action shows the exact command before it runs.

The failure this is for

Consumer groups give Redis streams at-least-once delivery. XREADGROUP hands an entry to one consumer and records it in the group's pending entries list (the PEL) until that consumer calls XACK. If the consumer crashes between reading and acknowledging, or its pod is rescheduled with a new name, the entry stays pending under the old name. Nobody else gets it. Your other workers stay healthy, throughput looks fine, and a handful of orders quietly never ship.

Finding this from a prompt means chaining XINFO GROUPS, XINFO CONSUMERS and XPENDING, then building an XAUTOCLAIM or XCLAIM by hand with the right group, consumer, idle threshold and ids. It's not hard, but it's easy to get wrong under pressure, and it's exactly the moment someone pastes a command into the wrong terminal. The key view puts each of those answers in a tab, and each fix in a form that shows the command it will run.

Build a stuck consumer yourself

Paste this into a SQLly query tab on a scratch Redis 7 database, or into redis-cli. It creates a six-entry stream with explicit ids so the later commands line up, two groups, and one consumer called dead that reads two entries and never acknowledges them.

DEL orders:events
XADD orders:events 1-0 n 00001
XADD orders:events 2-0 n 00002
XADD orders:events 3-0 n 00003
XADD orders:events 4-0 n 00004
XADD orders:events 5-0 n 00005
XADD orders:events 6-0 n 00006
XGROUP CREATE orders:events workers 0
XGROUP CREATE orders:events audit $
XREADGROUP GROUP workers dead COUNT 2 STREAMS orders:events >

XGROUP CREATE … workers 0 starts the workers group at the beginning of the stream. audit starts at $, so it only sees entries added from now on. XREADGROUP delivers entries 1-0 and 2-0 to dead. That's the crash: nothing ever acknowledges them. Wait a minute so the idle time is worth looking at, then double-click orders:events in the explorer, or right-click it and choose Open Value.

Reading the four tabs

Entries

The stream's entries, oldest first, 1,000 at a time, with showing N of M entries and Load more. Click a row to see its fields under the grid with the value-format picker. Field values are passed through as their exact bytes, so 00001 is shown as 00001, not 1.

Groups

One row per consumer group: consumer count, pending count, last delivered id, entries read, lag and the stream's length. Lag is the number of entries added after the last one the group read. In the scenario, workers shows 1 consumer, 2 pending and a lag of 4: it has read two of six entries and has four still to read. Lag and entries read need Redis 7 or later, and they are blank before that or when the server can't tell.

Consumers

Click a group (or use the group chips beside the tabs) to see its consumers, with how many entries each one holds pending and its idle time in milliseconds since it last read. On Redis 7.2 and later there's also inactive: how long since it last made progress. dead holds 2 and its idle time only goes up.

Pending

Every entry the chosen group delivered and nobody acknowledged: entry id, the consumer holding it, its idle time in milliseconds and its delivery count, 1,000 at a time with showing N of M pending. This is where a dead consumer is obvious. Two rows, both owned by dead, both with idle times that keep growing every time you press Refresh.

Screenshot to comeredis-streams-groups-lag.webp — the Groups tab of orders:events showing the workers group with 1 consumer, 2 pending, last delivered 2-0 and lag 4, and the audit group with lag 0
The Groups tab: workers has two pending entries and is four entries behind.
Screenshot to comeredis-streams-pending-autoclaim.webp — the Pending tab with the workers chip chosen, entries 1-0 and 2-0 held by consumer dead with growing idle ms, and the Auto-claim form with alive typed as the consumer showing the exact XAUTOCLAIM above Run
The Pending tab makes a dead consumer obvious, and Auto-claim shows the exact XAUTOCLAIM before it runs.

Fixing it

Auto-claim everything the dead consumer holds

On the Pending tab, press Auto-claim… and type the consumer that should take over. The form builds:

XAUTOCLAIM orders:events workers alive 60000 0-0 COUNT 100

That moves every pending entry idle for at least 60,000 ms (a minute, unless you change it) to alive, starting from the beginning of the PEL, 100 at a time. The idle threshold is what makes this safe to run while workers are healthy. An entry a live consumer picked up a few seconds ago is younger than the threshold, so it stays where it is. After the run the key reloads, you stay on the Pending tab and group you were on, and both rows now show alive.

Claiming doesn't acknowledge anything. The entries are still pending, now under a consumer that exists. Your alive worker sees them the next time it reads its own pending entries (XREADGROUP … STREAMS orders:events 0) and processes them.

Acknowledge what you've handled

If you've confirmed an entry was processed, or decided it shouldn't be, select it and press Acknowledge selected…:

XACK orders:events workers 1-0

The entry leaves the PEL and the group's pending count drops.

Claim one entry

For a single entry, Claim selected… builds an XCLAIM. Its idle threshold starts at the entry's current idle time, for example:

XCLAIM orders:events workers alive 93000 2-0

If another consumer picks up the entry between you looking and you pressing Run, its idle time resets below 93,000, the claim matches nothing, and that consumer keeps it. You don't take an entry away from a worker that's actively processing it.

Groups and trimming

On the Groups tab, Create group… builds XGROUP CREATE key group $. Start at $ to deliver only new entries, or type 0 for the whole stream. Destroy selected group… builds XGROUP DESTROY key group.

On the Entries tab, Append entry… builds XADD key * field value … (or with an id you type), Delete selected… builds XDEL key id, and Trim… builds either of:

XTRIM orders:events MAXLEN 3
XTRIM orders:events MINID ~ 3-0

MAXLEN n keeps the newest n, MINID id drops lower ids, and ~ lets Redis trim approximately, which is cheaper.

Trimming and destroying a group can't be undone. XTRIM deletes entries, and XGROUP DESTROY deletes the group with all its consumers and pending entries. So on top of the usual review, those two forms ask you to type the trim count (or id), or the group's name, and Run stays disabled until what you typed matches exactly. The trim form also says roughly how many entries the count would remove. A binary group's name is confirmed by typing its hex.

Screenshot to comeredis-streams-trim-typed-confirm.webp — the Trim form on orders:events with MAXLEN 3, the note saying roughly how many entries would be removed, the exact XTRIM preview, and Run disabled until the count is typed into the confirmation box
Trim asks you to type the count before Run is enabled. Destroying a group asks for its name.

The same guards as everything else

Stream actions go through the same checks as every key view edit and every query tab:

  • A read-only connection refuses them and says why.
  • A production connection whose writes are locked refuses them until you unlock writes from the status-bar chip, and the lock is checked again right before the command runs.
  • A connection that confirms data changes turns Run into Confirm Run, a second click on the command shown.

Field names, values, group and consumer names go to the server as their exact bytes. A binary group or consumer name picked from the grid is used as is, and Value is hex in Append entry writes binary values. We tested the whole loop live: seed a stream, have a consumer called dead read two entries, check that Pending shows both under dead with growing idle times and the group's lag at 4, auto-claim them for alive through the confirming click, and check that both now belong to alive. Then trim, which is refused until the count is typed.

A checklist for the next incident

  1. Open the stream. On Groups, look for a pending count that isn't falling and a lag that is growing.
  2. On Consumers, find the consumer whose idle time is much larger than its siblings'. That's usually a worker that no longer exists.
  3. On Pending, confirm the entries belong to it and check their delivery counts. An entry delivered many times may be a poison message, not a dead worker.
  4. Auto-claim… to a live consumer with an idle threshold longer than your slowest normal processing time.
  5. Acknowledge selected… only for entries you know were handled or have decided to drop.

Limits

  • Lag and entries read need Redis 7, and the consumers' inactive time needs Redis 7.2.
  • The view works one stream key at a time. There's no overview of consumer health across many streams.
  • Export as Command File… writes a stream's entries with their ids, but not its consumer groups.

The full reference is on Streams and consumer groups, and the other types are on Key view. Download SQLly, paste the scenario above into a scratch database, and open orders:events.