Redis streams: find and fix a dead consumer
Redis consumer groups in SQLly, hands-on: spot a dead consumer by lag and idle time, then XAUTOCLAIM, XACK, XCLAIM and trim, exact command shown first.
The short version
A dead consumer leaves entries pending forever. Here's how to find them and hand them to a live one.
A stream key in SQLly opens with four tabs: Entries, Groups, Consumers and Pending. Together they show you who reads the stream, how far behind each group is, and which entries were delivered to a consumer that never acknowledged them. From the same view you can auto-claim those entries for a live consumer, acknowledge, claim, create or destroy groups, and trim. Each action shows the exact command before it runs.
The failure this is for
Consumer groups give Redis streams at-least-once delivery. XREADGROUP hands an entry
to one consumer and records it in the group's pending entries list (the PEL) until that consumer
calls XACK. If the consumer crashes between reading and acknowledging, or its pod is
rescheduled with a new name, the entry stays pending under the old name. Nobody else gets it.
Your other workers stay healthy, throughput looks fine, and a handful of orders quietly never
ship.
Finding this from a prompt means chaining XINFO GROUPS, XINFO CONSUMERS
and XPENDING, then building an XAUTOCLAIM or XCLAIM by hand
with the right group, consumer, idle threshold and ids. It's not hard, but it's easy to get wrong
under pressure, and it's exactly the moment someone pastes a command into the wrong terminal. The
key view puts each of those answers in a tab, and each fix in a form that shows the command it
will run.
Build a stuck consumer yourself
Paste this into a SQLly query tab on a scratch Redis 7 database, or into redis-cli.
It creates a six-entry stream with explicit ids so the later commands line up, two groups, and
one consumer called dead that reads two entries and never acknowledges them.
DEL orders:events
XADD orders:events 1-0 n 00001
XADD orders:events 2-0 n 00002
XADD orders:events 3-0 n 00003
XADD orders:events 4-0 n 00004
XADD orders:events 5-0 n 00005
XADD orders:events 6-0 n 00006
XGROUP CREATE orders:events workers 0
XGROUP CREATE orders:events audit $
XREADGROUP GROUP workers dead COUNT 2 STREAMS orders:events >
XGROUP CREATE … workers 0 starts the workers group at the beginning of
the stream. audit starts at $, so it only sees entries added from now
on. XREADGROUP delivers entries 1-0 and 2-0 to
dead. That's the crash: nothing ever acknowledges them. Wait a minute so the idle
time is worth looking at, then double-click orders:events in the explorer, or
right-click it and choose Open Value.
Reading the four tabs
Entries
The stream's entries, oldest first, 1,000 at a time, with showing N of M entries and
Load more. Click a row to see its fields under the grid with the value-format
picker. Field values are passed through as their exact bytes, so 00001 is shown as
00001, not 1.
Groups
One row per consumer group: consumer count, pending count, last delivered id, entries read,
lag and the stream's length. Lag is the number of entries added after the last
one the group read. In the scenario, workers shows 1 consumer, 2 pending and a lag
of 4: it has read two of six entries and has four still to read. Lag and entries read need Redis 7
or later, and they are blank before that or when the server can't tell.
Consumers
Click a group (or use the group chips beside the tabs) to see its consumers, with how many
entries each one holds pending and its idle time in milliseconds since it last
read. On Redis 7.2 and later there's also inactive: how long since it last made
progress. dead holds 2 and its idle time only goes up.
Pending
Every entry the chosen group delivered and nobody acknowledged: entry id, the consumer holding
it, its idle time in milliseconds and its delivery count, 1,000 at a time with showing N of M
pending. This is where a dead consumer is obvious. Two rows, both owned by
dead, both with idle times that keep growing every time you press Refresh.
redis-streams-groups-lag.webp — the Groups tab of orders:events showing the workers group with 1 consumer, 2 pending, last delivered 2-0 and lag 4, and the audit group with lag 0workers has two pending entries and is four entries behind.redis-streams-pending-autoclaim.webp — the Pending tab with the workers chip chosen, entries 1-0 and 2-0 held by consumer dead with growing idle ms, and the Auto-claim form with alive typed as the consumer showing the exact XAUTOCLAIM above RunXAUTOCLAIM before it runs.Fixing it
Auto-claim everything the dead consumer holds
On the Pending tab, press Auto-claim… and type the consumer that should take over. The form builds:
XAUTOCLAIM orders:events workers alive 60000 0-0 COUNT 100
That moves every pending entry idle for at least 60,000 ms (a minute, unless you change it) to
alive, starting from the beginning of the PEL, 100 at a time. The idle threshold is
what makes this safe to run while workers are healthy. An entry a live consumer picked up a few
seconds ago is younger than the threshold, so it stays where it is. After the run the key reloads,
you stay on the Pending tab and group you were on, and both rows now show alive.
Claiming doesn't acknowledge anything. The entries are still pending, now under a consumer that
exists. Your alive worker sees them the next time it reads its own pending entries
(XREADGROUP … STREAMS orders:events 0) and processes them.
Acknowledge what you've handled
If you've confirmed an entry was processed, or decided it shouldn't be, select it and press Acknowledge selected…:
XACK orders:events workers 1-0
The entry leaves the PEL and the group's pending count drops.
Claim one entry
For a single entry, Claim selected… builds an XCLAIM. Its idle
threshold starts at the entry's current idle time, for example:
XCLAIM orders:events workers alive 93000 2-0
If another consumer picks up the entry between you looking and you pressing Run, its idle time resets below 93,000, the claim matches nothing, and that consumer keeps it. You don't take an entry away from a worker that's actively processing it.
Groups and trimming
On the Groups tab, Create group… builds XGROUP CREATE key group $.
Start at $ to deliver only new entries, or type 0 for the whole stream.
Destroy selected group… builds XGROUP DESTROY key group.
On the Entries tab, Append entry… builds XADD key * field value …
(or with an id you type), Delete selected… builds XDEL key id, and
Trim… builds either of:
XTRIM orders:events MAXLEN 3
XTRIM orders:events MINID ~ 3-0
MAXLEN n keeps the newest n, MINID id drops lower ids, and
~ lets Redis trim approximately, which is cheaper.
Trimming and destroying a group can't be undone. XTRIM deletes entries, and
XGROUP DESTROY deletes the group with all its consumers and pending entries. So on
top of the usual review, those two forms ask you to type the trim count (or id), or the
group's name, and Run stays disabled until what you typed matches exactly. The trim form
also says roughly how many entries the count would remove. A binary group's name is confirmed by
typing its hex.
redis-streams-trim-typed-confirm.webp — the Trim form on orders:events with MAXLEN 3, the note saying roughly how many entries would be removed, the exact XTRIM preview, and Run disabled until the count is typed into the confirmation boxThe same guards as everything else
Stream actions go through the same checks as every key view edit and every query tab:
- A read-only connection refuses them and says why.
- A production connection whose writes are locked refuses them until you unlock writes from the status-bar chip, and the lock is checked again right before the command runs.
- A connection that confirms data changes turns Run into Confirm Run, a second click on the command shown.
Field names, values, group and consumer names go to the server as their exact bytes. A binary
group or consumer name picked from the grid is used as is, and Value is hex in Append
entry writes binary values. We tested the whole loop live: seed a stream, have a consumer called
dead read two entries, check that Pending shows both under dead with
growing idle times and the group's lag at 4, auto-claim them for alive through the
confirming click, and check that both now belong to alive. Then trim, which is
refused until the count is typed.
A checklist for the next incident
- Open the stream. On Groups, look for a pending count that isn't falling and a lag that is growing.
- On Consumers, find the consumer whose idle time is much larger than its siblings'. That's usually a worker that no longer exists.
- On Pending, confirm the entries belong to it and check their delivery counts. An entry delivered many times may be a poison message, not a dead worker.
- Auto-claim… to a live consumer with an idle threshold longer than your slowest normal processing time.
- Acknowledge selected… only for entries you know were handled or have decided to drop.
Limits
- Lag and entries read need Redis 7, and the consumers' inactive time needs Redis 7.2.
- The view works one stream key at a time. There's no overview of consumer health across many streams.
- Export as Command File… writes a stream's entries with their ids, but not its consumer groups.
The full reference is on Streams and consumer groups,
and the other types are on Key view.
Download SQLly, paste the scenario above into a scratch database, and
open orders:events.