Hard20 minDistributed Systems
UpdatedAug 6, 2026
Edit

Service Bus: Replay Dead Letters

Question Variations

  • "Why should dead-letter replay not be automatic?"
  • "What distinguishes a poison message from a transient failure?"
  • "How do you verify a DLQ drain did not lose work?"

Why This Is Asked

A schema bug has placed thousands of valid orders in an Azure Service Bus dead-letter queue. This tests how a candidate would diagnose, correct, replay, and observe recovery without duplicating already processed orders.

Key Concepts

  • Reason inspection: Dead-letter reason, description, delivery count, and application metadata identify the failure class.
  • Fix before replay: Correct the producer, consumer, or contract issue before returning messages to the main path.
  • Idempotent replay: Reuse stable message or business IDs so previously completed work is harmless on replay.
  • Controlled recovery: Throttle and monitor replay to avoid recreating the outage or overwhelming downstream services.

Question Variations

  • “Why should dead-letter replay not be automatic?”
  • “What distinguishes a poison message from a transient failure?”
  • “How do you verify a DLQ drain did not lose work?”

Answers by Technology

+ Add Variant
Azure Service BusImprove this answer ✏️

Expected Answer

First inspect the dead-letter reason, description, delivery count, schema version, and correlation IDs to establish whether this is a bug, expired message, or permanently invalid input. Do not automatically replay a DLQ: fix and deploy the producer or consumer problem first, then test the corrected handler with representative messages. Build a controlled replay tool that reads messages, preserves the business and idempotency IDs, sends to the intended active entity, completes the DLQ message only after the send succeeds, and limits throughput. Monitor normal-queue backlog, downstream error rate, duplicates, and the remaining DLQ count. Finally reconcile the affected order IDs against the source of truth so recovery proves business completion rather than just an empty queue.

Why It Matters

A DLQ is evidence of an uncompleted workflow. Blind replay can duplicate completed work or overload the same broken consumer; deletion destroys the forensic trail needed to recover safely.

Example Code

for (const message of await deadLetterReceiver.receiveMessages(20)) {
  await sender.sendMessages({ body: message.body, messageId: message.messageId, applicationProperties: message.applicationProperties });
  await deadLetterReceiver.completeMessage(message);
}

Common Mistakes

  • Completing a DLQ message before the resend succeeds: A send failure loses the only retained copy.
  • Replaying before the consumer fix is deployed: The same messages return to the DLQ and increase operational load.

Follow-up Questions

  • Why retain the message ID? (Answer: It enables idempotent consumers to recognize a prior successful effect.)
  • How do you prove recovery? (Answer: Reconcile source records, successful processing metrics, and the controlled DLQ drain.)

Related Questions

References