Message Queues and Delivery Guarantees
Why systems put a queue between services and what delivery guarantees mean. At-least-once delivery caused 11 duplicate side effects in 20 messages; an idempotency key fixed it.
A message queue sits between a producer and a consumer so bursts and failures do not cascade between services. Delivery is usually at-least-once: if a consumer crashes before acknowledging, the message is redelivered and may be processed twice. Interviewers expect the fix: make consumers idempotent with an idempotency key, since true exactly-once delivery is not achievable over an unreliable network.
A message queue decouples a producer from a consumer so bursts and failures do not cascade. Delivery is usually at-least-once: if a consumer crashes before acknowledging, the message is redelivered. In a simulation with 25% crashes, 20 messages caused 31 side effects; an idempotency check brought it back to 20.
- At-most-once delivery acknowledges a message before processing and can lose it on a crash; at-least-once processes first and acknowledges after, so it can duplicate.
- In a simulation with a 25% crash rate before acknowledgement, 20 messages caused 31 side effects under at-least-once delivery, 11 duplicates.
- An idempotency check on a stable message id brought the same at-least-once simulation back to exactly 20 side effects for 20 messages.
- True exactly-once delivery is not achievable in general over a network that loses messages; systems offer exactly-once effects from at-least-once delivery plus deduplication.
- A classic queue such as SQS removes a message once it is acknowledged, while a log such as Kafka retains messages and each consumer group tracks its own offset.
- A poison message that fails on every retry should be retried a limited number of times with backoff, then moved to a dead-letter queue for inspection.
01What a queue gives you
A producer writes a message to a queue and carries on. A consumer reads it when ready. This separates the two sides in several ways. In time: the consumer can be down or slow and messages wait. In load: a traffic spike fills the queue while consumers work at their own steady pace (load leveling). In failure: a crash in the consumer does not fail the user's request. And in scale: add more consumers to drain faster, since each message goes to one of them (competing consumers).
Typical uses are sending emails, resizing images, processing payments after checkout, and passing events between services. The cost is that the work is now asynchronous: the user's request returns before the work is done, results arrive later, and you gain a new component to run.
02Queue versus log
In a classic broker queue (RabbitMQ queues, SQS), a message is removed once a consumer acknowledges it (in SQS, by explicitly deleting it; otherwise it reappears after the visibility timeout). In a log-based system (Kafka, Pulsar, RabbitMQ Streams), messages are appended to a partitioned log and retained for a period. Each consumer group tracks its own position (offset), so many independent consumers can read the same stream, and a consumer can rewind to reprocess old messages.
Ordering is usually guaranteed only within a partition or queue, not across all of them. If order matters per user or per order, route all messages with the same key to the same partition.
03Delivery guarantees
At-most-once: acknowledge first, then process. A crash after the acknowledgement loses the message. At-least-once: process first, acknowledge after. A crash between processing and acknowledging causes a redelivery, so the message may be processed twice. Exactly-once: each message affects the outcome exactly once. Over a network that can lose messages and acknowledgements, true exactly-once delivery is not achievable in general. What systems offer is exactly-once processing effects, built from at-least-once delivery plus deduplication or transactional writes.
Most production systems choose at-least-once and make consumers safe to repeat.
04Simulating duplicates
import random, collections
random.seed(4)
queue = collections.deque(range(20)) # 20 messages
effects = collections.Counter() # how often each message's side effect ran
while queue:
m = queue.popleft()
effects[m] += 1 # side effect, e.g. charge a card
if random.random() < 0.25: # crash before ack
queue.append(m) # visibility timeout expires -> redeliveredOutcome: 31 side effects for 20 messages, so 11 duplicates. Every message was processed at least once, none were lost, and some ran more than once. If the side effect were "charge the customer", that would be 11 double charges. The crash probability here is exaggerated, but any nonzero chance produces duplicates eventually, and a busy system will hit it.
05Idempotent consumers
seen = set()
while queue:
m = queue.popleft()
if m not in seen: # skip work already done
do_side_effect(m)
seen.add(m)
if random.random() < 0.25:
queue.append(m); continueWith the same crashes, the side effects ran exactly 20 times. The key (m here) is an idempotency key: a stable id carried by the message, such as an order id or a UUID generated by the producer. In real systems the "seen" record must live in the same database transaction as the side effect, or the crash window just moves. Other idempotent designs need no table at all: SET status = 'shipped' is safe to repeat, while balance = balance - 10 is not. Prefer operations of the first kind.
06Failures and poison messages
Not every failure is temporary. A malformed message will fail every time it is retried and, if the queue redelivers it forever, blocks progress or wastes the consumer. Standard practice: retry a limited number of times with backoff, then move the message to a dead-letter queue (DLQ) for inspection, and alert on its size.
Watch queue depth and the age of the oldest message. A growing queue means producers outpace consumers. Responses are to scale consumers out, slow the producers, or shed low-priority work. Set a visibility timeout (the time a received message is hidden before redelivery) longer than normal processing time, otherwise slow-but-healthy consumers cause duplicate work.
07Common interview questions
Q: Why use a message queue?
A: To decouple services, absorb bursts, retry failed work and scale consumers independently.
Q: At-least-once versus at-most-once?
A: At-least-once may deliver duplicates but never loses messages. At-most-once may lose messages but never duplicates.
Q: Can you get exactly-once delivery?
A: Not generally on an unreliable network. You get effectively-once by combining at-least-once delivery with idempotent processing.
Q: How do you make a consumer idempotent?
A: Use an idempotency key checked atomically with the side effect, or use naturally repeatable operations.
Q: What is a dead-letter queue?
A: A holding queue for messages that failed after the maximum retries, kept for inspection.
Q: How do you keep ordering?
A: Send related messages to the same partition using a key.