Mediumidempotencyretriesexponential backoffjitterreliabilityAPI design

Step 8 of 13 in System DesignIntermediateRead first: Message Queues and Delivery Guarantees

Idempotency, Retries and Backoff

Retry safely after a timeout: idempotency keys stop double charges, jittered backoff stops retry storms, and budgets cap the load. In simulation, jitter cut 5050 calls to about 600.

Yeshwanth Reddy Vallela

Cybersecurity, LLMs, Agentic AI, Full-stack, Java, Python

Published 23 Oct 2026

Short answer

An idempotency key is a unique ID the client sends with a request and reuses on every retry, so the server runs the operation once and replays the saved response for repeats. Interviewers expect the details: store the key and the effect in one transaction, add jitter to exponential backoff, and cap retries with a budget.

A timeout does not say whether the request ran, so a retry needs an idempotency key or the operation can happen twice. Retries also need jittered backoff and a budget, or they amplify an outage: in a simulation, 100 synchronised clients made 5050 calls against 576 to 657 with jitter.

Key facts

RFC 9110 section 9.2.2 defines a method as idempotent when several identical requests have the same intended effect as one, and counts PUT, DELETE, GET, HEAD, OPTIONS and TRACE as idempotent.
POST is not idempotent by specification, so payment and signup endpoints add an application-level idempotency key that the server uses to treat repeated requests as one.
In a simulation where 100 clients contend for a resource, plain exponential backoff made 5050 calls, the same as no backoff, while full jitter made 643.
When five service layers each try a failing call up to 3 times, the bottom service receives 3^5 = 243 attempts for one user request.
In a simulated total outage, a token-bucket retry budget cut backend load from 4.00 times the first attempts to 1.10 times.
An idempotency key and the effect it protects must be written in one database transaction, or a crash between the two steps can repeat or lose the charge.

Stripe: Idempotent requests AWS Architecture Blog: Exponential Backoff and Jitter RFC 9110 section 9.2.2: Idempotent methods

01The timeout that does not tell you what happened

A timeout says only that no answer arrived, so the request may have failed or may have succeeded with its reply lost on the way back.

A client sends POST /charge for $500 and waits. Five seconds pass, nothing comes back, the client gives up. Three things could have happened: the request never reached the server, the server crashed halfway, or the server finished the charge and the response was dropped. From the client's side these look identical.

If the client never retries, the first two cases leave the customer without a purchase. If it retries blindly, the third case charges them twice. The script below reproduces that third case in one process, with SQLite as the server's database: the charge runs, the reply is thrown away and the client sees a timeout, then it sends the same request again. With no protection the account ended at a balance of 1000 after 2 executions. With an idempotency key on both attempts it ended at 500 after 1 execution, and the retry received the original reply (charge_id 1). These are simulation results, not measurements of a real payment system.

A retry is only safe when repeating the operation cannot do damage twice, and that property has to be built, not assumed. The prerequisite is the at-least-once idea from message queues and delivery guarantees, because a retrying client is exactly an at-least-once sender.

The server in the script checks the key, runs the charge and saves the key inside one transaction (BEGIN IMMEDIATE). That is deliberate: a version that saved the key in a separate step could crash in between and leave a charge with no record. More on that in the section on key stores.

import sqlite3, threading, tempfile, os, time, json

class Timeout(Exception): pass

def make_db():
    path = os.path.join(tempfile.mkdtemp(), "bank.db")
    c = sqlite3.connect(path)
    c.execute("CREATE TABLE account (balance INTEGER, charges INTEGER)")
    c.execute("INSERT INTO account VALUES (0, 0)")
    c.execute("CREATE TABLE idem (key TEXT PRIMARY KEY, fingerprint TEXT, status INTEGER, body TEXT)")
    c.commit(); c.close()
    return path

def charge(path, amount, key=None, slow=0.0):
    """Server side. Returns (status, body). Key check, charge and key save share one transaction."""
    db = sqlite3.connect(path, timeout=5, isolation_level=None)
    try:
        db.execute("BEGIN IMMEDIATE")             # one writer at a time
        if key is not None:
            row = db.execute("SELECT fingerprint, status, body FROM idem WHERE key=?", (key,)).fetchone()
            if row:
                db.execute("COMMIT")
                if row[0] != str(amount):
                    return 422, "key reused with different amount"
                return row[1], row[2]             # replay the saved reply
        time.sleep(slow)                          # pretend the payment processor is slow
        db.execute("UPDATE account SET balance = balance + ?, charges = charges + 1", (amount,))
        n = db.execute("SELECT charges FROM account").fetchone()[0]
        body = json.dumps({"charge_id": n})
        if key is not None:
            db.execute("INSERT INTO idem VALUES (?,?,?,?)", (key, str(amount), 200, body))
        db.execute("COMMIT")
        return 200, body
    finally:
        db.close()

def lossy_post(path, amount, key=None, lose=False, slow=0.0):
    status, body = charge(path, amount, key, slow)
    if lose:
        raise Timeout()                           # the reply never reaches the client
    return status, body

def state(path):
    c = sqlite3.connect(path); r = c.execute("SELECT balance, charges FROM account").fetchone(); c.close(); return r

def client(path, key):
    try:
        return lossy_post(path, 500, key, lose=True)
    except Timeout:
        return lossy_post(path, 500, key)         # retry, same key

# 1. no key: the response is lost, the client retries
p = make_db()
try: lossy_post(p, 500, None, lose=True)
except Timeout: print("no key: attempt 1 timed out")
print("no key: retry ->", lossy_post(p, 500), "state (balance, executions):", state(p))

# 2. same flow with a key
p = make_db()
print("with key:", client(p, "K1"), "state:", state(p))

# 3. same key, different amount
print("same key, amount 900 ->", charge(p, 900, "K1"), "state:", state(p))

# 4. two threads, one key, slow processor
p = make_db(); out = []
def worker(): out.append(charge(p, 500, "K2", slow=0.2))
ts = [threading.Thread(target=worker) for _ in range(2)]
t0 = time.time(); [t.start() for t in ts]; [t.join() for t in ts]
print("two threads, one key ->", sorted(out), "state:", state(p), "elapsed %.2fs" % (time.time() - t0))

Sequence diagram of a client, server and key store: a charge runs, its response is lost, and a retry with the same key gets the stored response — The retry carries the same key, so the server replays the stored reply and the charge runs once.

02Idempotent is not the same as safe

An operation is idempotent when repeating it leaves the same final state as doing it once, which is different from being read-only (safe).

The HTTP specification defines the term plainly: a method is idempotent if the intended effect on the server of several identical requests is the same as the effect of one. RFC 9110, section 9.2.2, counts PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) as idempotent. Note the word intended. Nothing stops a server from writing a log line on every GET; the promise is about the effect the method is meant to have.

Idempotent does not mean harmless. DELETE /orders/7 changes state, but the second DELETE leaves the world the same as the first: the order is gone. PUT /users/3 with a full body behaves the same way. Compare "set the balance to 500" (idempotent) with "add 500 to the balance" (not). The difference lives in the operation, not in the verb, so idempotence is a promise the implementer keeps, and an endpoint can break it by accident.

Note what idempotent does not require: the same response each time. A repeated DELETE may return 204 the first time and 404 the second, and it is still idempotent because the state afterwards is identical.

POST is not idempotent by specification. Creating an order twice creates two orders. That is a problem, because POST is the verb used for payments, signups and anything else that creates something. The fix is an application-level contract layered on top: the client attaches a unique key, and the server promises to treat all requests carrying that key as one.

03Idempotency keys: what the server stores

A key store maps each client-chosen key to the request fingerprint and the saved response, so a repeat returns the stored reply instead of re-running the work.

The client generates the key, typically a random UUID, once per logical operation, and reuses it on every retry of that operation. A new purchase gets a new key. Stripe's documented design is a good reference: keys up to 255 characters, the status code and body of the first execution saved and replayed (even for a 500), parameters compared against the original with an error on mismatch, and keys that may be removed after at least 24 hours. Requests that fail validation, or that collide with one still running, are not saved, so they can be retried.

The server logic in the script has four outcomes:

Key unseen: run the work, save the response against the key, return it.
Key seen, same parameters: return the saved response. After a lost reply and a retry, the balance stayed at 500.
Key seen, different parameters: reject. Sending K1 again with amount 900 after a 500 charge returned 422 and left the state at (500, 1).
Key seen while the first request is still running: either refuse with a conflict (Stripe-style) or make the second request wait. The script waits, because BEGIN IMMEDIATE serialises writers. Two threads sent K2 at once with a 0.2 s processor delay; both got 200 with charge_id 1, one execution happened, and the run took about a quarter of a second (0.2 s of that is the simulated processor delay). A design that stores an "in progress" marker instead would return 409 to the loser.

The detail that bites in real systems is atomicity. If the server does the work and then saves the key as two separate steps, a crash between them leaves a charge with no record, and the retry runs it again. Reversing the order has the opposite flaw: a crash after saving the key but before the work leaves a key that claims success. So write the key and the effect in one database transaction (see the ACID and Transactions topic, acid-and-transactions), or make the key a unique constraint that the effect depends on. Scope keys per account or user so one client cannot replay another's key. Keys expire, so a retry after the window is a brand-new request. And keys must not contain personal data, since they get logged and stored.

04Retries that help and retries that kill

Each layer that retries multiplies the load on the layer below it, so three attempts at each of five layers means 243 calls to the bottom for one request.

Idempotency makes a retry safe. It does not make it useful. A retry helps when the failure was brief: a dropped packet, a restarting pod, a leader election. It hurts when the failure is overload, because the retry adds work to a system that is already failing to keep up.

Layers make this worse. Suppose a request passes through five services, each of which tries a failing downstream call up to 3 times. The bottom service sees 3 x 3 x 3 x 3 x 3 = 243 attempts for one user click. The script printed 3, 9, 27, 81 and 243 bottom calls for one to five layers (arithmetic plus a nested simulation where every call fails).

This is also why error classification matters. Retry on timeouts, connection resets, 429 and 5xx responses, since those can be transient. Do not retry a 400 or 404: the same request fails the same way and you have only added load. A 429 deserves extra care, because the server is explicitly telling you to slow down, so honour Retry-After if it is present.

Every retry also needs a deadline. A request the user abandoned 30 seconds ago still consumes capacity if some layer keeps retrying it. Pass the remaining time budget down the call chain and stop when it runs out.

# layered retries
counter = [0]
def call(depth, attempts=3):
    if depth == 0:
        counter[0] += 1; return False
    return any(call(depth - 1) for _ in range(attempts))
for d in range(1, 6):
    counter[0] = 0; call(d); print("layers", d, "-> bottom calls", counter[0])

05Backoff and the three jitter variants

Exponential backoff spaces retries out, but only random jitter breaks up clients that fail together, cutting 5050 calls to about 600 in a 100-client simulation.

Waiting between attempts is the first fix, and doubling the wait each time is the usual form: capped exponential backoff, sleep = min(cap, base x 2^attempt). On its own it has a flaw. Clients that failed at the same moment all wait the same time and collide again.

The script simulates that. A resource serves exactly one request per tick; 100 clients start together and every loser retries. Results are the average of 20 seeded runs, with base 2 ticks and cap 64 ticks (arbitrary values), where e = min(cap, base x 2^attempt):

No backoff (retry next tick): 5050 calls, 100 ticks.
Exponential, no jitter: 5050 calls, 6079 ticks.
Full jitter, sleep = random(0, e): 643 calls, 167 ticks.
Equal jitter, sleep = e/2 + random(0, e/2): 657 calls, 223 ticks.
Decorrelated, sleep = min(cap, random(base, previous sleep x 3)): 576 calls, 194 ticks.

Plain exponential made exactly as many calls as no backoff because the clients never desynchronised: the whole crowd retried together each round, one got through, the rest waited again. 5050 is 100 + 99 + ... + 1. Every jitter variant cut calls by close to nine tenths. Among the jitter variants, full jitter finished first and equal jitter finished last; decorrelated made the fewest calls. No backoff finishes in the fewest ticks of all, but only by hammering the resource with 5050 calls.

The AWS Architecture Blog article on exponential backoff and jitter runs a comparable contention simulation. Its broad result matches: plain backoff helps little, jitter removes most of the work. Full jitter beating equal jitter also matches. The difference is decorrelated jitter, which that article reports as making more calls, while this simulation made the fewest. The model here is simpler (one winner per tick, a different cap, 20 runs), so compare shapes, not exact numbers.

import random, statistics

BASE, CAP = 2, 64

def sleep_for(policy, attempt, prev):
    e = min(CAP, BASE * 2 ** attempt)
    if policy == "none":         return 1
    if policy == "exp":          return e
    if policy == "full":         return random.uniform(0, e)
    if policy == "equal":        return e / 2 + random.uniform(0, e / 2)
    if policy == "decorrelated": return min(CAP, random.uniform(BASE, prev * 3))

def storm(policy, n=100, seed=0):
    random.seed(seed)
    due = {c: 0.0 for c in range(n)}      # next attempt time per client
    fails = {c: 0 for c in range(n)}
    prev = {c: BASE for c in range(n)}
    calls, tick = 0, 0
    while due:
        tick += 1
        ready = [c for c, t in due.items() if t <= tick]
        if not ready: continue
        calls += len(ready)
        winner = random.choice(ready)      # the resource serves one call per tick
        del due[winner]
        for c in ready:
            if c == winner: continue
            s = max(1, sleep_for(policy, fails[c], prev[c]))
            fails[c] += 1; prev[c] = s
            due[c] = tick + s
    return calls, tick

for p in ["none", "exp", "full", "equal", "decorrelated"]:
    runs = [storm(p, seed=s) for s in range(20)]
    print("%-13s calls %7.0f  ticks %7.0f" % (p, statistics.mean(r[0] for r in runs), statistics.mean(r[1] for r in runs)))

06Retry budgets and where to retry

Retry in one layer only and cap total retries with a token bucket; in a total-outage simulation this held load at 1.10x instead of 4x.

Backoff controls when a client retries. A budget controls how much retrying the client is allowed at all. A common form is a token bucket (the same idea as in the rate limiting topic, rate-limiting-algorithms, pointed at your own retries): each retry costs a token, each first attempt refunds a small fraction, and when the bucket is empty the client fails fast instead of retrying. The bucket is kept per client or process, not globally, so each caller limits itself without coordination.

The script simulates a total outage: 100 new requests per second for 60 seconds, every call failing, up to 3 retries each. Without a budget the backend saw 24000 calls for 6000 first attempts, a 4.00x load. With a bucket of 10 tokens and a refund of 0.1 per first attempt, it saw 6609 calls, 609 of them retries, a 1.10x load. In a partial failure the budget should still allow some retrying (expected behaviour, not simulated here); in a full outage it shuts the tap.

Where to put the retry is the other half of the decision. Choose one layer, usually the one closest to the user or the one that knows the operation is idempotent, and make every other layer try once. Pair that with a short timeout at each hop that is lower than the caller's, so the inner call fails while the outer one still has time to react.

A circuit breaker is the related tool: after enough failures it stops calls entirely for a while. It adds state and a sudden change of behaviour, so many teams start with a budget and add a breaker only where needed.

# retry budget, total outage
def outage(budget, seconds=60, rps=100, max_retries=3):
    tokens, backend, first = 10.0, 0, 0
    for _ in range(seconds * rps):
        first += 1; backend += 1
        tokens = min(10.0, tokens + 0.1)
        for _ in range(max_retries):
            if budget:
                if tokens < 1: break
                tokens -= 1
            backend += 1
    return first, backend
for b in (False, True):
    f, be = outage(b); print("budget" if b else "no budget", "first", f, "backend", be, "load %.2fx" % (be / f), "retries", be - f)

07Common interview questions

Expect to design an idempotent payment API, explain why jitter exists, say which errors to retry, and handle two requests with one key.

Q: Design an idempotent payment endpoint.
A: The client generates a unique key per purchase and sends it in a header. The server, in one transaction, checks the key, runs the charge and stores the key with a fingerprint of the request and the response. A repeat with the same key and body returns the saved response; the same key with a different body gets an error. Keys are scoped to the account and expire after a retention window such as 24 hours.

Q: Is POST idempotent?
A: Not by specification. Two identical POSTs can create two resources. Idempotency keys add an application-level guarantee on top of it. GET, PUT, DELETE and HEAD are idempotent by definition, though the server still has to implement them that way.

Q: Why add jitter to exponential backoff?
A: Without randomness, clients that failed together retry together and collide again. In the 100-client simulation, plain exponential backoff made 5050 calls, the same as no backoff, while full jitter made about 643.

Q: Which failures should you retry?
A: Transient ones: timeouts, connection resets, 429 and most 5xx, and only for idempotent operations or ones protected by a key. Do not retry validation errors (400, 422) or not-found (404), which fail identically.

Q: Two requests with the same key arrive at once. What happens?
A: The key check and the work must be atomic, for example inside one transaction or behind a unique constraint. The second request then either waits and replays the first result (what the SQLite script does) or gets a 409 and retries later. Without atomicity, both would run.

Up next · step 9 of 13

Distributed Cache Design: Sharding, Stampedes and Staleness

Design notes for a cache spread over many nodes: cache-aside, why consistent hashing moves 22% of keys instead of 80%, preventing a cache stampede (50 DB calls down to 1), TTL jitter, and staleness.

Previous: The API Gateway Pattern