Idempotency in Distributed Systems

Overview

An idempotency key turns an otherwise unsafe-to-retry operation, a payment creation, an order submission, into one a client can safely retry after an uncertain outcome (a timeout, a dropped connection) without risking a duplicate side effect: the server records the key alongside the operation's result, and if that same key shows up again, it returns the exact original result, error included, rather than re-executing anything. Reusing a key with different parameters is rejected as an error rather than silently processed, because an idempotency key is a promise that one specific operation happened once, and mismatched parameters mean either that promise no longer holds or the client made a mistake generating the key. Keys are most commonly needed for genuinely non-idempotent operations like POST; GET is already safely repeatable by definition and never needs one, while whether DELETE needs one is API- and version-specific, since a DELETE can still carry non-idempotent side effects some APIs choose to protect with a key.

Quick Reference

Syntax

curl https://api.example.com/v1/charges \
  -H "Idempotency-Key: 4f3a9c2e-8b1d-4e6a-9f2c-1a2b3c4d5e6f" \
  -d amount=2000 -d currency=usd

Examples

# Client-side: generate the key once per logical operation,
# reuse it across retries of that same attempt only.
import uuid

idempotency_key = str(uuid.uuid4())
for attempt in range(3):
    try:
        response = create_charge(amount=2000, idempotency_key=idempotency_key)
        break
    except TimeoutError:
        continue  # safe to retry with the same key

Visual Diagram

Common Mistakes

• Generating a new idempotency key on every retry attempt, which defeats the purpose entirely, each "retry" becomes a genuinely new operation the server has never seen.
• Reusing the same key across requests with different parameters, expecting the new parameters to be processed rather than rejected.
• Adding idempotency keys to GET requests, which are already safely repeatable and don't need this mechanism; assuming the same is automatically true of every DELETE without checking that endpoint's documented contract.
• Assuming idempotency keys are retained forever; most systems prune them after a bounded window, after which reuse is treated as a brand-new request.

Performance

• Looking up whether an idempotency key has already been seen is a fast, indexed operation, negligible overhead compared to the operation it protects against duplicating.
• Idempotency key storage has a real, bounded retention window; this is a deliberate trade-off between protecting against reasonably-timed retries and not storing request history indefinitely.

Best Practices

• Generate exactly one idempotency key per logical operation, and reuse that same key for every retry attempt of that same operation.
• Use a sufficiently random identifier (a UUID) as the key, and avoid embedding sensitive data in it.
• Apply idempotency keys based on the endpoint's documented retry contract rather than the HTTP verb alone; POST-style state changes almost always need one, GET never does, and DELETE depends on the specific API and its side effects.
• Design the client to treat a mismatched-parameters error as a real bug to fix, not something to retry past.