Idempotency: Designing APIs That Survive Retries

A customer submits an order. The server commits it to the database, but the connection drops before the response arrives. The customer sees a timeout and tries again.

Should the server create another order?

The difficult part is that a timeout does not reveal whether the first request failed, succeeded, or is still running. The client knows that it did not receive an answer. It does not know what happened to the business operation.

Idempotency makes repeated attempts at the same operation safe: repeating the request does not multiply its intended effect. It is a foundational skill for engineers building APIs, background jobs, integrations, and distributed systems because all of these encounter uncertain outcomes.

This article moves from the definition to a concrete database design, then examines concurrent requests, external services, event consumers, retention, and failure testing. The designs below are illustrative engineering choices; their assumptions and limits are part of the design.

What idempotency actually means

In mathematics, an idempotent function satisfies:

f(f(x)) = f(x)

Applying it again has the same effect as applying it once. For a state-changing operation, think of x as the relevant system state and f as applying one fixed command.

The scope matters. Setting a row to the same value might be idempotent for that row while a trigger sends another email on every update. Review the complete business effect, including downstream actions.

Idempotency also does not mean that the response must be identical. Deleting a resource might return 204 first and 404 on a later attempt while leaving the resource absent in both cases. HTTP explicitly distinguishes the intended effect from incidental logging and allows responses to differ. RFC 9110: Idempotent Methods

Some APIs offer the stronger convenience of replaying a stored response. That is an API contract layered on top of idempotent effects.

Why every backend engineer should understand it

Retries are an ordinary recovery mechanism. Without a way to recognise repeated intent, recovery can create a second problem: duplicated orders, repeated inventory deductions, redundant infrastructure, or duplicate notifications.

The practical benefits extend beyond preventing duplicates:

• Safer recovery: clients can repeat an interrupted operation without inventing a different recovery procedure for every endpoint.
• Simpler incident handling: operators can replay a failed job with a known operation identity and inspect its recorded outcome.
• More reliable integrations: a sender and receiver can recover independently when a response or acknowledgement disappears.
• Clearer product behaviour: a repeated submission can return the original result instead of leaving the user to guess what happened.

AWS describes caller-provided request identifiers as a way to express intent and make retries easier to reason about. Identical parameters alone are insufficient: two identical requests can represent two intentional purchases or two desired resources. Amazon Builders’ Library: Making retries safe with idempotent APIs

Knowing the term is useful in interviews. Knowing where to put the transaction boundary is what prevents production mistakes.

HTTP methods, safety, and application guarantees

HTTP defines GET, HEAD, OPTIONS, and TRACE as safe methods, and safe methods are idempotent. PUT and DELETE are also idempotent, although they can change server state. POST has no general idempotency guarantee. Safe means the client does not request a state change; it does not mean the implementation performs no logging. RFC 9110: Common Method Properties

For application design, ask what the endpoint actually does:

• A PUT that replaces a preference should preserve its intended effect on repetition.
• A POST /orders can support safe retries through an explicit idempotency contract.
• A PATCH needs its own analysis: setting a field and incrementing a field have different repetition behaviour.

Adding a header named Idempotency-Key does not implement anything by itself. The server must recognise it, coordinate competing requests, and preserve the result according to a documented policy.

Disabling a submit button helps the interface, but cannot coordinate another browser tab, a mobile reconnect, a retrying client, or a background worker.

Design the operation identity before the storage

Consider this illustrative order API:

POST /v1/orders HTTP/1.1
Content-Type: application/json
Idempotency-Key: 735632fe-f9ba-40b3-bd1e-7f2504b1289a

{"cart_id":"cart_123","cart_version":7,"delivery_option":"standard"}

The client creates a random operation key before the first attempt and retains it for every retry of that logical submission. A new intentional order gets a new key. A request or trace identifier can change per network attempt; the operation key must remain stable.

The server resolves the authenticated tenant and uses an identity such as:

(authenticated tenant, operation name and version, idempotency key)

This separates tenants and prevents accidental collisions between unrelated endpoints. Authenticate and authorise every attempt, including a replay. Possessing a key must never grant access to another user’s stored response.

The server also stores a request fingerprint: a hash of a canonical representation of the validated, meaningful input. Include the resource identity, relevant version, and parameters that change the operation. Define how omitted defaults, numeric representations, and object field order are normalised. Exclude incidental metadata such as trace IDs.

The fingerprint answers whether the caller reused a key with different input. It is not the operation identity: two legitimate operations may have the same fingerprint.

For this order example, cart_version prevents a retry from silently ordering a changed cart. The server still calculates authoritative prices and validates availability; it does not trust client-provided totals.

Write down the retry contract

The following is a proposed contract for this article’s API, not a universal status-code standard:

Store the response fields needed by the contract, including the application body and relevant headers such as Location. Do not blindly persist or replay cookies, credentials, or every transport header.

Provider contracts differ. Stripe stores the initial status and body after endpoint execution begins, including 500 responses; it does not store results for validation failures or concurrent execution conflicts. It also compares parameters and permits pruning keys after at least 24 hours. These are Stripe-specific rules, not defaults to assume for another service. Stripe: Idempotent requests

A PostgreSQL design for one atomic database operation

For a short operation whose business changes all live in one PostgreSQL database, place the operation record, business mutation, and stored result in one transaction.

Here is a minimal schema for that design:

CREATE TABLE api_operations (
    tenant_id uuid NOT NULL,
    operation_name text NOT NULL,
    idempotency_key text NOT NULL
        CHECK (octet_length(idempotency_key) BETWEEN 1 AND 255),
    request_hash bytea NOT NULL
        CHECK (octet_length(request_hash) = 32),
    response_status smallint,
    response_body jsonb,
    created_at timestamptz NOT NULL DEFAULT now(),
    completed_at timestamptz,
    PRIMARY KEY (tenant_id, operation_name, idempotency_key),
    CHECK (
        (completed_at IS NULL
         AND response_status IS NULL
         AND response_body IS NULL)
        OR
        (completed_at IS NOT NULL
         AND response_status IS NOT NULL
         AND response_status BETWEEN 200 AND 599
         AND response_body IS NOT NULL)
    )
);

The 255-byte key limit and 32-byte hash are example API choices. The latter assumes a SHA-256 fingerprint. JSONB stores a semantic JSON result; choose a byte-preserving representation if exact response bytes are part of your contract.

The primary key arbitrates ownership. A separate lookup followed by an unprotected write cannot do that: two requests can both observe that the key is missing.

Start by attempting an insert with bound parameters:

BEGIN ISOLATION LEVEL READ COMMITTED;

INSERT INTO api_operations (
    tenant_id, operation_name, idempotency_key, request_hash
)
VALUES ($1, $2, $3, $4)
ON CONFLICT (tenant_id, operation_name, idempotency_key)
DO NOTHING
RETURNING idempotency_key;

The application then branches on whether the insert returned a row. This is transaction pseudocode, not a complete HTTP handler:

if insert returned a row:
    validate business preconditions inside the transaction
    create the order using this same transaction
    store response status, body, and completed_at
    commit
    return the stored response only after commit succeeds
else:
    select the existing operation in a NEW SQL statement
    if missing or incomplete:
        abort and investigate or retry within a bounded policy
        never execute the business mutation on this branch
    if request_hash differs:
        end transaction and return the mismatch error
    end transaction and replay the completed result

on database error:
    roll back if possible
    classify the outcome before deciding how to retry

At PostgreSQL’s READ COMMITTED isolation level, a competing insert can encounter a uniqueness conflict from a transaction that was not visible in the statement’s original snapshot. A subsequent statement gets a fresh snapshot. That is why the replay lookup is a separate statement, rather than a same-statement query assumed to see the winning row. PostgreSQL: Transaction Isolation

ON CONFLICT DO NOTHING suppresses the conflicting insert, and RETURNING reports rows actually inserted. Only the request that inserts the operation record enters the mutation branch. PostgreSQL: INSERT

The design assumes that operation records are retained during the supported retry window and that every handler commits them only with a completed result. The schema allows a temporary incomplete record inside the transaction; the handler must never commit that intermediate state. Restrict write access and alert on any incomplete committed row.

This transaction still needs the application’s ordinary constraints: inventory availability, ownership, valid state transitions, and any durable uniqueness rule such as one order per checkout intent. Different keys do not bypass those business rules.

Walk through the crash windows

The strength of the design becomes clearer when execution stops at inconvenient moments:

Two concurrent requests may block on the unique constraint. Configure bounded lock and statement waits, and roll back failed transactions. A wait timeout means the caller must retry according to the contract; it does not permit bypassing the operation record.

If the operation store is unavailable, this endpoint should reject or defer execution. Continuing without deduplication would abandon the guarantee precisely when failures make retries likely.

For a definite transient rollback, retry the whole transaction with the original key and bounded backoff. When commit acknowledgement is lost, retain that key and resolve the outcome. Avoid a lagging replica for this decision.

External calls require another boundary

The single-database design does not make this sequence atomic:

Call a payment provider successfully
Process crashes
Local database never records the result

A database rollback cannot undo an independently committed provider action. Holding a database lock while making the call also does not close that failure window.

For an asynchronous workflow, an outbox can atomically store the order and a durable instruction to perform the next action. A worker delivers it later. Delivery may repeat, so the receiver still needs duplicate protection. AWS Prescriptive Guidance: Transactional outbox pattern

A possible operation lifecycle is:

accepted -> executing -> succeeded
                     -> failed with a known outcome
                     -> outcome unknown -> reconciliation

Give each downstream action a stable identity derived from the durable workflow operation and step. Reuse that identity with the provider’s supported idempotency mechanism. If the provider outcome is unknown, query by the known reference or reconcile before attempting an action under a new identity.

Long-running workers also need ownership rules. A lease expiry does not prove the previous worker stopped; it might resume after a pause. Use conditional state transitions and, where supported, fencing tokens that cause the protected resource to reject stale owners. A token checked only in the worker’s memory cannot fence an external service.

If a provider offers neither idempotent execution nor a reliable way to look up an uncertain outcome, document that limitation. Some cases require manual reconciliation. No local middleware can manufacture an end-to-end guarantee across that boundary.

Queues and webhooks need idempotent consumers

A worker can commit its database changes and crash before acknowledging a message. Redelivery then repeats an already completed operation.

A consumer inbox uses the same transactional idea:

begin transaction
    insert (consumer_name, event_id) with a unique constraint
    if newly inserted:
        apply the event's local database changes
commit
acknowledge the message

Recording the event as processed before committing its effect can lose work. Committing the effect before recording the event can duplicate work. Keeping both in one transaction closes that local gap. If processing calls another service, apply the external-boundary reasoning from the previous section.

Include consumer identity because several independent consumers may legitimately process the same event. Preserve stable event IDs during redelivery. Deliberate projection rebuilds need an explicit replay policy, often a separate projection generation, so old inbox records do not suppress required work.

For webhooks, verify authenticity before accepting an event, durably enqueue or record it before acknowledging acceptance, and deduplicate the business processing. Stripe documents duplicate deliveries and possible out-of-order events; it also notes that distinct event objects can describe the same underlying object and event type. Choose business deduplication rules carefully so legitimate later changes are not discarded. Stripe: Webhook best practices

The journal’s guide to event-driven boundaries explains how ownership and event contracts shape this design.

Idempotency does not solve every correctness problem

Ordering: applying an old address update twice may be harmless by itself, but applying it after a newer address update can overwrite current data. Use an explicit version policy, sequence checks, or appropriate per-entity ordering.

Lost updates: two different valid commands can still overwrite each other. Optimistic concurrency control protects changes relative to a version; idempotency recognises repetition of one command. They solve different problems.

Business duplicates: two keys can describe the same checkout intent. Keep durable domain constraints independent of the temporary retry record.

Exactly-once delivery: a handler may run repeatedly while its committed business effect happens once. Always identify the system boundary behind an exactly-once claim. Kafka documents transactional guarantees for Kafka processing and explains that output to external systems requires cooperation with those systems. A database write or email is not automatically covered by a Kafka transaction. Apache Kafka: Message Delivery Semantics

Overload: duplicate requests still consume network, authentication, and database capacity. Pair safe retries with deadlines, retry limits, and backoff with jitter. Designing Go services that fail gracefully covers those complementary controls.

Retention, failover, and operational visibility

A deduplication guarantee lasts only as long as the information needed to enforce it survives.

Choose retention from the longest supported retry and replay horizon, including offline clients, delayed queues, incident recovery, and downstream-provider policies. Document what happens after expiry. A short retention window can allow a delayed retry to create a new effect; retaining every full response forever creates storage and privacy costs.

One option is to expire bulky response data while retaining a smaller operation identity or a durable business uniqueness constraint. The contract must explain whether an old request returns a resource reference, an expiry error, or another documented result.

Multi-region deployments need an explicit ownership strategy. Independent regional stores that accept the same key before replication catches up can both execute the operation. Route an operation to one authoritative owner or use coordination that enforces the required uniqueness across writers.

Database recovery also needs attention. A restored operation table might forget work that an external service already completed. Reconcile that external history before replaying uncertain actions.

Monitor new operations, replay counts, fingerprint mismatches, lock waits, transaction failures, unknown outcomes, and the age of unfinished workflows. Correlate attempts through a protected operation reference, but avoid raw keys and payloads in routine logs and avoid per-key metric labels. Alert on growing reconciliation backlogs, not only HTTP errors.

Test the guarantee under failure

Sending the same request twice in sequence is a useful first check. It is insufficient evidence for production readiness.

Use integration tests against the actual database behaviour and inject failures around the commit boundary:

Assert durable business state as well as HTTP responses. An endpoint can return the same response twice while accidentally creating two orders underneath.

For external integrations, use a controllable test provider that can succeed and then drop the connection. That scenario reveals whether the workflow can recover an unknown outcome instead of treating every timeout as a failure.

Questions to answer before enabling retries

A reviewable design should explain which logical operation the key identifies, who owns execution, which writes commit together, how a lost response is recovered, and how long the guarantee lasts. It should also identify every side effect outside that transaction and its recovery mechanism.

Start with the business invariant: one accepted checkout intent must not create two orders merely because its request was repeated. Then choose the operation identity, atomic storage boundary, replay contract, and failure tests that enforce it.

That is why idempotency deserves a place in every backend engineer’s working knowledge. It turns an ambiguous retry into a deliberate recovery step whose effects can be explained, observed, and tested.

Sources

• RFC 9110: HTTP method safety and idempotency
• Amazon Builders’ Library: Making retries safe with idempotent APIs
• Stripe API: Idempotent requests
• PostgreSQL 18: Transaction isolation and concurrent statement visibility
• PostgreSQL 18: INSERT, ON CONFLICT, and RETURNING
• AWS Prescriptive Guidance: Transactional outbox pattern
• Stripe: Webhook delivery, duplicate events, and signature verification
• Apache Kafka 4.1: Message delivery semantics