Why a Timeout Makes Retrying Dangerous

Retries are easy to configure in a microservice architecture. Most frameworks can repeat a failed HTTP call with a small amount of code, exponential backoff, and a maximum attempt count. The ease of implementation often leads teams to configure retries before answering the more important architectural question: is the business operation safe to execute again?

In a distributed system, a timeout is not a failure result. It is an absence of certainty.

Consider an Order Service that sends a payment instruction to a Payment Service. The Payment Service may validate the request, authorize the charge, commit the transaction, and return a successful response. If that response is delayed or lost, the Order Service records only a timeout. It cannot determine whether the payment was rejected, is still being processed, or has already completed.

Repeating the request without resolving that ambiguity can convert a temporary communication failure into a duplicate business transaction.

The problem is not solved merely by limiting the retry count. Backoff and jitter can protect downstream capacity, but they do not protect business state. The Payment Service must be able to recognize that two technically separate HTTP requests represent the same payment instruction and ensure that only one of them produces the financial side effect.

Making that retry safe requires the Payment Service to govern the logical business operation rather than treat every HTTP request as an independent transaction.

Idempotency Must Identify the Business Operation

For idempotency to work, every attempt to complete the same payment must carry the same key. The key identifies the payment itself, not the individual HTTP request used to submit it.

For example, the Order Service may send:

POST /payments Idempotency-Key: order-8472-payment-1 Content-Type: application/json { "orderId": "8472", "amount": 100.00, "currency": "USD" }
If the request times out, the Order Service must retry with order-8472-payment-1. Creating a new key would make the retry appear to be a new payment, leaving the Payment Service with no reliable way to distinguish recovery from a second transaction.

The key cannot be evaluated independently of the request it represents. The Payment Service should associate it with the caller, the operation, and the relevant request data. A request hash is commonly stored for this purpose. If the same key later arrives with a different amount or currency, the service should reject it as a conflict rather than return an unrelated earlier response or process another payment.

The idempotency record must also remain available long enough to cover delayed responses, client retries, and later attempts to verify an uncertain result. If it expires too early, a legitimate retry may once again appear to be a new payment.

The key format is not the important design decision. What matters is that every attempt to complete one payment carries one identity that remains valid throughout the recovery period.

The Check and the Transaction Must Be Atomic

Recognizing the same idempotency key is only the first step. The service must also prevent two requests carrying that key from executing the payment at the same time.

Consider two retries reaching different Payment Service instances within milliseconds of each other. If both instances first query the database and neither finds an existing record, both may conclude that the payment is new and continue processing it. This race condition between checking for the record and creating it can produce a duplicate charge even though both requests carry the correct key

The service must therefore claim the payment atomically before initiating the protected action. In PostgreSQL, this can be enforced with a unique constraint and a conditional insert.

INSERT INTO idempotency_record (idempotency_key, request_hash, status) VALUES (:key, :requestHash, 'PROCESSING') ON CONFLICT DO NOTHING;
The service must inspect the insert result. The request that creates the record owns the payment attempt. A competing request that encounters the existing key must read its current state instead of initiating another payment.

An application-level lock is not sufficient in a horizontally scaled service. Each instance maintains its own memory, so a lock held by one instance is invisible to another. The ownership decision must be enforced through a database or another coordination mechanism shared by every instance.

When the idempotency record and the business update are stored in the same database, they should be committed in the same transaction. Otherwise, the service can enter one of two inconsistent states: the key is recorded, but the payment is not completed, or the payment is completed without retaining the information required to recognize a later retry.

A remote payment processor introduces a different boundary because it cannot participate in the local database transaction. The Payment Service should propagate the same idempotency key downstream, retain a durable PROCESSING state, and verify the provider’s result when the outcome remains uncertain. For asynchronous interactions, a transactional outbox can coordinate the local state change with message publication.

Atomicity protects the part of the operation controlled by the service. Beyond that boundary, correctness depends on downstream idempotency and an explicit recovery process for uncertain outcomes.

The Original Outcome Must Be Preserved

Once the service has claimed the operation, it must define what later requests receive. Preventing concurrent execution is not enough if a retry cannot determine whether the original payment is still running, completed, or failed.

The idempotency record should therefore preserve both the operation state and the result of the original execution. A practical state model may include PROCESSING, COMPLETED, and FAILED, together with the HTTP status, response body, and the payment identifier created by the transaction.

When the state is COMPLETED, the service should return the stored response. It should not execute the payment again, nor should it rebuild the response from the current state of the order.

That distinction matters because the order may have changed after the payment completed. Its status may have advanced, or related data may have been updated. Returning the current representation would describe the order as it exists now, not the result produced by the original payment request.

A retry received while the state remains PROCESSING also requires a defined API response. The service may return an in-progress status, provide an operation identifier for later lookup, or specify when the caller should retry. The absence of a completed result must never be interpreted as permission to start the payment again.

Confirmed validation or business failures may be preserved as terminal results so that the same invalid request receives the same response on subsequent attempts. An uncertain external outcome, however, should not be recorded as FAILED merely because the caller did not receive a response. It must remain unresolved until the downstream result is verified.

Preserving the original outcome allows the API to answer repeated requests consistently without executing the underlying business transaction again.

Retry Behavior Must Be Governed End to End

Idempotency makes repetition safe, but retry behavior can still damage a system when several layers repeat the same call independently.

The application service, HTTP client, API gateway, service mesh, and downstream library may each have their own retry configuration. Without coordination, no layer knows how many attempts have already occurred elsewhere in the request path.

The amplification is multiplicative. If the calling layer makes three total attempts and a downstream library makes three attempts for each invocation, one business request can produce as many as nine downstream executions. During a partial outage, those attempts consume connections, worker threads, database capacity, and processing time precisely when the dependency is least able to absorb additional load.

The architecture should define one authoritative retry policy for each interaction. Other layers should either disable retries or operate within explicitly coordinated limits. The owner of the policy must understand the dependency’s failure behavior and whether the business operation is safe to repeat.

Retry decisions should also be based on the actual outcome. Connection failures, temporary unavailability, rate limiting with an explicit retry interval, and selected server errors may justify another attempt. Validation errors, authentication or authorization failures, malformed requests, and business rule rejections are terminal and should return immediately.

The complete retry sequence must fit within the end-to-end latency budget. Per-attempt timeouts, backoff intervals, and maximum attempts cannot be configured independently. A caller with a two-second response objective cannot safely invoke a dependency whose internal retries continue after the caller has already abandoned the request unless the eventual result can be retrieved through a durable operation record.

Every attempt should preserve the original idempotency key and emit correlated operational evidence. Attempt number, initiating layer, elapsed time, remaining time budget, downstream status, and final exhaustion reason should be visible in logs and metrics. This allows teams to distinguish a recovered transient failure from an operation that still requires investigation or reconciliation.

Conclusion

Retries are a recovery mechanism, not a correctness guarantee. They can improve availability during transient failure, but they cannot determine whether an earlier invocation has already changed business state.

Idempotency closes that gap by ensuring that repeated delivery does not create another business effect. When the outcome remains uncertain, the system must verify or reconcile it rather than execute the operation again.

Retries address availability. Idempotency protects transaction correctness. Reliable microservices require both because they solve different failure modes.