Your Retry Policy May Be Feeding the Outage

A retry looks harmless when viewed from one service. A timeout occurs, a client waits briefly, then tries again. In a dependency graph, that local decision is a claim on shared capacity: more work is sent toward a component that may already be failing or overloaded. If every layer makes the same reasonable decision independently, the system creates the exact traffic surge that prevents recovery.

The useful rule is simple: spend a retry once, at the layer nearest to the failure that has evidence the failure is transient and can be recovered locally. A retry is not merely a resilience parameter attached to every client. It is a recovery action with an owner.

One user request can become an expanding tree

Consider a request that crosses three dependency levels before reaching a failing service. If each caller retries once, Uber’s example shows that the failing service can receive eight times its baseline traffic. The arithmetic is not subtle: each retry recreates downstream work, and each recreated call gives lower layers another chance to retry. A small local policy becomes multiplicative behavior across the call chain.

A 10% retry budget at every layer constrains the rate at which this happens, but it does not resolve the underlying ambiguity. Which layer should use the scarce extra attempt? Per-service budgets, exponential backoff and jitter reduce the force of amplification. They do not establish whether an upstream service is seeing a new recoverable fault or merely relaying the same failure that a downstream caller has already tried to resolve.

Google’s SRE guidance reaches the same conclusion: in a deep stack, a failed request should be retried only by the layer immediately above the rejecting layer. Retrying at multiple layers creates combinatorial amplification. That is an architectural statement, not a tuning recommendation.

The hard problem is identifying the error owner

The service that returns an error is not necessarily the service that caused it. A service may fail because one of its own outbound calls failed; its error is then a propagated symptom. Treating every returned error as locally owned gives each upstream layer permission to begin another recovery loop.

Uber’s September 17 production account describes an error-ownership mechanism operating in the service mesh for user-facing APIs. It correlates a service’s inbound failure with failures among the outbound calls made while that request was processed. When a service determines that it is propagating a downstream failure, it can return the error as unclaimed. Higher layers then do not retry it again.

Uber reports that this mechanism prevented an estimated 9.5 million spurious requests during a November 18, 2025 outage. It also reports reducing the maximum retry-storm radius across user-facing APIs from as high as 25 dependency levels to 3, and the average from 20 to 2. Those numbers are evidence from one production environment, not a transferable forecast. Their importance is more basic: they show that retry ownership can be implemented and operated across a large service mesh.

Make retry provenance part of the request contract

A system cannot enforce a single recovery point if the context of earlier attempts disappears at each hop. The practical answer is a retry-provenance contract in shared RPC or mesh infrastructure, rather than an informal convention in application code.

For each request and returned error, carry enough context to distinguish an original operation from its recovery history: an original-request identifier, cumulative attempt count, remaining deadline, error classification, retry-eligibility signal, and an ownership state that distinguishes a causal error from a propagated symptom. The exact wire format can vary. The operational requirement does not: an upstream caller must be able to see that the retry opportunity was already considered below it.

• The immediate caller of the error-owning boundary normally gets the recovery decision.
• Services propagating a downstream failure return its ownership state instead of treating it as a new local fault.
• Upstream services propagate that result without starting another retry loop.
• Missing, contradictory or dropped provenance takes a deliberately bounded fallback path rather than silently restoring unrestricted retries.

This is not a claim that failures can always be attributed perfectly. Uber explicitly addresses coincidental errors and dropped context. A system therefore needs to expose uncertainty and decide what happens when attribution is unavailable. The dangerous default is not caution; it is allowing every layer to infer a new independent retry right from incomplete context.

Supporting controls still matter

Retry ownership does not replace the familiar reliability controls. gRPC provides per-method retry policies, exponential backoff with jitter, retry throttling, server pushback and retry-attempt telemetry. Envoy retry budgets can bound retry volume relative to active and pending requests. Google recommends per-request attempt limits, per-client budgets and metadata containing prior-attempt counts. These are useful mechanisms around the recovery point.

Nor does ownership make retries safe for operations with external effects. AWS guidance is clear that automated retries require idempotent API contracts when repeating an operation could create a duplicate effect. An idempotency key answers whether another attempt can safely repeat an effect. Retry ownership answers which layer is permitted to make that attempt. A deadline limits how long the operation may consume resources. Each solves a different problem.

Measure the operation, not the configuration

Founders and platform teams should stop treating this as a checklist of sensible client settings. A service can have a modest retry budget, jitter and a short attempt limit—and still participate in a destructive retry tree when every dependency applies the same policy independently.

Test a dependency failure by tracing one user operation. Count total attempts generated across the graph, identify the component receiving them, and record where each retry was authorized. Then break provenance deliberately: remove an ownership signal, drop attempt metadata, or introduce a coincidental local error. The expected result should be a bounded fallback, not a return to uncoordinated retry behavior.

The design goal is not to eliminate retries. A well-placed retry can preserve availability during a transient fault. The goal is to ensure that a failure produces one informed recovery decision, rather than a stack of independent guesses. In a deep system, recovery capacity is shared. Treat the retry accordingly.