RA

Draft · Distributed systems

Retry amplification: when “resilience” becomes the outage.

Retries are useful only when the whole request path has capacity for the additional work they create.

Published · Updated · 6 min read

Editorial note: This is a vendor-neutral educational draft. It does not describe a specific employer, service or incident.

A timeout is ambiguous. The request may have failed before execution, during execution or after the work succeeded but before the caller received a response. Retrying can therefore recover a transient failure. It can also magnify a slow dependency into a widespread outage.

A small policy can create a large multiplier #

Consider a request that travels through three layers. If each layer tries an operation once and then makes two retries, the final dependency can receive up to 27 attempts for one original customer request. In reality, timeouts, hedging and fan-out make the arithmetic less tidy—but not less dangerous.

attempts at dependency = client × gateway × service = 3 × 3 × 3 = 27

The dependency is usually already slow or overloaded when this multiplier appears. More attempts increase queue depth, extend latency and cause more callers to reach their own timeout thresholds. The system has accidentally created a positive feedback loop.

Give retries an owner #

Retrying should have a clear owner in the call chain. That owner needs enough context to decide whether an operation is safe to repeat, whether the remaining deadline can accommodate another attempt and whether the dependency is healthy enough to justify more traffic.

  • Propagate deadlines. A downstream call should receive the remaining budget, not a fresh timeout that ignores time already spent.
  • Use exponential backoff with jitter. This avoids synchronised retry waves after a shared failure.
  • Set a retry budget. A service should limit retry traffic to a small fraction of normal request volume.
  • Make idempotency explicit. A state-changing operation needs a defensible replay strategy before automatic retry is safe.

Reduce work while the system is unhealthy #

The best retry is sometimes no retry. When a dependency is demonstrably unhealthy, the caller needs a product-safe alternative: serve a cached value, omit a non-critical enrichment, return a clear partial response or shed work before the queue becomes the incident.

This is where circuit breaking, concurrency limits and load shedding belong. They are choices about which work the system is permitted to do when capacity is scarce.

Observe the extra work, not only the final error #

A dashboard that counts only final request success can hide the growing retry storm. Track attempts per original request, remaining deadline at each hop, queue depth, dependency saturation and the percentage of traffic served by a degraded path. Those signals reveal whether a resilience mechanism is helping or adding fuel.

The operating rule #

Before adding retries, ask one question: if this dependency is slow, can it afford the extra demand? If the answer is unknown, first define the deadline, failure mode and fallback. Reliability is not the number of attempts you make—it is the amount of customer value the system can preserve under stress.