Learning goal: assign retry ownership and a shared attempt budget rather than multiplying requests to an overloaded dependency.

Scenario: nine calls instead of one

This is a fictional educational example, not a report of a production incident. An API calls an intermediary service, which reads a catalogue. Both calling layers allow up to three attempts, including the first request. If the catalogue keeps failing, one user operation can generate nine catalogue calls (3 × 3). Extra traffic hits the dependency precisely when it has the least spare capacity.

Three design options

  1. A — retry at every layer. Simple locally, but attempts multiply across layers and tail latency can increase.
  2. B — one retry owner. The catalogue adapter retries safe reads; upstream callers propagate one deadline and never restart the entire chain.
  3. C — no retry. Appropriate when repeating an operation can cause an irreversible side effect or the previous outcome cannot be determined.

Conditional decision

Choose B for reads, provided we verify the behavior of the SDK, HTTP client, gateway and intermediaries. The attempt budget applies to the user operation, not independently to each layer. Capped exponential backoff with jitter reduces synchronized retry storms; concurrency limits protect the catalogue from queued demand. Once the shared deadline expires, no further attempt is allowed. Writes require an idempotency key or a reliable outcome-lookup contract; otherwise automatic retries remain disabled.

How do we prove it works?

  1. Trace the real API → intermediary → catalogue path and inventory retry behavior in every SDK and gateway.
  2. Inject HTTP 429 and 503 responses, both with and without Retry-After. This header suggests a delay; it does not authorize retries beyond the operation deadline.
  3. Measure catalogue calls per user operation, p95/p99 latency, concurrent requests and recovery time after failures stop.
  4. Verify that attempts stop at the budget and unsafe writes are not repeated without a safe contract.

Architect's question: if the gateway and SDK both retry independently, which layer owns the decision and who accounts for the total attempt cost?

Primary references

Related topics: idempotency and outcome lookup, bulkhead, and SLO error budgets.