Fictional teaching scenario. This is not an account of a particular deployment.
You design an API that accepts payments for orders. The application calls POST /payments. The server sends the request to a payment provider and records the result. Sometimes the provider charges the customer, but the response is lost to a timeout and the application retries.
The team proposes retrying every error after checking whether the order is marked paid. Early tests pass, but production sees concurrent requests for the same order and delayed provider confirmations.
How will you make retries safe and avoid starting a second charge? What happens when the first call has an unknown outcome?
Guiding questions
- Are the status check and provider call atomic?
- How do you distinguish a retry from a deliberate new attempt?
- Does the provider support idempotency keys, and for how long?
- What do you return when a charge may have succeeded but its response is lost?