The puzzle: a missing response

In a fictional distributed system, Service A asks Service B to execute a state-changing business operation. No reply arrives before the timeout. B may have committed the change, may still be processing it, or may never have received the command.

A records UNKNOWN. This need not be the operation's state in B; it describes what A knows. Sending a fresh command might duplicate the business effect. Calling the request a failure just because it timed out could be equally wrong.

Your task: design a system that detects ambiguous outcomes, determines the result where possible, and escalates unresolved cases with clear ownership — including cases nobody reports.

This fictional teaching scenario is not a description of a real organization.

Three possible designs

A. Status lookup

A assigns a stable operation identifier and queries B for the outcome. B exposes a status API distinguishing in-progress and terminal states.

Advantages: relatively simple implementation, clear accountability and easier auditing. Costs: polling traffic, dependence on B's availability, rate limits and delayed updates. Unless the contract makes that guarantee, HTTP 404 alone does not establish that an operation was never performed.

B. Events with reconciliation

After durably committing the result, B publishes an outcome event for A. Keeping the committed outcome and the intention to publish in sync may require a transactional outbox or an equivalent mechanism. A separate reconciliation process scans operations without terminal confirmation, aligns the two views and detects lost or delayed messages.

Advantages: reduced dependence on synchronous responses and improved scalability at higher volumes. Costs: extra components, event duplication and ordering, deduplication and durable event publication. An event must not become the only evidence of truth.

C. Exception queue with accountable operators

After bounded attempts, unresolved cases move to an operational exception queue. Each case has an owner, deadline, history and escalation policy.

Advantages: exceptional cases cannot silently disappear. Costs: manual effort, delay and a dependency on operational discipline. Without a technical way to establish the outcome, support teams are still unable to decide. A customer complaint must not be the only trigger.

The hidden catch

You need not choose exactly one. A sound design may combine stable identifiers and idempotent execution with polling or events, periodic reconciliation and exception escalation.

Do not promise universal cross-service “exactly once.” Instead, prevent repeated business effects at the executing service: the same idempotency key must refer to the same intent, and its history must be retained long enough for delayed retries. Identical payloads can sometimes represent two distinct, legitimate requests.

When an idempotency record and a business effect share a transactional store, they should be committed atomically. If they span independent resources, explicitly design recovery from partial failures.

Your architecture contract

  1. Source of truth: B or a named authoritative register determines whether the business effect occurred. UNKNOWN in A indicates uncertainty, not proof of non-execution.
  2. Identity: a durable operation ID, idempotency key and correlation ID; a retry must preserve the original intent.
  3. State model: for example RECEIVED, PROCESSING, SUCCEEDED and REJECTED, with UNKNOWN representing the caller's knowledge. Transitions and semantics belong in the contract.
  4. Time and ownership: define response timeouts separately from final-resolution targets, a named owner and an escalation policy.
  5. Missing outcomes: proactively scan records with no verified terminal result, regardless of customer complaints.
  6. Audit trail: record when the operation was requested, what was confirmed, who resolved it and what evidence supported the conclusion.

Anti-pattern: “a timeout means failure; support will handle the rest”

A communication timeout does not prove a business rejection. Customer support is a user-facing process, not a data-consistency mechanism; it does not automatically discover cases nobody reports.

Likewise, do not launch a compensating transaction solely because the reply is missing. First establish what actually happened and whether the effects can be safely reversed.

A sample mini-ADR

Decision: B executes commands idempotently and maintains an authoritative outcome register. A retains its register of pending operations. A status API and periodic reconciliation detect unresolved outcomes; events may accelerate updates. Cases exceeding the resolution window enter an audited exception queue.

Rationale: verifiable outcomes and prevention of duplicate effects matter more than a superficial synchronous success response.

Consequences: idempotency-state storage, an API, scheduled reconciliation, monitoring and operational handling. Scale the solution to the risk and transaction volume.

Acceptance tests: a lost reply after successful execution; repeated delivery of the same command; delayed and duplicated events; a missing completion signal; status API downtime; UNKNOWN with no customer complaint; detection and escalation within the target window.

Your decision

What is the minimum viable architecture, and what would you add for a critical workload? Who owns the outcome, reconciliation, and exceptions?

Architecture pattern references

These sources describe technical patterns, not the fictional case. The scenario and sample ADR are original editorial material.

Fictional teaching scenario. Architecture pattern reference: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/. This link does not document the described events.

Test your decision

This learning material also has an interactive exercise: three options in two minutes or a workshop with the AI mentor.

Open the UNKNOWN transaction exercise →