Learning goal: define service availability through the outcome promised to the user, rather than the number of green components.

Fictional educational scenario. A portal accepts a request and returns an identifier. The API, database and broker respond, but a stopped consumer never takes the process to a terminal state. This is not an account of a particular organisation or incident.

Two truthful answers can contradict each other

The command-acceptance service may be available while the business process is not. These are two different commitments:

  • acceptance — the system durably accepted the work and can identify it;
  • completion — the work reached an agreed terminal state within the agreed time.

If a dashboard combines those answers, green is not necessarily a lie. It is an answer to a question that is too small.

An evidence ladder

LayerQuestionExample evidenceWhat it does not prove
ProcessIs the component alive?Liveness, restart and resource useReadiness to serve traffic
DependencyCan the component perform its step?Readiness, connectivity and permissionsCompletion of the whole process
ProgressIs accepted work moving?Oldest-work age, completion rate and permanent failuresCorrectness of the user outcome
OutcomeDoes the user receive the promised service?Terminal-state SLI and a controlled synthetic journeyThe root cause of a failure

All layers are useful. Outcome signals expose the failure; component signals help locate it. One chart need not pretend to be the whole architecture.

Start with the process contract

  1. Name the accepted state and terminal states: success, rejection, cancellation or human intervention.
  2. Agree the measurement start, window and population. Do not invent a duration merely to give a dashboard a threshold.
  3. Carry a correlation identifier through queues, consumers and state stores. Keep confidential content out of metric labels.
  4. Measure latency and lack of progress. Queue depth alone can also rise during healthy traffic bursts.
  5. Define what happens after the threshold: limit intake, disclose delay, route to manual handling or stop the process.

A safe end-to-end probe

A synthetic journey needs a separate identity, harmless data, bounded frequency and controlled cleanup. It must verify the terminal state, not merely an HTTP response. For operations that create obligations, use an isolated test path instead of generating “monitoring orders” in production. Monitoring can become a customer too; it merely tends not to pay the invoices.

The negative test

Stop a consumer in a controlled environment and expect:

  • component checks to state precisely what still works;
  • progress measures to detect ageing work;
  • the outcome SLO to consume budget according to the contract;
  • recovery not to create duplicate effects;
  • operators to account for every accepted item.

Boundary of the technique

An end-to-end probe does not replace diagnostics, security tests or queue-capacity decisions. An SLO does not repair a process; it tells you when the observed outcome departs from the accepted promise. That promise has to exist first.

Practise the decision on a stalled process →

Sources and method basis: Google SRE — Monitoring Distributed Systems; Google SRE Workbook — Implementing SLOs; OpenTelemetry — traces. The evidence ladder and scenario are original ArchitectMentor work. Sources checked 19 October 2026.