A backup nobody has restored

Fictional teaching example. No particular organisation or incident is described.

Situation

An operator shows a green report for nightly backups. Asked about restoring the application, files and queue, the team replies, “The backups are there.” The service owner assumes a 30-minute recovery time. Nobody has measured download time, access to encryption keys or whether the stores form a consistent recovered state.

Warning signs

• Monitoring checks that the backup job finished but never reads from the backup.
• The runbook names a tool, not the order of recovery, dependencies or who can approve resuming traffic.
• RPO and RTO describe “the system” although database, files and messages follow separate schedules.
• The last drill used a small old sample and never tested the full user journey.

Consequences

A backup may be unreadable, a key unavailable or restored attachments may not match database records. Even a good backup of one component cannot restore the whole service. A green job status then measures the scheduler, not architectural resilience.

A better decision

1. Define what “service restored” means to a user; set RPO and RTO for that journey.
2. Check backup retention, integrity, availability and keys when the primary region is unavailable. Restrict access and rehearse in isolation.
3. Rehearse the entire runbook on representative data volume: database, files, messages, configuration and routing. Restore the function and run a transactional test.
4. Measure achieved targets, record gaps and owners for remediation. Repeat after material changes and periodically verify that recent backups can be read.

A drill consumes time and infrastructure. Skipping it has a cost too; it merely sends the bill at the worst moment.

A backup has value when you can use it to restore the required service within the agreed time.