A Runbook Template for a Known Failure
A short runbook template: the symptom, the checks in order, the action, and when to stop. Plus a filled-in example.
Long runbook templates start with a service overview, an architecture sketch, and an escalation matrix. The steps come last. The person who opened the page already has the alert. They need the checks and the action.
The template
One file per failure. If a section would be empty, delete it.
# Failure, in a few words
Alert:
Owner:
Last used:
## Check
1. First thing you look at, and what the result means.
2. Second check.
## Do this
The action that ends it, and who is allowed to take it.
## Stop and wake someone if
The point where these steps are no longer enough.
## After
What to write down once it is over.
Link the postmortem if there is one.A filled-in example
# Nightly billing job started twice
Alert: two "job started" lines for the same night
Owner: billing
Last used: 2026-03-02
## Check
1. Count "job started" lines for tonight. One line means this
page does not apply.
2. See whether a deploy finished during the cron window.
## Do this
Do not start another copy. If customers were charged twice,
refunds are a billing-team action. The lock in NightlyBilling.php
should make the second copy exit. If it did not, wake billing.
## Stop and wake someone if
Charges went out and you cannot tell which customers, or the
lock query errors instead of exiting.
## After
Note whether the second copy exited on the lock.
Link the postmortem if customers were charged."Last used" is the line that tells you whether to trust the page. If it is blank, or the service has changed since that date, treat the steps as unverified. Say so at the top rather than discovering it on the second check.
Keep the steps on the service
Prodigent ties the runbook to the service it covers, so a change to that path is a prompt to reread the steps.
Try Prodigent free