Incident Management Ready Reckoner

Severity, roles, terms, and timings. Adjust the thresholds to your service before you need them.

Vigiles

Severity by impact

Level Typical impact Response
P1 / Critical Major customer-facing outage, payment failure, widespread production failure, serious security or data risk Immediate response, incident command, full escalation
P2 / High Major feature unavailable, significant degradation, limited regional impact Immediate investigation; escalate if impact grows
P3 / Medium Limited impact, workaround available, non-critical service affected Normal engineering response
P4 / Low Minor issue with little or no operational impact Track as normal work

Roles

Incident CommanderOwns coordination, severity, assignments, and when recovery can be declared.
Technical responderInvestigates and remediates. Reports back instead of running a second incident in chat.
Communications leadOwns status page, support macros, and stakeholder updates.
Subject matter expertJoins for a named system. Do not invite twenty people because the incident is serious.

Event, alert, incident, problem

EventSomething happened. CPU reached 82%.
AlertA condition worth notifying. CPU stayed above 80% for 10 minutes.
IncidentMeaningful or threatened service impact that needs coordinated response.
ProblemThe underlying condition behind one or more incidents.

Example P1 escalation

0 minPage primary on-call.
5 minPage secondary.
10 minPage engineering lead.
15 minWiden the incident response.

Metrics to define in writing

MTTDTime from customer impact to detection.
MTTATime from alert to acknowledgement.
MTTRDefine restore, recover, resolve, or repair before you report it. Time to restore service is usually the useful one.
Time to communicateTime from confirmed customer impact to the first status update.