Incident reports
Every Critical and Major incident gets a blameless incident report (severity levels). Write the report within 24 hours after the incident, and post it in the incident channel. The report describes what happened, what it cost and what changes, never who is at fault. The people involved then go through the report in the incident report meeting.
Start from the template. The examples show a finished report.
Writing the report
The owner, usually the incident lead, writes the report. Start from the incident channel: the scribe's notes give you the timeline, and the status updates give you most of the summary.
Blameless
Keep the report blameless. Describe what people did and what they knew at that moment, and assume everyone acted with good intentions.
Involvement
Refer to people with an @-mention instead of their name, so nobody mixes up two people with the same name.
Impact
Give the business impact in exact numbers: orders, customers, revenue and support requests.
Timeline
Write the timeline with a 24-hour clock and state the time zone. Mark four moments, as the example does: impact starts, detected, mitigated and impact ends. The gaps between those moments show how long the team took to notice the problem and to fix it. Explain the first gap in the Detection section. If a customer noticed the problem before an alert did, add an alerting improvement to Learnings and improvements.
Actions
Give every pending action and improvement an owner and a ticket, and link the ticket in the report. Improvement tickets get the highest priority, and the team starts on them within a week.
Incident report meeting
Hold the incident report meeting within 48 to 72 hours after the report is written. Invite everyone in the Involved list and the owners of the pending actions.
Agenda
- Correct the timeline.
- Check that the business impact is complete.
- Check that the root cause goes deep enough.
- Agree on the improvements, their owners and their tickets.
- Review the severity.
Outcome
Write the date, the attendees and the conclusions in the report's last section, Incident report meeting. Then post the final report in the incident channel and archive the channel.