Incident response
An incident is production downtime, full or partial, or a bug that stops users from doing what they came for, such as placing an order. Handle each incident in its own chat channel. Everyone who helps joins that channel and posts their findings and decisions there, so afterwards the channel holds the timeline for the incident report. The examples use Slack; Microsoft Teams and Mattermost have most of these features under other names. Adapt the channel names, the severity examples and the on-call hours to your organization.
Declaring
Declare an incident when any of these is true:
- Customers can see the problem.
- Fixing the problem needs a second team.
- The problem is still unsolved after an hour of focused work.
Declare early. If the problem turns out to be Minor, file a bug ticket and close the incident. Closing an incident you didn't need costs less than starting the procedure hours into a growing problem.
Procedure
The person who declares the incident is the incident lead until they hand the role over.
During the incident
- Create a public channel named
incident-<date>-<summary>, or a private one for a security incident. - Mention
@incidentin the incident channel to bring in the stakeholders. - Post a summary in the incident channel: what users see, since when, the severity and the incident lead.
- Start a video call.
- Pin the call link in the incident channel.
- Assign the roles of communications lead and scribe.
- Stop the impact before you look for the cause (see Mitigation).
- Post every finding, decision and action in the incident channel as it happens.
- Post a status update in the incident channel every 30 minutes.
- When the impact ends, post the end time in the incident channel.
After the incident
- Write a blameless incident report within 24 hours.
- Post the incident report in the incident channel.
- Hold the incident report meeting within 48 to 72 hours after the write-up.
- Post the final incident report in the incident channel.
- Archive the incident channel.
Roles
Hand out roles, so the people who fix the problem don't also have to coordinate and keep others informed. The incident lead holds every role they haven't handed out yet.
| Role | What they do |
|---|---|
| Incident lead | Coordinates the response, decides what happens next and hands out the work |
| Communications lead | Posts the status updates and keeps customer service, management and customers informed |
| Scribe | Posts findings, decisions and actions with their times in the channel, and collects the timeline for the report |
Responders
The incident lead doesn't fix things. The other responders investigate and fix the problem, and only they change production during the incident.
Small teams
In a small team or organization, one person can hold all three roles. That person then also fixes the problem. Keep the channel, the status updates and the report anyway, so anyone who joins later can read what happened. Hand out roles as soon as a second person joins.
Handover
An incident lead who hands over the role posts the handover in the incident channel, and stays until the new lead confirms it.
Severity levels
Set a severity when you declare the incident. The severity decides who gets paged and informed, and whether the incident gets a report.
| Severity | Impact | Examples | Response |
|---|---|---|---|
| Critical (outage) | Most customers can't use a core function, or data is lost, corrupted or exposed | Checkout is down, orders get wrong prices, customer data leaks | Page right away, inform customers and management, write an incident report |
| Major (degradation) | A core function fails for some customers, or works badly for everyone | One payment method fails, search takes 20 seconds | Page right away, write an incident report |
| Minor (bug) | Small impact that customers can work around | Product images load slowly, a nightly export runs late | No incident: file a bug ticket with high priority, and the owning team fixes it during business hours |
When you're unsure between two levels, pick the higher one. Don't discuss the severity during the incident: review it in the incident report meeting.
Mitigation
Stop the impact first and find the root cause later.
Rollback
Roll back the latest release or configuration change, or turn off a feature flag you just turned on, before you start debugging. Before you restart or recreate anything, save the logs and data you need to find the root cause.
Changes
To find what to roll back, look in the metrics for the moment the problem started. Then look for releases, data migrations, experiments and configuration changes around that time.
Help
Ask people for help early. More people can find the cause faster, and anyone you don't need can leave the channel again.
Communication
Discuss the incident only in its channel, never in private messages. People who join late read the channel history instead of asking, and the report gets a complete timeline. Decisions made on a call go in the channel too. In the example incident, the talks with customer service happened in private messages, and the report has no record of them.
Channel
Name the channel incident-<date>-<summary>, with the date as YYYY-MM-DD and a few words about what users see, for example incident-2026-03-14-checkout-errors. Name it after the symptom, because the cause is usually unknown when you create the channel.
Make the channel public, so anyone in the organization can follow it without asking for an invite. Make a security incident's channel private, because it can contain details of a vulnerability or personal data.
Pin the call link and the latest status update in the channel. Put the severity and who holds which role in the channel topic or header, and update the topic when the severity or a role changes. Use the call to work through the problem together, and the channel for the record.
Status updates
The communications lead posts a status update every 30 minutes, even when nothing has changed. Each update says what users see, what is known about the cause, what happens next and when the next update comes. If the cause is unknown, say so instead of leaving it out:
Status 14:30 (Major): card payments fail for about 1 in 3 orders since 13:55.
Cause: unknown. The 13:40 release is the main suspect.
Next: rolling back the 13:40 release.
Next update: 15:00.Automation
A bot or chat workflow can do the first three steps at once: create the channel, mention @incident and post a summary template. Incident management tools automate these steps, and so do Slack workflows (create_channel) and Mattermost Playbooks (Playbooks). The procedure works by hand too, so start without automation.
Preparation
Make these decisions and write them down where everyone can find them.
| Decision | Example |
|---|---|
Who is in the @incident group | Team leads, product, customer support and the on-call engineers |
| Where production alerts go | #alerts-production, from uptime checks on every domain and application, around the clock |
| Where application errors go | Each team's own channel, from the error tracker |
| Who runs the infrastructure | The cloud team runs and supports the databases and Kubernetes |
| On-call hours and response time | Outside business hours from 07:00 to 23:59, with a response within 30 minutes; nobody on call at night |
Tools
In Slack, @incident is a user group, which needs a paid plan. Teams has tags and Mattermost has custom user groups for the same job. For the alerts themselves, see Monitoring.
Practice
Use the procedure for every Major incident too, so it's routine when a Critical one comes.