Incident report examples
Each example is a finished report, written with the template.
Database integrity issue
This report is based on a real incident, with the company, the people, the numbers and the dates changed. The example company is a web shop that sells products from partner merchants. Each person appears as their role instead of an @-mention.
- Date: 2026-03-12
- Severity: Critical
- Duration: 3 hours and 10 minutes of failed orders, recovery until the next afternoon
- Owner: incident lead
- Involved: incident lead, backend developer, database engineer, cloud engineer
- Chat channel:
#incident-2026-03-12-failed-orders
Summary
A CMS user changed a discount code, and a seven-year-old bug in the CMS deleted a discount label that other discount codes still used. Replication to the read replicas stopped. For over three hours, orders paid in advance failed to complete, and merchants received order confirmations with a price of 0 EUR. The team restored integrity on the primary database, recreated the replicas, recovered the deleted labels from a point-in-time backup and fixed the CMS bug.
Business impact
- 38 merchants received 76 confirmations for orders paid at pickup that showed a price of 0 EUR.
- Between 12:50 and 16:00, the payment didn't complete for 1,860 orders paid fully or partly in advance, and customers had to order again.
Detection
The on-call engineer created the incident channel at 14:11, 81 minutes after the impact started. There was no alert on replication lag, and the cloud team owns the improvement that adds one. At 14:30, the database engineer found the first hint of the cause in the error tracker.
Root cause
When a CMS user sets a discount code's label to none, the CMS deletes the label, even when other codes still use it. Deleting the label's rows from the discount_code_label table breaks the database's integrity constraints. The bug was seven years old.
The broken data stopped replication to the read replicas. Apps that read from the replicas, such as the CMS and the merchant portal, got outdated data.
Actions taken
- Restored the integrity of the primary database (database engineer)
- Recreated the replicas to get replication working again (cloud engineer)
- Turned off editing of discount codes in the CMS during the investigation (backend developer)
- Ran a point-in-time recovery to get the deleted labels back (cloud engineer)
- Inserted the recovered labels into the primary database (database engineer)
- Shared the list of affected orders with customer service (database engineer)
- Emailed all affected customers to apologize and ask them to order again (customer service)
- Resent the order confirmations to the merchants on the evening of the incident (customer service)
Actions pending
- Clean up the duplicate order numbers, urgent (backend developer): done
- Fix the CMS bug (backend developer): done
- Allow creating and updating discount codes again (backend developer): done
- Tell CMS users how to use discount labels until the bug is fixed (backend developer): done
- Go through the bug and the next steps with the CMS product owner (backend developer): to be scheduled
Learnings and improvements
- Keep all communication in the incident channel: the talks with customer service happened in private messages and are missing from this report (everyone)
- Make apps read from the primary database when the data has to be current, for example right before a write (team leads)
- Alert on replication lag (cloud team)
- Add replication lag to the uptime monitor (cloud team)
Timeline
All times CET. On 2026-03-12:
- 12:50 A CMS user saves a change to a discount code.
- 12:50 The CMS deletes rows from the
discount_code_labeltable. - 12:50 Replication lag starts on the replicas (impact starts).
- 14:11 The on-call engineer creates the incident channel (detected).
- 14:30 The database engineer finds the first hint in the error tracker: a
QueryExceptionwith "Integrity constraint violation". - 14:33 The incident lead starts the video call.
- 16:00 Orders complete again (mitigated, impact ends).
- 18:00 Customer service and the database engineer email the customers whose orders didn't complete.
- 18:00 Customer service sends the merchants new confirmations for orders completed during the incident, because the first ones showed 0 EUR.
On 2026-03-13:
- 10:38 The cloud engineer starts a point-in-time restore to 2026-03-12 12:50.
- 11:23 The point-in-time restore completes.
- 12:57 The database engineer recovers the deleted discount labels.
- 13:35 The database engineer regenerates the deleted labels and the ones missing since 12:50 the day before, and inserts them into the primary database.
- 15:00 The team meets to go through the latest status and the search for the CMS bug.
- 16:50 The team reproduces the bug and agrees on the cause.
Extra
Point-in-time restore target: 2026-03-12 12:50:27 CET.