Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Incident postmortem collector series ~6 min read

An incident postmortem collector on AWS for a few dollars a month

Something broke on a Tuesday, four people spent two hours fixing it, somebody wrote it up on Thursday from memory, and the document went into a folder. Eleven months later something similar happens and nobody remembers the first one existed. The write-up was not the problem; nothing was ever going to bring it back at the moment it mattered.

Two men and a dog silhouetted in a dark doorway
Photo by LOGAN WEAVER on Unsplash

Key takeaways

  • Capture the timeline during the incident; memory degrades within hours.
  • Contributing conditions, not a single root cause.
  • Track actions to closure or to an honest declined.
  • Surface relevant past incidents when a new one starts. That is the payoff.
  • Designed on AWS for about $2 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: incident timelines captured, written up, and resurfacedThree boxes across the top sit outside the AWS account. On the left, An incident happening now. In the middle, The people in it, busy and later tired. On the right, Everybody else, now and in two years. Each connects by an arrow to the AWS account container below. Events as they happen flow down into the account. What they knew and when feeds in. A document that resurfaces goes back out. Inside the AWS account are three components in a row. On the left, the Timeline, captured live rather than remembered. In the middle, the Write-up, describing conditions rather than a single cause. On the right, Actions and recall, followed and found again. A note at the bottom says the last four words are the point and everything before them is preparation.AWS ACCOUNTAn incidenthappening nowThe people in itbusy, and later,tiredEverybody elsenow, and in two yearsTimelinecaptured live,not rememberedWrite-upconditions, nota single causeActions and recallfollowed, andfound againevents as theyhappenwhat they knew,whena document thatresurfacesThe last four words are the point. Everything before them is preparation.
Fig 1. Three things outside the account, three pieces inside it. The value is created on the right, months after the incident.
  • Machine learning
  • Management
  • Analytics
  • People

The value is in the reading

Writing a postmortem teaches the author something and occasionally teaches the people who attend the review. That is a modest return on several hours of work by several people, and it is the entire return most organisations get.

The larger return comes when somebody encounters a similar situation later and finds the earlier write-up. That happens by accident or not at all, unless something makes it happen, which is the third component and the one that is usually missing.

What runs (the inside)

  • The timeline. Captured during, because it cannot be reconstructed afterwards. Part 2.
  • The write-up. Structured around contributing conditions rather than a cause. Part 3.
  • Actions and recall. Following the actions honestly, and putting old incidents in front of people at the moment they are relevant. Parts 4 and 5.

One incident, end to end

One incident from declaration through to being resurfaced laterA horizontal row of five boxes joined by arrows. Declared at two minutes past two: a channel opens. Timeline builds as people talk. Resolved at ten past four: the timeline already exists. Written up on Thursday, from the timeline. Resurfaced in June, when it nearly recurred. A note says the fifth box is where the four hours spent on the fourth box gets paid back.ONE INCIDENT, END TO ENDDeclared 14:02a channel opensTimeline buildsas people talkResolved 16:10timeline already existsWritten up Thursdayfrom the timelineResurfaced in Junewhen it nearly recurredThe fifth box is where the four hours spent on the fourth box gets paid back.
Fig 2. The same system as one line. Everything is preparation for a moment nine months later.
  • Machine learning
  • Management
  • Analytics

In plain words

Somebody declares an incident at two minutes past two. A channel opens and from that moment everything said in it is timestamped and kept. People are not asked to write a timeline; they are asked to work, and the timeline is a by-product of them talking to each other.

It resolves at ten past four. On Thursday somebody writes it up, and the hardest part — what happened when, and what people believed at the time — is already there, with timestamps, rather than being reconstructed from four people’s memories of a stressful afternoon.

Three actions come out of it. Two are done within a month. The third is closed as declined in July with a stated reason, which is a better outcome than it sitting open for three years. And in June, when a similar alert fires, the earlier write-up appears next to it automatically.

Design rules that shaped every decision

  • Capture the timeline during, never after. Memory is the problem.
  • Contributing conditions, plural. No single root cause field.
  • No names in the causal chain. Roles and systems, not people.
  • Every action has an owner, a date and an honest end state.
  • Old incidents surface automatically when something similar happens.
  • A near miss is worth writing up and almost never is.

Why this shape

Postmortem practice fails in three well-known ways. The timeline is written from memory and is therefore wrong in the specific places that matter. The analysis stops at the last thing that changed. And the actions accumulate unclosed until nobody believes the process produces anything.

None of those are solved by a better template. They are solved by capturing at the right moment, structuring the write-up so that a single cause is not the obvious answer, and treating an action item as something with an end state rather than a wish.

The next four posts walk through each piece: how the timeline gets assembled while it is fresh, why there is rarely one root cause, what happens to the action items, and how old postmortems get read again. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts