Skip to content
allanninal.dev/build
Collaborate

Part 5 of 7 · Incident postmortem collector series ~5 min read

How old postmortems get read again

This is the last post in this series and it is about the part of postmortem practice that almost nobody builds, which is a shame because it is where the entire return sits.

Key takeaways

  • Surface old incidents automatically when a new one starts.
  • Match on the systems involved and the alert, not on the words in the write-up.
  • The near miss is the cheapest possible incident to learn from and is rarely recorded.
  • Count recurrences by contributing condition, not by incident.
  • An annual read-through of the year’s incidents finds things no individual review did.

Surfacing at the right moment

How relevant past incidents are surfaced during a new oneA vertical chain of five steps entered by a box labelled An incident is declared, right now. Step one identifies which systems, from the declaration. Step two identifies which alert fired, the signature. Step three asks whether there is any past match on the same systems or alert; if not it exits to Nothing to show, stay quiet. Step four posts the summaries into the channel, now. Step five includes the actions, especially the declined ones. A note says the last box is uncomfortable and is frequently the most useful thing on the screen.AWS ACCOUNTAn incident is declaredright nowWhich systems?from the declarationWhich alert fired?the signatureAny past match?same systems or alertNothing to showstay quietnoPost the summariesinto the channel, nowIncluding the actionsespecially the declined onesThe last box is uncomfortable and is frequently the most useful thing on the screen.
Fig 1. How past incidents reach the people handling a new one. Posting into the channel at declaration is the only moment anybody will read them.
  • Compute
  • App integration
  • Machine learning
  • Management
  • Analytics
  • Front-end & mobile

The declined action

When an incident recurs and the earlier postmortem contains an action that was declined as not worth doing, that is exactly the information the current responders need, and it is precisely what a well-run process would prefer not to display.

Showing it anyway is what makes the declining decision honest. A risk accepted with a stated reason is a defensible decision; the same risk accepted and then quietly hidden when it materialises is not.

Matching on structure, not words

Full text search over postmortems works badly, because the words describing an incident are written after the fact and vary enormously. Matching on which systems were involved and which alert fired is cruder and considerably more effective.

That requires the declaration to capture the systems, which is one field at the moment an incident opens. It is the smallest possible ask at the busiest possible time, which is why it is a picklist and not free text.

The near miss

Why a near miss is worth writing up despite having no impactA horizontal row of five boxes. Nearly an incident, caught in time. Everyone relieved and moves on. Nothing written, because there was no impact. Same conditions as a real one. And it was free, because nobody was affected. A note says a near miss teaches the same thing as an incident and costs nothing to have had.THE CHEAPEST LESSON AVAILABLENearly an incidentcaught in timeEveryone relievedand moves onNothing writtenthere was no impactSame conditionsas a real oneAnd it was freenobody was affectedA near miss teaches the same thing as an incident and costs nothing to have had.
Fig 2. Why near misses are worth writing up. They contain the same contributing conditions as a real incident with none of the damage.
  • Compute
  • Machine learning
  • Management
  • Analytics
  • People

The barrier is that a near miss has no natural trigger for a review: nothing broke, nobody was paged for long, and writing it up feels like manufacturing work.

A short form — what nearly happened, what caught it, what would have happened if it had not — is ten minutes and produces most of the value of a full postmortem. Making it explicitly short is what gets it filled in.

Patterns across incidents

Contributing conditions counted across a year of incidentsA bar chart with four bars showing incidents each condition contributed to per year. No gradual rollout: seven. Alert pointed at the wrong component: five. Rollback needed an approval: four. Config unvalidated: three. A note says no single postmortem showed this, and it is only visible by counting conditions.02.557.510~7No gradual rollout~5Alert pointed wrong~4Rollback needed approval~3Config unvalidatedIncidents this condition contributed to, per yearNo single postmortem showed this. It is only visible by counting conditions.
Fig 3. A year of contributing conditions counted across all incidents. Each individual postmortem mentioned its condition once; only the aggregate shows which ones recur.

This is the argument for tagging contributing conditions with a controlled vocabulary rather than leaving them as prose. It is slightly more work per postmortem and it is the only way this chart exists.

Seven incidents in a year sharing one condition is a prioritisation argument that no individual review could have made, and it points at work that would have prevented seven incidents rather than one.

The annual read-through

Once a year, somebody reads all of the year’s postmortems in one sitting. It takes an afternoon, it is not enjoyable, and it reliably produces two or three findings that no individual review contained.

It is also the only realistic way to notice that the same three sentences appear in six documents, which is the strongest available signal about where to spend effort.

What this system does not do

It does not write the postmortem, it does not identify causes, and it does not decide which actions matter. Those are the parts that require somebody who understands what happened, and a generated write-up would be plausible, fluent, and would remove the thinking that is the entire point of the exercise.

What it does is capture what would otherwise be lost, keep the actions honest, and put the right old document in front of the right person at the right moment. That is enough.

Next: what all of this costs to run.

All posts