Skip to content
allanninal.dev/build
Collaborate

Part 2 of 7 · Incident postmortem collector series ~5 min read

How the timeline gets assembled while it is fresh

The timeline is the part of a postmortem that cannot be recovered later, and it is the part most processes leave until later.

Key takeaways

  • Hindsight rewrites memory towards what turned out to be true.
  • Capture as a by-product of the work: the channel is the timeline.
  • Record what people believed at the time, explicitly.
  • Automatic events — alerts, deploys, restarts — go in with timestamps.
  • One prompt at resolution catches what the channel missed.

How memory fails

How hindsight distorts a remembered incident timelineA vertical chain of five steps entered by a box labelled Writing up on Thursday about Tuesday. Step one notes that you know the answer now and did not then. Step two says the wrong paths vanish, phrased as we quickly identified, with a side box noting that forty minutes were spent there. Step three says the times compress, so two hours feels like one. Step four says the order shifts towards a clean story. Step five produces a tidy, false timeline, and nobody lied. A note says every one of these is automatic and unconscious, and effort does not fix it.AWS ACCOUNTWriting up on Thursdayabout TuesdayYou know the answer nowyou did not thenSo the wrong paths vanish'we quickly identified...'But40 minutes were spent thereAnd the times compresstwo hours feels like oneAnd the order shiftstowards a clean storyA tidy, false timelinenobody liedEvery one of these is automatic and unconscious. Effort does not fix it.
Fig 1. How an accurate memory becomes an inaccurate timeline. None of it is dishonesty and none of it is avoidable by trying harder.
  • Compute
  • Management
  • Analytics

The wrong paths matter most

Forty minutes spent investigating a component that turned out to be fine is the most useful information in the whole incident, because it says something about what the system looks like from the outside during a failure.

It is also the first thing hindsight deletes. The write-up says the cause was identified at twenty past two, and the forty minutes before it disappear, taking with them the finding that a misleading dashboard sent four people in the wrong direction.

Capture as a by-product

Three approaches to building an incident timelineThree boxes stacked on the left. Ask people to log during the incident, labelled does not happen. Reconstruct after from memory, labelled distorted. And Use what they said in the channel, timestamped, labelled free and accurate. All three converge on A raw timeline before anybody writes, and that leads down to Edited down later, removing rather than recalling. A note says editing a real record down is a different task from remembering and a much easier one.Ask people to logduring the incidentdoes not happenReconstruct afterfrom memorydistortedUse what they saidin the channel, timestampedfree and accurateA raw timelinebefore anybody writesEdited down laterremoving, not recallingEditing a real record down is a different task from remembering, and a much easier one.
Fig 2. Three ways of getting a timeline. Only the third produces an accurate one and it costs nothing during the incident.
  • Compute
  • App integration
  • Management
  • Front-end & mobile

The channel is the timeline

People in an incident talk to each other, and that conversation is already timestamped, already contains what they were trying, and already records the wrong turns. Treating it as the raw material rather than asking anybody for anything extra is the whole mechanism.

The write-up then becomes an editing task: take four hundred messages, keep thirty, add structure. That is achievable on a Thursday afternoon in a way that reconstruction is not.

Beliefs, not facts

The two columns that make a timeline useful

  • 14:02 — Alert fires. Believed: a database problem, because that alert has meant that twice before.
  • 14:11 — Database checked, healthy. Believed: probably the network then.
  • 14:24 — Network fine. Believed: nobody has a hypothesis.
  • 14:38 — Someone mentions the deploy at 13:50. Believed: possibly related.
  • 14:52 — Rolled back. Recovery begins.
  • The italics are the postmortem. Without them this is a list of times.

The belief column is what turns a timeline into something you can learn from. The gap between 14:02 and 14:38 is not incompetence; it is a system whose failure signature pointed confidently at the wrong component, and that is a fixable thing.

Capturing beliefs requires a small nudge during the incident: a prompt in the channel every so often asking what people currently think is happening. It is cheap and it produces the most valuable rows in the eventual document.

Automatic events

Automatic events collected into an incident timelineA horizontal row of five boxes. Alerts, fired and resolved. Deploys and rollbacks. Restarts and scaling events. Config changes, with who made them. Interleaved with what people said. A note says the deploy at thirteen fifty was in the timeline before anybody thought to look for it.WHAT GOES IN WITHOUT ANYBODY TYPINGAlertsfired and resolvedDeploysand rollbacksRestartsand scaling eventsConfig changeswith whoInterleavedwith what people saidThe deploy at 13:50 was in the timeline before anybody thought to look for it.
Fig 3. The events that can be collected automatically. Interleaving them with the conversation is what makes the connection visible.
  • Compute
  • Management
  • Front-end & mobile

The value is in the interleaving. A deploy at ten to two sitting in a deployment log is information nobody connects; the same deploy appearing twelve minutes before the alert in a single timeline is the connection making itself.

One prompt at resolution

When the incident closes, a single prompt to everybody involved: anything the channel missed? Something you tried privately, a phone call, a thing you noticed and did not mention. Two minutes, while it is still fresh, and it catches the material that was never written down.

Next: what caused it.

All posts