Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Incident postmortem collector series ~6 min read

Why there is rarely one root cause

A root cause field on a postmortem template produces a shallow answer every time, because there is always a last thing that changed and writing it down feels like completing the analysis.

Key takeaways

  • The trigger is the last thing that changed. The causes are what made it an incident.
  • Ask what made it possible, what made it worse, and what delayed detection.
  • No names in the causal chain. A person’s action is a system that allowed it.
  • ’Human error’ is where analysis stops rather than where it should.
  • A write-up with one cause and one action is usually incomplete.

Trigger against causes

How contributing conditions are identified rather than a single root causeA vertical chain of five steps entered by a box labelled The deploy at thirteen fifty, the trigger. Step one asks what made it possible: a config with no validation. Step two asks what made it worse: no gradual rollout. Step three asks what delayed detection: the alert pointed elsewhere. Step four asks what delayed recovery: the rollback needed an approval. Step five concludes with four conditions and one trigger. A note says fixing the trigger prevents this incident while fixing the conditions prevents a class.AWS ACCOUNTThe deploy at 13:50the triggerWhat made it possible?a config with no validationWhat made it worse?no gradual rolloutWhat delayed detection?the alert pointed elsewhereWhat delayed recovery?rollback needed an approvalFour conditionsand one triggerFixing the trigger prevents this incident. Fixing the conditions prevents a class.
Fig 1. The four questions that replace a root cause field. Each one produces a different and independently fixable finding.
  • Compute
  • Security & identity
  • Management
  • Analytics

Why the distinction matters

The trigger is specific and will not recur in the same form. Somebody will not make that exact configuration mistake again, and a postmortem whose only action is “be more careful with that config” has prevented one incident.

The conditions are general. A config path with no validation will admit a different mistake next time. An alert that points at the wrong component will mislead a different team. A rollback that needs an approval will be slow in every future incident.

The write-up structure

Sections that produce a useful document

  • What happened, in two sentences, for somebody who was not there.
  • Impact: who was affected, how, and for how long. Specific.
  • Timeline, with the belief column from Part 2.
  • Contributing conditions, plural, each independently stated.
  • What went well. Genuinely, and it is not padding: the things that limited the damage are worth protecting.
  • Actions, each tied to a condition rather than to the trigger.
  • No section called root cause, because the field creates the answer.

The fifth section is skipped in most templates and it earns its place. An incident that was detected in four minutes because somebody had built a good alert last year is telling you something about what to keep doing, and the alert’s author will otherwise never hear about it.

Blame is an analytical failure

Findings produced by blaming human error versus analysing conditionsA bar chart with two bars showing independently fixable findings. Human error: one. Conditions analysis: four. A note says blame is not just unkind, it stops the analysis at the first plausible answer.02468~1'Human error'~4Conditions analysisIndependently fixable findingsBlame is not just unkind. It stops the analysis at the first plausible answer.
Fig 2. The same incident analysed two ways. Attributing it to a person’s mistake yields one finding and it is not actionable.

The cultural argument for blameless postmortems is well made and there is a second argument that persuades different people: blame produces worse analysis. “Somebody typed the wrong value” is a complete-sounding explanation that generates one action, and the action is a reminder.

“A production configuration change was possible without validation, review or a gradual rollout” describes the same event and generates three actions, none of which depend on anybody being more careful.

No names in the causal chain

A practical rule that is easy to apply: names appear in the timeline where they are needed for clarity, and never in the contributing conditions. Where a person’s action is part of the story, the condition is stated as the system that permitted it.

This is not a euphemism. It is the more accurate statement, because the same action taken by anybody else would have had the same effect, which means the action is not what distinguishes this incident.

How deep to go

How many levels deep to analyse an incident's causesA horizontal row of five boxes. One level: the deploy. Two levels: no validation. Three levels: why no validation. Four levels: a decision made in 2023. Stop when actionable, not when profound. A note says it is always possible to go one level deeper and reach something nobody can act on.HOW FAR TO KEEP ASKINGOne levelthe deployTwo levelsno validationThree levelswhy no validation?Four levelsa decision in 2023Stop when actionablenot when profoundIt is always possible to go one level deeper and reach something nobody can act on.
Fig 3. How deep an analysis should go. The stopping condition is actionability rather than philosophical completeness.
  • Compute
  • Machine learning
  • Management
  • Analytics

There is a failure mode at the other end from blame, which is an analysis that arrives at organisational culture, resourcing decisions from three years ago, or the general difficulty of distributed systems. All true, and none of it produces anything anybody can do on Monday.

The stopping rule is practical: keep asking until each condition has an action somebody could actually take, and then stop. If a level produces no actionable finding, the previous level was the right depth.

Next: what happens to those actions.

All posts