Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Uptime status poster series ~5 min read

How a check becomes an outage

A single checker reporting a failure has told you one of two things and cannot say which: either your site is down, or the path between that checker and your site is. Those need completely different responses and telling them apart is what the third region is for.

Key takeaways

  • Three regions, and a journey is down only when all three agree.
  • Two of three is a network finding, recorded and not published.
  • Two consecutive agreeing rounds before a state change. One round is a blip.
  • Recovery needs only one agreeing round, because being wrong about up is cheaper.
  • If all journeys fail from all regions at once, suspect the checkers before the site.

Agreement

How three regional check results become an outage decisionA vertical chain of five steps entered by a box labelled Three results, for one journey in one minute. Step one asks whether all three failed; one or two failing exits to record it as a network finding. Step two asks whether they also failed last round, requiring two consecutive; if not it exits to Hold, because one round is a blip. Step three asks whether all journeys are failing from all regions at once; if so it exits to Suspect the checkers and alarms differently. Step four changes the status, published automatically. Step five drafts the note for a person to publish. A note says the third branch is the one people leave out and then get woken by at three in the morning.AWS ACCOUNTThree resultsone journey, one minuteAll three fail?One or two failrecord as a network findingpartialAlso last round?two consecutiveHoldone round is a blipnoAll journeys, all regions?everything at onceSuspect the checkersalarm differentlyyesChange the statuspublished automaticallyDraft the notefor a person to publishThe third branch is the one people leave out and then get woken by at 3am.
Fig 1. How three results become one decision. The everything-at-once branch is the check on the checkers, and it is what stops a monitoring failure being reported as a total outage.
  • App integration
  • Management
  • People
  • Outside AWS

Two of three is information

A journey that fails from two regions and succeeds from a third is almost never your site being down. It is usually a network path problem, occasionally a CDN edge having a bad time, and once in a while a genuine regional problem in your own infrastructure.

None of those is a status page event and all of them are worth recording. So a partial failure is written to the history with which regions disagreed, and a pattern of the same region disagreeing repeatedly is raised weekly — because that is either a checker with a problem or a real geographic issue, and both are worth knowing without either being an outage.

Two rounds down, one round up

The asymmetry is deliberate. Publishing an outage that was not one costs credibility and generates support contacts, so it is worth a minute of delay to be sure. Publishing a recovery that turns out to be premature costs very little — the status goes back down a minute later and the incident note explains it.

So going down needs two consecutive agreeing rounds and coming back up needs one. In practice that means an outage is published about two minutes after it starts and marked resolved about one minute after it ends, which is the right way round.

When the checkers are the problem

Why simultaneous total failure suggests a monitoring problemA horizontal row of five boxes. All four journeys: failing. All three regions: at the same minute. Suspicious: real outages are messier. Check the checker: credentials, quota, a deploy. Different alarm: saying monitoring may be broken. A note says genuine total outages exist but are rarer than expired check credentials.TOTAL FAILURE IS USUALLY YOUR MONITORINGAll 4 journeysfailingAll 3 regionsat the same minuteSuspiciousreal outages are messierCheck the checkercredentials, quota, deployDifferent alarm'monitoring may be broken'Genuine total outages exist. They are rarer than expired check credentials.
Fig 2. The pattern that suggests the monitoring rather than the site. Simultaneous total failure across every journey and every region is a shape real outages rarely have.
  • App integration
  • Networking
  • Management
  • Analytics

The most common causes are mundane: the check account’s password expired, an API key rotated, a deploy changed the string the assertion looks for, or the checkers were themselves deployed with a bug. Every one of those looks exactly like a total outage and none of them is.

So that pattern raises a distinct alarm with different wording — “all checks failing from all regions; this may be a monitoring problem” — and it does not automatically publish a total outage to the status page. A person confirms before telling every customer the business is down.

Next: how the page is served independently.

All posts