Skip to content
allanninal.dev/build
Collaborate

Part 4 of 7 · Broken link reporter series ~5 min read

How the three-run rule works

The three-run rule is the least sophisticated idea in this system and does more for the quality of the output than anything else in it. Most things that fail once are fine, and a report that includes them is a report that is wrong often enough to be discounted.

Key takeaways

  • A failure must appear in three consecutive weekly runs before it is reported.
  • That removes the large majority of transient failures at a cost of two weeks.
  • Internal 404s skip the rule, because your own site being down is a different alarm.
  • A link that starts failing and then recovers is recorded as flaky, not forgotten.
  • A link that disappears from the site is closed rather than counted as fixed.

Three runs

How the three-run confirmation rule worksA vertical chain of five steps entered by a box labelled A failed check, this week. Step one asks whether it is an internal 404 on your own site; if so it exits to Report now, skipping the rule entirely. Step two asks whether it also failed last week; if not it exits to Hold, strike one. Step three asks whether it failed the week before that; if not it exits to Hold, strike two. Step four confirms three failures in a row with the same status. Step five reports it, with the page and the traffic. A note says internal 404s skip everything, because a broken link on your own site is your own bug.AWS ACCOUNTA failed checkthis weekInternal 404?your own siteReport nowskip the rule entirelyyesFailed last week?Holdstrike onenoAnd the week before?Holdstrike twonoThree in a rowwith the same statusReport itwith the page and the trafficInternal 404s skip everything. A broken link on your own site is your own bug.
Fig 1. The three-run rule and its one exception. An internal 404 is not a transient network condition; it is a page that is missing from a site you control.
  • App integration
  • Management
  • Analytics

What two weeks costs

It is worth being explicit: a link that genuinely breaks on the Monday after a run will not be reported for nearly three weeks. That is a real cost and it is the right trade for external links, where the alternative is a report containing dozens of sites that happened to be down for ten minutes.

For internal links it is not the right trade at all, which is why they skip the rule. A 404 on your own site is either a page you deleted or a link you typed wrong, and neither gets better by waiting a fortnight.

Same status, not just failing

Three failures in a row only counts when they are the same kind of failure. A URL that returns a timeout, then a 500, then a 403 is behaving oddly rather than being consistently gone, and it is more likely to be a struggling server than a dead page.

That distinction is recorded: a link with three different failure modes across three runs goes into a small “unstable” list rather than the main report. In practice those are frequently the most interesting entries, because a site that is intermittently failing is a site that is about to disappear.

Flaky and disappeared

How an intermittently failing link is eventually reportedA horizontal row of five boxes. Fails: on run one. Recovers: on run two. Fails again: on run three. Flaky: counted rather than reported. Four times in ten: now it is reported. A note says resetting the counter on every recovery means a genuinely flaky link is never reported.A LINK THAT KEEPS RECOVERINGFailsrun 1Recoversrun 2Fails againrun 3Flakycounted, not reportedFour times in tennow it is reportedResetting the counter on every recovery means a genuinely flaky link is never reported.
Fig 2. Why recoveries do not simply reset the count. A link that fails four weeks out of ten is broken for a lot of readers and a naive three-in-a-row rule would never surface it.
  • Machine learning
  • Management
  • Analytics

So there are two counters: consecutive failures, which drives the main report, and failures in the last ten runs, which catches the flaky ones. Four in ten is a reasonable threshold and it surfaces a category that a strict consecutive rule misses entirely.

A broken link that stops appearing in the crawl has been removed from the site, which means somebody fixed it — or means the page containing it was deleted. Both are resolutions and neither should be counted as a fix in a way that implies somebody acted.

So it closes as gone rather than fixed, with a note of which page it was on and whether that page still exists. The distinction matters for the one statistic worth watching, which Part 5 covers.

Next: what the report says, and what it deliberately leaves out.

All posts