Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Broken link reporter series ~5 min read

How a link gets checked politely

Checking external links means pointing a robot at several hundred other people’s servers every week. Doing that carelessly gets you blocked, which breaks the checker, and it is also just rude. This post is about doing it properly, and about the status codes that lie.

Key takeaways

  • One request per unique URL per run, with the result shared by every page linking to it.
  • HEAD first, GET only if HEAD is refused. Many servers dislike HEAD.
  • Rate limit per host, not globally. One slow host must not stall the run.
  • 403 and 429 usually mean ’not to a robot’, not ’broken’.
  • A soft 404 — a 200 that says the page is gone — needs its own check.

The request

How a single link is checked politelyA vertical chain of five steps entered by a box labelled A unique URL, from the crawl. Step one asks whether it is in the cache for this run; if so it exits to Reuse, so a footer link is checked once. Step two applies the per-host rate limit, waiting if needed. Step three sends a HEAD request as the cheapest option; a 405 or 501 exits to try GET instead. Step four interprets the status, since not all failures are failures. Step five caches the result for every page linking there. A note says a site-wide footer link is one request per run rather than one per page.AWS ACCOUNTA unique URLfrom the crawlIn the cache?this runReusea footer link is checked onceyesHost rate limitwait if neededHEADcheapest possible405 or 501try GET insteadrefusedInterpret the statusnot all failures areCache the resultfor every page linking hereA site-wide footer link is one request per run, not one per page.
Fig 1. How one URL is checked. The cache is the difference between a few thousand requests and a few hundred thousand.
  • Database
  • App integration
  • Machine learning
  • Networking
  • Management
  • Analytics

Rate limiting per host

A global rate limit is the obvious implementation and it has the wrong shape: it either hammers a single small host that happens to be linked forty times, or it slows the whole run to the speed of the slowest server.

Per-host limits — say one request a second to any single host, with several hosts in flight — solve both. It also means one unresponsive server delays only its own links, and a timeout budget per host keeps a dead domain from consuming the run.

Status codes that lie

CodeLooks likeUsually means
403ForbiddenThe server does not serve robots. Not broken.
429Too many requestsYou went too fast. Back off and retry.
405Method not allowedHEAD is refused. Try GET.
999NonsenseOne large social network’s way of saying no. Not broken.
503UnavailableOften temporary, sometimes a bot wall. Retry next run.
200FineSometimes a soft 404. Check the content.

The first four are the reason a naive checker produces a report full of links that are perfectly fine. A 403 from a site that blocks automated requests is not a broken link and reporting it teaches whoever reads the report that the report is wrong.

So those codes are recorded as unverifiable rather than broken. They appear in a separate short section of the report, once, with an explanation — because a person checking one by hand is a perfectly reasonable resolution and telling them twenty times is not.

Soft 404s

A page that returns 200 and says “this page no longer exists” is broken in every way that matters to a reader and invisible to a status check. They are common on sites that have been migrated, where the new platform serves a friendly page instead of a 404.

Three ways to detect a soft 404 behind a 200 responseA horizontal row of five boxes. 200 OK: looks fine. Body says gone: containing text like page not found. Or very short: under a length threshold. Or redirected: to the home page. Soft 404: reported as broken. A note says the third is the strongest signal, because a redirect to the home page is almost always a dead page.THREE SIGNS OF A SOFT 404200 OKlooks fineBody says gone'page not found'Or: very shortunder a thresholdOr: redirectedto the home pageSoft 404reported as brokenThe third is the strongest: a redirect to the home page is almost always a dead page.
Fig 2. How a soft 404 is detected. A redirect to the site root is the most reliable of the three signals and the cheapest to check.
  • Machine learning
  • Management
  • Analytics

Checking for soft 404s means fetching the body rather than just a HEAD, which costs more. So it is done only for external links that have redirected, and for internal links always — where it is cheap and where a migrated site is most likely to have them.

Next: the three-run rule, which is what keeps the report short.

All posts