Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Broken link reporter series ~6 min read

A broken link reporter on AWS for a few dollars a month

Every broken-link tool produces the same output and the same outcome: a list of four hundred URLs, an intention to work through it, and nothing. The list is correct and unusable, because four hundred items with no priority is not a task. This post walks through a small system whose main design decision is what order to put things in.

A man sitting in a yellow bus with his head out of the window
Photo by Kseniia Ilinykh on Unsplash

Key takeaways

  • Crawl weekly, check every link, but only report failures that persist for three runs.
  • Order by the traffic on the page containing the link, not by the link.
  • Report the page to edit, not the URL that is broken. Those are different things.
  • Internal and external breakage are different problems and go to different lists.
  • Designed on AWS for about $2 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: a site crawled, links checked, pages to edit reportedThree boxes across the top sit outside the AWS account. On the left, Your site: crawled weekly. In the middle, Analytics: pageviews per page. On the right, Whoever edits: the person who receives a short ordered list. Each connects by an arrow to the AWS account container below. Pages and links flow down into the account. Analytics feeds in how much each page matters. A list of which pages to edit goes back out. Inside the AWS account are three components in a row. On the left, the Crawler, which fetches pages and every link on them. In the middle, the Checker, which makes one request per unique URL and caches the result. On the right, the Reporter, which surfaces persistent failures ordered by traffic. A note at the bottom says the output is a list of pages to edit, ordered by how many people see them.AWS ACCOUNTYour sitecrawled weeklyAnalyticspageviews per pageWhoever editsa short ordered listCrawlerpages, and everylink on themCheckerone request per URL,cachedReporterpersistent failures,by trafficpages and linkshow much itmatterswhich pages toeditThe output is a list of pages to edit, ordered by how many people see them.
Fig 1. Three things outside the account, three pieces inside it. Analytics is what turns a correct list into a useful one, and it is the input most link checkers do not have.
  • Networking
  • Analytics
  • Front-end & mobile
  • People

Order by the page, not the link

This is the whole design. A conventional report lists broken URLs, and a broken URL that appears in a footer template is four hundred rows. Ordered that way it dominates the list; grouped by page it is one fix.

And the pages are not equal. A dead link on a page nobody visits costs approximately nothing; the same dead link on your third most-visited page costs a customer. Ordering by pageviews turns four hundred rows into a first line that says “six broken links on your pricing page, which had 4,100 views last month”, which is a thing somebody does on a Tuesday morning.

What runs weekly (the inside)

  • The crawler. Starts from the sitemap and the home page, follows internal links, and records every link it finds along with the page it was on. Part 2 covers the boundaries: what it follows, what it records without following, and where it stops.
  • The checker. One request per unique URL, cached, with results shared across every page that links to it. A URL in a site-wide footer is checked once, not four hundred times, which matters both for cost and for not hammering somebody else’s server.
  • The reporter. Applies the three-run rule, joins to traffic, groups by page, and produces a list short enough to act on. Part 5 is about what it leaves out.
One broken link from discovery to report, in five stagesA horizontal row of five boxes joined by arrows. Found: on a page, while crawling. Checked: once per unique URL. Failed: on run one. Failed again: on runs two and three. Reported: with its page and that page's traffic. A note says three weeks before reporting, and almost everything transient resolves itself by then.ONE BROKEN LINK, END TO ENDFoundon a page, while crawlingCheckedonce per unique URLFailedrun 1Failed againruns 2 and 3Reportedwith its page and trafficThree weeks before reporting. Almost everything transient resolves itself by then.
Fig 2. The same system as one line. The deliberate delay between the first failure and the report is what keeps the list short enough to be worth reading.
  • Networking
  • Management
  • Analytics

In plain words

A site has about two thousand pages accumulated over eight years. The first full run finds four hundred and eleven links that fail. A conventional tool would send that list and nothing would happen, because four hundred and eleven is not a number anybody starts.

This one waits three weeks. By the third run, sixty-odd have resolved themselves — sites that were down, rate limits, a certificate that was renewed. Three hundred and forty-seven remain. Grouped by containing page that is a hundred and twelve pages, and ordered by traffic the first six pages account for eleven thousand views a month and twenty-two broken links between them.

The report leads with those six. Somebody spends forty minutes on a Tuesday and fixes the links that most of the site’s visitors would have hit. The remaining hundred and six pages, with about eight hundred views a month between them, are on page two of the report and will get done eventually or will not, and either outcome is fine. That is the difference between a list and a priority.

Design rules that shaped every decision

  • Report pages, not URLs. A footer link is one fix, not four hundred rows.
  • Order by the traffic on the containing page. A broken link nobody sees is not a problem worth anybody’s Tuesday.
  • Three runs before reporting. Almost all transient failures resolve within a fortnight.
  • One request per unique URL, cached. Politeness to other people’s servers is not optional.
  • Internal and external breakage are separate lists, because they have different fixes and different owners.
  • It never edits a page. The output is a list; the fix is a person’s.

Why this shape

There is no shortage of link checkers and most of them are technically fine. What they share is an output shaped by what is easy to produce rather than by what somebody can act on, and the result is that broken-link reports are famously generated and famously ignored.

So this design spends its effort on three things a generic tool cannot do: it knows which of your pages people actually read, it waits long enough to be confident, and it groups by the unit of work — a page somebody opens in an editor — rather than by the unit of failure. The crawling itself is the least interesting part.

The next four posts walk through each piece: how the crawl is bounded, how a link is checked politely, the three-run rule, and what the report leaves out. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts