Broken link reporter
A small system that crawls your site, follows every internal and external link, and reports what is broken, ordered by how much traffic the containing page gets. It distinguishes a page that is gone from a site that is briefly down, and it never edits anything. Seven posts on the same system — one diagram at a time — with a cost breakdown and an engineering reference at the end.
- 01
A broken link reporter on AWS for a few dollars a month
The whole system on one page — a crawler, a checker and a reporter — with the ordering that turns an unreadable list into a morning’s work.
- 02
How the crawl is bounded
Where a crawl starts, what it follows versus records, the four traps that produce an unbounded crawl, and the cap that ends the argument.
- 03
How a link gets checked politely
HEAD before GET, one request per unique URL with a shared cache, per-host rate limits, and the status codes that do not mean what they appear to.
- 04
How the three-run rule works
Why a failure must persist across three weekly runs, what that delay costs, and the one class of failure that bypasses it entirely.
- 05
How the link report is ordered
Grouping by page, ordering by traffic, why the report has a hard length limit, and the one number that says whether the site is improving.
- 06
What the broken link reporter costs
About $13 a month. One Bedrock read per page crawled is the only line that grows; the queue, the table, the mail and the storage are rounding errors. Plus the three ways the bill could surprise you.
- 07
Engineering reference: the broken link reporter architecture
Same system, drawn purely for engineers. Service names, region, Lambda inventory, IAM scopes, the schemas and the exact model id.
Frequently asked questions
- What is a broken link reporter?
- A small serverless system that crawls your own site weekly, checks every link on every page, and reports the broken ones with the page they are on and how much traffic that page gets. It reports; it never edits a page.
- Why not one of the many existing tools?
- Plenty are good and this is genuinely a build-or-buy decision. What a small custom one gives you is ordering by your own traffic data, a report shaped like your own site, and no per-page pricing on a site that has grown to a few thousand pages.
- How does it tell a dead link from a site being down?
- It does not, on the first check. A failure has to persist across three consecutive weekly runs before it is reported, which removes almost all transient noise at the cost of a fortnight’s delay on genuinely new breakage.
- Does it respect robots.txt?
- On your own site it reads it and follows it, because a path you have disallowed is a path you do not want crawled. For external link checks it makes a single HEAD request per URL and caches the result, which is about as polite as checking can be.
- What does it cost to run?
- A couple of dollars a month for a site of a few thousand pages. See part six.