Skip to content
allanninal.dev/build
Collaborate

Part 2 of 7 · Broken link reporter series ~5 min read

How the crawl is bounded

Crawling your own site is a solved problem right up until it is not, and the way it stops being solved is always the same: something on the site generates links, and the crawler follows them forever.

Key takeaways

  • Start from the sitemap, and separately from the home page, because they disagree.
  • Follow internal HTML links. Record everything else without following it.
  • Four traps: calendars, faceted filters, session parameters, and infinite pagination.
  • Normalise URLs before deduplicating, or the same page is crawled forty times.
  • A hard page cap, because no bound is perfect and an unbounded crawl is expensive.

Two starting points

The sitemap and the home page find different things, and the difference between them is itself useful. Pages in the sitemap that are not reachable by following links are orphans. Pages reachable by links that are not in the sitemap are missing from it. Both are worth knowing and neither is visible from a single starting point.

How the site crawl is boundedA vertical chain of five steps entered by a box labelled Start, from the sitemap and the home page. Step one reads robots.txt and obeys it on your own site. Step two asks whether this is an internal HTML link on the same host; if not it exits to Record but do not follow, covering PDFs, images and external links. Step three normalises the URL before deduplicating. Step four asks whether it has been seen after normalising; if so it exits to Skip, the usual outcome. Step five fetches and parses, up to the cap. A note says normalising before deduplicating is the difference between two thousand pages and forty thousand.AWS ACCOUNTStartsitemap and homeRead robots.txtand obey it on your own siteInternal HTML link?same host, text/htmlRecord, do not followPDFs, images, externalsnoNormalise the URLbefore deduplicatingSeen it?after normalisingSkipthe usual outcomeyesFetch and parseuntil the capNormalising before deduplicating is the difference between 2,000 pages and 40,000.
Fig 1. The crawl loop. Normalisation before deduplication is the single line that decides whether the crawl finishes.
  • App integration
  • Machine learning
  • Management
  • Analytics
  • Front-end & mobile

Normalisation

The same page is reachable as /about, /about/, /About, /about?utm_source=x and /about#team. Without normalisation the crawler treats those as five pages, fetches each, and finds the same links five times.

  • Drop the fragment. It never changes what the server returns.
  • Drop known tracking parameters. The utm family, click identifiers, and whatever your own analytics adds.
  • Sort remaining query parameters. ?a=1&b=2 and ?b=2&a=1 are the same page.
  • Lowercase the host, keep the path’s case. Hosts are case-insensitive and paths frequently are not.
  • Resolve the trailing slash according to what your server actually does, which you can determine once by asking it.

The four traps

Four crawl traps and the page cap that bounds themA horizontal row of five boxes. Calendars: a link to next month, forever. Faceted filters: every combination of facets. Session parameters: a new URL on each visit. Infinite pagination: page equals nine thousand still returns a page. A page cap: because none of these is fully solved. A note says the cap is not a fallback for bad rules but the acknowledgement that rules leak.FOUR WAYS A CRAWL NEVER ENDSCalendarsnext month, foreverFaceted filtersevery combinationSession parametersa new URL each visitInfinite paginationpage=9999 returns a pageA page capbecause none of these is fully solvedThe cap is not a fallback for bad rules. It is the acknowledgement that rules leak.
Fig 2. The four patterns that generate infinite URL space, and the cap that acknowledges no pattern list is complete.
  • Security & identity
  • Management
  • Analytics

Each trap has a rule that mostly works — skip URLs with a date parameter beyond some horizon, cap the number of query parameters, drop session-looking parameters, stop paginating when a page has no new links. None of them is complete, and the fifth measure is the one that actually guarantees termination: a hard cap on pages fetched per run, with the cap reported when it is hit.

Hitting the cap is information rather than a failure. “Stopped at 5,000 pages” on a site you believe has two thousand means something is generating URLs, and finding out what is usually worth more than the link report.

Record without following

A PDF, an image, an external link and a mailto: are all recorded as links to check and never fetched as pages to parse. That distinction keeps the crawl inside your own site while still checking everything that leaves it, which is where most breakage lives.

Next: how a link gets checked without being rude about it.

All posts