Engineering reference: the broken link reporter architecture

Key takeaways

Single region, single account. Every resource is regional; nothing is global except the IAM roles.
3 Lambda functions, each with its own execution role. No shared role, no wildcards on resources.
2 DynamoDB tables, each keyed so the concurrency story is a condition expression rather than a lock.
One Bedrock model, called once, with a JSON schema it must fill or leave null.
Nothing always-on: no instance, no container, no provisioned capacity.

The system, by service name

Fig 1. The same shape as Part 1 with the service names filled in. Nothing here is new; it is the same three groups, named.

Compute
Database
App integration
Analytics
Front-end & mobile

Region and account

Region: us-east-1. Chosen because SES inbound receipt rules exist in only a subset of regions and this one has the widest Bedrock model availability. If your data has to stay elsewhere, check both constraints before moving: inbound SES is the binding one.
Account: one. This is a small system, and a separate account per environment costs more in wiring than it saves. A dev and a prod stack in the same account, with distinct resource prefixes, is the right size here.
Everything is regional. The only global resources are the IAM roles and policies. There is no CloudFront, no global table and no cross-region replication, because nothing here has a latency or durability requirement that would justify them.

Lambda inventory

Function	Trigger	Does	Timeout / memory
`bl-crawl`	SQS frontier queue	Fetches one page, extracts links, enqueues new internal pages	30s / 1024 MB
`bl-check`	SQS check queue	HEAD then GET per unique URL, with per-host rate limiting	60s / 512 MB
`bl-report`	EventBridge weekly	Applies the three-run rule, joins traffic, builds the page	60s / 1024 MB

Splitting this into separate functions is not about modularity. It is that only one of them needs Bedrock permissions and only one is reachable from the public internet, and neither of those is true if it is one handler behind a router.

IAM, scoped

Role	Allowed	On
`bl-crawl-role`	`dynamodb:PutItem`, `sqs:SendMessage`	The links table; the frontier and check queues
`bl-check-role`	`dynamodb:UpdateItem`	The links table only
`bl-report-role`	`dynamodb:Query`, `ses:SendEmail`, `secretsmanager:GetSecretValue`	Links and runs, read; one identity; the analytics credential

No role has a Resource: “*” on anything that writes, and every GetSecretValue grant names a single secret arn. That is why there is more than one secret rather than one JSON blob with everything in it.

DynamoDB schemas

Table: links

PK   url_hash          S   sha256 of the normalised URL
     url               S   the normalised URL
     internal          BOOL true
     found_on          L   [normalised page URLs that link here]
     last_status       S   200 | 404 | timeout | 403 | soft404
     consecutive_fails N   3
     fails_in_last_10  N   4
     first_failed_run  S   2026-07-14
     state             S   ok | failing | reported | gone | unverifiable
     last_run          S   2026-07-28

`found_on` is the field that makes the report groupable by page. It is a
list because a URL in a footer appears on every page on the site.

Table: runs

PK   run_id            S   2026-07-28
     pages_crawled     N   1987
     cap_hit           BOOL false   — true is itself a finding
     unique_urls       N   4412
     requests_made     N   4412     — should equal unique_urls, not links found
     duration_s        N   412
     broken_confirmed  N   347
     top20_broken      N   4        — the number worth trending

`requests_made` equalling `unique_urls` is the assertion that the cache is
working. If it drifts upward, a footer link is being checked per page.

Inbound and outbound

The crawl is a queue, not a loop. Each page is one SQS message and one invocation, which means the crawl parallelises, survives a failure on one page, and never hits a Lambda timeout on a large site.
robots.txt is fetched once per run and obeyed on your own site. Disallowed paths are not crawled.
Per-host rate limiting uses a small DynamoDB counter with a conditional write, so several concurrent checkers cannot collectively exceed one request a second to any single host.
The user agent names the business and links to a page explaining what the crawler is. A blocked crawler is a broken crawler, and being identifiable is how you get unblocked.

The model call

There is no model in this system. Crawling, checking and ordering are all mechanical.
The one plausible use would be judging whether a 200 response is a soft 404, and simple heuristics — a redirect to the root, a very short body, known phrases — do it well enough and for nothing.
Suggesting replacement URLs is the other tempting use, and it is worse than it sounds: a plausible-looking wrong replacement inserted into a page is a more expensive error than the broken link was.
The report wording is fixed. There is nothing per-finding to generate.
The cost page assumes none, which is why there is no read band on it.

Things worth knowing before you build it

Normalise before deduplicating. It is the difference between crawling two thousand pages and forty thousand.
Cache check results per run. A site-wide footer link checked once per page will get you rate limited by somebody, deservedly.
Treat 403 and 429 as unverifiable, not broken. Reporting them teaches the reader that the report is wrong.
Skip the three-run rule for internal 404s. Your own missing page is not a transient network condition.
Track broken links on the top twenty pages, not the total. The total on an old site never reaches zero and setting it as a target is how the whole effort gets abandoned.

That is the whole system. Seven posts, one diagram at a time, and nothing in it that needs a server.

All posts