Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Page speed watcher series ~6 min read

A page speed watcher on AWS for a few dollars a month

Almost every page speed setup fails the same way. Somebody runs a tool, gets a score, shares it, and three weeks later runs it again and gets a different score. Nobody knows whether the site got slower or the measurement did, so nothing happens, and eventually the checking stops. This post walks through a small system built around the one property that fixes it: measuring the same thing, the same way, enough times to know what noise looks like.

red and white concrete building
Photo by Raymond Kotewicz on Unsplash

Key takeaways

  • A short list of pages, chosen deliberately. Not every page, and not the home page only.
  • One device and network profile, fixed forever. Changing it invalidates the whole history.
  • Several samples per run, because a single page load is not a measurement.
  • A regression is reported only when it exceeds the noise the site’s own history shows.
  • Designed on AWS for about $4 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: pages measured on a fixed profile, regressions reportedThree boxes across the top sit outside the AWS account. On the left, Your pages: a deliberately short list of six or eight. In the middle, Deploy events: what shipped and when. On the right, Whoever owns the site: the person who hears about regressions. Each connects by an arrow to the AWS account container below. Loads and timings flow down into the account. Deploy events feed in so a change can be correlated. A regression, explained, goes back out. Inside the AWS account are three components in a row. On the left, the Measurer, using a fixed profile and taking several samples. In the middle, the Comparer, judging against the site's own noise. On the right, the Reporter, saying what got heavier and after what. A note at the bottom says it never blocks a deploy, because a noisy metric that gates releases gets bypassed.AWS ACCOUNTYour pagesa list of six or eightDeploy eventswhat shipped, whenWhoever owns the sitehears about regressionsMeasurerfixed profile,several samplesCompareragainst the site'sown noiseReporterwhat got heavier,and after whatloads and timingscorrelate to adeploya regression,explainedIt never blocks a deploy. A noisy metric that gates releases gets bypassed.
Fig 1. Three things outside the account, three pieces inside it. The deploy feed is what turns “this page got slower” into “this page got slower after Tuesday’s release”, which is the difference between a fact and a lead.
  • Management
  • Analytics
  • Front-end & mobile
  • People

A single page load is not a measurement

Load the same page ten times on the same connection and the numbers will differ by twenty per cent or more. Server warm-up, network variance, CDN cache state, a third-party script that is slow this minute. None of that is your site changing, and all of it looks exactly like your site changing if you measure once.

So every run takes several samples of every page and keeps the median. That single decision is what makes everything downstream possible: with a median of nine samples you can say what normal variation looks like, and once you know that, you can say when something is outside it.

What runs daily (the inside)

  • The measurer. Loads each page on the list several times in a headless browser on a fixed CPU and network profile, and records the metrics plus the full resource list. Part 2 covers the profile and why it must never change.
  • The comparer. Takes the median for each page and compares it against a rolling baseline of the last few weeks, using the spread of that baseline to decide what counts as a change. Part 3 is entirely about this.
  • The reporter. When something regresses, works out what got heavier by diffing the resource lists, and correlates the timing against the deploy feed. The output is “the hero image is now 1.4MB rather than 180KB”, not “LCP increased”.

One regression, end to end

One page speed regression from measurement to explanation, in five stagesA horizontal row of five boxes joined by arrows. Measured: nine samples on a fixed profile. Median: one number per page. Compared: against the last twenty-one days. Outside the noise: not merely higher. Explained: what changed, and after what. A note says steps two and four are the whole design and everything else is a headless browser.ONE REGRESSION, END TO ENDMeasured9 samples, fixed profileMedianone number per pageComparedto the last 21 daysOutside the noisenot just higherExplainedwhat changed, after whatSteps two and four are the whole design. Everything else is a headless browser.
Fig 2. The same system as one line. Taking a median and comparing against the site’s own spread are the two steps that separate this from a tool that reports a different score every week.

In plain words

The pricing page has had a largest-contentful-paint median between 1.8 and 2.2 seconds for three weeks, measured nine times a day on a throttled mid-range mobile profile. On Wednesday the median is 3.9 seconds. That is well outside the range the page has shown across sixty-three previous runs, so it is a regression rather than a bad day.

The reporter diffs Wednesday’s resource list against Tuesday’s. One image on the page is now 1.4 megabytes where it was 180 kilobytes. The deploy feed shows a release at 09:12 on Wednesday, and the first slow run was 09:40. The message says all of that in three lines: the page, the number, the resource, and the release.

Somebody looks and finds that a new hero image was uploaded through the CMS without being resized. It is a five-minute fix, and the value of the system is entirely in somebody finding out on Wednesday rather than in three months when a customer mentions the site feels slow on their phone.

Design rules that shaped every decision

  • One profile, forever. The absolute number is far less important than comparability.
  • Several samples, always. A single load is not a measurement and treating it as one produces a tool nobody trusts.
  • Compare against the site’s own spread, not a fixed threshold. Every page has a different amount of natural variation.
  • Name the resource, not the metric. “LCP up 1.7s” is a fact; “the hero image is now 1.4MB” is a fix.
  • Never block a deploy. A noisy gate is a gate people learn to bypass.
  • A short page list. Eight pages measured properly beats four hundred measured once.

Why this shape

Web performance monitoring has a credibility problem, and it is self-inflicted. Tools report a score, the score moves for reasons unrelated to the site, and everybody learns to discount it. By the time something genuinely regresses, the signal has no audience.

So this design spends nearly all of its effort on making the number trustworthy: a fixed profile, many samples, and a comparison that understands its own noise. The measuring is a headless browser and a library, which is the least interesting part, and the reporting is short because a trustworthy number does not need much explanation.

The next four posts walk through each piece: how a page gets measured, how a regression is distinguished from noise, how a budget is set, and what the report says. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts