Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Page speed watcher series ~5 min read

How a speed regression is detected

A fixed threshold — alert if LCP goes above two and a half seconds — is the obvious approach and it is wrong for a specific reason: your pages are not equally variable, and a threshold tuned to the steadiest one will fire constantly on the noisiest.

Key takeaways

  • Compare against the page’s own rolling baseline, not a global threshold.
  • The trigger is a median outside the baseline’s spread, sustained for two runs.
  • The baseline rolls, so a deliberate improvement becomes the new normal automatically.
  • A rolling baseline cannot see slow drift, so a separate quarterly comparison does.
  • A page that has never been measured has no baseline and reports nothing for three weeks.

The page’s own spread

How a page speed regression is distinguished from normal variationA vertical chain of five steps entered by a box labelled Today's median, for one page. Step one asks whether there is enough history, with a minimum of twenty-one runs; if not it exits to No baseline yet, which measures and reports nothing. Step two builds the baseline from the median and spread of the last twenty-one runs, held per page and per metric. Step three asks whether today is outside the spread by a set multiple; if not it exits to Normal day, the usual outcome. Step four asks whether it is outside again tomorrow, requiring two runs rather than one; if not it exits to Hold, because one bad day happens. Step five is A regression, which gets explained. A note says two consecutive runs outside the spread, because one is a bad afternoon.AWS ACCOUNTToday's medianone pageEnough history?21 runs minimumNo baseline yetmeasure, report nothingnoBuild the baselinemedian and spreadof the last 21Historyper page, per metricOutside the spread?by a set multipleNormal daythe usual outcomenoAgain tomorrow?two runs, not oneHoldone bad day happensnoA regressionexplain itTwo consecutive runs outside the spread. One is a bad afternoon.
Fig 1. How a regression is confirmed. Requiring two consecutive runs costs a day of latency and removes nearly every false alarm.
  • Database
  • App integration
  • Machine learning
  • Management
  • Analytics

Why a multiple of the spread

A page whose median has sat between 1.8 and 2.2 seconds for three weeks has a spread of about 0.2 seconds, and 2.9 seconds is a long way outside it. A page that has ranged between 3.1 and 5.4 seconds — because it embeds a map, say — has a spread of over two seconds, and 5.6 is unremarkable.

One fixed threshold cannot serve both. A multiple of each page’s own spread serves both with a single setting, and it self-adjusts: a page that becomes more variable automatically becomes harder to alarm on, which is correct.

Why the baseline rolls

A rolling baseline of the last twenty-one runs means a deliberate improvement becomes the new normal within three weeks with nobody doing anything. Somebody optimises the pricing page from 2.0 to 1.3 seconds, and from then on the system is watching for regressions from 1.3.

A fixed baseline would keep congratulating you on being under a target set a year ago, and would miss a regression from 1.3 back to 1.9 entirely, because 1.9 is still under the target.

The drift a rolling baseline cannot see

How slow drift escapes a rolling baselineA horizontal row of five boxes. One point eight seconds in January: the baseline. Plus forty milliseconds a month: never outside the spread. No alarm, ever: and correctly so. Two point three seconds by August: nobody noticed. Quarterly compare: this quarter against last. A note says a rolling baseline follows drift by construction, which is both a feature and a blind spot.THE FAILURE MODE OF A ROLLING BASELINE1.8s in Januarybaseline+40ms a monthnever outside the spreadNo alarm, evercorrectly2.3s by Augustnobody noticedQuarterly comparethis quarter vs lastA rolling baseline follows drift by construction. That is a feature and a blind spot.
Fig 2. The one thing a rolling baseline structurally cannot detect, and the separate comparison that covers it.
  • Machine learning
  • Management
  • Analytics

Forty milliseconds a month is invisible to any comparison against the last three weeks, and it is exactly how sites get slow. It is not one bad deploy; it is eleven small ones, each of which was individually reasonable.

So there is a second, much simpler comparison that runs once a quarter: this quarter’s median against last quarter’s, per page, with no cleverness at all. A page that is twenty per cent slower than three months ago appears there whether or not any single day ever triggered anything.

New pages

A page added to the list has no baseline and reports nothing for three weeks, which is worth saying explicitly in the report so its absence is not mistaken for good news. During those three weeks it is measured normally; it simply has nothing to be compared against yet.

Next: budgets, which are the one place a fixed threshold does belong.

All posts