Skip to content
allanninal.dev/build
Collaborate

Part 4 of 7 · Log anomaly spotter series ~5 min read

How a log rate change is judged

A shape that appears eleven times at three on a Sunday morning and four hundred times at eleven on a Tuesday is behaving completely normally. Any comparison that ignores that will report a four-hundred-fold increase every Tuesday morning.

Key takeaways

  • Compare an hour against the same hour of the week, across several weeks.
  • Shapes seen fewer than a handful of times a week are too rare to judge on rate.
  • Compare proportion of traffic as well as absolute count, because volume moves.
  • A shape that always appears and has stopped is often the more urgent finding.
  • One report per shape per day, however many hours it keeps being unusual.

Same hour of the week

How a change in a log shape's rate is judgedA vertical chain of five steps entered by a box labelled This hour's count, for one shape. Step one asks whether it is a rare shape seen under twenty times a week; if so it exits to Only new-shape checks, because rate is meaningless. Step two compares against the same hour over the last four weeks, Tuesday eleven against Tuesday eleven, using history held per shape per hour slot. Step three asks whether this hour is outside that range by a multiple; if not it exits to Normal, the usual outcome. Step four asks whether the proportion of total traffic moved too, or whether all traffic simply rose; if only volume rose it exits to Traffic rather than the shape, noted quietly. Step five is a real rate change, reported once today. A note says the fourth check stops a busy morning being reported as forty separate anomalies.AWS ACCOUNTThis hour's countone shapeRare shape?under 20 a weekOnly new-shape checksrate is meaninglessyesSame hour, last 4 weeksTuesday 11am vs Tuesday 11amHistoryper shape, per hour slotOutside that range?by a multipleNormalthe usual outcomenoProportion moved too?or did all traffic rise?Traffic, not the shapenote it, quietlynoA real rate changereport once todayThe fourth check stops a busy morning being reported as forty separate anomalies.
Fig 1. How a rate change is judged. The proportion check is what distinguishes a shape behaving differently from the whole system simply being busier.
  • Database
  • App integration
  • Machine learning
  • Security & identity
  • Management
  • Analytics

Rare shapes cannot be judged on rate

A shape that occurs three times one week and eleven the next has tripled, and that means nothing at all. Small counts move around by large multiples for entirely uninteresting reasons, and any rate rule applied to them produces constant noise.

So shapes below a weekly floor are exempt from rate checking entirely. They are still covered by the new-shape rule, which is the useful check for something that rare: an error that happens three times a week is interesting the first time it appears and not particularly interesting thereafter.

Proportion, not just count

The check that saves the most false reports. On a morning when overall traffic doubles — a campaign, a news mention, a batch job — every shape’s count doubles, and a pure count comparison reports forty simultaneous anomalies.

Comparing each shape’s share of total log volume as well as its absolute count separates the two: a shape whose count doubled while its share stayed constant is just busier, and one whose share also moved is genuinely behaving differently. The first gets one quiet note about traffic; the second gets reported.

Disappearance

How a log shape that has stopped appearing is detectedA horizontal row of five boxes. Batch completed: logged every night, four hundred times. Tonight: zero. No error logged: nothing failed loudly. The job did not run: or died silently. Reported: as an absence. A note says absence produces no log line, which is why nothing else will ever tell you.THE SHAPE THAT STOPPED'Batch completed'every night, 400 timesTonightzeroNo error loggednothing failed loudlyThe job did not runor died silentlyReportedas an absenceAbsence produces no log line, which is why nothing else will ever tell you.
Fig 2. The finding that only a system counting shapes can produce. Nothing logs the absence of a log line, so no error-based alerting will ever surface it.
  • App integration
  • Machine learning
  • Security & identity
  • Management
  • Analytics

This is frequently the most valuable finding the system produces and it is invisible to every other kind of monitoring. A scheduled job that stops being scheduled, a consumer that stops consuming, a function that is no longer being invoked because a trigger was deleted — none of those produce an error, and all of them produce a shape that used to appear four hundred times and now appears zero times.

The rule is deliberately conservative: a shape that has appeared in the same hour slot in each of the last four weeks, with a count above the rare floor, and appears zero times this week, is reported as an absence. That is narrow enough to almost never fire spuriously and it catches the case that matters.

Once per shape per day

A shape that is genuinely running at ten times its normal rate will be outside its range for every hour of the incident, and reporting it hourly turns one finding into twelve. So each shape is reported once a day, with the subsequent hours added to the same entry as a duration rather than as new findings.

Next: what the digest says.

All posts