Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Log anomaly spotter series ~5 min read

How a new shape is handled

Reporting a brand-new log shape on its first occurrence is the single most valuable thing this system does, and it is the thing a rate-based alarm structurally cannot do: one occurrence never exceeds any threshold.

Key takeaways

  • A shape never seen before is reported on its first occurrence, at any count.
  • The first two weeks are a warm-up: everything is new, so nothing is reported.
  • After a deploy, new shapes are grouped and labelled rather than suppressed.
  • A new shape stops being new after it has appeared in three separate hours.
  • A shape that reappears after 60 days of absence is treated as new again.

First occurrence

How a previously unseen log shape is handledA vertical chain of five steps entered by a box labelled A shape this hour, with a count. Step one asks whether it has been seen before, ever, against a known shapes table holding first and last seen; a known shape exits to the rate check. Step two asks whether the system is still warming up in its first fourteen days; if so it exits to Record only, because everything is new. Step three asks whether there was a deploy in the last hour according to the deploy feed; if so it exits to Group and label as new since the release. Step four reports it with a real example line. Step five marks it as seen, keeping it new for three more hours. A note says deploys are labelled rather than suppressed, because a deploy is when new errors appear.AWS ACCOUNTA shape this hourwith a countSeen before?everKnown shapeswith first and last seenKnowngo to the rate checkyesStill warming up?first 14 daysRecord onlyeverything is newyesA deploy in the last hour?from the deploy feedGroup and label'new since the release'yesReport itwith a real example lineMark as seennew for three more hoursDeploys are labelled rather than suppressed. A deploy is when new errors appear.
Fig 1. How a new shape is handled. The deploy branch is the one worth getting right: grouping is useful and suppressing is exactly wrong.
  • Database
  • App integration
  • Machine learning
  • Management
  • Analytics

The warm-up

On the first day, every shape is new and the digest would contain three hundred and forty entries. So the first fourteen days record shapes without reporting them, and the system says clearly that it is warming up and when it will start.

Fourteen days is chosen to cover the weekly cycle twice, which matters because a substantial number of shapes only occur on a particular day — a Sunday batch job, a Monday morning report, a month-end process that will not appear at all in the first two weeks and will be reported as new when it does. That last case is worth expecting rather than being surprised by.

Deploys: label, do not suppress

The obvious handling is to suppress new shapes for an hour after a deploy, since a release naturally produces new log messages. It is also precisely backwards: a deploy is the single most likely moment for a genuinely new error to appear, and suppressing exactly then removes the system’s best opportunity.

So new shapes after a deploy are grouped under a heading — “7 new shapes since release a3f21c” — and reported together. A person scanning that sees six that are obviously new informational messages and one that is a stack trace, which is a much better outcome than either suppressing all seven or listing them individually.

When a shape stops being new

The lifecycle of a log shape from new to baselineA horizontal row of five boxes. First seen: reported. Hour two: still labelled new. Hour three: the last time it is treated as new. Now baseline: rate checks apply. Absent sixty days: it becomes new again if it returns. A note says three hours rather than three occurrences, because a burst in one hour is still one event.THE LIFE OF A SHAPEFirst seenreportedHour twostill labelled newHour threelast time as newNow baselinerate checks applyAbsent 60 daysnew again if it returnsThree hours, not three occurrences. A burst in one hour is still one event.
Fig 2. How a shape transitions from new to baseline. Counting hours rather than occurrences means a single burst does not exhaust the new window.

Counting hours rather than occurrences matters. A new shape that appears four hundred times in its first hour is one event, and treating it as three occurrences’ worth of novelty would move it to baseline before anybody had seen it in the digest.

Reappearance after absence

A shape that has not been seen for sixty days and then returns is treated as new again, which catches a specific and common case: an error that was fixed months ago and has come back. Under a pure seen-before rule it would be recognised as familiar and never mentioned, which is the wrong answer for something that was absent for two months.

Next: how a rate change is judged.

All posts