Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Log anomaly spotter series ~6 min read

A log anomaly spotter on AWS for a few dollars a month

Every small system produces more log lines than anybody will read and a handful of them matter. The usual responses both fail: alarming on the word ERROR produces a channel full of the same four exceptions you decided months ago were fine, and alarming on nothing means the genuinely new failure appears at the same volume as everything else. This post walks through a small system that needs no threshold and no opinion about which errors are important.

a person in a chefs uniform preparing food on a table
Photo by Lucas Law on Unsplash

Key takeaways

  • Lines become fingerprints: the same message with different values is one shape.
  • Every shape gets an hourly count, and its own history is its baseline.
  • Two findings: a shape nobody has seen before, and a shape far outside its own pattern.
  • Nothing pages. An hourly digest and a daily summary, because logs are not an alarm signal.
  • Designed on AWS for about $4 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: log lines fingerprinted, counted and compared to their own historyThree boxes across the top sit outside the AWS account. On the left, Log groups: your functions and services. In the middle, Known shapes: the record of what has been seen before. On the right, Whoever is on call: the person who receives an hourly digest. Each connects by an arrow to the AWS account container below. Lines arrive through a subscription filter. The known shapes feed in the question of whether something has been seen before. A short digest goes back out. Inside the AWS account are three components in a row. On the left, the Fingerprinter, turning lines into shapes with values stripped. In the middle, the Counter, counting per shape per hour. On the right, the Comparer, looking for shapes that are new or far outside their own history. A note at the bottom says there are no thresholds anywhere and each shape is measured against its own history.AWS ACCOUNTLog groupsyour functions and servicesKnown shapeswhat has been seenWhoever is on callan hourly digestFingerprinterlines to shapes,values strippedCounterper shape,per hourComparernew, or far outsideits own historysubscriptionfilterseen before?a short digestNo thresholds anywhere. Each shape is measured against its own history.
Fig 1. Three things outside the account, three pieces inside it. The absence of thresholds is the design: nobody has to predict in advance which errors will matter.
  • Database
  • Management
  • Analytics
  • People

Shapes, not lines

Two log lines that differ only in an order id, a timestamp and a duration are the same event happening twice. Collapsing them into one shape is what turns a stream of millions into a set of a few hundred, and a set of a few hundred is small enough to reason about completely.

That collapse is the whole idea. Once you have a bounded set of shapes with counts, the two interesting questions become trivial: which shapes are new, and which shapes are happening at a rate they have never happened at before. Neither requires anybody to have predicted anything.

What runs each hour (the inside)

  • The fingerprinter. Strips the variable parts of a line — numbers, identifiers, timestamps, paths, quoted strings — and hashes what is left. Part 2 is about doing that well enough that the same error always produces the same shape.
  • The counter. One count per shape per hour, plus the first example line seen for that shape in that hour, so a report can show a real line rather than a fingerprint.
  • The comparer. Two questions per shape. Is this shape new? And is this hour’s count far outside what this shape usually does at this hour of the week? Part 4 covers why the hour of the week matters.

One shape, end to end

One hour of logs from arrival to digest, in five stagesA horizontal row of five boxes joined by arrows. Lines arrive: millions of them. Fingerprinted: into a few hundred shapes. Counted: per shape, per hour. Compared: for new shapes or moved ones. Digested: into a few lines an hour. A note says the second step is the whole reduction and everything after it is arithmetic on hundreds.ONE HOUR OF LOGS, END TO ENDLines arrivemillionsFingerprinteda few hundred shapesCountedper shape, per hourComparednew, or movedDigesteda few lines an hourThe second step is the whole reduction. Everything after it is arithmetic on hundreds.
Fig 2. The same system as one line. The reduction from millions to hundreds is what makes the rest possible; without it every subsequent step is a data engineering problem.

In plain words

A small system produces about two million log lines a week across a dozen functions. Fingerprinted, those collapse to about three hundred and forty distinct shapes. Most are routine: a request completed, a batch processed, a scheduled job started.

About twenty are errors, and eleven of those have been happening steadily for months — a third-party timeout that retries successfully, a validation failure on badly formatted input, a warning from a library nobody can silence. Under an error-word alarm those eleven produce four hundred notifications a day and everybody has stopped looking.

Here they produce nothing, because their rate is exactly what it always is. On Thursday a new shape appears — a serialisation error nobody has seen before — eleven times in one hour. It is in the digest within the hour with the first real line and the function it came from, and it is the only thing in the digest, because everything else is behaving normally. That is the difference between a log alarm and this.

Design rules that shaped every decision

  • No thresholds. Every shape is measured against its own history rather than a number somebody guessed.
  • A new shape is interesting once. After it has been seen and not acted on, it becomes baseline like everything else.
  • Never page. Logs are a diagnostic signal, not an alarm signal, and treating them as one produces a muted channel.
  • Always show a real line. A fingerprint is not readable; the example line it came from is.
  • Compare like hours. Three in the morning on a Sunday is not comparable with eleven on a Tuesday.
  • Report absence too. A shape that always appears and has stopped is frequently the more urgent finding.

Why this shape

The reason log alerting fails is that it asks the wrong person at the wrong time. It asks somebody, in advance, to write down which strings matter and how often is too often — a prediction about failures that have not happened yet, made by somebody who does not yet know what they will look like.

Measuring each shape against itself removes the prediction entirely. A new failure is interesting because it is new, not because somebody guessed its wording; and a familiar one is uninteresting because it is familiar, not because somebody remembered to filter it out.

The next four posts walk through each piece: how a line becomes a shape, how a new shape is handled, how a rate change is judged, and what the digest says. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts