Skip to content
allanninal.dev/build
Collaborate

Part 5 of 7 · Log anomaly spotter series ~5 min read

How the log digest reads

The digest is short by construction and empty most hours, which is a property worth protecting. A channel that carries something every hour is a channel where an empty hour means nothing, and an empty hour is the most common true state of a small system.

Key takeaways

  • Most hours the digest is empty and it is not sent at all.
  • An entry is: the example line, the counts, the function, and why it is here.
  • The example line is a real line, not a fingerprint, and it is the first thing shown.
  • Two buttons: expected, and worth looking at. Both are recorded.
  • The daily summary carries shape counts; the hourly digest never does.

An hour with something in it

10:00–11:00

  • New shape — 11 occurrences, order-worker
    Order 84412 failed after 1203ms: serialisation error in line item 3
    First seen 10:14. Not present in the previous 30 days.
  • Rate change — api
    Retrying upstream request (attempt 2)
    412 this hour, usually 20–40 at this hour on a Tuesday. Share of total logs up from 0.4% to 6%.
  • Two buttons on each: expected · worth looking at

Two entries and that is the whole message. The example line comes first in each because it is the thing that lets somebody recognise the problem in a second; the counts and the history are context underneath it.

Why the real line goes first

A fingerprint is unreadable and a count is uninterpretable without knowing what is being counted. “Shape a3f21c is up 10x” requires a lookup before it means anything, and a digest that requires a lookup does not get read on a phone.

The first real line seen for that shape in that hour costs one extra field to store and turns the entry into something a person understands immediately, including a person who has never heard of this system.

The empty hours

How the hourly log digest is assembled and sentA vertical chain of five steps entered by a box labelled The hourly run, every hour. Step one asks whether there is anything to report; no exits to Send nothing, which is most hours. Step two builds the entries with the example line first. Step three asks whether there are more than five, which means something is wrong; if so it exits to Send a summary saying twenty-three anomalies and pointing at the daily. Step four sends the digest with two or three entries. Step five records the answers, expected or not. A note says a digest with twenty-three entries is not a digest but its own finding.AWS ACCOUNTThe hourly runevery hourAnything to report?Send nothingmost hoursnoBuild the entriesexample line firstMore than five?something is wrongSend a summary'23 anomalies -- see the daily'yesSend the digesttwo or three entriesRecord the answersexpected, or notA digest with 23 entries is not a digest. It is its own finding.
Fig 1. The hourly run and its two unusual paths: sending nothing, which is most hours, and refusing to send twenty-three entries, which is itself a signal.
  • App integration
  • Machine learning
  • Management

Refusing to send a long digest

Twenty-three anomalous shapes in one hour is either a genuine major incident, in which case somebody already knows, or a problem with the system itself — a deploy that changed every log message, a fingerprinting change, a log group that started including something new.

In both cases a list of twenty-three entries is the wrong output. So above a threshold the digest collapses to one line saying how many and pointing at the daily summary, which is both more readable and more accurate about what is actually being said.

The two buttons

“Expected” and “worth looking at”, both recorded against the shape. Neither changes the system’s behaviour automatically, and that restraint is deliberate: a shape marked expected once should not be permanently silenced, because the same shape at a hundred times the rate next month is a different event.

What the answers do is accumulate. A shape marked expected on eight separate occasions is a shape whose rate check is set too tight, and that appears in the monthly review as a specific suggestion rather than as something somebody has to notice.

The daily summary

One day of log analysis summarised in five numbersA horizontal row of five boxes. Shapes seen: three hundred and forty-one. New today: two. Rate changes: three. Disappeared: one. Marked expected: four of six. A note says the last number is the tuning signal, and four expected out of six is too tight.THE DAILY SUMMARYShapes seen341New today2Rate changes3Disappeared1Marked expected4 of 6The last number is the tuning signal: four expected out of six is too tight.
Fig 2. The daily summary in five numbers. The ratio of findings marked expected is what says whether the thresholds need loosening.
  • Machine learning
  • Management
  • Analytics

Four findings out of six marked expected means two thirds of what the system reported was not worth reporting, and that is a bad ratio that will end with the digest being ignored. One or two out of six is healthy; the multiplier on the rate check is the knob that moves it.

Next: what all of this costs to run.

All posts