Log anomaly spotter
A small system that turns log lines into fingerprints, counts each fingerprint per hour, and reports the ones that are new or that have moved far outside their own history. It never alerts on a threshold you would have to guess at, and it never wakes anybody. Seven posts on the same system — one diagram at a time — with a cost breakdown and an engineering reference at the end.
- 01
A log anomaly spotter on AWS for a few dollars a month
The whole system on one page — a fingerprinter, a counter and a comparer — built so that nobody has to decide in advance which errors matter.
- 02
How a log line becomes a shape
What gets stripped from a line and what must not be, why structured logs make this nearly free, and the two opposite failure modes of fingerprinting.
- 03
How a new shape is handled
Why a first occurrence is worth reporting at all, the warm-up that stops the first week being noise, and how a deploy is handled without suppressing real findings.
- 04
How a log rate change is judged
Comparing an hour against the same hour of the week, the shapes too rare to judge at all, why proportion beats absolute count, and the disappearance that matters more than any spike.
- 05
How the log digest reads
The hourly digest that is usually empty, what a single entry contains, why the example line matters more than the count, and the daily summary.
- 06
What the log anomaly spotter costs
About $6 a month. One Bedrock read per million lines is the only line that grows; the queue, the table, the mail and the storage are rounding errors. Plus the three ways the bill could surprise you.
- 07
Engineering reference: the log anomaly spotter architecture
Same system, drawn purely for engineers. Service names, region, Lambda inventory, IAM scopes, the schemas and the exact model id.
Frequently asked questions
- What is a log anomaly spotter?
- A small serverless system that reduces log lines to fingerprints — the same message with different values collapses to one shape — counts each shape per hour, and reports shapes that are new or that have moved far outside their own established pattern.
- How is this different from an error alarm?
- An error alarm needs somebody to decide in advance which errors matter and at what rate. This needs no thresholds: it learns each shape’s normal rate from its own history, so a new error is noticed on its first occurrence and a familiar one is not reported at all.
- Does it page anybody?
- No. It produces an hourly digest and a daily summary. Anything that genuinely needs waking somebody should have a real alarm on a real signal, not a log heuristic.
- What about the errors we already ignore?
- They become part of the baseline, which is the point. A stack trace that appears four hundred times an hour every hour is noise; the system reports it once, when it first appears, and then never again unless the rate changes.
- What does it cost to run?
- A few dollars a month for a small system’s logs. See part six.