Engineering reference: the log anomaly spotter architecture

Key takeaways

Single region, single account. Every resource is regional; nothing is global except the IAM roles.
3 Lambda functions, each with its own execution role. No shared role, no wildcards on resources.
2 DynamoDB tables, each keyed so the concurrency story is a condition expression rather than a lock.
One Bedrock model, called once, with a JSON schema it must fill or leave null.
Nothing always-on: no instance, no container, no provisioned capacity.

The system, by service name

Fig 1. The same shape as Part 1 with the service names filled in. Nothing here is new; it is the same three groups, named.

Compute
Database
App integration
Management

Region and account

Region: us-east-1. Chosen because SES inbound receipt rules exist in only a subset of regions and this one has the widest Bedrock model availability. If your data has to stay elsewhere, check both constraints before moving: inbound SES is the binding one.
Account: one. This is a small system, and a separate account per environment costs more in wiring than it saves. A dev and a prod stack in the same account, with distinct resource prefixes, is the right size here.
Everything is regional. The only global resources are the IAM roles and policies. There is no CloudFront, no global table and no cross-region replication, because nothing here has a latency or durability requirement that would justify them.

Lambda inventory

Function	Trigger	Does	Timeout / memory
`la-fingerprint`	Kinesis, batched	Strips variables, hashes, increments an in-memory tally per batch	60s / 1024 MB
`la-rollup`	EventBridge hourly	Flushes tallies into hourly counts; records first example lines	60s / 1024 MB
`la-compare`	EventBridge hourly + daily	New-shape and rate checks; builds and sends the digests	30s / 512 MB

Splitting this into separate functions is not about modularity. It is that only one of them needs Bedrock permissions and only one is reachable from the public internet, and neither of those is true if it is one handler behind a router.

IAM, scoped

Role	Allowed	On
`la-fingerprint-role`	`kinesis:GetRecords`, `dynamodb:UpdateItem`	The log stream; the counts table
`la-rollup-role`	`dynamodb:UpdateItem`/`Query`	Shapes and counts
`la-compare-role`	`dynamodb:Query`, `ses:SendEmail`	Shapes and counts, read; one verified identity

No role has a Resource: “*” on anything that writes, and every GetSecretValue grant names a single secret arn. That is why there is more than one secret rather than one JSON blob with everything in it.

DynamoDB schemas

Table: shapes

PK   shape             S   sha256 of the stripped line
     example           S   a real line, the most recent one seen
     source            S   which log group and function
     level             S   ERROR | WARN | INFO
     first_seen        S   2026-08-02T10:14:00Z
     last_seen         S   2026-08-02T10:59:00Z
     new_until         S   first_seen + 3 reporting hours
     weekly_count      N   used for the rare-shape floor
     marked_expected   N   how many times a person said ’expected’

`marked_expected` accumulating is the tuning signal: a shape somebody has
dismissed eight times has a rate check that is set too tight.

Table: counts

PK   shape             S   sha256
SK   hour              S   2026-08-02T10
     count             N   412
     share             N   0.06   — proportion of all lines that hour
     ttl               N   epoch, +35 days

35 days is deliberate: the baseline compares against the same hour slot in
the last four weeks, so nothing older than that is ever read. Keeping more
would be storage nobody queries.

Inbound and outbound

CloudWatch subscription filters on each log group feed a single Kinesis stream. A filter pattern can pre-drop obvious noise before it is billed to this system, but not before CloudWatch ingestion, which has already been charged.
Batches, not lines. The fingerprint function receives a batch of records and tallies in memory before one write, which is what makes fifty million lines a week affordable.
The deploy feed is a timestamped identifier written to a small table, used only to group new shapes under a release heading.
No model, no external calls. The entire pipeline is regex, hashing and counting, which is also why it can keep up with a Kinesis stream.

The model call

There is no model in this system. Fingerprinting is stripping and hashing, and the comparisons are arithmetic.
The tempting use is asking a model to summarise a new stack trace, and it is worth resisting: the example line is already the summary, and a paraphrase adds a way to be misleading about an error.
A second tempting use is clustering shapes semantically rather than structurally, which has the same problem as clustering search queries: the groups shift and the history stops being comparable.
If you want one, use it interactively on a specific new shape rather than in the pipeline.
The cost page assumes none, which is why ingestion dominates it entirely.

Things worth knowing before you build it

Watch the shape count. Thousands of shapes mostly seen once means something variable is not being stripped; too few means two errors have collapsed, which is silent and worse.
Do not suppress new shapes after a deploy. A deploy is exactly when new errors appear; group and label them instead.
Compare like hours. A shape’s Tuesday-morning rate has nothing to do with its Sunday-night rate, and ignoring that reports an anomaly every week.
Check proportion as well as count, or a busy morning is reported as forty separate anomalies.
Report disappearance. A shape that always appeared and has stopped is often the most urgent finding, and nothing else in your stack will ever tell you.

That is the whole system. Seven posts, one diagram at a time, and nothing in it that needs a server.

All posts