Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Cost anomaly alerter series ~6 min read

A cost anomaly alerter on AWS for a few dollars a month

Cloud bills go wrong in a particular shape. Nothing happens for months, then one change — a recursive trigger, a log group with no retention, a NAT gateway somebody added for one thing — starts costing forty pounds a day, and the first anybody knows is the invoice on the third of next month. By then it has cost twelve hundred pounds and it is still running. This post walks through a small system that notices on day one.

Two colleagues discussing documents at an office desk
Photo by Md Ishak Rahman on Unsplash

Key takeaways

  • Daily comparison per service, not a monthly total against a budget.
  • Each service is compared against its own recent pattern, so no thresholds are guessed.
  • The report names the resource, not just the service, which is what makes it actionable.
  • It cannot stop or delete anything, and that is a deliberate safety decision.
  • Designed on AWS for under $1 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: daily cost data compared per service and attributed to resourcesThree boxes across the top sit outside the AWS account. On the left, Cost data: daily, by service. In the middle, Resource tags: the record of what belongs to what. On the right, Whoever pays and whoever built it. Each connects by an arrow to the AWS account container below. Yesterday's spend flows down into the account. Tags feed in who owns what. The named resource goes back out. Inside the AWS account are three components in a row. On the left, the Fetcher, which pulls yesterday by service and by resource. In the middle, the Comparer, which measures each line against its own pattern. On the right, the Attributor, which works out which resource and what changed. A note at the bottom says the system has no permission to stop, delete or scale anything, and tells a person fast instead.AWS ACCOUNTCost datadaily, by serviceResource tagswhat belongs to whatWhoever paysand whoever built itFetcheryesterday, by serviceand by resourceComparereach line againstits own patternAttributorwhich resource,and what changedyesterday's spendwho owns whatthe resource,namedNo permission to stop, delete or scale anything. It tells a person, fast.
Fig 1. Three things outside the account, three pieces inside it. The note at the bottom is a capability decision rather than a limitation, and Part 5 argues for it.
  • Database
  • Analytics
  • People

Daily, per service

A monthly budget alert is a single number crossing a single line, and by construction it fires late: a runaway that starts on the 2nd will not push the month over budget until somewhere around the 24th, by which point it has cost twenty-two days of whatever it costs.

Comparing each service daily against its own recent daily spend catches the same event on the second or third day, when it has cost two or three days. That is the entire value proposition, and it needs no budget, no forecast and no threshold — only the observation that this service does not normally cost this.

What runs daily (the inside)

  • The fetcher. Pulls yesterday’s cost grouped by service, and separately by resource where the data supports it. Part 2 covers the delay, the granularity, and the services where resource-level attribution is not available.
  • The comparer. Each service against its own last few weeks, with the same day-of-week awareness as the log spotter, because weekend spend genuinely differs. Part 3 covers the two shapes a cost anomaly takes.
  • The attributor. Turns “Lambda is up £31” into "the image-resize function ran 400,000 times yesterday against a normal 2,000", which is the difference between a number and a cause.

One anomaly, end to end

One cost anomaly from daily data to an attributed report, in five stagesA horizontal row of five boxes joined by arrows. Yesterday's spend: by service. Compared: against its own twenty-eight days. One service moved: Lambda, up thirty-one pounds. Attributed: one function running two hundred times its normal invocations. Told: with the projection. A note says the fourth box is what turns a bill into a bug report.ONE COST ANOMALY, END TO ENDYesterday's spendby serviceComparedto its own 28 daysOne service movedLambda, +£31Attributedone function, 200x invocationsToldwith the projectionThe fourth box is what turns a bill into a bug report.
Fig 2. The same system as one line. Attribution is what changes the message from a finance observation into something an engineer can act on immediately.

In plain words

A small business’s AWS bill runs at about eleven pounds a day, and Lambda is normally about ninety pence of that. On Tuesday Lambda is thirty-two pounds. Nothing else moved.

The attributor looks at the resource-level data and finds one function, image-resize, with four hundred thousand invocations against a normal two thousand. It also notices that the function writes to the same S3 prefix that triggers it — which the message states as a fact rather than a diagnosis, but which is enough for anybody who has seen a recursive trigger before.

The message goes out on Wednesday morning: "Lambda £32 yesterday, normally £0.90. One function, image-resize, 400k invocations against a normal 2k. At this rate the month will be about £960 rather than £330." That is fixed before lunch and it has cost about sixty pounds. The version of this story without the system ends on the third of next month and costs nine hundred.

Design rules that shaped every decision

  • Compare daily, per service, against each service’s own history. No budgets, no thresholds.
  • Always attribute. A service name is a question; a resource name is an answer.
  • Project the month. The daily number is small and the projection is what gets attention.
  • It cannot stop anything. A cost metric is not a safe trigger for deleting infrastructure.
  • Tell the person who built it as well as the person who pays for it.
  • Report a fall as well as a rise. A service that stopped costing anything usually stopped working.

Why this shape

AWS has a native anomaly detection service and it is good; for many businesses that plus a budget is genuinely enough. The reason to build something small alongside it is the attribution and the wording: a native alert tells you a service moved, and the thing somebody needs is which resource and what to do about it.

So this design spends almost nothing on detection — a per-service comparison is a dozen lines — and almost everything on turning a detection into a sentence with a resource name and a monthly projection in it. That sentence is what gets the problem fixed the same morning.

The next four posts walk through each piece: how the cost data arrives, how an anomaly is judged, how it gets attributed to a resource, and why the system cannot act. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts