Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Cost anomaly alerter series ~5 min read

How a cost anomaly is judged

Cost anomalies come in two shapes and most detection only catches one. A step change is obvious the next morning. A ramp — something growing ten per cent a day — never triggers a day-on-day comparison and is how the genuinely expensive surprises happen.

Key takeaways

  • Two shapes: a step change, and a ramp that no daily comparison will catch.
  • Absolute money gates everything. A 100x increase on four pence is four pence.
  • A new service with no history is reported once, on its first day of spend.
  • Weekend and weekday spend differ, so compare like days.
  • A service that stopped costing anything usually stopped working.

Two shapes

How a daily cost figure is judged for anomaliesA vertical chain of five steps entered by a box labelled A service, yesterday, with a figure. Step one asks whether the movement is over the money floor, for example five pounds; if not it exits to Ignore, however large the multiple. Step two asks whether there is any history, with fourteen days minimum; if not it exits to New service, reported once on day one. Step three asks whether it is a step change against the same weekday; if so it exits to Report it and attribute. Step four asks whether it is ramping, comparing the last seven days against the twenty-one before; if so it exits to Report the trend with a projection. Step five is Normal, which is most services on most days. A note says the ramp check is the one most implementations omit and it is where the money is.AWS ACCOUNTA service, yesterdaywith a figureOver the money floor?e.g. £5 of movementIgnorehowever large the multiplenoAny history?14 days minimumNew servicereport once, on day onenoStep change?vs the same weekdayReport itand attributeyesRamping?7-day trend vs the 21 beforeReport the trendwith the projectionyesNormalmost services, most daysThe ramp check is the one most implementations omit, and it is where the money is.
Fig 1. How a service’s daily figure is judged. The two detection shapes catch different failures and the second one is the expensive one.
  • App integration
  • Machine learning
  • Security & identity
  • Management
  • Analytics

The ramp

A service growing eight per cent a day never triggers a day-on-day comparison: today is always within a sensible band of yesterday. After a month it costs ten times what it did, and every individual day looked fine.

The usual causes are storage that is never expired, a log group filling up, a table growing without a TTL, or a workload that genuinely is growing and nobody has noticed how fast. All of them are cheap to fix early and expensive to discover late.

The check is deliberately simple: the last seven days’ average against the twenty-one before it. A twenty-five per cent increase between those two windows is a ramp, and the report carries the projection rather than the percentage, because “on this trend the month will be £430 rather than £180” is what gets attention.

The money floor

The single most important guard. Relative comparisons on small numbers are meaningless: a service that cost four pence yesterday and four pounds today has increased a hundredfold and is not worth a message.

So a movement must exceed an absolute amount before any relative test applies. Five pounds a day is a reasonable floor for a business whose bill is eleven pounds a day, and it should be set as a fraction of the total bill rather than as a fixed number, so it scales without anybody remembering to change it.

New services and falls

Two cost findings that are not spikesA horizontal row of five boxes. New service appears: with no history. Report once: saying this is new. Fourteen days later: it has a baseline. A service falls to zero: having been six pounds a day. Usually something broke: rather than a saving. A note says a service that stopped costing money has usually stopped doing its job.TWO CASES THAT ARE NOT SPIKESNew service appearsno historyReport once'this is new'14 days laterit has a baselineA service falls to zerowas £6 a dayUsually something brokenot a savingA service that stopped costing money has usually stopped doing its job.
Fig 2. Two findings that a spike detector misses entirely. The fall is the one people are surprised to see reported and it is frequently the more urgent of the two.
  • App integration
  • Security & identity
  • Management
  • Analytics

The new-service report is worth having because a service appearing on the bill for the first time is frequently somebody trying something, and the useful moment to ask whether it is going to stay is the day it appears rather than the month it becomes significant.

The fall is the one that surprises people in a cost alerter and it earns its place. A service that was costing six pounds a day and is now costing nothing has almost never become efficient overnight. A queue with no messages, a function with no invocations, a database with no reads — each of those is a saving on the bill and an outage in the business.

Next: attribution.

All posts