Skip to content
allanninal.dev/build
Collaborate

Part 5 of 7 · Cost anomaly alerter series ~5 min read

Why the alerter cannot act

The obvious next feature, requested within a week of this system existing, is for it to stop the runaway rather than telling somebody about it. It is a reasonable request with a bad answer, and the reasoning is worth setting out properly rather than asserting.

Key takeaways

  • A cost signal is a lagging, aggregated, occasionally misattributed metric.
  • Acting on it means giving a system permission to delete production infrastructure.
  • The failure mode is asymmetric: a wrong action costs far more than a delayed fix.
  • The one defensible exception is a hard concurrency cap set in advance, not in response.
  • What the message carries instead: the projection, the resource, and the likely fix.

Three properties of the signal

Why a cost anomaly is not a safe trigger for automatic actionA vertical chain of five steps entered by a box labelled Should it act, the obvious request. Step one asks whether the signal is timely, and it is a day late, so the answer exits to No, because acting on stale data. Step two asks whether it is precise, and it is aggregated and sometimes wrong, exiting to No because misattribution happens. Step three asks whether the action is safe, covering stop, delete and scale down, and exits to No because that is production. Step four concludes: tell a person, fast, and with the diagnosis. Step five notes that speed is in the message rather than the action. A note says every one of the three would have to be yes for automatic action and none of them is.AWS ACCOUNTShould it act?the obvious requestIs the signal timely?a day lateNoacting on stale datanoIs it precise?aggregated, sometimes wrongNomisattribution happensnoIs the action safe?stop, delete, scale downNothat is productionnoSo: tell a personfast, and with the diagnosisSpeed is in the messagenot the actionEvery one of the three has to be yes for automatic action, and none of them is.
Fig 1. The three properties an automation would need and does not have. The conclusion is not caution for its own sake; it follows from what the signal actually is.
  • App integration
  • Security & identity
  • Management
  • Analytics
  • People

The asymmetry

A runaway that is fixed four hours later than it could have been costs a few more hours of whatever it costs, which is usually tens of pounds. A system that stops the wrong resource because a cost line was misattributed takes a production service down for as long as it takes somebody to work out what happened.

Those two are not comparable, and the comparison does not become closer as the bill gets bigger — a business with a larger bill also has a more expensive outage. The arithmetic points the same way at every scale.

The one defensible exception

A concurrency cap set in advance, as a permanent configuration rather than as a response. A Lambda function whose reserved concurrency is capped at fifty cannot run four hundred thousand times an hour whatever goes wrong, and the cap was set on a quiet Tuesday by somebody thinking about the function rather than by an alerter at three in the morning.

That is genuinely worth doing and it is not this system doing it. It is a standing design decision that limits the blast radius of every future mistake, and the useful thing this system can contribute is a monthly note about which functions have no cap and how much they could theoretically cost.

What the message carries instead

Four lines, sent by 09:15

  • Lambda, £32 yesterday. Normally £0.90 on a Tuesday.
  • One function: image-resize, 400,000 invocations against a normal 2,000. It writes to the same S3 prefix that triggers it.
  • On this rate the month will be about £960 rather than £330.
  • It is still running. Invocations in the last hour: 16,800.
  • To: the account owner and the team tagged on the function.

The fourth line is the one that produces action within minutes rather than within the day. Cost data is a day late, but the usage metric is current, and saying plainly that the thing is happening right now converts a finance message into an incident.

The monthly numbers

One month of cost anomaly detection summarised in five numbersA horizontal row of five boxes. Spend: three hundred and forty pounds, up four per cent. Anomalies: two, both fixed. Caught early: an estimated saving of about eleven hundred pounds. Untagged: twelve per cent of spend. Uncapped functions: four. A note says the third number is an estimate and should be labelled as one, and that it is still the point.ONE MONTH OF COST WATCHINGSpend£340, +4%Anomalies2, both fixedCaught earlysaved ~£1,100Untagged12% of spendUncapped functions4The third number is an estimate and should be labelled as one. It is still the point.
Fig 2. A month of cost watching in five numbers. The saving estimate is the one that justifies the system and the one to be careful about claiming precisely.

The saving figure is a projection of what the anomaly would have cost if it had run to the end of the month, minus what it did cost. It is an estimate, it should be presented as one, and it is also the only number that makes the case for keeping a cost monitor running in a month where nothing went wrong.

Next: what all of this costs to run.

All posts