Skip to content

Part 2 of 7 · Cost anomaly alerter series ~5 min read

How cost data arrives

Cost data is a day late and sometimes two, and every design in this space has to start by accepting that. What is worth knowing is exactly how late, what granularity you can get, and which faster signals exist for the handful of services that can run away faster than a day.

Key takeaways

  • Billing data settles about 24 hours behind, and some lines take longer.
  • Two sources: a daily API for service totals, and detailed reports for resource level.
  • Some services have no resource-level attribution at all, and the report says so.
  • Yesterday’s figure can still change; re-fetch the last three days each run.
  • For the fastest-moving services, a usage metric is a same-hour proxy worth adding.

The delay

How daily cost data is fetched and reconciledA vertical chain of five steps entered by a box labelled The daily run at nine in the morning. Step one fetches the last three days rather than just yesterday, from the cost API by service and daily. Step two asks whether any day changed, since restatements happen; if so it exits to Update the history and re-compare. Step three asks whether resource-level data is available for each service; if not it exits to Service level only, and the report says so. Step four fetches by resource for the services that support it. Step five is Ready to compare, with one day at two granularities. A note says re-fetching three days is what stops a restated figure producing a phantom anomaly.AWS ACCOUNTThe daily run09:00Fetch the last 3 daysnot just yesterdayCost APIby service, dailyAny day changed?restatements happenUpdate the historyand re-compareyesResource level available?per serviceService level onlysay so in the reportnoFetch by resourcefor the services that support itReady to compareone day, two granularitiesRe-fetching three days is what stops a restated figure producing a phantom anomaly.
Fig 1. The daily fetch. Re-reading the previous three days rather than only yesterday is what keeps the history honest as billing figures settle.
  • Database
  • App integration
  • Machine learning
  • Management
  • Analytics

Why three days

Yesterday’s figure is provisional. Some usage is reported late, some is reallocated between services, and credits and discounts land afterwards. A figure fetched on Wednesday for Tuesday can be meaningfully different when fetched again on Friday.

So each run re-fetches the last three days and updates the history rather than only appending. Without that, a day that was restated upwards stays in the baseline at its provisional value, and the baseline slowly drifts away from what the bill actually says.

Two granularities

LevelWhat you getLatency
Service, dailyTotal per service per dayAbout 24 hours
Resource, dailyPer function, bucket, table, instance24–48 hours, and not for everything
Usage metricsInvocations, requests, GB storedMinutes

The third row is the interesting one. Usage metrics are not cost data, they are available almost immediately, and for the services that can run away fastest they are a very good proxy. A Lambda function running four hundred thousand times in an hour is visible in metrics within minutes and in cost data tomorrow.

Where resource attribution does not exist

Several services report only at the account level, or aggregate resources in ways that make attribution impossible. Data transfer is the classic example: a large egress charge is real, attributable to nothing specific in the billing data, and frequently the hardest anomaly to chase.

The honest handling is to say so in the report rather than to guess. “Data transfer up £18; this service does not report at resource level, so the likely candidates by volume are these three buckets” is more useful than either silence or a confident wrong attribution.

The fast proxies

Three usage metrics that move before cost data doesA horizontal row of five boxes. Lambda invocations: available in minutes. S3 request count: minutes. NAT gateway bytes: minutes. Cost data: tomorrow. Both: a fast signal and a slow truth. A note says these three cover almost every runaway that can cost real money inside a day.THREE METRICS THAT MOVE BEFORE THE BILL DOESLambda invocationsminutesS3 request countminutesNAT bytesminutesCost datatomorrowBothfast signal, slow truthThese three cover almost every runaway that can cost real money inside a day.
Fig 2. The three usage metrics worth watching alongside the billing data. All three are available in minutes and all three precede the cost lines that most often run away.
  • Compute
  • Storage
  • Machine learning

Adding those three as separate hourly checks, compared against their own patterns exactly like the cost lines, turns a one-day detection into a one-hour one for the three services that most often produce a genuinely expensive surprise. They are not cost data and the report says so — “invocations are 200x normal; the cost impact will appear tomorrow” — which is both honest and completely actionable.

Next: how an anomaly is judged.

All posts