Diagnostic LLM APIs
529 overloaded errors arrive in clusters and get dropped
Three minutes on a Tuesday afternoon, and about fourteen hundred jobs are simply not in the output table. Nothing crashed. The worker's error counter shows a spike of something it logged as unexpected status, because the client special-cases 429 and 500 and lets everything else fall through to a generic failure path that drops the work. The status was 529, the platform was over capacity for four minutes, and the only trace left is that Anthropic did no work in those minutes while your client believed it was sending plenty.
Two numbers, and Anthropic can only supply one of them. Read the per-minute usage buckets with an Admin API key: GET /v1/organizations/usage_report/messages?starting_at={now-4h}&bucket_width=1m&group_by[]=service_tier. Then hand the script your own count of requests attempted, per minute, because that number exists nowhere in the API.
The report has no request-count field at all — only token sums — so served requests have to be inferred from the work that was done. Take the median tokens per attempt across the window as a baseline, divide each minute's billed tokens by it to estimate how many attempts were served, and subtract. The remainder is requests that produced nothing.
Then look only at contiguous runs of such minutes. A single bad minute is a bucket-boundary artefact: a call that starts at 14:03:58 and finishes at 14:04:06 lands its attempt in one minute and its tokens in the next. A platform capacity condition affects everyone at once and lasts, so it arrives as a run. That is why this note is about clusters rather than about a rate.
The problem in plain words
529 is overloaded_error, and the docs are explicit that it happens when the API is under high traffic across all users. It is not caused by your traffic and it is not something a smaller prompt or a lower concurrency avoids. It is retryable, it clusters in time, and the official SDKs already retry it. Hand-rolled clients frequently do not, because the error class list in the code was written from the errors that had been seen so far, and 529 is unusual enough that it was not one of them.
What that produces is silent data loss with a perfect-looking error rate. The failures are counted somewhere as a generic exception, the queue moves on, and the work is gone. Because the loss is measured in requests and Anthropic reports only tokens, there is no report you can pull that names it — which is exactly why teams find it months later, in a reconciliation between what was submitted and what came back.
Why it happens
Anthropic's messages usage report has no request count. This is the single fact that shapes the whole script. The buckets carry uncached_input_tokens, cache_read_input_tokens, the nested cache_creation pair and output_tokens, and nothing that counts calls. So "billed requests" cannot be read; it can only be estimated from tokens, and every number this script prints is an estimate that says so out loud.
The baseline has to be a median, not a mean. The minutes being hunted are exactly the ones that would drag a mean tokens-per-attempt down, so a mean baseline quietly absorbs the loss it was computed to reveal. The median over the window survives a cluster covering up to half of it, which is a much better failure mode than the alternative.
Contiguity separates a real cluster from bucket arithmetic. A request spans a minute boundary all the time, so isolated minutes with a shortfall are noise and always will be. Requiring three or more adjacent minutes is what turns a noisy residual into a finding, and it matches the mechanism: a capacity condition on the platform is not a coin flip per request.
This is the mirror image of the streaming reconciliation, not a copy of it. The streaming note finds the provider's number larger than yours and concludes your pipeline is failing to record tokens you were billed for. This one finds your number larger than the provider's and concludes the platform never did the work. Same style of comparison, opposite sign, and opposite owner: there the bug is in your telemetry, here the telemetry is right and the requests are gone. The script reports an excess as its own state and hands that reader straight to the other note rather than calling it overload.
It is also not the retry-storm read. The divergence note runs on OpenAI, needs nothing from you, and looks for requests the provider did count. This one cannot run at all without a number you supply, and looks for requests the provider never counted. A retry storm inflates the provider's own request count; a 529 never reaches it. If your client retries 529 successfully, both are true at once and the residual measures only the attempts that were never served.
Priority Tier is not the escape hatch it used to be. Grouping by service_tier tells you whether any of your traffic was served as priority, and on most organizations the answer is now none, because capacity commitments are no longer sold. That makes 529 handling everybody's problem rather than something a purchase order solves, which is worth printing next to the clusters.
The fix, as a flow
The messages usage report has no request count, so there is nothing to subtract your attempts from. Served requests have to be inferred from the work that was done, against a median baseline that a cluster cannot drag down, and only runs of adjacent minutes count: one bad minute is a call that straddled a bucket boundary.
How to fix it
Export your own attempt counter, per minute
A JSON object keyed by minute: {"2026-08-30T14:03Z": 900}. Count attempts, including retries, at the point the request leaves your process — not logical jobs, and not successes. This is the half of the comparison the API cannot give you, and getting it wrong in either direction moves the whole finding.
Read one-minute usage buckets for the same window
bucket_width=1m with group_by[]=service_tier, paging on next_page. Sum every token field, cache reads and both cache-creation fields included: the question is whether the platform did work in that minute, not what that work cost. A parser that looks for a flat cache_creation_input_tokens finds nothing, because it is a nested object.
Establish a baseline with the median, never the mean
Median tokens per attempt across the minutes both sources cover. Take the median because a mean would be pulled down by the very minutes you are looking for, which is how a first draft of this check comes back clean during an actual outage.
Estimate served attempts and keep only the adjacent minutes
Billed tokens divided by the baseline is roughly how many attempts were served; attempts minus that is the residual. Group the minutes whose residual share crosses the floor into contiguous runs and discard the runs shorter than three minutes. Those are boundary artefacts, and reporting them is how a check gets muted.
Print the retry class, and never retry anything
The repair is one retryable class containing 429, every 5xx and 529, with exponential backoff and jitter — or simply the SDK's own retry rather than a hand-rolled except. Print the affected minutes, the estimated loss and the request-id values your logs should already be capturing from every response, error responses included. An audit script that starts re-sending traffic into a platform that is over capacity is the one thing worse than the bug.
How to check it worked
Re-run after the retry class is widened. The clusters should still appear in the platform's bad minutes and no longer correspond to lost work, because the attempts now succeed on the second try.
python3 anthropic_overload_residual.py --attempts attempts.json --minutes 240
# overload-cluster 2026-08-30T14:04Z through 2026-08-30T14:06Z: 1800 attempt(s) over 3 minute(s), about 1440 of them produced no billed tokens (80%)
# baseline 5000 token(s) per attempt, taken as the median across 240 minute(s)
# no traffic in this window was served as priority
# 240 minute(s) compared, 1 cluster(s)The full code
One GET and a file you supply. The Admin key reads the per-minute buckets; the attempt counts come from you, because the report has no request-count field to compare against. Eight pure functions: the minute normaliser, which has to make your timestamps and Anthropic's agree; the minute index that makes adjacency testable; the token sum, which reaches into the nested cache-creation object; the attempt reader; the median baseline; the per-minute residual; the clustering, which is where the finding actually lives; and the classifier that throws away everything shorter than three minutes.
"""Size the Anthropic requests that were attempted and never served.
Read only. One GET against the Admin API, which needs an Admin API key
(sk-ant-admin...); a workspace key is rejected by every /v1/organizations/*
path, and an Admin key can be provisioned read-only.
The messages usage report carries token sums and no request count at all, so
"requests the platform served" cannot be read. It is estimated from the work
that was done: median tokens per attempt as a baseline, billed tokens divided
by it, subtracted from your own attempt counter. Every number here is an
estimate and the output says so.
Nothing is retried and nothing is sent. A script that starts re-issuing traffic
into a platform that is over capacity is worse than the bug it found.
"""
import argparse
import datetime as dt
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_overload_residual")
API = "https://api.anthropic.com/v1"
VERSION = "2023-06-01"
# Every token field the messages usage report returns. cache_creation is a
# nested object; a parser looking for a flat cache_creation_input_tokens sums
# zero and reports a heavily cached minute as one where nothing happened.
TOKEN_FIELDS = ("uncached_input_tokens", "input_tokens",
"cache_read_input_tokens", "output_tokens")
CACHE_CREATION_FIELDS = ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens")
FINDINGS = ("overload-cluster",)
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def minute_key(stamp):
"""Normalise a timestamp to a UTC minute key. Pure. None if unreadable.
Accepts the RFC 3339 strings the usage report returns and the shapes your
own counter is likely to emit: with or without seconds, with a space
instead of a T, or as epoch seconds. Two sources that disagree about
timestamp format produce a comparison with no overlap and a clean bill of
health, which is the worst possible failure for this check.
"""
if isinstance(stamp, bool):
return None
if isinstance(stamp, (int, float)):
try:
when = dt.datetime.fromtimestamp(int(stamp), dt.timezone.utc)
except (ValueError, OSError, OverflowError):
return None
return when.strftime("%Y-%m-%dT%H:%MZ")
text = str(stamp or "").strip().replace(" ", "T")
if len(text) < 16:
return None
head = text[:16]
if head[4] != "-" or head[7] != "-" or head[10] != "T" or head[13] != ":":
return None
for part in (head[0:4], head[5:7], head[8:10], head[11:13], head[14:16]):
if not part.isdigit():
return None
return head + "Z"
def minute_index(key):
"""Minutes since the epoch for a minute key. Pure. None if unreadable.
Adjacency is the whole finding, so it needs to be arithmetic on integers
rather than string comparison, which gets 14:59 and 15:00 wrong.
"""
normalised = minute_key(key)
if normalised is None:
return None
try:
when = dt.datetime(int(normalised[0:4]), int(normalised[5:7]),
int(normalised[8:10]), int(normalised[11:13]),
int(normalised[14:16]), tzinfo=dt.timezone.utc)
except ValueError:
return None
return int(when.timestamp()) // 60
def tokens_by_minute(buckets):
"""Total billed tokens per minute. Pure.
Every field is summed, cache reads and cache creation included, because the
question is whether the platform did any work in that minute and not what
the work cost.
"""
out = {}
for bucket in buckets or []:
key = minute_key(bucket.get("starting_at") or bucket.get("start_time"))
if key is None:
continue
total = 0
for result in bucket.get("results") or []:
for field in TOKEN_FIELDS:
total += _int(result.get(field))
creation = result.get("cache_creation") or {}
for field in CACHE_CREATION_FIELDS:
total += _int(creation.get(field))
out[key] = out.get(key, 0) + total
return out
def attempts_by_minute(raw):
"""Read your own attempt counter into minute keys. Pure.
Accepts {"2026-08-30T14:03Z": 900} or {"...": {"attempts": 900}}. Minutes
that cannot be parsed are dropped rather than folded into a neighbour: an
attempt attributed to the wrong minute breaks the contiguity test, which is
the only thing separating a finding from noise.
"""
out = {}
for stamp, value in (raw or {}).items():
key = minute_key(stamp)
if key is None:
continue
if isinstance(value, dict):
count = _int(value.get("attempts"))
elif isinstance(value, bool):
count = 0
else:
count = _int(value)
out[key] = out.get(key, 0) + count
return out
def _median(values):
"""Median of a list of numbers. Pure. None when empty."""
ordered = sorted(values or [])
if not ordered:
return None
middle = len(ordered) // 2
if len(ordered) % 2:
return float(ordered[middle])
return (float(ordered[middle - 1]) + float(ordered[middle])) / 2.0
def baseline_tokens_per_attempt(tokens, attempts, min_minutes=5, min_attempts=1):
"""Median tokens per attempt across the covered minutes. Pure.
The median, never the mean. The minutes this script is hunting are exactly
the ones that would drag a mean down, so a mean baseline absorbs the loss
it was computed to reveal and the check comes back clean during an outage.
"""
ratios = []
for key, made in (attempts or {}).items():
made = _int(made)
if made < min_attempts:
continue
ratios.append(_int((tokens or {}).get(key)) / float(made))
if len(ratios) < min_minutes:
return None
value = _median(ratios)
return value if value and value > 0 else None
def residual_rows(tokens, attempts, baseline):
"""One row per minute: attempts, tokens, estimated served and residual. Pure.
served is tokens / baseline, which is an estimate and the only one
available: the report has no request count to read instead.
"""
out = []
if not baseline or baseline <= 0:
return out
for key in sorted(attempts or {}):
made = _int(attempts.get(key))
if made <= 0:
continue
billed = _int((tokens or {}).get(key))
served = billed / float(baseline)
residual = max(0.0, made - served)
out.append({"minute": key, "index": minute_index(key), "attempts": made,
"tokens": billed, "served": served, "residual": residual,
"share": residual / float(made)})
return out
def clusters(rows, floor=0.3, min_attempts=20):
"""Group the shortfall minutes into contiguous runs. Pure.
Contiguity is the finding. A request that starts at 14:03:58 and finishes
at 14:04:06 lands its attempt in one minute and its tokens in the next, so
isolated minutes are bucket arithmetic. A platform capacity condition is
not a coin flip per request and arrives as a run.
"""
bad = [r for r in rows or []
if r.get("index") is not None
and _int(r.get("attempts")) >= min_attempts
and float(r.get("share") or 0.0) >= floor]
bad.sort(key=lambda r: r["index"])
runs = []
for row in bad:
if runs and row["index"] == runs[-1][-1]["index"] + 1:
runs[-1].append(row)
else:
runs.append([row])
return runs
def classify(cluster, min_minutes=3):
"""Classify one run of minutes. Pure. Returns (state, detail)."""
cluster = cluster or []
if not cluster:
return ("no-cluster", "nothing to classify")
attempts = sum(_int(r.get("attempts")) for r in cluster)
lost = sum(float(r.get("residual") or 0.0) for r in cluster)
share = (lost / attempts) if attempts else 0.0
detail = ("%s through %s: %d attempt(s) over %d minute(s), about %d of them "
"produced no billed tokens (%.0f%%)"
% (cluster[0]["minute"], cluster[-1]["minute"], attempts,
len(cluster), int(lost), share * 100))
if len(cluster) < min_minutes:
return ("single-minute-dip",
detail + ". Shorter than the %d minute floor, so this is most "
"likely a request that straddled a bucket boundary rather than "
"a capacity condition." % min_minutes)
return ("overload-cluster",
detail + ". A run this long is a platform capacity condition, "
"which is what 529 is, and it is retryable.")
def excess_minutes(rows, tolerance=0.25):
"""Minutes where far more work was billed than the attempts explain. Pure.
The opposite sign, and a different note. Being billed for tokens your own
counter cannot account for is a recording gap in your telemetry, not
requests the platform failed to serve.
"""
out = []
for row in rows or []:
made = _int(row.get("attempts"))
if made <= 0:
continue
if float(row.get("served") or 0.0) > made * (1.0 + tolerance):
out.append(row["minute"])
return out
def tiers_seen(buckets):
"""Every service_tier value present in the window. Pure.
Priority Tier used to be the answer to 529 and capacity commitments are no
longer sold, so "none of your traffic was served as priority" is usually
the true and useful thing to print.
"""
out = set()
for bucket in buckets or []:
for result in bucket.get("results") or []:
tier = str(result.get("service_tier") or "").strip()
if tier:
out.add(tier)
return out
def window_start(minutes):
"""Floor to the minute: starting_at has to sit on a bucket boundary."""
now = dt.datetime.now(dt.timezone.utc).replace(second=0, microsecond=0)
return (now - dt.timedelta(minutes=minutes)).strftime("%Y-%m-%dT%H:%M:%SZ")
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from Anthropic: /v1/organizations/* needs an Admin "
"API key (sk-ant-admin...), not a workspace key"
% r.status_code)
r.raise_for_status()
return r.json()
def read_buckets(session, path, params):
"""Walk the paginated usage report."""
params = dict(params)
while True:
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--attempts", required=True,
help="JSON file of the requests your client attempted, "
"keyed by minute")
ap.add_argument("--minutes", type=int, default=240,
help="minutes of one-minute buckets to read (max 1440)")
ap.add_argument("--floor", type=float, default=0.3,
help="residual share above which a minute joins a cluster "
"(default 0.3)")
ap.add_argument("--min-cluster", type=int, default=3,
help="adjacent minutes needed to call it a cluster "
"(default 3)")
args = ap.parse_args()
admin = os.environ.get("ANTHROPIC_ADMIN_KEY")
if not admin:
log.error("set ANTHROPIC_ADMIN_KEY to an Admin API key "
"(sk-ant-admin...); a workspace key cannot read "
"/v1/organizations/*")
return 2
try:
with open(args.attempts, "r", encoding="utf-8") as fh:
raw = json.load(fh)
except (OSError, ValueError) as exc:
log.error("could not read %s: %s", args.attempts, exc)
return 2
if not isinstance(raw, dict):
log.error("%s should be a JSON object keyed by minute", args.attempts)
return 2
minutes = max(1, min(int(args.minutes), 1440))
session = requests.Session()
session.headers.update({"x-api-key": admin, "anthropic-version": VERSION})
buckets = list(read_buckets(session, "/organizations/usage_report/messages", {
"starting_at": window_start(minutes),
"bucket_width": "1m",
"limit": minutes,
"group_by[]": ["service_tier"],
}))
tokens = tokens_by_minute(buckets)
attempts = attempts_by_minute(raw)
if not attempts:
log.error("no readable minutes in %s. Keys should look like "
"2026-08-30T14:03Z", args.attempts)
return 2
baseline = baseline_tokens_per_attempt(tokens, attempts)
if baseline is None:
log.info("not enough overlapping minutes to establish a baseline; "
"nothing can be said about loss in this window")
return 0
log.info("baseline %d token(s) per attempt, taken as the median across "
"%d minute(s)", int(baseline), len(attempts))
rows = residual_rows(tokens, attempts, baseline)
found = 0
for cluster in clusters(rows, args.floor):
state, detail = classify(cluster, args.min_cluster)
if state in FINDINGS:
found += 1
log.warning("%-18s %s", state, detail)
else:
log.info("%-18s %s", state, detail)
over = excess_minutes(rows)
if over:
log.warning(" %d minute(s) billed far more work than your attempts "
"explain, starting at %s. That is the opposite sign and a "
"different problem: tokens you were billed for and did not "
"record.", len(over), over[0])
tiers = tiers_seen(buckets)
if tiers and "priority" not in tiers:
log.info(" no traffic in this window was served as priority (%s)",
", ".join(sorted(tiers)))
if found:
log.warning(" repair: put 429, every 5xx and 529 in one retryable "
"class with exponential backoff and jitter, or use the "
"SDK's own retry instead of a hand-rolled except. 529 is "
"overloaded_error and is a platform capacity condition, "
"not something your request caused.")
log.warning(" repair: capture the request-id header from every "
"response including errors. It is the only identifier "
"support can act on, and this report cannot recover it "
"after the fact.")
log.info("%d minute(s) compared, %d cluster(s)", len(rows), found)
return 1 if found else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Size the Anthropic requests that were attempted and never served.
*
* Read only. One GET against the Admin API, which needs an Admin API key
* (sk-ant-admin...). The messages usage report carries token sums and no
* request count, so served requests are estimated from the work that was done:
* median tokens per attempt, billed tokens divided by it, subtracted from your
* own attempt counter.
*
* Nothing is retried and nothing is sent.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.anthropic.com/v1';
const VERSION = '2023-06-01';
// cache_creation is a nested object; a flat reader sums zero.
const TOKEN_FIELDS = ['uncached_input_tokens', 'input_tokens',
'cache_read_input_tokens', 'output_tokens'];
const CACHE_CREATION_FIELDS = ['ephemeral_5m_input_tokens', 'ephemeral_1h_input_tokens'];
const FINDINGS = new Set(['overload-cluster']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Normalise a timestamp to a UTC minute key. Pure. Null if unreadable.
* Two sources that disagree about timestamp format produce a comparison with
* no overlap and a clean bill of health, which is the worst failure here.
*/
export function minuteKey(stamp) {
if (typeof stamp === 'boolean') return null;
if (typeof stamp === 'number') {
if (!Number.isFinite(stamp)) return null;
const when = new Date(Math.trunc(stamp) * 1000);
if (Number.isNaN(when.getTime())) return null;
return `${when.toISOString().slice(0, 16)}Z`;
}
const text = String(stamp ?? '').trim().replace(' ', 'T');
if (text.length < 16) return null;
const head = text.slice(0, 16);
if (head[4] !== '-' || head[7] !== '-' || head[10] !== 'T' || head[13] !== ':') return null;
for (const part of [head.slice(0, 4), head.slice(5, 7), head.slice(8, 10),
head.slice(11, 13), head.slice(14, 16)]) {
if (!/^[0-9]+$/.test(part)) return null;
}
return `${head}Z`;
}
/**
* Minutes since the epoch for a minute key. Pure. Null if unreadable.
* Adjacency has to be integer arithmetic; string comparison gets 14:59 and
* 15:00 wrong.
*/
export function minuteIndex(key) {
const normalised = minuteKey(key);
if (normalised === null) return null;
const ms = Date.UTC(Number(normalised.slice(0, 4)), Number(normalised.slice(5, 7)) - 1,
Number(normalised.slice(8, 10)), Number(normalised.slice(11, 13)),
Number(normalised.slice(14, 16)));
return Number.isFinite(ms) ? Math.floor(ms / 60000) : null;
}
/** Total billed tokens per minute. Pure. Every field, cache included. */
export function tokensByMinute(buckets) {
const out = new Map();
for (const bucket of buckets ?? []) {
const key = minuteKey(bucket?.starting_at ?? bucket?.start_time);
if (key === null) continue;
let total = 0;
for (const result of bucket?.results ?? []) {
for (const field of TOKEN_FIELDS) total += readInt(result?.[field]);
const creation = result?.cache_creation ?? {};
for (const field of CACHE_CREATION_FIELDS) total += readInt(creation?.[field]);
}
out.set(key, (out.get(key) ?? 0) + total);
}
return out;
}
/**
* Read your own attempt counter into minute keys. Pure.
* Unparseable minutes are dropped rather than folded into a neighbour: a
* misattributed attempt breaks the contiguity test.
*/
export function attemptsByMinute(raw) {
const out = new Map();
for (const [stamp, value] of Object.entries(raw ?? {})) {
const key = minuteKey(stamp);
if (key === null) continue;
let count;
if (value !== null && typeof value === 'object') count = readInt(value.attempts);
else if (typeof value === 'boolean') count = 0;
else count = readInt(value);
out.set(key, (out.get(key) ?? 0) + count);
}
return out;
}
function median(values) {
const ordered = [...(values ?? [])].sort((a, b) => a - b);
if (ordered.length === 0) return null;
const middle = Math.floor(ordered.length / 2);
if (ordered.length % 2) return ordered[middle];
return (ordered[middle - 1] + ordered[middle]) / 2;
}
/**
* Median tokens per attempt across the covered minutes. Pure.
* The median, never the mean: a mean would be dragged down by the very minutes
* this is meant to find, so it would come back clean during an outage.
*/
export function baselineTokensPerAttempt(tokens, attempts, minMinutes = 5, minAttempts = 1) {
const ratios = [];
for (const [key, value] of attempts ?? new Map()) {
const made = readInt(value);
if (made < minAttempts) continue;
ratios.push(readInt(tokens?.get(key)) / made);
}
if (ratios.length < minMinutes) return null;
const value = median(ratios);
return value && value > 0 ? value : null;
}
/** One row per minute: attempts, tokens, estimated served and residual. Pure. */
export function residualRows(tokens, attempts, baseline) {
const out = [];
if (!baseline || baseline <= 0) return out;
for (const key of [...(attempts?.keys() ?? [])].sort()) {
const made = readInt(attempts.get(key));
if (made <= 0) continue;
const billed = readInt(tokens?.get(key));
const served = billed / baseline;
const residual = Math.max(0, made - served);
out.push({ minute: key, index: minuteIndex(key), attempts: made, tokens: billed,
served, residual, share: residual / made });
}
return out;
}
/**
* Group the shortfall minutes into contiguous runs. Pure.
* Contiguity is the finding: a call spanning a bucket boundary lands its
* attempt in one minute and its tokens in the next, so isolated minutes are
* arithmetic rather than overload.
*/
export function clusters(rows, floor = 0.3, minAttempts = 20) {
const bad = (rows ?? [])
.filter((r) => r?.index !== null && r?.index !== undefined
&& readInt(r?.attempts) >= minAttempts && Number(r?.share ?? 0) >= floor)
.sort((a, b) => a.index - b.index);
const runs = [];
for (const row of bad) {
const last = runs[runs.length - 1];
if (last && row.index === last[last.length - 1].index + 1) last.push(row);
else runs.push([row]);
}
return runs;
}
/** Classify one run of minutes. Pure. Returns [state, detail]. */
export function classify(cluster, minMinutes = 3) {
const run = cluster ?? [];
if (run.length === 0) return ['no-cluster', 'nothing to classify'];
const attempts = run.reduce((sum, r) => sum + readInt(r?.attempts), 0);
const lost = run.reduce((sum, r) => sum + Number(r?.residual ?? 0), 0);
const share = attempts ? lost / attempts : 0;
const detail = `${run[0].minute} through ${run[run.length - 1].minute}: ` +
`${attempts} attempt(s) over ${run.length} minute(s), about ` +
`${Math.trunc(lost)} of them produced no billed tokens ` +
`(${(share * 100).toFixed(0)}%)`;
if (run.length < minMinutes) {
return ['single-minute-dip',
`${detail}. Shorter than the ${minMinutes} minute floor, so this is most ` +
'likely a request that straddled a bucket boundary rather than a ' +
'capacity condition.'];
}
return ['overload-cluster',
`${detail}. A run this long is a platform capacity condition, which is ` +
'what 529 is, and it is retryable.'];
}
/**
* Minutes where far more work was billed than the attempts explain. Pure.
* The opposite sign and a different note: a recording gap in your telemetry.
*/
export function excessMinutes(rows, tolerance = 0.25) {
const out = [];
for (const row of rows ?? []) {
const made = readInt(row?.attempts);
if (made <= 0) continue;
if (Number(row?.served ?? 0) > made * (1 + tolerance)) out.push(row.minute);
}
return out;
}
/** Every service_tier value present in the window. Pure. */
export function tiersSeen(buckets) {
const out = new Set();
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const tier = String(result?.service_tier ?? '').trim();
if (tier) out.add(tier);
}
}
return out;
}
function windowStart(minutes) {
const now = new Date();
now.setUTCSeconds(0, 0);
return `${new Date(now.getTime() - minutes * 60000).toISOString().slice(0, 19)}Z`;
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, {
headers: { 'x-api-key': key, 'anthropic-version': VERSION },
});
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Anthropic: /v1/organizations/* needs an ` +
'Admin API key (sk-ant-admin...), not a workspace key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* readBuckets(key, path, params) {
let query = { ...params };
for (;;) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...query, page: page.next_page };
}
}
async function main() {
const admin = process.env.ANTHROPIC_ADMIN_KEY;
if (!admin) {
console.error('set ANTHROPIC_ADMIN_KEY to an Admin API key (sk-ant-admin...); ' +
'a workspace key cannot read /v1/organizations/*');
process.exitCode = 2;
return;
}
const file = process.env.ATTEMPTS;
if (!file) {
console.error('set ATTEMPTS to a JSON file of the requests your client ' +
'attempted, keyed by minute');
process.exitCode = 2;
return;
}
const minutes = Math.max(1, Math.min(Number(process.env.MINUTES ?? 240), 1440));
const floor = Number(process.env.FLOOR ?? 0.3);
const minCluster = Number(process.env.MIN_CLUSTER ?? 3);
let raw;
try {
raw = JSON.parse(await readFile(file, 'utf8'));
} catch (err) {
console.error(`could not read ${file}: ${err.message}`);
process.exitCode = 2;
return;
}
const buckets = [];
for await (const bucket of readBuckets(admin, '/organizations/usage_report/messages', {
starting_at: windowStart(minutes),
bucket_width: '1m',
limit: minutes,
'group_by[]': ['service_tier'],
})) buckets.push(bucket);
const tokens = tokensByMinute(buckets);
const attempts = attemptsByMinute(raw);
if (attempts.size === 0) {
console.error(`no readable minutes in ${file}. Keys should look like ` +
'2026-08-30T14:03Z');
process.exitCode = 2;
return;
}
const baseline = baselineTokensPerAttempt(tokens, attempts);
if (baseline === null) {
console.log('not enough overlapping minutes to establish a baseline; nothing ' +
'can be said about loss in this window');
return;
}
console.log(`baseline ${Math.trunc(baseline)} token(s) per attempt, taken as ` +
`the median across ${attempts.size} minute(s)`);
const rows = residualRows(tokens, attempts, baseline);
let found = 0;
for (const cluster of clusters(rows, floor)) {
const [state, detail] = classify(cluster, minCluster);
if (FINDINGS.has(state)) { found += 1; console.warn(`${state.padEnd(18)} ${detail}`); }
else console.log(`${state.padEnd(18)} ${detail}`);
}
const over = excessMinutes(rows);
if (over.length > 0) {
console.warn(` ${over.length} minute(s) billed far more work than your ` +
`attempts explain, starting at ${over[0]}. That is the opposite ` +
'sign and a different problem: tokens you were billed for and ' +
'did not record.');
}
const tiers = tiersSeen(buckets);
if (tiers.size > 0 && !tiers.has('priority')) {
console.log(` no traffic in this window was served as priority ` +
`(${[...tiers].sort().join(', ')})`);
}
if (found) {
console.warn(' repair: put 429, every 5xx and 529 in one retryable class ' +
'with exponential backoff and jitter, or use the SDK own retry ' +
'instead of a hand-rolled catch. 529 is overloaded_error and is ' +
'a platform capacity condition, not something your request caused.');
console.warn(' repair: capture the request-id header from every response ' +
'including errors. It is the only identifier support can act on, ' +
'and this report cannot recover it after the fact.');
}
console.log(`${rows.length} minute(s) compared, ${found} cluster(s)`);
process.exitCode = found ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is a ten-minute window in which three adjacent minutes billed a fifth of the tokens their attempt counts should have produced. It asserts the cluster covers exactly those three minutes and no others, and it asserts the baseline is still five thousand tokens per attempt — which is the point of using a median, since a mean over the same data comes out low enough to make the outage look like a normal afternoon. The rest defend the two things that quietly break this check: an isolated bad minute has to come back as bucket arithmetic rather than as overload, and a minute key has to normalise your timestamps and Anthropic's to the same string, because two formats that never match produce a clean report during a real incident.
from anthropic_overload_residual import (attempts_by_minute,
baseline_tokens_per_attempt,
classify, clusters, excess_minutes,
minute_index, minute_key,
residual_rows, tiers_seen,
tokens_by_minute)
def minute(n):
return "2026-08-30T14:%02dZ" % n
# Ten minutes at 600 attempts each. Seven of them do the full 3,000,000 tokens
# of work that 600 calls at 5000 tokens implies; minutes 4, 5 and 6 do a fifth
# of it, because the platform was over capacity and served 120 of the 600.
ATTEMPTS = {minute(n): 600 for n in range(10)}
TOKENS = {minute(n): (600_000 if n in (4, 5, 6) else 3_000_000) for n in range(10)}
def test_three_adjacent_bad_minutes_are_one_overload_cluster():
baseline = baseline_tokens_per_attempt(TOKENS, ATTEMPTS)
# The median survives the outage. A mean over the same data is 3800, which
# would hide most of the loss it was computed to find.
assert baseline == 5000.0
rows = residual_rows(TOKENS, ATTEMPTS, baseline)
assert len(rows) == 10
bad = [r for r in rows if r["share"] > 0.5]
assert [r["minute"] for r in bad] == [minute(4), minute(5), minute(6)]
assert round(bad[0]["residual"]) == 480
runs = clusters(rows)
assert len(runs) == 1
state, detail = classify(runs[0])
assert state == "overload-cluster"
assert "2026-08-30T14:04Z through 2026-08-30T14:06Z" in detail
assert "1800 attempt(s) over 3 minute(s)" in detail
assert "about 1440 of them produced no billed tokens (80%)" in detail
def test_one_bad_minute_on_its_own_is_bucket_arithmetic():
attempts = dict(ATTEMPTS)
tokens = {k: 3_000_000 for k in attempts}
tokens[minute(4)] = 600_000
rows = residual_rows(tokens, attempts,
baseline_tokens_per_attempt(tokens, attempts))
runs = clusters(rows)
assert len(runs) == 1 and len(runs[0]) == 1
state, detail = classify(runs[0])
assert state == "single-minute-dip"
assert "straddled a bucket boundary" in detail
def test_minutes_that_are_not_adjacent_do_not_become_one_cluster():
tokens = {k: 3_000_000 for k in ATTEMPTS}
for n in (1, 5, 9):
tokens[minute(n)] = 100_000
rows = residual_rows(tokens, ATTEMPTS,
baseline_tokens_per_attempt(tokens, ATTEMPTS))
assert [len(run) for run in clusters(rows)] == [1, 1, 1]
def test_the_two_clocks_are_normalised_to_the_same_minute():
# Anthropic returns full RFC 3339; your counter probably does not. Two
# formats that never match produce a clean report during a real incident.
for stamp in ("2026-08-30T14:03:27Z", "2026-08-30T14:03Z",
"2026-08-30 14:03:00+00:00", "2026-08-30T14:03:59.512Z"):
assert minute_key(stamp) == "2026-08-30T14:03Z"
assert minute_key(1788098580) == "2026-08-30T14:03Z"
assert minute_key("last tuesday") is None
assert minute_key("") is None
assert minute_key(None) is None
assert minute_key(True) is None
# Adjacency is arithmetic, not string order: 14:59 and 15:00 are neighbours.
assert minute_index("2026-08-30T15:00Z") - minute_index("2026-08-30T14:59Z") == 1
def test_the_nested_cache_creation_object_is_counted_as_work():
buckets = [{"starting_at": "2026-08-30T14:03:00Z",
"results": [{"uncached_input_tokens": 10,
"cache_read_input_tokens": 20,
"output_tokens": 5,
"service_tier": "standard",
"cache_creation": {"ephemeral_5m_input_tokens": 100,
"ephemeral_1h_input_tokens": 65}}]}]
assert tokens_by_minute(buckets) == {"2026-08-30T14:03Z": 200}
assert tiers_seen(buckets) == {"standard"}
assert tokens_by_minute([]) == {}
def test_an_attempt_file_is_read_leniently_and_bad_keys_are_dropped():
assert attempts_by_minute({"2026-08-30T14:03:00Z": 900}) == {"2026-08-30T14:03Z": 900}
assert attempts_by_minute({"2026-08-30T14:03Z": {"attempts": 900}}) \
== {"2026-08-30T14:03Z": 900}
assert attempts_by_minute({"whenever": 900}) == {}
assert attempts_by_minute(None) == {}
def test_more_work_than_the_attempts_explain_is_the_other_note():
attempts = {minute(n): 100 for n in range(10)}
tokens = {minute(n): 500_000 for n in range(10)}
tokens[minute(3)] = 2_000_000
baseline = baseline_tokens_per_attempt(tokens, attempts)
rows = residual_rows(tokens, attempts, baseline)
assert excess_minutes(rows) == [minute(3)]
assert clusters(rows) == []
def test_too_little_overlap_produces_no_baseline_rather_than_a_guess():
assert baseline_tokens_per_attempt({}, {}) is None
assert baseline_tokens_per_attempt({minute(0): 5000}, {minute(0): 1}) is None
assert baseline_tokens_per_attempt({}, ATTEMPTS) is None
assert residual_rows(TOKENS, ATTEMPTS, None) == []
assert classify([])[0] == "no-cluster"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { attemptsByMinute, baselineTokensPerAttempt, classify, clusters,
excessMinutes, minuteIndex, minuteKey, residualRows, tiersSeen,
tokensByMinute } from './anthropic-overload-residual.mjs';
const minute = (n) => `2026-08-30T14:${String(n).padStart(2, '0')}Z`;
// Ten minutes at 600 attempts each; minutes 4, 5 and 6 did a fifth of the work.
const ATTEMPTS = new Map(Array.from({ length: 10 }, (_, n) => [minute(n), 600]));
const TOKENS = new Map(Array.from({ length: 10 },
(_, n) => [minute(n), [4, 5, 6].includes(n) ? 600000 : 3000000]));
test('three adjacent bad minutes are one overload cluster', () => {
const baseline = baselineTokensPerAttempt(TOKENS, ATTEMPTS);
// The median survives the outage; a mean over the same data is 3800.
assert.equal(baseline, 5000);
const rows = residualRows(TOKENS, ATTEMPTS, baseline);
assert.equal(rows.length, 10);
const bad = rows.filter((r) => r.share > 0.5);
assert.deepEqual(bad.map((r) => r.minute), [minute(4), minute(5), minute(6)]);
assert.equal(Math.round(bad[0].residual), 480);
const runs = clusters(rows);
assert.equal(runs.length, 1);
const [state, detail] = classify(runs[0]);
assert.equal(state, 'overload-cluster');
assert.match(detail, /2026-08-30T14:04Z through 2026-08-30T14:06Z/);
assert.match(detail, /1800 attempt\(s\) over 3 minute\(s\)/);
assert.match(detail, /about 1440 of them produced no billed tokens \(80%\)/);
});
test('one bad minute on its own is bucket arithmetic', () => {
const tokens = new Map([...ATTEMPTS.keys()].map((k) => [k, 3000000]));
tokens.set(minute(4), 600000);
const rows = residualRows(tokens, ATTEMPTS,
baselineTokensPerAttempt(tokens, ATTEMPTS));
const runs = clusters(rows);
assert.equal(runs.length, 1);
assert.equal(runs[0].length, 1);
const [state, detail] = classify(runs[0]);
assert.equal(state, 'single-minute-dip');
assert.match(detail, /straddled a bucket boundary/);
});
test('minutes that are not adjacent do not become one cluster', () => {
const tokens = new Map([...ATTEMPTS.keys()].map((k) => [k, 3000000]));
for (const n of [1, 5, 9]) tokens.set(minute(n), 100000);
const rows = residualRows(tokens, ATTEMPTS,
baselineTokensPerAttempt(tokens, ATTEMPTS));
assert.deepEqual(clusters(rows).map((run) => run.length), [1, 1, 1]);
});
test('the two clocks are normalised to the same minute', () => {
for (const stamp of ['2026-08-30T14:03:27Z', '2026-08-30T14:03Z',
'2026-08-30 14:03:00+00:00', '2026-08-30T14:03:59.512Z']) {
assert.equal(minuteKey(stamp), '2026-08-30T14:03Z');
}
assert.equal(minuteKey(1788098580), '2026-08-30T14:03Z');
assert.equal(minuteKey('last tuesday'), null);
assert.equal(minuteKey(''), null);
assert.equal(minuteKey(null), null);
assert.equal(minuteKey(true), null);
assert.equal(minuteIndex('2026-08-30T15:00Z') - minuteIndex('2026-08-30T14:59Z'), 1);
});
test('the nested cache creation object is counted as work', () => {
const buckets = [{ starting_at: '2026-08-30T14:03:00Z',
results: [{ uncached_input_tokens: 10, cache_read_input_tokens: 20,
output_tokens: 5, service_tier: 'standard',
cache_creation: { ephemeral_5m_input_tokens: 100,
ephemeral_1h_input_tokens: 65 } }] }];
assert.deepEqual([...tokensByMinute(buckets)], [['2026-08-30T14:03Z', 200]]);
assert.deepEqual([...tiersSeen(buckets)], ['standard']);
assert.equal(tokensByMinute([]).size, 0);
});
test('an attempt file is read leniently and bad keys are dropped', () => {
assert.deepEqual([...attemptsByMinute({ '2026-08-30T14:03:00Z': 900 })],
[['2026-08-30T14:03Z', 900]]);
assert.deepEqual([...attemptsByMinute({ '2026-08-30T14:03Z': { attempts: 900 } })],
[['2026-08-30T14:03Z', 900]]);
assert.equal(attemptsByMinute({ whenever: 900 }).size, 0);
assert.equal(attemptsByMinute(null).size, 0);
});
test('more work than the attempts explain is the other note', () => {
const attempts = new Map(Array.from({ length: 10 }, (_, n) => [minute(n), 100]));
const tokens = new Map(Array.from({ length: 10 }, (_, n) => [minute(n), 500000]));
tokens.set(minute(3), 2000000);
const rows = residualRows(tokens, attempts,
baselineTokensPerAttempt(tokens, attempts));
assert.deepEqual(excessMinutes(rows), [minute(3)]);
assert.deepEqual(clusters(rows), []);
});
test('too little overlap produces no baseline rather than a guess', () => {
assert.equal(baselineTokensPerAttempt(new Map(), new Map()), null);
assert.equal(baselineTokensPerAttempt(new Map([[minute(0), 5000]]),
new Map([[minute(0), 1]])), null);
assert.equal(baselineTokensPerAttempt(new Map(), ATTEMPTS), null);
assert.deepEqual(residualRows(TOKENS, ATTEMPTS, null), []);
assert.equal(classify([])[0], 'no-cluster');
});
FAQ
Why do I have to supply the attempt count myself?
Because Anthropic does not publish one. The messages usage report returns token sums per bucket and has no request-count field at all, which is a documented difference from OpenAI's usage endpoints. Without a count from your side there is no subtraction to do, and the best the API can offer on its own is that some minutes look quieter than others.
How accurate is the estimate of served requests?
It is an estimate and the script never pretends otherwise. Dividing billed tokens by a median tokens-per-attempt assumes the requests in a bad minute would have been about the same size as the ones around it, which holds for a homogeneous workload and does not hold if your traffic mixes tiny classifications with enormous summaries. Read the finding as a shape and a rough magnitude, not as an accounting record.
Is this the same check as reconciling streamed usage?
No, and the sign is the difference. Reconciling streamed usage finds the provider's token count larger than your own and concludes your pipeline is not recording what you were billed for. This one finds your attempt count larger than the work the provider did and concludes the platform never served the requests. The script reports an excess as its own state and points you at the other note rather than calling it overload.
How is this different from spotting a retry storm?
A retry storm is visible in the provider's own numbers, because the attempts were served and counted. A 529 cluster never reaches the provider's numbers at all, which is why this check needs a number from you and the storm check does not. If your client retries 529 and eventually succeeds, both are happening and the residual measures only the attempts that were never served.
Should the script retry the lost requests?
No, and it will not. It holds a key that can spend money on inference, the condition it just detected is the platform being over capacity, and re-sending traffic into that is how a transient becomes an incident. It prints the affected minutes, the estimated loss and the retry class to adopt, and you decide what to replay.
Related field notes
- Tokens the provider billed that your telemetry never recorded
- Request count climbing while token volume stays flat
- Naming which of the three limiters actually emptied
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Errors — Claude API
- Get messages usage report — Claude Admin API
- Service tiers — Claude Docs
- Usage and Cost API — Claude Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.