Diagnostic LLM APIs
ITPM runs out because uncached input is never cached
The 429s arrive in the afternoon and the team does what teams do: fewer workers, longer sleeps, a queue in front. None of it moves. The request counter was never the thing that ran out. What ran out was input tokens per minute, and the reason is that the same forty thousand tokens of system prompt and tool schemas are sent uncached on every single call, and every one of those tokens is charged against the limiter.
Read the per-minute usage buckets with an Admin API key: GET /v1/organizations/usage_report/messages?starting_at={T-4h}&bucket_width=1m&limit=240&group_by[]=model. Per minute and per model, the number charged against ITPM is uncached_input_tokens + cache_creation.ephemeral_5m_input_tokens + cache_creation.ephemeral_1h_input_tokens.
cache_read_input_tokens is not in that sum. Cache reads do not count toward the input limiter on any current model — the one exception is Claude Haiku 3.5, where they do. Compare the peak minute against input_tokens_per_minute from GET /v1/organizations/rate_limits.
A peak at the ceiling with a cache-read share near zero is the finding, and the conclusion is not the one the caching notes reach. Caching here buys throughput: at an eighty percent read share the same ITPM ceiling carries five times the total input.
The problem in plain words
Rate limits are debugged with the knob everybody has, which is concurrency. It is the right knob when requests per minute is what emptied, and it does nothing whatsoever when input tokens per minute is. Six workers sending forty-thousand-token prompts and three workers sending the same prompts twice as often put identical pressure on ITPM; the queue in front of them just makes the pressure arrive more politely.
Underneath, the shape is almost always the same. A long, stable prefix — tool definitions, a system prompt, a few-shot block, a document that does not change between turns — is re-sent in full on every call. It is the part of the request nobody thinks about, because it was written once and works. And it is the part that dominates the token count, so the limiter is being spent almost entirely on text the model has already been shown.
Why it happens
Only uncached input is charged against ITPM. input_tokens and cache_creation_input_tokens count; cache_read_input_tokens does not. That single exclusion is why this note is about a limiter rather than about a bill. Every token you move behind a cache breakpoint is a token that stops competing for the ceiling, and the effect on throughput is a multiplier rather than a percentage.
Claude Haiku 3.5 is the exception and it inverts the advice. On that model cache reads do count toward ITPM. Caching still lowers the bill there and buys no headroom at all, so a script that applies one rule to every model tells half its readers to make a change that will not help them. The check has to branch on the model id, and this one does.
The peak minute is the finding; the hourly mean is not. Buckets a minute wide exist because the limiter is enforced by the minute. A workload that saturates ITPM for ninety seconds every hour looks entirely comfortable at hourly resolution and is 429ing its users twice an hour. Fold to the maximum, never to the average.
This is not the caching cost finding wearing different clothes. Caching never switched on and writes with no reads both end in a dollar figure and are true whether or not you are anywhere near a limit. This one is true whether or not caching would save you money: a workload can be comfortably profitable, already priced in, and still be unable to grow because its input limiter is full.
The usage report has no request count. GET /v1/organizations/usage_report/messages returns token sums per bucket and nothing else, so nothing here is expressed per request. That is a real limit on what can be claimed: the script can say the input limiter is full and cannot tell you how many calls filled it.
Note the input_tokens trap in the response object. On a live message, usage.input_tokens is only the tail after the last cache breakpoint. Total input is cache_read + cache_creation + input_tokens. Reading the field by its name and calling it the prompt size is how a caching workload gets reported as a tiny one.
The fix, as a flow
Only uncached input is charged against the input limiter: uncached tokens plus both cache creation fields, with cache reads excluded on every model but one. That exclusion is why this is a throughput note rather than a discount note, and why the multiplier is one over one minus the read share.
How to fix it
Pull minute buckets, not hour buckets
bucket_width=1m with starting_at floored to a minute boundary. Up to 1,440 buckets come back, and the endpoint paginates on next_page. Four hours across the busy part of the day is usually enough to find the peak; a full day is fine and is 1,440 buckets exactly.
Charge the right tokens against the limiter
uncached_input_tokens plus both members of the nested cache_creation object. cache_read_input_tokens is excluded — except on Claude Haiku 3.5, where it is included. A parser that looks for a flat cache_creation_input_tokens finds nothing and under-charges a caching workload into looking idle.
Take the maximum, and remember which minute it was
Per model, keep the largest charged minute and the cache-read count from that same minute. Both numbers have to come from one minute: a peak from Tuesday compared against a read share from Sunday produces a confident sentence about nothing.
Compare against the published ITPM for the model group
GET /v1/organizations/rate_limits returns model_group entries whose limits[] carry {type: "input_tokens_per_minute", value}. Match the model id to its group by longest prefix. A group with no published ITPM is reported as unpublished, not as unlimited — the limiter still exists, you were simply not told the number.
Say what caching would buy here, in throughput
The multiplier is 1 / (1 - read_share). Zero percent read share means the ceiling carries exactly your uncached input; eighty percent means it carries five times as much. Print that, print where the breakpoint goes — the render order is tools, then system, then messages — and stop. Moving a breakpoint changes what the model sees, and that is a deploy with an owner.
How to check it worked
Add the breakpoint, wait an hour, and read the same window again. The charged peak should fall while total input holds steady; that difference is the headroom you just bought.
python3 anthropic_itpm_headroom.py --minutes 240
# itpm-saturated-uncached claude-sonnet-5 peak minute charged 4,880,000 token(s)
# against an ITPM of 5,000,000 (98%); cache reads were 2% of that minute's input
# at this read share the ceiling carries 1.0x your total input; at 80% it carries 5.0x
# 3 model(s) checked, 1 finding(s)The full code
Two GETs against the Admin API and no writes. It wants ANTHROPIC_ADMIN_KEY, which can and should be provisioned read-only, because /v1/organizations/* rejects a workspace key outright. Six pure functions: the model test that decides whether cache reads are charged, the accumulator that reads the nested cache_creation object, the fold that keeps peaks rather than means, the group match, the multiplier, and a verdict that separates three different reasons an input limiter can be full.
"""Report an Anthropic input limiter that is full of uncached input.
Read only. Two GET requests and nothing else against the Admin API, which needs
an Admin API key (sk-ant-admin...); a workspace key is rejected by every
/v1/organizations/* path, and an Admin key can be provisioned read-only.
The repair is printed, never performed. Adding a cache_control breakpoint
changes what the model is shown on every request, which is a deploy.
The messages usage report carries token sums and no request count, so nothing
here is expressed per request. This script can say the input limiter is full.
It cannot say how many calls filled it.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_itpm_headroom")
API = "https://api.anthropic.com/v1"
VERSION = "2023-06-01"
# The one family where cache reads are charged against the input limiter. On
# every other current model a cache read is free of ITPM, which is the entire
# mechanism this script reports on, so the exception has to be explicit rather
# than a footnote in the prose.
CACHE_READS_CHARGED = ("claude-3-5-haiku",)
FINDINGS = ("itpm-saturated-uncached", "itpm-saturated-already-cached",
"itpm-saturated-cache-counts")
def cache_reads_count(model):
"""Do cache reads count toward this model's ITPM? Pure.
True only for Claude Haiku 3.5. Getting this backwards tells a reader to
add a breakpoint that will not buy them a single token of headroom.
"""
name = str(model or "").strip().lower()
return any(name.startswith(prefix) for prefix in CACHE_READS_CHARGED)
def chargeable_input(result, model):
"""Tokens in one usage result that count against ITPM. Pure.
cache_creation is a nested object holding ephemeral_5m_input_tokens and
ephemeral_1h_input_tokens. A parser looking for a flat
cache_creation_input_tokens sums zero and reports a heavily cached workload
as one that writes nothing.
"""
if not isinstance(result, dict):
return 0
total = 0
for field in ("uncached_input_tokens",):
try:
total += int(result.get(field) or 0)
except (TypeError, ValueError):
pass
creation = result.get("cache_creation") or {}
for field in ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"):
try:
total += int(creation.get(field) or 0)
except (TypeError, ValueError):
pass
if cache_reads_count(model):
try:
total += int(result.get("cache_read_input_tokens") or 0)
except (TypeError, ValueError):
pass
return total
def peaks(buckets):
"""Fold one-minute buckets into per-model peaks. Pure.
The peak is the finding and the mean is not. ITPM is enforced by the
minute, so a workload that saturates for ninety seconds an hour has a
comfortable hourly average and a queue of 429s inside it.
"""
per_minute = {}
for bucket in buckets or []:
stamp = str(bucket.get("starting_at") or bucket.get("start_time") or "")
for result in bucket.get("results") or []:
model = str(result.get("model") or "").strip() or "all models"
row = per_minute.setdefault((model, stamp), {"charged": 0, "read": 0})
row["charged"] += chargeable_input(result, model)
try:
row["read"] += int(result.get("cache_read_input_tokens") or 0)
except (TypeError, ValueError):
pass
out = {}
for (model, stamp), row in per_minute.items():
stats = out.setdefault(model, {"peak": 0, "peak_at": None, "peak_read": 0,
"minutes": 0, "charged": 0, "read": 0})
stats["minutes"] += 1
stats["charged"] += row["charged"]
stats["read"] += row["read"]
if row["charged"] > stats["peak"]:
stats["peak"] = row["charged"]
stats["peak_at"] = stamp
stats["peak_read"] = row["read"]
return out
def cache_read_share(stats, model):
"""Share of the peak minute's input that arrived as a cache read. Pure.
On a model where reads are charged the peak already contains them, so the
denominator differs. Using one denominator for both models reports the
Haiku 3.5 case at half its real share.
"""
if not isinstance(stats, dict):
return None
read = int(stats.get("peak_read") or 0)
charged = int(stats.get("peak") or 0)
total = charged if cache_reads_count(model) else charged + read
if total <= 0:
return None
return min(1.0, read / float(total))
def headroom_multiplier(share):
"""How much total input one ITPM ceiling carries at a given read share. Pure.
1 / (1 - share). At zero the ceiling carries exactly your uncached input;
at 0.8 it carries five times your total input. This is the throughput
argument for caching and it is a different argument from the discount.
"""
if share is None:
return None
bounded = max(0.0, min(0.99, float(share)))
return 1.0 / (1.0 - bounded)
def itpm_by_group(payload):
"""{model_group: input_tokens_per_minute} from the rate-limits response. Pure.
A group whose limits[] omits the type is recorded as None. Absent means it
inherits, never that it is unlimited, and reading a missing number as no
ceiling is how a team decides it has room nobody granted it.
"""
out = {}
for entry in (payload or {}).get("data") or []:
group = str(entry.get("model_group") or "").strip()
if not group:
continue
out.setdefault(group, None)
for limit in entry.get("limits") or []:
if str(limit.get("type") or "").strip() != "input_tokens_per_minute":
continue
try:
out[group] = int(limit.get("value"))
except (TypeError, ValueError):
out[group] = None
return out
def limit_for(groups, model):
"""The ITPM for the group a model id belongs to, or None. Pure.
Longest prefix wins, so a dated id resolves to the most specific group that
claims it rather than to whichever one the dict happened to yield first.
"""
name = str(model or "").strip().lower()
if not name:
return None
best_key, best_len = None, -1
for group in (groups or {}):
candidate = str(group).strip().lower()
if not candidate:
continue
if name == candidate or name.startswith(candidate):
if len(candidate) > best_len:
best_key, best_len = group, len(candidate)
if best_key is None:
return None
return (groups or {}).get(best_key)
def verdict(model, stats, limit, floor=0.9, watch=0.6, min_minutes=10,
cached_enough=0.15):
"""Classify one model's input limiter. Pure. Returns (state, detail).
Three ways an ITPM ceiling can be full, and they do not share a repair:
the prefix is uncached and caching buys headroom; the prefix is already
cached and only a limit increase is left; or the model charges cache reads
so caching was never going to buy headroom in the first place.
"""
minutes = int((stats or {}).get("minutes") or 0)
if minutes < min_minutes:
return ("too-few-buckets",
"%d minute(s) of traffic in the window, under the floor of %d. "
"A peak taken over this little is noise." % (minutes, min_minutes))
if limit is None or limit <= 0:
return ("no-limit-published",
"no input_tokens_per_minute is published for this model's group, "
"so there is no ceiling to compare the peak against. The limiter "
"still exists; the number was simply not returned.")
peak = int(stats.get("peak") or 0)
used = peak / float(limit)
share = cache_read_share(stats, model)
shape = ("peak minute charged %d token(s) against an ITPM of %d (%.0f%%); "
"cache reads were %.0f%% of that minute's input"
% (peak, limit, used * 100, (share or 0.0) * 100))
if used < watch:
return ("itpm-headroom", shape + ".")
if used < floor:
return ("itpm-approaching",
shape + ". Thin enough that an ordinary spike lands on the "
"input limiter rather than on the request limiter.")
if cache_reads_count(model):
return ("itpm-saturated-cache-counts",
shape + ". This model charges cache reads against ITPM, so "
"caching lowers the bill here and buys no headroom at all. The "
"levers are a shorter prefix or a higher limit.")
if share is not None and share >= cached_enough:
return ("itpm-saturated-already-cached",
shape + ". The prefix is already being read back, so a "
"breakpoint has little left to give. What remains is a limit "
"increase, or splitting the workload across model groups.")
return ("itpm-saturated-uncached",
shape + ". Cache reads are not charged against ITPM on this model, "
"so covering the stable prefix buys throughput and not only a "
"discount.")
def window_start(minutes):
"""Floor to the minute: starting_at must sit on a bucket boundary."""
now = dt.datetime.now(dt.timezone.utc).replace(second=0, microsecond=0)
return (now - dt.timedelta(minutes=minutes)).strftime("%Y-%m-%dT%H:%M:%SZ")
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from Anthropic: /v1/organizations/* needs an Admin "
"API key (sk-ant-admin...), not a workspace key"
% r.status_code)
r.raise_for_status()
return r.json()
def read_buckets(session, path, params):
"""Walk the paginated usage report."""
params = dict(params)
while True:
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--minutes", type=int, default=240,
help="minutes of one-minute buckets to read (max 1440)")
ap.add_argument("--target-share", type=float, default=0.8,
help="cache-read share to quote the multiplier at (default 0.8)")
ap.add_argument("--show-all", action="store_true",
help="also print models with headroom left")
args = ap.parse_args()
admin = os.environ.get("ANTHROPIC_ADMIN_KEY")
if not admin:
log.error("set ANTHROPIC_ADMIN_KEY to an Admin API key (sk-ant-admin...); "
"a workspace key cannot read /v1/organizations/*")
return 2
minutes = max(1, min(int(args.minutes), 1440))
session = requests.Session()
session.headers.update({"x-api-key": admin, "anthropic-version": VERSION})
params = {"starting_at": window_start(minutes), "bucket_width": "1m",
"limit": minutes, "group_by[]": ["model"]}
stats = peaks(read_buckets(session, "/organizations/usage_report/messages", params))
if not stats:
log.info("no message usage in the last %d minute(s)", minutes)
return 0
groups = itpm_by_group(get(session, "/organizations/rate_limits"))
checked = 0
bad = 0
for model in sorted(stats, key=lambda m: -stats[m]["peak"]):
row = stats[model]
limit = limit_for(groups, model)
state, detail = verdict(model, row, limit)
checked += 1
line = "%-30s %-28s %s" % (state, model, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
if state == "itpm-saturated-uncached":
share = cache_read_share(row, model) or 0.0
now_x = headroom_multiplier(share)
then_x = headroom_multiplier(args.target_share)
log.warning(" at this read share the ceiling carries %.1fx your "
"total input; at %.0f%% it would carry %.1fx",
now_x, args.target_share * 100, then_x)
log.warning(" repair: put a cache_control breakpoint at the end "
"of the stable prefix. The render order is tools, "
"then system, then messages, so the breakpoint goes "
"after the last thing that never changes.")
else:
log.warning(" repair: request an input_tokens_per_minute "
"increase for this model group, or move latency "
"tolerant work onto the Message Batches API, which "
"is metered by its own limiter group.")
elif state in ("itpm-approaching", "no-limit-published"):
log.warning(line)
elif args.show_all:
log.info(line)
log.info("%d model(s) checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report an Anthropic input limiter that is full of uncached input.
*
* Read only. Two GET requests and nothing else against the Admin API, which
* needs an Admin API key (sk-ant-admin...); a workspace key is rejected by
* every /v1/organizations/* path. The repair is printed, never performed.
*
* The messages usage report carries token sums and no request count, so
* nothing here is expressed per request.
*/
const API = 'https://api.anthropic.com/v1';
const VERSION = '2023-06-01';
// The one family where cache reads are charged against the input limiter.
const CACHE_READS_CHARGED = ['claude-3-5-haiku'];
const FINDINGS = new Set(['itpm-saturated-uncached', 'itpm-saturated-already-cached',
'itpm-saturated-cache-counts']);
/**
* Do cache reads count toward this model's ITPM? Pure.
* True only for Claude Haiku 3.5; getting it backwards tells a reader to add a
* breakpoint that buys them no headroom.
*/
export function cacheReadsCount(model) {
const name = String(model ?? '').trim().toLowerCase();
return CACHE_READS_CHARGED.some((prefix) => name.startsWith(prefix));
}
/**
* Tokens in one usage result that count against ITPM. Pure.
* cache_creation is nested; a flat read sums zero and reports a heavily cached
* workload as one that writes nothing.
*/
export function chargeableInput(result, model) {
if (!result || typeof result !== 'object') return 0;
const num = (v) => (Number.isFinite(Number(v)) ? Math.trunc(Number(v)) : 0);
const creation = result.cache_creation ?? {};
let total = num(result.uncached_input_tokens)
+ num(creation.ephemeral_5m_input_tokens)
+ num(creation.ephemeral_1h_input_tokens);
if (cacheReadsCount(model)) total += num(result.cache_read_input_tokens);
return total;
}
/**
* Fold one-minute buckets into per-model peaks. Pure.
* The peak is the finding and the mean is not: ITPM is enforced by the minute.
*/
export function peaks(buckets) {
const perMinute = new Map();
for (const bucket of buckets ?? []) {
const stamp = String(bucket.starting_at ?? bucket.start_time ?? '');
for (const result of bucket.results ?? []) {
const model = String(result.model ?? '').trim() || 'all models';
const key = `${model}\u0000${stamp}`;
const row = perMinute.get(key) ?? { model, charged: 0, read: 0 };
row.charged += chargeableInput(result, model);
const read = Number(result.cache_read_input_tokens ?? 0);
row.read += Number.isFinite(read) ? Math.trunc(read) : 0;
perMinute.set(key, row);
}
}
const out = {};
for (const [key, row] of perMinute) {
const stamp = key.slice(key.indexOf('\u0000') + 1);
const stats = out[row.model] ?? { peak: 0, peak_at: null, peak_read: 0,
minutes: 0, charged: 0, read: 0 };
stats.minutes += 1;
stats.charged += row.charged;
stats.read += row.read;
if (row.charged > stats.peak) {
stats.peak = row.charged;
stats.peak_at = stamp;
stats.peak_read = row.read;
}
out[row.model] = stats;
}
return out;
}
/**
* Share of the peak minute's input that arrived as a cache read. Pure.
* The denominator differs on a model that charges reads, because the peak
* already contains them.
*/
export function cacheReadShare(stats, model) {
if (!stats || typeof stats !== 'object') return null;
const read = Number(stats.peak_read ?? 0);
const charged = Number(stats.peak ?? 0);
const total = cacheReadsCount(model) ? charged : charged + read;
if (!(total > 0)) return null;
return Math.min(1, read / total);
}
/**
* How much total input one ITPM ceiling carries at a given read share. Pure.
* 1 / (1 - share): at 0.8 the same ceiling carries five times the input.
*/
export function headroomMultiplier(share) {
if (share === null || share === undefined) return null;
const bounded = Math.max(0, Math.min(0.99, Number(share)));
return 1 / (1 - bounded);
}
/**
* {model_group: input_tokens_per_minute} from the rate-limits response. Pure.
* An omitted type is recorded as null: absent means it inherits, not unlimited.
*/
export function itpmByGroup(payload) {
const out = {};
for (const entry of (payload ?? {}).data ?? []) {
const group = String(entry.model_group ?? '').trim();
if (!group) continue;
if (!(group in out)) out[group] = null;
for (const limit of entry.limits ?? []) {
if (String(limit.type ?? '').trim() !== 'input_tokens_per_minute') continue;
const value = Number(limit.value);
out[group] = Number.isInteger(value) ? value : null;
}
}
return out;
}
/** The ITPM for the group a model id belongs to, or null. Pure. Longest prefix wins. */
export function limitFor(groups, model) {
const name = String(model ?? '').trim().toLowerCase();
if (!name) return null;
let bestKey = null;
let bestLen = -1;
for (const group of Object.keys(groups ?? {})) {
const candidate = group.trim().toLowerCase();
if (!candidate) continue;
if (name === candidate || name.startsWith(candidate)) {
if (candidate.length > bestLen) { bestKey = group; bestLen = candidate.length; }
}
}
if (bestKey === null) return null;
const value = groups[bestKey];
return value === undefined ? null : value;
}
/**
* Classify one model's input limiter. Pure. Returns [state, detail].
* Three ways an ITPM ceiling can be full, and they do not share a repair.
*/
export function verdict(model, stats, limit, {
floor = 0.9, watch = 0.6, minMinutes = 10, cachedEnough = 0.15,
} = {}) {
const minutes = Number((stats ?? {}).minutes ?? 0);
if (minutes < minMinutes) {
return ['too-few-buckets',
`${minutes} minute(s) of traffic in the window, under the floor of ` +
`${minMinutes}. A peak taken over this little is noise.`];
}
if (limit === null || limit === undefined || limit <= 0) {
return ['no-limit-published',
"no input_tokens_per_minute is published for this model's group, so there " +
'is no ceiling to compare the peak against. The limiter still exists; the ' +
'number was simply not returned.'];
}
const peak = Number(stats.peak ?? 0);
const used = peak / limit;
const share = cacheReadShare(stats, model);
const shape = `peak minute charged ${peak} token(s) against an ITPM of ` +
`${limit} (${(used * 100).toFixed(0)}%); cache reads were ` +
`${((share ?? 0) * 100).toFixed(0)}% of that minute's input`;
if (used < watch) return ['itpm-headroom', `${shape}.`];
if (used < floor) {
return ['itpm-approaching',
`${shape}. Thin enough that an ordinary spike lands on the input limiter ` +
'rather than on the request limiter.'];
}
if (cacheReadsCount(model)) {
return ['itpm-saturated-cache-counts',
`${shape}. This model charges cache reads against ITPM, so caching lowers ` +
'the bill here and buys no headroom at all. The levers are a shorter ' +
'prefix or a higher limit.'];
}
if (share !== null && share >= cachedEnough) {
return ['itpm-saturated-already-cached',
`${shape}. The prefix is already being read back, so a breakpoint has ` +
'little left to give. What remains is a limit increase, or splitting the ' +
'workload across model groups.'];
}
return ['itpm-saturated-uncached',
`${shape}. Cache reads are not charged against ITPM on this model, so ` +
'covering the stable prefix buys throughput and not only a discount.'];
}
/** Floor to the minute: starting_at must sit on a bucket boundary. */
export function windowStart(minutes, now = new Date()) {
const floored = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate(),
now.getUTCHours(), now.getUTCMinutes());
return new Date(floored - minutes * 60000).toISOString().replace(/\.\d{3}Z$/, 'Z');
}
async function get(adminKey, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) {
for (const one of Array.isArray(v) ? v : [v]) url.searchParams.append(k, one);
}
const res = await fetch(url, {
headers: { 'x-api-key': adminKey, 'anthropic-version': VERSION },
});
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Anthropic: /v1/organizations/* needs an ` +
'Admin API key (sk-ant-admin...), not a workspace key');
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
async function* readBuckets(adminKey, path, params) {
const q = { ...params };
for (;;) {
const page = await get(adminKey, path, q);
for (const bucket of page.data ?? []) yield bucket;
if (!page.has_more || !page.next_page) return;
q.page = page.next_page;
}
}
async function main() {
const adminKey = process.env.ANTHROPIC_ADMIN_KEY;
if (!adminKey) {
console.error('set ANTHROPIC_ADMIN_KEY to an Admin API key (sk-ant-admin...); ' +
'a workspace key cannot read /v1/organizations/*');
process.exitCode = 2;
return;
}
const minutes = Math.max(1, Math.min(Number(process.env.MINUTES ?? 240), 1440));
const targetShare = Number(process.env.TARGET_SHARE ?? 0.8);
const showAll = process.env.SHOW_ALL === '1';
const collected = [];
for await (const bucket of readBuckets(adminKey, '/organizations/usage_report/messages',
{ starting_at: windowStart(minutes), bucket_width: '1m', limit: minutes,
'group_by[]': ['model'] })) {
collected.push(bucket);
}
const stats = peaks(collected);
const models = Object.keys(stats);
if (models.length === 0) {
console.log(`no message usage in the last ${minutes} minute(s)`);
return;
}
const groups = itpmByGroup(await get(adminKey, '/organizations/rate_limits'));
let bad = 0;
models.sort((a, b) => stats[b].peak - stats[a].peak);
for (const model of models) {
const row = stats[model];
const limit = limitFor(groups, model);
const [state, detail] = verdict(model, row, limit);
const line = `${state.padEnd(30)} ${model.padEnd(28)} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
if (state === 'itpm-saturated-uncached') {
const share = cacheReadShare(row, model) ?? 0;
console.warn(` at this read share the ceiling carries ` +
`${headroomMultiplier(share).toFixed(1)}x your total input; at ` +
`${(targetShare * 100).toFixed(0)}% it would carry ` +
`${headroomMultiplier(targetShare).toFixed(1)}x`);
console.warn(' repair: put a cache_control breakpoint at the end of the ' +
'stable prefix. The render order is tools, then system, then ' +
'messages, so the breakpoint goes after the last thing that ' +
'never changes.');
} else {
console.warn(' repair: request an input_tokens_per_minute increase for this ' +
'model group, or move latency tolerant work onto the Message ' +
'Batches API, which is metered by its own limiter group.');
}
} else if (state === 'itpm-approaching' || state === 'no-limit-published') {
console.warn(line);
} else if (showAll) {
console.log(line);
}
}
console.log(`${models.length} model(s) checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first two tests are the note and its opposite: an ITPM ceiling at ninety-eight percent with a two percent cache-read share is a caching finding, and the same ceiling at the same ninety-eight percent with an eighty percent read share is not — there the prefix is already covered and the only lever left is the limit. The third is the exception that would otherwise make the advice wrong for one family: on Claude Haiku 3.5 cache reads are charged against ITPM, so the script has to change both its arithmetic and its recommendation. The rest pin the nested cache_creation read, the fact that peaks are maxima rather than means, and a group with no published number staying unpublished rather than becoming unlimited.
from anthropic_itpm_headroom import (cache_read_share, cache_reads_count,
chargeable_input, headroom_multiplier,
itpm_by_group, limit_for, peaks, verdict)
def minute(stamp, model, uncached=0, write_5m=0, write_1h=0, read=0):
"""One 1m bucket from GET /v1/organizations/usage_report/messages."""
return {"starting_at": stamp, "results": [{
"model": model,
"uncached_input_tokens": uncached,
"cache_read_input_tokens": read,
"cache_creation": {"ephemeral_5m_input_tokens": write_5m,
"ephemeral_1h_input_tokens": write_1h},
"output_tokens": 12000,
}]}
def test_a_full_input_limiter_with_no_cache_reads_is_the_finding():
stats = peaks([minute("2026-08-30T14:%02d:00Z" % i, "claude-sonnet-5",
uncached=4_880_000 if i == 7 else 900_000,
read=100_000 if i == 7 else 0)
for i in range(20)])
state, detail = verdict("claude-sonnet-5", stats["claude-sonnet-5"], 5_000_000)
assert state == "itpm-saturated-uncached"
assert "against an ITPM of 5000000 (98%)" in detail
assert "cache reads were 2% of that minute's input" in detail
assert "buys throughput and not only a discount" in detail
# The throughput argument, which is the whole point of the note.
assert round(headroom_multiplier(0.8), 1) == 5.0
assert round(headroom_multiplier(0.0), 1) == 1.0
def test_the_same_full_ceiling_with_a_cached_prefix_is_a_different_finding():
# 98% of ITPM again, but the prefix is already being read back. Telling this
# reader to add a breakpoint sends them to do work that is already done.
stats = peaks([minute("2026-08-30T14:%02d:00Z" % i, "claude-sonnet-5",
uncached=4_880_000 if i == 3 else 100_000,
read=19_520_000 if i == 3 else 0)
for i in range(20)])
state, detail = verdict("claude-sonnet-5", stats["claude-sonnet-5"], 5_000_000)
assert state == "itpm-saturated-already-cached"
assert "cache reads were 80% of that minute's input" in detail
assert "limit increase" in detail
def test_haiku_35_charges_cache_reads_so_caching_buys_no_headroom():
assert cache_reads_count("claude-3-5-haiku-20241022") is True
assert cache_reads_count("claude-haiku-4-5-20251001") is False
assert cache_reads_count("claude-opus-5") is False
# The read is inside the charged number on that model and outside it here.
result = {"uncached_input_tokens": 1000, "cache_read_input_tokens": 4000,
"cache_creation": {"ephemeral_5m_input_tokens": 500}}
assert chargeable_input(result, "claude-sonnet-5") == 1500
assert chargeable_input(result, "claude-3-5-haiku-20241022") == 5500
stats = peaks([minute("2026-08-30T14:%02d:00Z" % i, "claude-3-5-haiku-20241022",
uncached=200_000, read=1_800_000) for i in range(20)])
state, detail = verdict("claude-3-5-haiku-20241022",
stats["claude-3-5-haiku-20241022"], 2_000_000)
assert state == "itpm-saturated-cache-counts"
assert "buys no headroom at all" in detail
def test_chargeable_input_reads_the_nested_cache_creation_object():
# The trap: these two fields live inside cache_creation, not at the top.
assert chargeable_input({"uncached_input_tokens": 100,
"cache_creation": {"ephemeral_5m_input_tokens": 7,
"ephemeral_1h_input_tokens": 3}},
"claude-opus-5") == 110
assert chargeable_input({"cache_creation_input_tokens": 999}, "claude-opus-5") == 0
assert chargeable_input({"uncached_input_tokens": None}, "claude-opus-5") == 0
assert chargeable_input(None, "claude-opus-5") == 0
def test_the_peak_minute_survives_an_otherwise_quiet_window():
# One saturated minute in twenty. The mean would read 24% of the ceiling and
# report a comfortable workload that is 429ing every hour.
stats = peaks([minute("2026-08-30T14:%02d:00Z" % i, "claude-opus-5",
uncached=4_800_000 if i == 11 else 200_000)
for i in range(20)])
row = stats["claude-opus-5"]
assert row["peak"] == 4_800_000
assert row["peak_at"] == "2026-08-30T14:11:00Z"
assert row["minutes"] == 20
assert verdict("claude-opus-5", row, 5_000_000)[0] == "itpm-saturated-uncached"
def test_a_window_too_short_to_have_a_peak_gets_no_verdict():
stats = peaks([minute("2026-08-30T14:00:00Z", "claude-opus-5", uncached=9_000_000)])
assert verdict("claude-opus-5", stats["claude-opus-5"], 5_000_000)[0] == "too-few-buckets"
def test_an_unpublished_ceiling_is_not_an_absent_one():
groups = itpm_by_group({"data": [
{"model_group": "claude-sonnet-5",
"limits": [{"type": "input_tokens_per_minute", "value": 5000000},
{"type": "output_tokens_per_minute", "value": 1000000}]},
{"model_group": "claude-fable-5",
"limits": [{"type": "output_tokens_per_minute", "value": 300000}]},
]})
assert groups["claude-sonnet-5"] == 5000000
assert groups["claude-fable-5"] is None
assert limit_for(groups, "claude-sonnet-5-20260101") == 5000000
assert limit_for(groups, "claude-fable-5") is None
assert limit_for(groups, "claude-opus-5") is None
assert verdict("claude-fable-5", {"minutes": 60, "peak": 9}, None)[0] == "no-limit-published"
def test_longest_prefix_wins_when_two_groups_could_claim_a_model():
groups = {"claude-haiku": 1000, "claude-haiku-4-5": 5_000_000}
assert limit_for(groups, "claude-haiku-4-5-20251001") == 5_000_000
assert limit_for(groups, "") is None
assert limit_for({}, "claude-opus-5") is None
assert cache_read_share({"peak": 0, "peak_read": 0}, "claude-opus-5") is None
assert headroom_multiplier(None) is None
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cacheReadShare, cacheReadsCount, chargeableInput, headroomMultiplier,
itpmByGroup, limitFor, peaks, verdict }
from './anthropic-itpm-headroom.mjs';
/** One 1m bucket from GET /v1/organizations/usage_report/messages. */
function minute(stamp, model, { uncached = 0, write5m = 0, write1h = 0, read = 0 } = {}) {
return { starting_at: stamp, results: [{
model,
uncached_input_tokens: uncached,
cache_read_input_tokens: read,
cache_creation: { ephemeral_5m_input_tokens: write5m,
ephemeral_1h_input_tokens: write1h },
output_tokens: 12000,
}] };
}
const stamp = (i) => `2026-08-30T14:${String(i).padStart(2, '0')}:00Z`;
test('a full input limiter with no cache reads is the finding', () => {
const stats = peaks([...Array(20).keys()].map((i) =>
minute(stamp(i), 'claude-sonnet-5',
{ uncached: i === 7 ? 4880000 : 900000, read: i === 7 ? 100000 : 0 })));
const [state, detail] = verdict('claude-sonnet-5', stats['claude-sonnet-5'], 5000000);
assert.equal(state, 'itpm-saturated-uncached');
assert.match(detail, /against an ITPM of 5000000 \(98%\)/);
assert.match(detail, /cache reads were 2% of that minute's input/);
assert.match(detail, /buys throughput and not only a discount/);
assert.equal(headroomMultiplier(0.8).toFixed(1), '5.0');
assert.equal(headroomMultiplier(0).toFixed(1), '1.0');
});
test('the same full ceiling with a cached prefix is a different finding', () => {
const stats = peaks([...Array(20).keys()].map((i) =>
minute(stamp(i), 'claude-sonnet-5',
{ uncached: i === 3 ? 4880000 : 100000, read: i === 3 ? 19520000 : 0 })));
const [state, detail] = verdict('claude-sonnet-5', stats['claude-sonnet-5'], 5000000);
assert.equal(state, 'itpm-saturated-already-cached');
assert.match(detail, /cache reads were 80% of that minute's input/);
assert.match(detail, /limit increase/);
});
test('haiku 3.5 charges cache reads so caching buys no headroom', () => {
assert.equal(cacheReadsCount('claude-3-5-haiku-20241022'), true);
assert.equal(cacheReadsCount('claude-haiku-4-5-20251001'), false);
assert.equal(cacheReadsCount('claude-opus-5'), false);
const result = { uncached_input_tokens: 1000, cache_read_input_tokens: 4000,
cache_creation: { ephemeral_5m_input_tokens: 500 } };
assert.equal(chargeableInput(result, 'claude-sonnet-5'), 1500);
assert.equal(chargeableInput(result, 'claude-3-5-haiku-20241022'), 5500);
const stats = peaks([...Array(20).keys()].map((i) =>
minute(stamp(i), 'claude-3-5-haiku-20241022', { uncached: 200000, read: 1800000 })));
const [state, detail] = verdict('claude-3-5-haiku-20241022',
stats['claude-3-5-haiku-20241022'], 2000000);
assert.equal(state, 'itpm-saturated-cache-counts');
assert.match(detail, /buys no headroom at all/);
});
test('chargeableInput reads the nested cache_creation object', () => {
assert.equal(chargeableInput({ uncached_input_tokens: 100,
cache_creation: { ephemeral_5m_input_tokens: 7, ephemeral_1h_input_tokens: 3 } },
'claude-opus-5'), 110);
assert.equal(chargeableInput({ cache_creation_input_tokens: 999 }, 'claude-opus-5'), 0);
assert.equal(chargeableInput({ uncached_input_tokens: null }, 'claude-opus-5'), 0);
assert.equal(chargeableInput(null, 'claude-opus-5'), 0);
});
test('the peak minute survives an otherwise quiet window', () => {
const stats = peaks([...Array(20).keys()].map((i) =>
minute(stamp(i), 'claude-opus-5', { uncached: i === 11 ? 4800000 : 200000 })));
const row = stats['claude-opus-5'];
assert.equal(row.peak, 4800000);
assert.equal(row.peak_at, '2026-08-30T14:11:00Z');
assert.equal(row.minutes, 20);
assert.equal(verdict('claude-opus-5', row, 5000000)[0], 'itpm-saturated-uncached');
});
test('a window too short to have a peak gets no verdict', () => {
const stats = peaks([minute('2026-08-30T14:00:00Z', 'claude-opus-5',
{ uncached: 9000000 })]);
assert.equal(verdict('claude-opus-5', stats['claude-opus-5'], 5000000)[0],
'too-few-buckets');
});
test('an unpublished ceiling is not an absent one', () => {
const groups = itpmByGroup({ data: [
{ model_group: 'claude-sonnet-5', limits: [
{ type: 'input_tokens_per_minute', value: 5000000 },
{ type: 'output_tokens_per_minute', value: 1000000 }] },
{ model_group: 'claude-fable-5', limits: [
{ type: 'output_tokens_per_minute', value: 300000 }] },
] });
assert.equal(groups['claude-sonnet-5'], 5000000);
assert.equal(groups['claude-fable-5'], null);
assert.equal(limitFor(groups, 'claude-sonnet-5-20260101'), 5000000);
assert.equal(limitFor(groups, 'claude-fable-5'), null);
assert.equal(limitFor(groups, 'claude-opus-5'), null);
assert.equal(verdict('claude-fable-5', { minutes: 60, peak: 9 }, null)[0],
'no-limit-published');
});
test('longest prefix wins when two groups could claim a model', () => {
const groups = { 'claude-haiku': 1000, 'claude-haiku-4-5': 5000000 };
assert.equal(limitFor(groups, 'claude-haiku-4-5-20251001'), 5000000);
assert.equal(limitFor(groups, ''), null);
assert.equal(limitFor({}, 'claude-opus-5'), null);
assert.equal(cacheReadShare({ peak: 0, peak_read: 0 }, 'claude-opus-5'), null);
assert.equal(headroomMultiplier(null), null);
});
FAQ
Why does caching help a rate limit at all?
Because the limiter is charged on uncached input only. cache_read_input_tokens is excluded from ITPM on every current model except Claude Haiku 3.5, so tokens you move behind a breakpoint stop competing for the ceiling entirely. At an eighty percent read share the same ITPM number carries five times the total input, which is a throughput change rather than a percentage saved.
Is this the same as the prompt caching cost notes?
No, and the difference is worth holding onto. Those notes end in a dollar figure and are true whether or not you are near any limit. This one ends in a ceiling and is true whether or not caching would save you money: a workload can be comfortably profitable, already budgeted, and still unable to grow because its input limiter is full every afternoon.
Will lowering concurrency help?
Almost never. ITPM is charged on tokens, not on connections, so the same volume of prompt spread over fewer workers arrives at the same rate. Concurrency is the right lever when requests per minute is the bucket that emptied, which is a different finding from a different header. Naming which one you hit is the sibling note on limiter identification.
Why minute buckets rather than hourly?
Because the limiter is enforced by the minute and an hourly mean hides exactly the shape you are looking for. A workload saturating ITPM for ninety seconds an hour reads as a quarter of its ceiling on an hourly graph and is rejecting requests twice an hour. The usage report allows up to 1,440 one-minute buckets, which is a full day.
What about Claude Haiku 3.5?
It is the one model family where cache reads are charged against ITPM. Caching still lowers the bill on it, and buys no headroom, so the script reports it as its own state rather than folding it in with the rest. If it is in your mix, the levers there are a shorter prefix, a different model, or a limit increase.
Related field notes
- Output per minute is the ceiling, so concurrency will not help
- A stable prefix reprocessed at full price on every call
- Cache writes paid for and never read back
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits — Claude Docs
- Get messages usage report — Claude Admin API
- Prompt caching — Claude Docs
- Rate Limits API — Claude Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.