Diagnostic LLM APIs
Request count tripled while token volume stayed flat
Latency got worse over about a fortnight, in the way that nobody can date precisely. Then 429s started arriving at a volume that used to be comfortable, and the obvious explanation was growth, except that the invoice barely moved. Two numbers sit in the same usage report and they have come apart: requests are up three times and tokens are up not at all. Two thirds of the extra calls landed in seventeen hours out of a hundred and sixty-eight, which is not what a new customer looks like.
Read both series off one call and divide. With an organization admin key: GET /v1/organization/usage/completions?start_time={now-14d}&bucket_width=1h&group_by=model&group_by=project_id. Each result carries num_model_requests alongside input_tokens and output_tokens, so the whole finding is available without any telemetry of your own.
Compare the last seven days against the seven before. Real growth moves both series by roughly the same factor. A retry storm moves the request count and leaves the token count almost where it was, so (input + output) / num_model_requests collapses: many short calls that failed early, retried, and eventually succeeded.
Then ask where in the week the extra calls landed, because that is the only independent evidence available. Retries amplify during the incidents that caused them, so a storm piles its surplus requests into a handful of hours. A workload that genuinely got shorter spreads them evenly across all of them.
Then confirm which ceiling you are pressed against: GET /v1/organization/projects/{project_id}/rate_limits. A retry storm sits near max_requests_per_1_minute while nowhere near max_tokens_per_1_minute, which is itself diagnostic, because genuine traffic growth pushes both up together.
The problem in plain words
Every OpenAI SDK retries on 429 and 5xx by default. Application code then adds a second layer — a decorator, a queue redelivery, a job runner that re-enqueues on failure — and the layers multiply rather than add. Three attempts inside three attempts is nine requests for one logical call, and each of the nine consumes request-rate budget. The work still gets done, so nothing raises, so nobody looks.
What it costs is capacity rather than money. Rate limits start binding at volumes that were fine a month ago, tail latency climbs because every logical call now carries the time of its failed attempts, and the system becomes less stable exactly when it is under load, since the retries amplify precisely when failures are most likely. The bill is a poor alarm here: retried attempts that failed early generated few tokens, so a storm can triple your request rate and move the invoice by single digits.
Why it happens
The two series come apart in only one way. Genuine growth is more calls of the same shape, so requests and tokens rise together. A prompt that grew moves tokens and leaves requests alone. Only re-issuing the same work moves requests without moving tokens, which is why the ratio between the two growth rates is a fingerprint rather than a heuristic.
Tokens per request is not corroboration, and this note was rewritten when that became obvious. The first draft flagged a storm when request growth outran token growth and the mean call size collapsed, as though those were two agreeing witnesses. They are one witness: request growth divided by token growth is exactly the reciprocal of the change in tokens per request, so the second test can never disagree with the first. It is arithmetic, not evidence. The script now says so in the function that computes it.
The independent signal is where in the week the extra calls landed. Retries amplify during the failures that caused them, so a storm concentrates its surplus requests into the hours when something was wrong. A new short workload — a classifier endpoint, a health check that started calling the model — produces the same weekly ratios spread evenly across every hour. Take the busiest tenth of the recent hours: even traffic puts about a tenth of its requests there, and a storm puts most of them there. That comparison needs the hourly buckets kept rather than summed, which is why the script does not fold the window until the last moment.
The RPM ceiling fills while the TPM ceiling stays empty. Retried attempts are charged against the request limiter whether or not they generate anything. A project pinned near max_requests_per_1_minute with its token limit barely touched is the same finding from the other side, and it is the reason raising the rate limit feels like it works: it does, for a fortnight, and then the amplification catches up.
Hourly buckets cannot resolve a minute. The rate-limit comparison here is an hourly mean spread across sixty minutes, which is a floor on the real peak and never the peak itself. That is fine for this purpose: if even the mean is close to the ceiling, the peak went past it long ago. It is not fine as a capacity plan, and the script says so rather than quietly implying a precision it does not have.
This is not the same read as a spend spike. A week-over-week cost check folds one series — money — and classifies its shape as a spike, a step or a ramp. This note ignores money entirely and compares two token-and-request series against each other. A storm that triples requests can be invisible to the cost check and unmissable here, which is the whole reason both exist.
It is also not the same read as a 529 cluster. Both notes are about requests that did not do the work you expected, but the arithmetic runs in opposite directions and needs different inputs. Here every number is the provider's: requests it counted and tokens it billed. In the overload note the requests were never counted at all, so the only way to see them is to bring your own attempt counter. A retry storm inflates the provider's request count; a 529 never reaches it.
The fix, as a flow
Two series arrive on the same result object and they have stopped agreeing. The trap is that the obvious second opinion is not one: request growth over token growth is the change in tokens per request inverted, so it can never disagree. The independent evidence is where in the week the extra calls landed.
How to fix it
Pull fourteen days of hourly buckets, grouped
bucket_width=1h with group_by=model and group_by=project_id, paging on next_page until it stops. Hourly rather than daily because the ratio you are about to compute is meaningless if a busy hour and an idle one are averaged together first, and grouped because a storm in one worker is invisible in an organization total.
Throw away the hour the clock is still inside
The newest bucket is always partial. Leave it in and every run before the hour is up reports a decline, which is the fastest way to teach a team to ignore a scheduled report. Drop it, and compare whole hours only.
Compute both growth rates, then the ratio between them
Requests last week over requests the week before; the same for input plus output tokens. Then divide one growth rate by the other. A factor above two — request growth outpacing token growth by more than double — is the threshold this note is built on, and it is a starting point rather than a law.
Look at where in the week the surplus requests landed
Report (input + output) / num_model_requests for each window, because it is the number a reader recognises — but do not treat it as a second opinion, since it is the first one inverted. The independent test is concentration: the share of the recent window's requests falling in its busiest tenth of hours. Around a tenth means even traffic. Most of them means the surplus arrived in bursts, which is what retries do.
Print the retry layering, not a rate-limit increase
The repair is to collapse to a single layer: set max_retries explicitly on the SDK client and delete the outer wrapper, or set it to zero and keep the wrapper, with exponential backoff, jitter, and a circuit breaker so a sustained failure stops re-amplifying. Raising the project's RPM is printed too, second and clearly marked, because doing it first buys a fortnight and hides the cause.
How to check it worked
Re-run a week after the retry layers are collapsed. Requests should fall back toward the token series and tokens per request should climb back to roughly where it was.
python3 openai_retry_storm_shape.py --days 14
# retry-storm proj_ingest / gpt-5.1 requests x3.04, tokens x1.00, tokens per request 5000 then 1647; 67% of the surplus landed in the busiest 10% of hours
# hourly mean sits at 82% of the RPM ceiling and 9% of the TPM ceiling
# repair: collapse to one retry layer. Set max_retries on the client and remove the outer wrapper.
# 7 model/project series checked, 1 finding(s)The full code
One GET for both series, and a second only when there is a finding to attribute. Eight pure functions: the series builder, which keeps each bucket rather than a total; the window fold, which drops the partial hour and splits on a cutoff; the growth ratio, which returns nothing rather than infinity when there is no prior week; the mean call size; the divergence ratio, whose docstring admits it is the same statement as the mean call size inverted; the concentration measure that is therefore the only independent evidence; the classifier; and the limiter comparison, which is honest that an hourly mean is a floor on the peak and not the peak.
"""Report OpenAI request volume growing faster than the tokens it carries.
Read only. One GET against the organization usage report, plus one per finding
against the project rate limits. Both need an organization admin key
(sk-admin-), which can be provisioned read-only.
Everything here comes from the provider's own numbers: num_model_requests and
the token counts arrive on the same result object, so no telemetry of your own
is required. Requests climbing while tokens stay flat, with the mean call size
collapsing underneath, is the retry-storm signature and nothing else makes it.
The repair is printed, never performed. Retry layering lives in your client.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_retry_storm_shape")
API = "https://api.openai.com/v1"
FINDINGS = ("retry-storm", "requests-outpacing-tokens")
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def series(buckets):
"""Per (project, model), the hourly points. Pure.
Buckets are kept rather than totalled, because the whole method is a
comparison between two halves of the window and a sum has already thrown
the halves away.
"""
out = {}
for bucket in buckets or []:
start = _int(bucket.get("start_time"))
for result in bucket.get("results") or []:
key = (str(result.get("project_id") or "unknown"),
str(result.get("model") or "unknown"))
out.setdefault(key, []).append({
"start": start,
"requests": _int(result.get("num_model_requests")),
"tokens": (_int(result.get("input_tokens"))
+ _int(result.get("output_tokens"))),
})
for points in out.values():
points.sort(key=lambda p: p["start"])
return out
def fold_windows(points, cutoff, partial_after=None):
"""Sum one series into (prior, recent) either side of a cutoff. Pure.
Points at or after partial_after are dropped. The hour the clock is still
inside is always short, and a growth ratio computed with it in reports a
decline every single time the job runs before the hour is up.
"""
prior = {"requests": 0, "tokens": 0, "buckets": 0}
recent = {"requests": 0, "tokens": 0, "buckets": 0}
for point in points or []:
start = _int(point.get("start"))
if partial_after is not None and start >= partial_after:
continue
window = recent if start >= cutoff else prior
window["requests"] += _int(point.get("requests"))
window["tokens"] += _int(point.get("tokens"))
window["buckets"] += 1
return prior, recent
def growth(prior_value, recent_value):
"""recent / prior, or None when there is nothing to divide by. Pure.
None rather than infinity. A workload that did not exist last week has no
growth rate, and reporting one as an enormous number puts every new
deployment at the top of the report.
"""
prior_value = float(prior_value or 0)
if prior_value <= 0:
return None
return float(recent_value or 0) / prior_value
def tokens_per_request(window):
"""Mean tokens per request in one window, or None. Pure."""
made = _int((window or {}).get("requests"))
if made <= 0:
return None
return _int(window.get("tokens")) / float(made)
def divergence_ratio(prior, recent):
"""Request growth divided by token growth. Pure. None when unavailable.
Worth stating in the code rather than only in the prose: this number is
exactly the reciprocal of the change in tokens per request. The first
version of this script tested both and read them as two agreeing witnesses.
They are one witness stated twice, so the corroboration has to come from
somewhere else, which is what burstiness() is for.
"""
request_growth = growth(_int((prior or {}).get("requests")),
_int((recent or {}).get("requests")))
token_growth = growth(_int((prior or {}).get("tokens")),
_int((recent or {}).get("tokens")))
if request_growth is None or token_growth is None or token_growth <= 0:
return None
return request_growth / token_growth
def burstiness(points, cutoff, partial_after=None, top_share=0.1, min_buckets=24):
"""Share of the recent window's requests in its busiest hours. Pure.
The busiest top_share of hours, by request count. Evenly spread traffic
puts about top_share of its requests there. A retry storm puts most of them
there, because retries amplify during the failures that caused them, and
that concentration is the only evidence in this report that is independent
of the growth ratio.
None when there are too few hours for the share to mean anything.
"""
recent = []
for point in points or []:
start = _int(point.get("start"))
if partial_after is not None and start >= partial_after:
continue
if start >= cutoff:
recent.append(_int(point.get("requests")))
if len(recent) < min_buckets:
return None
total = sum(recent)
if total <= 0:
return None
top = max(1, int(round(len(recent) * top_share)))
return sum(sorted(recent, reverse=True)[:top]) / float(total)
def classify(prior, recent, burst=None, divergence=2.0, min_requests=1000,
burst_floor=0.35):
"""Compare two windows of one series. Pure. Returns (state, detail).
Four ways two series can move relative to each other, and only one of them
is a retry storm. The divergence says the request count grew on its own;
the burst share says whether it grew in the shape retries have.
"""
prior = prior or {}
recent = recent or {}
prior_requests = _int(prior.get("requests"))
recent_requests = _int(recent.get("requests"))
if prior_requests < min_requests and recent_requests < min_requests:
return ("too-little-traffic",
"%d request(s) then %d, both under the floor of %d"
% (prior_requests, recent_requests, min_requests))
request_growth = growth(prior_requests, recent_requests)
token_growth = growth(_int(prior.get("tokens")), _int(recent.get("tokens")))
if request_growth is None or token_growth is None:
return ("new-workload",
"nothing in the prior window to compare against: %d request(s) "
"and %d token(s) appeared this week"
% (recent_requests, _int(recent.get("tokens"))))
before = tokens_per_request(prior) or 0.0
after = tokens_per_request(recent) or 0.0
shape = ("requests x%.2f, tokens x%.2f, tokens per request %d then %d"
% (request_growth, token_growth, before, after))
if burst is not None:
shape += ("; %.0f%% of the surplus landed in the busiest 10%% of hours"
% (burst * 100))
if request_growth >= divergence * token_growth:
if burst is None:
return ("retry-storm",
shape + ". Too few hourly buckets to measure how "
"concentrated the surplus was, so this rests on the growth "
"ratio alone.")
if burst < burst_floor:
return ("requests-outpacing-tokens",
shape + ". The extra calls are spread evenly across the "
"hours rather than piled into a few, which is a workload "
"that got shorter rather than one being retried.")
return ("retry-storm",
shape + ". The surplus arrived in bursts, which is what "
"retries do: they amplify during the failures that caused "
"them.")
if token_growth >= divergence * request_growth:
return ("prompts-grew",
shape + ". Tokens moved and the call count did not, so this is "
"prompt or answer length, not call volume.")
if request_growth >= 1.25 and token_growth >= 1.25:
return ("traffic-growth",
shape + ". Both series moved together, which is traffic rather "
"than amplification.")
if request_growth <= 0.75:
return ("quieter", shape + ". Fewer calls than the week before.")
return ("steady", shape + ".")
def rate_limit_values(payload, model):
"""The RPM and TPM this project publishes for a model. Pure.
Longest matching prefix wins, so a dated id resolves to the most specific
entry that claims it rather than to whichever one came back first.
"""
best_key, best_len = None, -1
name = str(model or "").strip().lower()
for entry in (payload or {}).get("data") or []:
candidate = str(entry.get("model") or "").strip().lower()
if not candidate:
continue
if name == candidate or name.startswith(candidate):
if len(candidate) > best_len:
best_key, best_len = entry, len(candidate)
if best_key is None:
return {"requests": None, "tokens": None}
out = {}
for field, key in (("max_requests_per_1_minute", "requests"),
("max_tokens_per_1_minute", "tokens")):
try:
out[key] = int(best_key.get(field))
except (TypeError, ValueError):
out[key] = None
return out
def limiter_pressure(window, hours, limits, near=0.7, idle=0.3):
"""Where a window's mean traffic sits against RPM and TPM. Pure.
Hourly buckets cannot resolve a minute, so this is the hourly mean spread
across sixty minutes: a floor on the real peak and never the peak itself.
If even the mean is near the ceiling then the peak went past it long ago,
which is all this needs to say.
"""
limits = limits or {}
rpm_limit = limits.get("requests")
tpm_limit = limits.get("tokens")
minutes = max(1, int(hours or 0) * 60)
if not rpm_limit and not tpm_limit:
return ("no-limits-published",
"this project publishes no rate limit for the model, so there "
"is no ceiling to compare the mean against")
rpm_used = (_int((window or {}).get("requests")) / float(minutes) / rpm_limit
if rpm_limit else None)
tpm_used = (_int((window or {}).get("tokens")) / float(minutes) / tpm_limit
if tpm_limit else None)
shape = "hourly mean sits at %s of the RPM ceiling and %s of the TPM ceiling" % (
"%.0f%%" % (rpm_used * 100) if rpm_used is not None else "an unpublished share",
"%.0f%%" % (tpm_used * 100) if tpm_used is not None else "an unpublished share")
if rpm_used is not None and tpm_used is not None:
if rpm_used >= near and tpm_used <= idle:
return ("rpm-bound-tpm-idle",
shape + ", which is what amplification looks like from the "
"limiter side: the request bucket fills and the token "
"bucket does not")
if rpm_used >= near and tpm_used >= near:
return ("both-near", shape + ", so both limiters are under pressure")
if tpm_used >= near:
return ("tpm-bound",
shape + ", so the token limiter is the binding one and this "
"is volume rather than retries")
return ("headroom", shape)
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key (sk-admin-), not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def pages(session, path, params, max_pages=40):
"""Walk the usage report, which paginates on an opaque page cursor."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params = dict(params)
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=14,
help="days to read, split into two halves (default 14)")
ap.add_argument("--divergence", type=float, default=2.0,
help="how far request growth must outpace token growth "
"(default 2.0)")
ap.add_argument("--min-requests", type=int, default=1000,
help="ignore series below this many requests (default 1000)")
ap.add_argument("--show-all", action="store_true",
help="also print series that moved together")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY (an organization admin key; read-only "
"scopes are enough)")
return 2
days = max(2, min(int(args.days), 30))
half = days // 2
now = int(time.time())
start = now - days * 86400
cutoff = now - half * 86400
partial_after = now - (now % 3600)
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
buckets = pages(session, "/organization/usage/completions", {
"start_time": start,
"bucket_width": "1h",
"limit": 168,
"group_by": ["model", "project_id"],
})
rows = series(buckets)
if not rows:
log.info("no completions usage in the last %d day(s)", days)
return 0
checked = 0
bad = 0
for project, model in sorted(rows):
points = rows[(project, model)]
prior, recent = fold_windows(points, cutoff, partial_after)
burst = burstiness(points, cutoff, partial_after)
state, detail = classify(prior, recent, burst, args.divergence,
min_requests=args.min_requests)
checked += 1
line = "%-26s %s / %s %s" % (state, project, model, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
limits = {"requests": None, "tokens": None}
if project != "unknown":
try:
limits = rate_limit_values(
get(session, "/organization/projects/%s/rate_limits" % project,
{"limit": 100}), model)
except (requests.RequestException, SystemExit):
limits = {"requests": None, "tokens": None}
_, pressure = limiter_pressure(recent, half * 24, limits)
log.warning(" %s", pressure)
if state == "retry-storm":
log.warning(" repair: collapse to one retry layer. Set "
"max_retries explicitly on the SDK client and "
"remove the outer wrapper, or set it to 0 and keep "
"the wrapper. Exponential backoff with jitter, and "
"a circuit breaker so a sustained failure stops "
"re-amplifying.")
log.warning(" repair: raising the project rate limit is the "
"second measure, not the first. An admin can call "
"POST /v1/organization/projects/{project_id}"
"/rate_limits/{rate_limit_id} once the layering is "
"fixed. It is printed here, not run.")
else:
log.warning(" repair: nothing yet. Confirm the shorter calls "
"are a real workload before changing any retry "
"policy, and re-run next week.")
elif args.show_all:
log.info(line)
log.info("%d model/project series checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report OpenAI request volume growing faster than the tokens it carries.
*
* Read only. One GET against the organization usage report, plus one per
* finding against the project rate limits. Both need an organization admin key
* (sk-admin-), which can be provisioned read-only.
*
* Both series come off the provider's own report, so no telemetry of your own
* is required. The repair is printed, never performed.
*/
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set(['retry-storm', 'requests-outpacing-tokens']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Per (project, model), the hourly points. Pure.
* Buckets are kept rather than totalled: the method is a comparison between
* two halves of the window, and a sum has already thrown the halves away.
*/
export function series(buckets) {
const out = new Map();
for (const bucket of buckets ?? []) {
const start = readInt(bucket?.start_time);
for (const result of bucket?.results ?? []) {
const key = `${result?.project_id ?? 'unknown'}\u0000${result?.model ?? 'unknown'}`;
if (!out.has(key)) {
out.set(key, { project: String(result?.project_id ?? 'unknown'),
model: String(result?.model ?? 'unknown'), points: [] });
}
out.get(key).points.push({
start,
requests: readInt(result?.num_model_requests),
tokens: readInt(result?.input_tokens) + readInt(result?.output_tokens),
});
}
}
for (const row of out.values()) row.points.sort((a, b) => a.start - b.start);
return out;
}
/**
* Sum one series into [prior, recent] either side of a cutoff. Pure.
* Points at or after partialAfter are dropped: the hour the clock is still
* inside is always short, and leaving it in reports a decline every run.
*/
export function foldWindows(points, cutoff, partialAfter = null) {
const prior = { requests: 0, tokens: 0, buckets: 0 };
const recent = { requests: 0, tokens: 0, buckets: 0 };
for (const point of points ?? []) {
const start = readInt(point?.start);
if (partialAfter !== null && partialAfter !== undefined && start >= partialAfter) continue;
const window = start >= cutoff ? recent : prior;
window.requests += readInt(point?.requests);
window.tokens += readInt(point?.tokens);
window.buckets += 1;
}
return [prior, recent];
}
/**
* recent / prior, or null when there is nothing to divide by. Pure.
* Null rather than Infinity: a workload that did not exist last week has no
* growth rate, and a huge number would put every new deployment on top.
*/
export function growth(priorValue, recentValue) {
const prior = Number(priorValue ?? 0);
if (!(prior > 0)) return null;
return Number(recentValue ?? 0) / prior;
}
/** Mean tokens per request in one window, or null. Pure. */
export function tokensPerRequest(window) {
const made = readInt(window?.requests);
if (made <= 0) return null;
return readInt(window?.tokens) / made;
}
/**
* Request growth divided by token growth. Pure. Null when unavailable.
* This number is exactly the reciprocal of the change in tokens per request.
* The first version of this script tested both and read them as two agreeing
* witnesses; they are one witness stated twice, which is why burstiness exists.
*/
export function divergenceRatio(prior, recent) {
const requestGrowth = growth(readInt(prior?.requests), readInt(recent?.requests));
const tokenGrowth = growth(readInt(prior?.tokens), readInt(recent?.tokens));
if (requestGrowth === null || tokenGrowth === null || tokenGrowth <= 0) return null;
return requestGrowth / tokenGrowth;
}
/**
* Share of the recent window's requests in its busiest hours. Pure.
* Evenly spread traffic puts about topShare of its requests there; a retry
* storm puts most of them there, because retries amplify during the failures
* that caused them. Null when there are too few hours to mean anything.
*/
export function burstiness(points, cutoff, partialAfter = null, topShare = 0.1,
minBuckets = 24) {
const recent = [];
for (const point of points ?? []) {
const start = readInt(point?.start);
if (partialAfter !== null && partialAfter !== undefined && start >= partialAfter) continue;
if (start >= cutoff) recent.push(readInt(point?.requests));
}
if (recent.length < minBuckets) return null;
const total = recent.reduce((a, b) => a + b, 0);
if (total <= 0) return null;
const top = Math.max(1, Math.round(recent.length * topShare));
const head = [...recent].sort((a, b) => b - a).slice(0, top);
return head.reduce((a, b) => a + b, 0) / total;
}
/**
* Compare two windows of one series. Pure. Returns [state, detail].
* The divergence says the request count grew on its own; the burst share says
* whether it grew in the shape retries have.
*/
export function classify(prior, recent, burst = null, divergence = 2.0,
minRequests = 1000, burstFloor = 0.35) {
const priorRequests = readInt(prior?.requests);
const recentRequests = readInt(recent?.requests);
if (priorRequests < minRequests && recentRequests < minRequests) {
return ['too-little-traffic',
`${priorRequests} request(s) then ${recentRequests}, both under the ` +
`floor of ${minRequests}`];
}
const requestGrowth = growth(priorRequests, recentRequests);
const tokenGrowth = growth(readInt(prior?.tokens), readInt(recent?.tokens));
if (requestGrowth === null || tokenGrowth === null) {
return ['new-workload',
`nothing in the prior window to compare against: ${recentRequests} ` +
`request(s) and ${readInt(recent?.tokens)} token(s) appeared this week`];
}
const before = tokensPerRequest(prior) ?? 0;
const after = tokensPerRequest(recent) ?? 0;
let shape = `requests x${requestGrowth.toFixed(2)}, tokens x` +
`${tokenGrowth.toFixed(2)}, tokens per request ${Math.trunc(before)} then ` +
`${Math.trunc(after)}`;
if (burst !== null && burst !== undefined) {
shape += `; ${(burst * 100).toFixed(0)}% of the surplus landed in the ` +
'busiest 10% of hours';
}
if (requestGrowth >= divergence * tokenGrowth) {
if (burst === null || burst === undefined) {
return ['retry-storm',
`${shape}. Too few hourly buckets to measure how concentrated the ` +
'surplus was, so this rests on the growth ratio alone.'];
}
if (burst < burstFloor) {
return ['requests-outpacing-tokens',
`${shape}. The extra calls are spread evenly across the hours rather ` +
'than piled into a few, which is a workload that got shorter rather ' +
'than one being retried.'];
}
return ['retry-storm',
`${shape}. The surplus arrived in bursts, which is what retries do: they ` +
'amplify during the failures that caused them.'];
}
if (tokenGrowth >= divergence * requestGrowth) {
return ['prompts-grew',
`${shape}. Tokens moved and the call count did not, so this is prompt ` +
'or answer length, not call volume.'];
}
if (requestGrowth >= 1.25 && tokenGrowth >= 1.25) {
return ['traffic-growth',
`${shape}. Both series moved together, which is traffic rather than ` +
'amplification.'];
}
if (requestGrowth <= 0.75) {
return ['quieter', `${shape}. Fewer calls than the week before.`];
}
return ['steady', `${shape}.`];
}
/** The RPM and TPM this project publishes for a model. Pure. Longest prefix wins. */
export function rateLimitValues(payload, model) {
const name = String(model ?? '').trim().toLowerCase();
let best = null;
let bestLen = -1;
for (const entry of payload?.data ?? []) {
const candidate = String(entry?.model ?? '').trim().toLowerCase();
if (!candidate) continue;
if ((name === candidate || name.startsWith(candidate)) && candidate.length > bestLen) {
best = entry;
bestLen = candidate.length;
}
}
if (best === null) return { requests: null, tokens: null };
const read = (field) => {
const n = Number(best[field]);
return Number.isFinite(n) ? Math.trunc(n) : null;
};
return { requests: read('max_requests_per_1_minute'),
tokens: read('max_tokens_per_1_minute') };
}
/**
* Where a window's mean traffic sits against RPM and TPM. Pure.
* An hourly mean spread across sixty minutes is a floor on the real peak and
* never the peak itself, which is all this needs to be.
*/
export function limiterPressure(window, hours, limits, near = 0.7, idle = 0.3) {
const rpmLimit = limits?.requests;
const tpmLimit = limits?.tokens;
const minutes = Math.max(1, readInt(hours) * 60);
if (!rpmLimit && !tpmLimit) {
return ['no-limits-published',
'this project publishes no rate limit for the model, so there is no ' +
'ceiling to compare the mean against'];
}
const rpmUsed = rpmLimit ? readInt(window?.requests) / minutes / rpmLimit : null;
const tpmUsed = tpmLimit ? readInt(window?.tokens) / minutes / tpmLimit : null;
const pct = (v) => (v === null ? 'an unpublished share' : `${(v * 100).toFixed(0)}%`);
const shape = `hourly mean sits at ${pct(rpmUsed)} of the RPM ceiling and ` +
`${pct(tpmUsed)} of the TPM ceiling`;
if (rpmUsed !== null && tpmUsed !== null) {
if (rpmUsed >= near && tpmUsed <= idle) {
return ['rpm-bound-tpm-idle',
`${shape}, which is what amplification looks like from the limiter ` +
'side: the request bucket fills and the token bucket does not'];
}
if (rpmUsed >= near && tpmUsed >= near) {
return ['both-near', `${shape}, so both limiters are under pressure`];
}
if (tpmUsed >= near) {
return ['tpm-bound',
`${shape}, so the token limiter is the binding one and this is volume ` +
'rather than retries'];
}
}
return ['headroom', shape];
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization/* needs an ` +
'organization admin key (sk-admin-), not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* pages(key, path, params, maxPages = 40) {
let query = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key; read-only ' +
'scopes are enough)');
process.exitCode = 2;
return;
}
const days = Math.max(2, Math.min(Number(process.env.DAYS ?? 14), 30));
const half = Math.floor(days / 2);
const now = Math.floor(Date.now() / 1000);
const cutoff = now - half * 86400;
const partialAfter = now - (now % 3600);
const showAll = process.env.SHOW_ALL === '1';
const buckets = [];
for await (const bucket of pages(admin, '/organization/usage/completions', {
start_time: now - days * 86400,
bucket_width: '1h',
limit: 168,
group_by: ['model', 'project_id'],
})) buckets.push(bucket);
const rows = series(buckets);
if (rows.size === 0) {
console.log(`no completions usage in the last ${days} day(s)`);
return;
}
let checked = 0;
let bad = 0;
for (const row of [...rows.values()].sort((a, b) =>
`${a.project}${a.model}`.localeCompare(`${b.project}${b.model}`))) {
const [prior, recent] = foldWindows(row.points, cutoff, partialAfter);
const burst = burstiness(row.points, cutoff, partialAfter);
const [state, detail] = classify(prior, recent, burst);
checked += 1;
const line = `${state.padEnd(26)} ${row.project} / ${row.model} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
let limits = { requests: null, tokens: null };
if (row.project !== 'unknown') {
try {
limits = rateLimitValues(
await get(admin, `/organization/projects/${row.project}/rate_limits`,
{ limit: 100 }), row.model);
} catch {
limits = { requests: null, tokens: null };
}
}
const [, pressure] = limiterPressure(recent, half * 24, limits);
console.warn(` ${pressure}`);
if (state === 'retry-storm') {
console.warn(' repair: collapse to one retry layer. Set max_retries ' +
'explicitly on the SDK client and remove the outer wrapper, ' +
'or set it to 0 and keep the wrapper. Exponential backoff ' +
'with jitter, and a circuit breaker so a sustained failure ' +
'stops re-amplifying.');
console.warn(' repair: raising the project rate limit is the second ' +
'measure, not the first. An admin can call POST ' +
'/v1/organization/projects/{project_id}/rate_limits/' +
'{rate_limit_id} once the layering is fixed. It is printed ' +
'here, not run.');
} else {
console.warn(' repair: nothing yet. Confirm the shorter calls are a ' +
'real workload before changing any retry policy, and ' +
're-run next week.');
}
} else if (showAll) {
console.log(line);
}
}
console.log(`${checked} model/project series checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Two tests carry this note and they are deliberately a pair: two fortnights with byte-identical weekly arithmetic — three times the requests, no more tokens, the mean call down from five thousand to sixteen hundred — where one piles its surplus into eighteen hours and the other spreads it across all hundred and sixty-eight. One is a retry storm and one is a workload that got shorter, and nothing in the weekly totals can tell them apart. A third test asserts the embarrassing identity that forced the rewrite: the divergence ratio is the change in mean call size inverted, so it can never be a second opinion. The rest pin the partial hour, the new workload that must not report infinite growth, and the limiter reading that names a full request bucket beside an empty token one.
from openai_retry_storm_shape import (burstiness, classify, divergence_ratio,
fold_windows, growth, limiter_pressure,
rate_limit_values, series,
tokens_per_request)
CUTOFF = 1_000_000
HOUR = 3600
def hours(start, count, requests_each, tokens_each):
return [{"start": start + i * HOUR, "requests": requests_each,
"tokens": tokens_each} for i in range(count)]
PRIOR_WEEK = hours(CUTOFF - 168 * HOUR, 168, 1000, 5_000_000)
# Same weekly totals, two different shapes. The storm piles its surplus into
# eighteen hours; the short workload spreads the identical surplus evenly.
STORM = (hours(CUTOFF, 150, 1000, 5_000_000)
+ hours(CUTOFF + 150 * HOUR, 18, 20_000, 5_000_000))
EVEN = hours(CUTOFF, 168, 3000, 5_000_000)
def test_requests_climb_in_bursts_while_tokens_stand_still():
# The note in one assertion. Three times the calls, no more tokens, the
# mean call down from 5000 to 1647, and two thirds of the surplus landing
# in the busiest tenth of the hours.
prior, recent = fold_windows(PRIOR_WEEK + STORM, CUTOFF)
assert prior["requests"] == 168_000 and prior["tokens"] == 840_000_000
assert recent["requests"] == 510_000 and recent["tokens"] == 840_000_000
assert round(growth(prior["requests"], recent["requests"]), 3) == 3.036
assert growth(prior["tokens"], recent["tokens"]) == 1.0
assert int(tokens_per_request(prior)) == 5000
assert int(tokens_per_request(recent)) == 1647
burst = burstiness(PRIOR_WEEK + STORM, CUTOFF)
assert round(burst, 3) == 0.667
state, detail = classify(prior, recent, burst)
assert state == "retry-storm"
assert "requests x3.04, tokens x1.00" in detail
assert "tokens per request 5000 then 1647" in detail
assert "67% of the surplus landed in the busiest 10% of hours" in detail
def test_the_same_ratios_spread_evenly_are_not_a_storm():
# Identical weekly arithmetic, opposite conclusion. This pair is the reason
# the concentration measure exists at all.
prior, recent = fold_windows(PRIOR_WEEK + EVEN, CUTOFF)
assert round(divergence_ratio(prior, recent), 2) == 3.0
burst = burstiness(PRIOR_WEEK + EVEN, CUTOFF)
assert round(burst, 3) == 0.101
state, detail = classify(prior, recent, burst)
assert state == "requests-outpacing-tokens"
assert "spread evenly across the hours" in detail
def test_the_divergence_ratio_is_the_mean_call_size_inverted():
# Stated as a test because the first draft treated these as two agreeing
# signals. They cannot disagree.
prior, recent = fold_windows(PRIOR_WEEK + STORM, CUTOFF)
identity = tokens_per_request(prior) / tokens_per_request(recent)
assert round(divergence_ratio(prior, recent), 9) == round(identity, 9)
def test_a_real_customer_moves_both_series_together():
state, detail = classify({"requests": 100_000, "tokens": 500_000_000},
{"requests": 300_000, "tokens": 1_500_000_000}, 0.1)
assert state == "traffic-growth"
assert "moved together" in detail
def test_a_prompt_that_grew_moves_only_the_token_series():
state, _ = classify({"requests": 100_000, "tokens": 200_000_000},
{"requests": 100_000, "tokens": 600_000_000}, 0.1)
assert state == "prompts-grew"
def test_the_partial_hour_is_dropped_before_anything_is_divided():
tail = [{"start": CUTOFF + 200 * HOUR, "requests": 1, "tokens": 10}]
prior, recent = fold_windows(PRIOR_WEEK + EVEN + tail, CUTOFF,
partial_after=CUTOFF + 200 * HOUR)
assert recent["buckets"] == 168 and recent["requests"] == 504_000
assert burstiness(EVEN + tail, CUTOFF, partial_after=CUTOFF + 200 * HOUR) \
== burstiness(EVEN, CUTOFF)
def test_a_workload_with_no_prior_week_has_no_growth_rate():
assert growth(0, 5000) is None
assert growth(None, 5000) is None
assert divergence_ratio({"requests": 1, "tokens": 0},
{"requests": 2, "tokens": 0}) is None
assert tokens_per_request({"requests": 0, "tokens": 0}) is None
state, _ = classify({"requests": 0, "tokens": 0},
{"requests": 40_000, "tokens": 90_000_000})
assert state == "new-workload"
assert classify({"requests": 10, "tokens": 900},
{"requests": 12, "tokens": 1000})[0] == "too-little-traffic"
def test_too_few_hours_reports_no_concentration_rather_than_a_wrong_one():
short = [{"start": CUTOFF + i * HOUR, "requests": 10, "tokens": 1}
for i in range(6)]
assert burstiness(short, CUTOFF) is None
assert burstiness([], CUTOFF) is None
state, detail = classify({"requests": 100_000, "tokens": 500_000_000},
{"requests": 400_000, "tokens": 520_000_000})
assert state == "retry-storm"
assert "Too few hourly buckets" in detail
def test_the_request_bucket_is_full_while_the_token_bucket_is_empty():
payload = {"data": [{"model": "gpt-5.1", "max_requests_per_1_minute": 10_000,
"max_tokens_per_1_minute": 20_000_000},
{"model": "gpt-5", "max_requests_per_1_minute": 1,
"max_tokens_per_1_minute": 1}]}
limits = rate_limit_values(payload, "gpt-5.1-2026-01-15")
assert limits == {"requests": 10_000, "tokens": 20_000_000}
state, detail = limiter_pressure(
{"requests": 82_656_000, "tokens": 18_144_000_000}, 168, limits)
assert state == "rpm-bound-tpm-idle"
assert "82% of the RPM ceiling and 9% of the TPM ceiling" in detail
def test_an_unpublished_limit_is_not_a_missing_one():
assert rate_limit_values({"data": []}, "gpt-5.1") == {"requests": None,
"tokens": None}
assert limiter_pressure({"requests": 1}, 24, None)[0] == "no-limits-published"
assert series([]) == {}
assert series(None) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { burstiness, classify, divergenceRatio, foldWindows, growth,
limiterPressure, rateLimitValues, series, tokensPerRequest }
from './openai-retry-storm-shape.mjs';
const CUTOFF = 1000000;
const HOUR = 3600;
const hours = (start, count, requests, tokens) =>
Array.from({ length: count }, (_, i) => ({ start: start + i * HOUR, requests, tokens }));
const PRIOR_WEEK = hours(CUTOFF - 168 * HOUR, 168, 1000, 5000000);
// Same weekly totals, two different shapes.
const STORM = [...hours(CUTOFF, 150, 1000, 5000000),
...hours(CUTOFF + 150 * HOUR, 18, 20000, 5000000)];
const EVEN = hours(CUTOFF, 168, 3000, 5000000);
test('requests climb in bursts while tokens stand still', () => {
const [prior, recent] = foldWindows([...PRIOR_WEEK, ...STORM], CUTOFF);
assert.equal(prior.requests, 168000);
assert.equal(prior.tokens, 840000000);
assert.equal(recent.requests, 510000);
assert.equal(recent.tokens, 840000000);
assert.equal(Number(growth(prior.requests, recent.requests).toFixed(3)), 3.036);
assert.equal(growth(prior.tokens, recent.tokens), 1);
assert.equal(Math.trunc(tokensPerRequest(prior)), 5000);
assert.equal(Math.trunc(tokensPerRequest(recent)), 1647);
const burst = burstiness([...PRIOR_WEEK, ...STORM], CUTOFF);
assert.equal(Number(burst.toFixed(3)), 0.667);
const [state, detail] = classify(prior, recent, burst);
assert.equal(state, 'retry-storm');
assert.match(detail, /requests x3\.04, tokens x1\.00/);
assert.match(detail, /tokens per request 5000 then 1647/);
assert.match(detail, /67% of the surplus landed in the busiest 10% of hours/);
});
test('the same ratios spread evenly are not a storm', () => {
const [prior, recent] = foldWindows([...PRIOR_WEEK, ...EVEN], CUTOFF);
assert.equal(Number(divergenceRatio(prior, recent).toFixed(2)), 3);
const burst = burstiness([...PRIOR_WEEK, ...EVEN], CUTOFF);
assert.equal(Number(burst.toFixed(3)), 0.101);
const [state, detail] = classify(prior, recent, burst);
assert.equal(state, 'requests-outpacing-tokens');
assert.match(detail, /spread evenly across the hours/);
});
test('the divergence ratio is the mean call size inverted', () => {
const [prior, recent] = foldWindows([...PRIOR_WEEK, ...STORM], CUTOFF);
const identity = tokensPerRequest(prior) / tokensPerRequest(recent);
assert.equal(divergenceRatio(prior, recent).toFixed(9), identity.toFixed(9));
});
test('a real customer moves both series together', () => {
const [state, detail] = classify({ requests: 100000, tokens: 500000000 },
{ requests: 300000, tokens: 1500000000 }, 0.1);
assert.equal(state, 'traffic-growth');
assert.match(detail, /moved together/);
});
test('a prompt that grew moves only the token series', () => {
const [state] = classify({ requests: 100000, tokens: 200000000 },
{ requests: 100000, tokens: 600000000 }, 0.1);
assert.equal(state, 'prompts-grew');
});
test('the partial hour is dropped before anything is divided', () => {
const tail = [{ start: CUTOFF + 200 * HOUR, requests: 1, tokens: 10 }];
const [, recent] = foldWindows([...PRIOR_WEEK, ...EVEN, ...tail], CUTOFF,
CUTOFF + 200 * HOUR);
assert.equal(recent.buckets, 168);
assert.equal(recent.requests, 504000);
assert.equal(burstiness([...EVEN, ...tail], CUTOFF, CUTOFF + 200 * HOUR),
burstiness(EVEN, CUTOFF));
});
test('a workload with no prior week has no growth rate', () => {
assert.equal(growth(0, 5000), null);
assert.equal(growth(null, 5000), null);
assert.equal(divergenceRatio({ requests: 1, tokens: 0 },
{ requests: 2, tokens: 0 }), null);
assert.equal(tokensPerRequest({ requests: 0, tokens: 0 }), null);
assert.equal(classify({ requests: 0, tokens: 0 },
{ requests: 40000, tokens: 90000000 })[0], 'new-workload');
assert.equal(classify({ requests: 10, tokens: 900 },
{ requests: 12, tokens: 1000 })[0], 'too-little-traffic');
});
test('too few hours reports no concentration rather than a wrong one', () => {
const short = Array.from({ length: 6 },
(_, i) => ({ start: CUTOFF + i * HOUR, requests: 10, tokens: 1 }));
assert.equal(burstiness(short, CUTOFF), null);
assert.equal(burstiness([], CUTOFF), null);
const [state, detail] = classify({ requests: 100000, tokens: 500000000 },
{ requests: 400000, tokens: 520000000 });
assert.equal(state, 'retry-storm');
assert.match(detail, /Too few hourly buckets/);
});
test('the request bucket is full while the token bucket is empty', () => {
const payload = { data: [
{ model: 'gpt-5.1', max_requests_per_1_minute: 10000,
max_tokens_per_1_minute: 20000000 },
{ model: 'gpt-5', max_requests_per_1_minute: 1, max_tokens_per_1_minute: 1 },
] };
const limits = rateLimitValues(payload, 'gpt-5.1-2026-01-15');
assert.deepEqual(limits, { requests: 10000, tokens: 20000000 });
const [state, detail] = limiterPressure(
{ requests: 82656000, tokens: 18144000000 }, 168, limits);
assert.equal(state, 'rpm-bound-tpm-idle');
assert.match(detail, /82% of the RPM ceiling and 9% of the TPM ceiling/);
});
test('an unpublished limit is not a missing one', () => {
assert.deepEqual(rateLimitValues({ data: [] }, 'gpt-5.1'),
{ requests: null, tokens: null });
assert.equal(limiterPressure({ requests: 1 }, 24, null)[0], 'no-limits-published');
assert.equal(series([]).size, 0);
assert.equal(series(null).size, 0);
});
FAQ
How is this different from watching spend go up?
A cost check folds one series and asks what shape its change is. This one carries two series and asks whether they still agree with each other. They catch different things: a retry storm that triples your request rate might move the invoice by a few percent, because the extra attempts failed early and generated almost nothing, and a prompt that doubled in size moves the invoice sharply without touching the request count at all.
Why does the token count barely move during a storm?
Because most retried attempts never got as far as generating. A 429 or a connection reset costs you a request against the limiter and produces no tokens, so nine attempts at one logical call bill roughly the tokens of the one that succeeded. That is exactly why the request series is the sensitive one and the token series is the control.
Could this be a legitimate change in traffic shape?
Yes, and the script keeps that as its own state rather than calling it a storm. A new endpoint doing many small classifications pushes requests up faster than tokens with no amplification anywhere. The weekly ratios cannot separate the two cases, because the drop in tokens per request is the growth divergence restated. What separates them is concentration: the script measures how much of the recent surplus landed in the busiest tenth of the hours, and even traffic puts about a tenth of it there while retries put most of it there.
Should I raise the rate limit?
Not first. A higher RPM absorbs the amplification for a while and leaves the multiplier in place, so the next incident is larger and arrives with less warning. Collapse the retry layers to one, verify the request series comes back down toward the token series, and then decide whether the ceiling was ever the problem.
Does Anthropic support the same check?
Not directly. Anthropic's messages usage report carries token sums and no request-count field at all, so there is no request series to compare a token series against. The nearest equivalent on that side is the overload note, which reconstructs the missing count from your own attempt counter instead.
Related field notes
- Attempts the platform never served, clustered by minute
- The shape of a change in spend: a spike, a step or a ramp
- How much request and token headroom is actually left
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Completions usage — OpenAI API reference
- Rate limits — OpenAI API
- Project rate limits — OpenAI API reference
- Administration and the Admin APIs — OpenAI developer docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.