Diagnostic LLM APIs
x-ratelimit-remaining sits near zero before any 429
Nothing has failed. There is no incident, no 429 in the logs, no page. There is a number that says you are running on four percent of your token budget at four in the afternoon, and it arrives attached to every successful response your application has ever received, and no code you own has ever looked at it.
Make one cheap real call and read the headers off it. GET /v1/models with a project key set to Read Only returns x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-reset-requests and the matching -tokens triple. The finding is remaining / limit, and anything under about 0.2 at steady state is a 429 waiting for a traffic spike.
Read both triples, not one. Token headroom and request headroom empty independently: a workload of many tiny calls runs out of requests with most of its tokens unspent, and a workload of few enormous calls does the reverse. Only the scarcer of the two is worth quoting.
If x-ratelimit-limit-project-tokens comes back as well, the project ceiling is lower than the organization ceiling and is the number that actually binds. If none of the headers come back at all, something between you and OpenAI is stripping them, and that is the finding.
The problem in plain words
OpenAI does not have a quota endpoint. There is no GET anywhere in the API that answers "how much of my rate limit is left", and the dashboard shows you what you have already spent rather than what you have left in the current window. The only forward-looking signal the platform emits is a set of response headers, and they arrive on successful responses — which is to say, on exactly the responses nobody inspects.
So the entire signal is discarded by construction. The SDK parses the body and hands you a typed object; the headers are on the raw response, one attribute deeper than anything the application touches. The result is a system that has no idea it is running at ninety-six percent utilisation, and finds out on the afternoon a customer sends twice the usual traffic, when a burst that would have been absorbed a month ago comes back 429 instead. Latency creeps first, because requests queue behind a bucket that is refilling as fast as it is being drained, and creeping latency is not a signal anyone traces back to a rate limit.
Why it happens
The headroom is not stored anywhere you can query. These numbers describe the state of a token bucket at the instant your call was served, at the scope that call was made in. There is no history and no aggregate. A read-only audit sees them by making a real call and looking, which is why this script probes rather than reports — and why running it once at three in the morning tells you nothing about four in the afternoon.
The two dimensions exhaust independently, and averaging them hides the finding. A classifier making four hundred tiny calls a second is limited by requests per minute long before it is limited by tokens per minute. A summariser sending sixty-thousand-token documents is the opposite. The dimension with the least headroom is the one that will produce your 429, and it is the only one whose number belongs in the report.
A project-scoped triple means the org ceiling is not your ceiling. When x-ratelimit-limit-project-tokens is present it is a separate, usually lower limit configured on the project. Reading the org triple and concluding you have room is the specific mistake this causes: the org has room, and your project does not. Which limit is smaller is a comparison, and the script does it rather than assuming.
Absent headers are worse news than low ones. Corporate proxies, API gateways and some LLM routers strip or rewrite response headers they do not recognise. If they never reach your process, you have no forward-looking signal and no diagnostic on the 429 when it comes, because Retry-After travels the same path. A clean run with no headers is not a clean bill of health and the script does not report it as one.
This is not the billing wall. A 429 that carries insufficient_quota or credit_balance_exhausted has nothing to do with these headers and will not clear when the window resets. That is a separate note with a separate remedy, and telling them apart is the difference between waiting sixty seconds and adding credits.
The fix, as a flow
There is no endpoint to ask. The only forward looking number OpenAI emits rides on responses that succeeded, which is to say on the ones nobody opens. The fix makes one cheap real call, reads both triples, and then refuses to average two buckets that empty independently.
How to fix it
Probe with the cheapest real call there is
GET /v1/models returns the model list and consumes no inference quota, and it carries the same x-ratelimit-* header set as a completion. A Read Only project key is enough. Do not manufacture a 429 to see what one looks like — deliberately draining a bucket on a production key is an outage you caused for a diagram.
Parse the triple, including the reset
Each dimension is three headers: -limit-, -remaining- and -reset-. The reset is a Go-style duration string, so 6m0s and 500ms and 1h2m3s are all valid and none of them is an integer. A parser that calls int() on it throws away the one number that tells you how long the pressure lasts.
Compute the ratio and name the scarcest dimension
remaining / limit per dimension, then report the minimum. Quoting the healthy dimension alongside the scarce one reads like reassurance; the report should say "tokens, four percent" and stop. Twenty percent is a reasonable floor to alert on, and it is a floor about your traffic shape rather than a rule.
Compare the project ceiling against the organization ceiling
If the project-scoped headers are present, put the two limits side by side. The smaller one is the real constraint and the larger one is decoration. This is also how a staging project's throttle gets discovered after it followed the project id into production.
Print the repair; do not apply it
Three repairs, and the script prints whichever fits: request a tier increase, pace the client with a token bucket sized to x-ratelimit-limit-tokens so bursts are spread rather than rejected, or raise the project's limit through the Admin API. The last of those is a write call against a limit your colleagues share, and an audit script holding a key is not who should make that decision.
How to check it worked
Run it on a schedule at your busiest hour rather than once. Headroom is a property of the moment, and a single sample at midnight is a number about midnight.
python3 openai_rate_limit_headroom.py
# near-exhaustion tokens 8000 of 200000 left (4%), resets in 47s
# binding dimension: tokens, at 4% of its ceiling
# note: project ceiling binds for tokens (150000 against an org 200000)
# 4 dimension(s) read, 1 finding(s)
The full code
One GET and nothing else. It wants OPENAI_API_KEY, a project key set to Read Only, and it never touches /v1/organization at all — the headers are project- and org-scoped by whatever the key is. Six pure functions carry every judgement: two parsers for the formats the headers actually use, the triple reader that is case-insensitive because gateways rewrite header casing, the ratio, the verdict, and the comparison that decides whether the project or the organization owns your real ceiling.
"""Report how much OpenAI rate-limit headroom is left, before anything 429s.
Read only. One GET request and nothing else: OPENAI_API_KEY should be a project
key set to Read Only. GET /v1/models consumes no inference quota and carries the
same x-ratelimit-* header set as a completion, which is the whole trick, because
OpenAI has no endpoint that returns remaining quota on request.
The repair is printed, never performed. Raising a project rate limit is a write
call against a ceiling your colleagues share.
This script never tries to provoke a 429. Draining a production token bucket to
see what the error looks like is an outage you caused on purpose.
"""
import argparse
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_rate_limit_headroom")
API = "https://api.openai.com/v1"
# The dimensions OpenAI reports. The project-scoped pair is present only when the
# project carries its own limit, and when it is present it is usually the lower
# of the two, which makes it the one that actually binds.
DIMENSIONS = ("requests", "tokens", "project-requests", "project-tokens")
# The reset headers are Go duration strings: "6m0s", "500ms", "1h2m3s". Ordered
# so that "ms" is tried before "m", because the other way round parses 500ms as
# 500 minutes and reports eight hours of pressure that does not exist.
_DURATION = re.compile(r"(\d+(?:\.\d+)?)(ms|us|ns|h|m|s)")
_UNITS = {"ns": 1e-9, "us": 1e-6, "ms": 1e-3, "s": 1.0, "m": 60.0, "h": 3600.0}
FINDINGS = ("exhausted", "near-exhaustion")
def header_names(dimension):
"""The limit/remaining/reset header triple for one dimension. Pure."""
return ("x-ratelimit-limit-" + dimension,
"x-ratelimit-remaining-" + dimension,
"x-ratelimit-reset-" + dimension)
def parse_count(value):
"""Read a limit or remaining header as an integer. Pure.
Returns None rather than zero when the value is missing or unreadable.
Zero is a real and important state here: it means the bucket is empty. A
parser that folds "absent" into "empty" reports a stripped header as an
exhausted limiter and sends somebody to the wrong console.
"""
if value is None:
return None
text = str(value).strip().replace(",", "").replace("_", "")
if not text:
return None
try:
return int(text)
except ValueError:
pass
try:
return int(float(text))
except ValueError:
return None
def parse_reset(value):
"""Read a reset header as seconds. Pure. Returns None if unreadable.
The whole string has to be consumed. A partial match on something like
"60 seconds" would return 60.0 from a format this parser does not actually
understand, and a reset window is exactly the number a reader will act on.
"""
if value is None:
return None
text = str(value).strip().lower()
if not text:
return None
parts = _DURATION.findall(text)
if not parts:
return None
if "".join(a + b for a, b in parts) != text:
return None
return sum(float(a) * _UNITS[b] for a, b in parts)
def triples(headers):
"""Parse the x-ratelimit-* triples off one response. Pure.
Matched case-insensitively because gateways and proxies rewrite header
casing freely, and a dict keyed on the exact casing OpenAI sends is how a
working probe starts reporting no headers the day a load balancer changes.
"""
lower = {}
for name, value in dict(headers or {}).items():
lower[str(name).strip().lower()] = value
out = {}
for dimension in DIMENSIONS:
limit_h, remaining_h, reset_h = header_names(dimension)
if limit_h not in lower and remaining_h not in lower:
continue
out[dimension] = {"limit": parse_count(lower.get(limit_h)),
"remaining": parse_count(lower.get(remaining_h)),
"reset": parse_reset(lower.get(reset_h))}
return out
def headroom(triple):
"""remaining / limit for one dimension, or None if it cannot be computed. Pure."""
if not isinstance(triple, dict):
return None
limit = triple.get("limit")
remaining = triple.get("remaining")
if limit is None or remaining is None or limit <= 0:
return None
return max(0.0, min(1.0, remaining / float(limit)))
def verdict(dimension, triple, floor=0.2):
"""Classify one dimension. Pure. Returns (state, detail)."""
share = headroom(triple)
if share is None:
return ("unreadable",
"the %s triple arrived without a usable limit and remaining "
"pair, so there is no ratio to read" % dimension)
remaining = triple.get("remaining")
limit = triple.get("limit")
reset = triple.get("reset")
window = ("resets in %.0fs" % reset) if reset is not None else "no readable reset"
shape = "%d of %d left (%.0f%%), %s" % (remaining, limit, share * 100, window)
if remaining == 0:
return ("exhausted",
"%s. This bucket is empty now, so the next call in this window "
"is a 429 no matter how small it is." % shape)
if share < floor:
return ("near-exhaustion",
"%s. Under the %.0f%% floor, which means the next traffic spike "
"converts this into a 429." % (shape, floor * 100))
return ("headroom", shape + ".")
def binding(parsed):
"""The dimension with the least headroom left. Pure. Returns (name, share).
Token headroom and request headroom empty independently, so the mean of the
two is a number about nothing. The minimum is the one that produces the 429
and the only one worth putting in a report.
"""
best = None
for dimension in sorted(parsed or {}):
share = headroom(parsed[dimension])
if share is None:
continue
if best is None or share < best[1]:
best = (dimension, share)
return best
def scope_note(parsed):
"""Which scope owns the real ceiling, per dimension. Pure.
Returns a list of (owner, dimension, binding_limit, other_limit). The
project-scoped headers are present only when the project carries its own
limit; when it is lower than the organization's, reading the org triple and
concluding there is room is the exact mistake this function exists to stop.
"""
out = []
for dimension in ("requests", "tokens"):
org = (parsed or {}).get(dimension) or {}
project = (parsed or {}).get("project-" + dimension) or {}
org_limit = org.get("limit")
project_limit = project.get("limit")
if org_limit is None or project_limit is None:
continue
if project_limit < org_limit:
out.append(("project", dimension, project_limit, org_limit))
elif org_limit < project_limit:
out.append(("organization", dimension, org_limit, project_limit))
else:
out.append(("equal", dimension, project_limit, org_limit))
return out
def probe(session):
"""One cheap real call. GET only, and it consumes no inference quota."""
r = session.get(API + "/models", timeout=30)
if r.status_code == 401:
raise SystemExit("401 from OpenAI: OPENAI_API_KEY is not a valid key")
if r.status_code == 429:
# Worth saying plainly: the headers are still on this response, and a
# 429 here is about the model-list endpoint, not about inference.
log.warning("the probe itself was rate limited; the headers below "
"describe the bucket that rejected it")
return r.headers
r.raise_for_status()
return r.headers
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--floor", type=float, default=0.2,
help="headroom share below which a dimension is a finding "
"(default 0.2)")
ap.add_argument("--show-all", action="store_true",
help="also print dimensions with plenty of headroom")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY (a project key set to Read Only)")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
parsed = triples(probe(session))
if not parsed:
log.warning("headers-missing no x-ratelimit-* headers reached this "
"process at all")
log.warning(" This is not a clean bill of health. Something between "
"you and OpenAI is stripping response headers, so you have "
"no forward-looking signal and no Retry-After on the 429 "
"when it arrives.")
log.warning(" repair: check the proxy, gateway or LLM router in front "
"of api.openai.com and allow the x-ratelimit-* and "
"retry-after headers through unmodified")
return 1
checked = 0
bad = 0
for dimension in sorted(parsed):
state, detail = verdict(dimension, parsed[dimension], args.floor)
checked += 1
line = "%-16s %-18s %s" % (state, dimension, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
elif state == "unreadable":
log.warning(line)
elif args.show_all:
log.info(line)
scarcest = binding(parsed)
if scarcest:
log.info("binding dimension: %s, at %.0f%% of its ceiling",
scarcest[0], scarcest[1] * 100)
for owner, dimension, low, high in scope_note(parsed):
if owner == "project":
log.warning(" note: the project ceiling binds for %s (%d against "
"an org %d), so org headroom is not your headroom",
dimension, low, high)
elif owner == "organization":
log.info(" note: the org ceiling binds for %s (%d against a "
"project %d)", dimension, low, high)
if bad:
log.warning(" repair: request a usage tier increase, or pace the "
"client with a token bucket sized to the limit above so "
"bursts are spread across the window instead of rejected")
log.warning(" repair: to raise the project ceiling instead, an admin "
"can call POST /v1/organization/projects/{project_id}"
"/rate_limits/{rate_limit_id}. That is a write against a "
"limit your colleagues share, so it is printed, not run.")
log.info("%d dimension(s) read, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report how much OpenAI rate-limit headroom is left, before anything 429s.
*
* Read only. One GET request and nothing else: OPENAI_API_KEY should be a
* project key set to Read Only. GET /v1/models consumes no inference quota and
* carries the same x-ratelimit-* header set as a completion, which is the whole
* trick, because OpenAI has no endpoint that returns remaining quota.
*
* The repair is printed, never performed, and the script never tries to provoke
* a 429.
*/
const API = 'https://api.openai.com/v1';
const DIMENSIONS = ['requests', 'tokens', 'project-requests', 'project-tokens'];
// "ms" before "m", because the other order parses 500ms as 500 minutes.
const DURATION = /(\d+(?:\.\d+)?)(ms|us|ns|h|m|s)/g;
const UNITS = { ns: 1e-9, us: 1e-6, ms: 1e-3, s: 1, m: 60, h: 3600 };
const FINDINGS = new Set(['exhausted', 'near-exhaustion']);
/** The limit/remaining/reset header triple for one dimension. Pure. */
export function headerNames(dimension) {
return [`x-ratelimit-limit-${dimension}`,
`x-ratelimit-remaining-${dimension}`,
`x-ratelimit-reset-${dimension}`];
}
/**
* Read a limit or remaining header as an integer. Pure.
* Returns null rather than 0 when absent: zero is a real state here and means
* the bucket is empty, so folding the two reports a stripped header as an
* exhausted limiter.
*/
export function parseCount(value) {
if (value === null || value === undefined) return null;
const text = String(value).trim().replace(/[,_]/g, '');
if (!text) return null;
const n = Number(text);
return Number.isFinite(n) ? Math.trunc(n) : null;
}
/**
* Read a reset header as seconds. Pure. Returns null if unreadable.
* The whole string must be consumed; a partial match on "60 seconds" would
* return a number from a format this parser does not understand.
*/
export function parseReset(value) {
if (value === null || value === undefined) return null;
const text = String(value).trim().toLowerCase();
if (!text) return null;
const parts = [...text.matchAll(DURATION)];
if (parts.length === 0) return null;
if (parts.map(([whole]) => whole).join('') !== text) return null;
return parts.reduce((sum, [, n, unit]) => sum + Number(n) * UNITS[unit], 0);
}
/**
* Parse the x-ratelimit-* triples off one response. Pure.
* Case-insensitive, because gateways rewrite header casing freely.
*/
export function triples(headers) {
const lower = new Map();
const entries = typeof headers?.entries === 'function'
? [...headers.entries()] : Object.entries(headers ?? {});
for (const [name, value] of entries) lower.set(String(name).trim().toLowerCase(), value);
const out = {};
for (const dimension of DIMENSIONS) {
const [limitH, remainingH, resetH] = headerNames(dimension);
if (!lower.has(limitH) && !lower.has(remainingH)) continue;
out[dimension] = {
limit: parseCount(lower.get(limitH)),
remaining: parseCount(lower.get(remainingH)),
reset: parseReset(lower.get(resetH)),
};
}
return out;
}
/** remaining / limit for one dimension, or null. Pure. */
export function headroom(triple) {
if (!triple || typeof triple !== 'object') return null;
const { limit, remaining } = triple;
if (limit === null || limit === undefined) return null;
if (remaining === null || remaining === undefined) return null;
if (limit <= 0) return null;
return Math.max(0, Math.min(1, remaining / limit));
}
/** Classify one dimension. Pure. Returns [state, detail]. */
export function verdict(dimension, triple, floor = 0.2) {
const share = headroom(triple);
if (share === null) {
return ['unreadable',
`the ${dimension} triple arrived without a usable limit and remaining ` +
'pair, so there is no ratio to read'];
}
const { remaining, limit, reset } = triple;
const window = reset === null || reset === undefined
? 'no readable reset' : `resets in ${reset.toFixed(0)}s`;
const shape = `${remaining} of ${limit} left (${(share * 100).toFixed(0)}%), ${window}`;
if (remaining === 0) {
return ['exhausted',
`${shape}. This bucket is empty now, so the next call in this window is ` +
'a 429 no matter how small it is.'];
}
if (share < floor) {
return ['near-exhaustion',
`${shape}. Under the ${(floor * 100).toFixed(0)}% floor, which means the ` +
'next traffic spike converts this into a 429.'];
}
return ['headroom', `${shape}.`];
}
/**
* The dimension with the least headroom left. Pure. Returns [name, share].
* The mean of two independently emptying buckets is a number about nothing.
*/
export function binding(parsed) {
let best = null;
for (const dimension of Object.keys(parsed ?? {}).sort()) {
const share = headroom(parsed[dimension]);
if (share === null) continue;
if (best === null || share < best[1]) best = [dimension, share];
}
return best;
}
/**
* Which scope owns the real ceiling, per dimension. Pure.
* Returns [owner, dimension, bindingLimit, otherLimit] rows.
*/
export function scopeNote(parsed) {
const out = [];
for (const dimension of ['requests', 'tokens']) {
const orgLimit = (parsed ?? {})[dimension]?.limit;
const projectLimit = (parsed ?? {})[`project-${dimension}`]?.limit;
if (orgLimit === null || orgLimit === undefined) continue;
if (projectLimit === null || projectLimit === undefined) continue;
if (projectLimit < orgLimit) out.push(['project', dimension, projectLimit, orgLimit]);
else if (orgLimit < projectLimit) out.push(['organization', dimension, orgLimit, projectLimit]);
else out.push(['equal', dimension, projectLimit, orgLimit]);
}
return out;
}
/** One cheap real call. GET only, and it consumes no inference quota. */
async function probe(key) {
const res = await fetch(`${API}/models`, {
headers: { Authorization: `Bearer ${key}` },
});
if (res.status === 401) throw new Error('401 from OpenAI: OPENAI_API_KEY is not a valid key');
if (res.status === 429) {
console.warn('the probe itself was rate limited; the headers below describe ' +
'the bucket that rejected it');
return res.headers;
}
if (!res.ok) throw new Error(`${res.status} from /v1/models`);
return res.headers;
}
async function main() {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY (a project key set to Read Only)');
process.exitCode = 2;
return;
}
const floor = Number(process.env.FLOOR ?? 0.2);
const showAll = process.env.SHOW_ALL === '1';
const parsed = triples(await probe(key));
if (Object.keys(parsed).length === 0) {
console.warn('headers-missing no x-ratelimit-* headers reached this process at all');
console.warn(' This is not a clean bill of health. Something between you and ' +
'OpenAI is stripping response headers, so you have no forward-looking ' +
'signal and no Retry-After on the 429 when it arrives.');
console.warn(' repair: check the proxy, gateway or LLM router in front of ' +
'api.openai.com and allow the x-ratelimit-* and retry-after headers ' +
'through unmodified');
process.exitCode = 1;
return;
}
let checked = 0;
let bad = 0;
for (const dimension of Object.keys(parsed).sort()) {
const [state, detail] = verdict(dimension, parsed[dimension], floor);
checked += 1;
const line = `${state.padEnd(16)} ${dimension.padEnd(18)} ${detail}`;
if (FINDINGS.has(state)) { bad += 1; console.warn(line); }
else if (state === 'unreadable') console.warn(line);
else if (showAll) console.log(line);
}
const scarcest = binding(parsed);
if (scarcest) {
console.log(`binding dimension: ${scarcest[0]}, at ` +
`${(scarcest[1] * 100).toFixed(0)}% of its ceiling`);
}
for (const [owner, dimension, low, high] of scopeNote(parsed)) {
if (owner === 'project') {
console.warn(` note: the project ceiling binds for ${dimension} (${low} ` +
`against an org ${high}), so org headroom is not your headroom`);
} else if (owner === 'organization') {
console.log(` note: the org ceiling binds for ${dimension} (${low} against ` +
`a project ${high})`);
}
}
if (bad) {
console.warn(' repair: request a usage tier increase, or pace the client with ' +
'a token bucket sized to the limit above so bursts are spread ' +
'across the window instead of rejected');
console.warn(' repair: to raise the project ceiling instead, an admin can call ' +
'POST /v1/organization/projects/{project_id}/rate_limits/{rate_limit_id}. ' +
'That is a write against a limit your colleagues share, so it is ' +
'printed, not run.');
}
console.log(`${checked} dimension(s) read, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is that four percent of token headroom is a finding while ninety-one percent of request headroom sits next to it — the two buckets empty independently and reporting the healthy one is how the finding gets lost. The rest pin the parsers, because the formats are not what a first draft assumes: the reset header is a Go duration string where 500ms must not become five hundred minutes, an absent header must stay absent rather than becoming a zero that reads as an empty bucket, and a probe that arrives with no headers at all has to come back as its own state instead of a clean bill of health.
from openai_rate_limit_headroom import (binding, headroom, parse_count,
parse_reset, scope_note, triples,
verdict)
def test_token_headroom_is_the_finding_while_requests_look_fine():
# The note in one assertion: 4% of tokens left, 91% of requests, and the
# report has to name tokens. An average of the two says 47% and says it
# about nothing. The mixed casing is deliberate: gateways rewrite it.
parsed = triples({
"X-RateLimit-Limit-Requests": "10000",
"X-RateLimit-Remaining-Requests": "9100",
"X-RateLimit-Reset-Requests": "6m0s",
"x-ratelimit-limit-tokens": "200000",
"x-ratelimit-remaining-tokens": "8000",
"x-ratelimit-reset-tokens": "47s",
})
assert verdict("requests", parsed["requests"])[0] == "headroom"
state, detail = verdict("tokens", parsed["tokens"])
assert state == "near-exhaustion"
assert "8000 of 200000 left (4%), resets in 47s" in detail
assert binding(parsed) == ("tokens", 0.04)
def test_an_empty_bucket_is_reported_before_any_429_arrives():
state, detail = verdict("tokens", {"limit": 200000, "remaining": 0, "reset": 12.0})
assert state == "exhausted"
assert "empty now" in detail
def test_absent_headers_are_not_an_empty_bucket():
# parse_count must distinguish "the gateway stripped it" from "you are out".
assert parse_count(None) is None
assert parse_count("") is None
assert parse_count("0") == 0
assert parse_count("1,500,000") == 1500000
assert parse_count("not a number") is None
assert headroom({"limit": 200000, "remaining": None}) is None
assert verdict("tokens", {"limit": 200000, "remaining": None})[0] == "unreadable"
def test_go_duration_resets_parse_and_ms_is_not_minutes():
assert parse_reset("500ms") == 0.5
assert parse_reset("6m0s") == 360.0
assert parse_reset("1h2m3s") == 3723.0
assert parse_reset("47s") == 47.0
# Formats this parser does not understand must not half-parse into a number
# a reader would then act on.
assert parse_reset("60 seconds") is None
assert parse_reset("soon") is None
assert parse_reset("") is None
assert parse_reset(None) is None
def test_a_probe_with_no_rate_limit_headers_parses_to_nothing():
# Which main() reports as its own finding rather than as a clean run.
assert triples({"content-type": "application/json"}) == {}
assert triples({}) == {}
assert triples(None) == {}
assert binding({}) is None
def test_the_project_ceiling_is_the_real_ceiling_when_it_is_lower():
parsed = triples({
"x-ratelimit-limit-tokens": "200000",
"x-ratelimit-remaining-tokens": "150000",
"x-ratelimit-limit-project-tokens": "150000",
"x-ratelimit-remaining-project-tokens": "12000",
"x-ratelimit-reset-project-tokens": "30s",
})
assert scope_note(parsed) == [("project", "tokens", 150000, 200000)]
# And the org triple, read alone, would have said everything was fine.
assert verdict("tokens", parsed["tokens"])[0] == "headroom"
assert verdict("project-tokens", parsed["project-tokens"])[0] == "near-exhaustion"
assert binding(parsed)[0] == "project-tokens"
def test_scope_note_says_nothing_when_there_is_nothing_to_compare():
assert scope_note(triples({
"x-ratelimit-limit-tokens": "200000",
"x-ratelimit-remaining-tokens": "150000",
})) == []
assert scope_note({}) == []
assert scope_note(None) == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { binding, headroom, parseCount, parseReset, scopeNote, triples, verdict }
from './openai-rate-limit-headroom.mjs';
test('token headroom is the finding while requests look fine', () => {
const parsed = triples({
'X-RateLimit-Limit-Requests': '10000',
'X-RateLimit-Remaining-Requests': '9100',
'X-RateLimit-Reset-Requests': '6m0s',
'x-ratelimit-limit-tokens': '200000',
'x-ratelimit-remaining-tokens': '8000',
'x-ratelimit-reset-tokens': '47s',
});
assert.equal(verdict('requests', parsed.requests)[0], 'headroom');
const [state, detail] = verdict('tokens', parsed.tokens);
assert.equal(state, 'near-exhaustion');
assert.match(detail, /8000 of 200000 left \(4%\), resets in 47s/);
assert.deepEqual(binding(parsed), ['tokens', 0.04]);
});
test('an empty bucket is reported before any 429 arrives', () => {
const [state, detail] = verdict('tokens', { limit: 200000, remaining: 0, reset: 12 });
assert.equal(state, 'exhausted');
assert.match(detail, /empty now/);
});
test('absent headers are not an empty bucket', () => {
assert.equal(parseCount(null), null);
assert.equal(parseCount(''), null);
assert.equal(parseCount('0'), 0);
assert.equal(parseCount('1,500,000'), 1500000);
assert.equal(parseCount('not a number'), null);
assert.equal(headroom({ limit: 200000, remaining: null }), null);
assert.equal(verdict('tokens', { limit: 200000, remaining: null })[0], 'unreadable');
});
test('go duration resets parse and ms is not minutes', () => {
assert.equal(parseReset('500ms'), 0.5);
assert.equal(parseReset('6m0s'), 360);
assert.equal(parseReset('1h2m3s'), 3723);
assert.equal(parseReset('47s'), 47);
assert.equal(parseReset('60 seconds'), null);
assert.equal(parseReset('soon'), null);
assert.equal(parseReset(''), null);
assert.equal(parseReset(null), null);
});
test('a probe with no rate limit headers parses to nothing', () => {
assert.deepEqual(triples({ 'content-type': 'application/json' }), {});
assert.deepEqual(triples({}), {});
assert.deepEqual(triples(null), {});
assert.equal(binding({}), null);
});
test('a fetch Headers object is read the same way as a plain object', () => {
const h = new Headers({ 'x-ratelimit-limit-tokens': '100',
'x-ratelimit-remaining-tokens': '5' });
assert.equal(verdict('tokens', triples(h).tokens)[0], 'near-exhaustion');
});
test('the project ceiling is the real ceiling when it is lower', () => {
const parsed = triples({
'x-ratelimit-limit-tokens': '200000',
'x-ratelimit-remaining-tokens': '150000',
'x-ratelimit-limit-project-tokens': '150000',
'x-ratelimit-remaining-project-tokens': '12000',
'x-ratelimit-reset-project-tokens': '30s',
});
assert.deepEqual(scopeNote(parsed), [['project', 'tokens', 150000, 200000]]);
assert.equal(verdict('tokens', parsed.tokens)[0], 'headroom');
assert.equal(verdict('project-tokens', parsed['project-tokens'])[0], 'near-exhaustion');
assert.equal(binding(parsed)[0], 'project-tokens');
});
test('scopeNote says nothing when there is nothing to compare', () => {
assert.deepEqual(scopeNote(triples({
'x-ratelimit-limit-tokens': '200000',
'x-ratelimit-remaining-tokens': '150000',
})), []);
assert.deepEqual(scopeNote({}), []);
assert.deepEqual(scopeNote(null), []);
});
FAQ
Does the probe itself use up rate limit?
It consumes a request against the model-list endpoint's limiter, and no inference quota at all: no tokens are generated and nothing is billed. That is why GET /v1/models is the probe rather than a one-token completion. Run it once a minute at most and it is invisible; run it in a tight loop and you are the traffic spike you were trying to predict.
Why not just catch the 429 and read the headers off that?
Because by then it has happened. The headers come back on successful responses too, which is the entire point of this note: the signal that would have told you a fortnight ago is attached to every 200 you have ever received. Logging them on the error path only is a system that can explain outages and never prevent them.
What is a sensible floor?
Twenty percent is a starting point, not a rule. The right number is a function of how spiky your traffic is: a workload whose peak is three times its mean needs far more headroom than one that runs flat all day. Watch the ratio at your busiest hour for a week and set the floor where a normal spike would still fit underneath it.
The headers are missing entirely. Is that good or bad?
Bad, and worth chasing before anything else here. Corporate proxies, API gateways and LLM routers strip response headers they do not recognise, and Retry-After travels the same path as x-ratelimit-*. If they are not reaching your process you have no forward-looking signal and no correct backoff either, so a 429 becomes a guess in both directions.
Does Anthropic work the same way?
The mechanism is the same and the shape is different. Anthropic returns four header triples rather than two, and it also publishes the configured limits on a GET, so you can read the ceilings without probing at all. Naming which of its three limiters emptied is a separate exercise, covered in the sibling note on limiter identification.
Related field notes
- Naming which of the three limiters actually emptied
- A 429 that is a billing wall rather than a throttle
- Nothing in the platform stops a runaway bill
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits — OpenAI API
- Models — OpenAI API reference
- Project rate limits — OpenAI API reference
- Administration and the Admin APIs — OpenAI developer docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.