Diagnostic LLM APIs
The gateway strips the header your backoff depends on
The backoff was reviewed, it is textbook, and it reads retry-after before it sleeps. It has also never once seen that header, because the reverse proxy in front of the API forwards a response header allowlist that somebody wrote in 2023 and it contains content-type. So the handler falls through to its default of one second, retries into a bucket that has not refilled, and every retry pushes the reset further out while the incident channel fills up with people saying the backoff must be broken.
retry-after only exists on a 429, so a script that refuses to cause one can never photograph it. What it can do is probe the header class that arrives on every response and prove that class survives the network path your application actually uses.
One GET /v1/models issued twice: once straight at the provider, once through the base URL your client is configured with. Then diff the headers. On Anthropic the always-present set is anthropic-ratelimit-requests-limit, -remaining and -reset, the matching input-token and output-token triples, and the aggregate anthropic-ratelimit-tokens-* triple. On OpenAI it is x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-reset-requests and the token equivalents. A path that drops those on a 200 will drop retry-after on a 429, and no amount of correct backoff code survives that.
Compare only the -limit- values across the two paths. -remaining and -reset are supposed to differ between two calls made a second apart; a limit that differs means something in the middle is inventing headers rather than forwarding them, which is worse than stripping because it looks like it is working.
Then check the clock. Anthropic's reset values are RFC 3339 timestamps, so a client that sleeps until that instant is trusting its own clock to agree with the server's; the script compares the server's date header against local time and reports the skew. OpenAI's reset values are durations, which are immune to this, and the asymmetry is worth knowing before you write one backoff for both.
What this note is not: it never grades how much headroom is left, and it never names which limiter emptied. One published note reads those values for headroom and another reads them to name the bucket. This one reads whether they arrive.
The problem in plain words
The documentation is unusually blunt about what retry-after is for. It is the number of seconds to wait until you can retry, and earlier retries will fail. The bucket refills continuously, so a retry sent before the stated delay is not a gamble that might pay off; it is a request that is guaranteed to be refused, and it spends a slot doing it.
So everybody writes the handler that reads the header. And then the header does not arrive, for reasons that have nothing to do with the handler. Corporate egress proxies strip unknown response headers. API gateways forward an allowlist. Service meshes normalise. CDN and WAF layers rewrite. Every one of those is a sensible default somewhere else, and every one of them turns a precise wait instruction into a missing key and a fallback constant.
The fallback is where it gets expensive. A missing header means sleep(1), which means a retry into an empty bucket, which means another 429, which means another slot burned and the reset pushed out again. From the outside this looks exactly like a rate limit that will not clear, and the usual response is to add more retries.
The second failure is quieter and only affects one of the two providers. Anthropic's reset headers are absolute timestamps. A client that computes its sleep from one of those is combining a server timestamp with a local clock, and if the two disagree by thirty seconds the sleep is thirty seconds wrong in whichever direction hurts. Containers with no time sync, virtual machines resuming from suspend, and anything behind a captive NTP source all do this, and it is invisible to code review because the code is correct.
And one case is not a transport failure at all: the 429 you get when the organization has crossed its monthly spend cap carries no retry-after, deliberately, because there is nothing to wait for. A client that treats a missing header as a proxy problem will retry that one forever.
Why it happens
The header you care about cannot be observed without causing the failure, so probe its class instead. This is the whole design. retry-after appears on 429 responses; the rate-limit triples appear on every response including successful ones. They are the same class of header, added by the same layer, and stripped or forwarded by the same middlebox rules. If the triples arrive intact on a 200, the wait instruction will arrive on a 429. Using the healthy call as a canary for the unhealthy one costs one GET and no capacity.
Two paths, because one path cannot tell you whose fault it is. A single probe that comes back with no rate-limit headers is genuinely ambiguous: it could be the provider, the endpoint, or three hops of your own infrastructure. Issuing the identical call directly and through the configured base URL turns that into an answer. Headers present direct and absent through the gateway is your gateway, in one line, with no argument possible.
Only the limit values are comparable. The two probes happen at different instants, so -remaining and -reset are expected to differ and comparing them would produce a permanent false positive. -limit- values are configuration and do not move between two calls a second apart. When they do differ across paths, something is synthesising headers rather than forwarding them — a caching layer replaying a stale response, or a gateway helpfully making up numbers — and that is a worse state than stripping, because the client believes what it is told.
Absolute resets and duration resets fail differently, and the script parses both. Anthropic returns RFC 3339 instants; OpenAI returns durations like 6m0s. A duration is skew-proof. An instant is only as good as the agreement between two clocks, so the script reads the server's own date header, compares it against local time, and reports the difference. A reset that has already passed according to the server is a separate finding again: any client sleeping until that instant sleeps for nothing.
It will not provoke a 429 to look at one. Driving traffic into a limiter to photograph its error is not a diagnostic. On a saturated organization it is a second outage; on a healthy one it spends capacity that belongs to production, and the thing it proves was already provable from a single successful call. If a 429 does arrive on its own during the probe, the script records the retry-after it carried, reports it as directly observed, and does not retry.
The fix, as a flow
The only chain in the batch with a middlebox in it, and the fault is drawn at the hop rather than at either end, because both ends are behaving correctly. The API sends the wait instruction and the client reads it; the allowlist between them was written before either existed. Since the header itself only appears on a 429, the probe watches the family that arrives on every response instead.
How to fix it
Point the script at the same path your application uses
Set ANTHROPIC_BASE_URL or OPENAI_BASE_URL to the gateway, proxy or mesh address your production client is configured with. Those are the same variables the official SDKs read, so if your application sets one, use the same value. Without a base URL the script still runs, but it can only check presence and clock agreement on one path, not attribute a loss to a hop.
Issue one GET /v1/models per path and keep every header
Both providers answer /v1/models with the same rate-limit header family they attach to inference calls, and it costs no tokens. Lower-case every header name before comparing: HTTP header names are case-insensitive and middleboxes rewrite their casing freely.
Diff the required set, then diff the limit values
Anything required and present on the direct path but absent through the gateway is stripped. Anything present on both is compared on its -limit- value only, because remaining and reset are supposed to move. A value that differs is a rewrite, and a header that appears only through the gateway is an invention.
Check the clock against the server's own date header
Parse the reset values. Durations need no clock. Absolute timestamps do, so compare the server's date header against local time and report the skew, and separately report any reset that is already in the past according to the server, which makes any sleep computed from it a no-op.
Read the output as a checklist for whoever owns the middlebox
The repair is a header allowlist, and the script prints the exact names to add. It changes no configuration, retries nothing, and never sends a second request to see whether the first one was unlucky.
How to check it worked
Add the header names to the gateway allowlist and re-run. Every required header should show intact on both paths, the limit values should agree, and the verdict should be headers-intact. A run that moves from headers-stripped to headers-rewritten means the allowlist now passes the names through but something upstream is still generating the values rather than forwarding them, which is the state that most looks like success and is not.
python3 retry_after_header_probe.py
# anthropic: direct api.anthropic.com, gateway llm-gw.internal
# headers-stripped 6 of 12 rate limit header(s) do not survive the gateway
# stripped anthropic-ratelimit-input-tokens-limit
# stripped anthropic-ratelimit-input-tokens-remaining
# stripped anthropic-ratelimit-input-tokens-reset
# stripped anthropic-ratelimit-output-tokens-limit ...
# intact anthropic-ratelimit-requests-limit 1000
# repair: retry-after travels with these. A path that drops them on a 200
# drops the wait instruction on a 429, and your backoff falls back
# to a constant. Add the names above to the gateway allowlist.
# clock-skew local clock is 42s behind the server's date header
# repair: anthropic reset values are RFC 3339 instants, so a sleep computed
# from one is only as good as clock agreement. Fix time sync, or
# prefer retry-after, which is relative and immune to this.
# openai: direct api.openai.com, no gateway configured
# headers-intact 6 of 6 rate limit header(s) present, resets are
# durations and need no clock
# 2 finding(s)The full code
One GET per path and every judgement in a pure function. lower_headers, because middleboxes rewrite casing; required and optional, which hold the documented header families per provider; compare, which classifies each name as intact, stripped, added or rewritten and only ever compares a -limit- value; parse_reset, which recognises an RFC 3339 instant and a Go-style duration and says which it got rather than guessing; clock_skew, which reads the server's own date header so the comparison is against the server and not against another server; and verdict, which orders the findings so a stripped header is reported before a skewed clock rather than alongside it.
"""Prove that retry-after can reach your client before you need it.
Read only, and deliberately small: one GET /v1/models per path. This script
will not drive traffic into a 429 in order to photograph one. Provoking the
failure you are investigating is not a diagnostic; on a saturated organization
it is a second outage, and on a healthy one it spends capacity that belongs to
production.
retry-after appears only on a 429, so its class is probed instead. The rate
limit triples arrive on every response, are added by the same layer, and are
forwarded or dropped by the same middlebox rules. If they survive the path on a
200, the wait instruction survives it on a 429.
Two paths, because one cannot attribute a loss: the same call straight at the
provider and through the base URL the application is configured with. Only the
-limit- values are compared, because remaining and reset are supposed to move
between two calls a second apart.
"""
import argparse
import datetime as dt
import email.utils
import logging
import os
import re
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("retry_after_header_probe")
DIRECT = {
"anthropic": "https://api.anthropic.com/v1",
"openai": "https://api.openai.com/v1",
}
# The headers documented as arriving on every response. These are the canary:
# retry-after belongs to the same family and is added by the same layer, so a
# path that keeps these keeps it too.
REQUIRED = {
"anthropic": (
"anthropic-ratelimit-requests-limit",
"anthropic-ratelimit-requests-remaining",
"anthropic-ratelimit-requests-reset",
"anthropic-ratelimit-input-tokens-limit",
"anthropic-ratelimit-input-tokens-remaining",
"anthropic-ratelimit-input-tokens-reset",
"anthropic-ratelimit-output-tokens-limit",
"anthropic-ratelimit-output-tokens-remaining",
"anthropic-ratelimit-output-tokens-reset",
"anthropic-ratelimit-tokens-limit",
"anthropic-ratelimit-tokens-remaining",
"anthropic-ratelimit-tokens-reset",
),
"openai": (
"x-ratelimit-limit-requests",
"x-ratelimit-remaining-requests",
"x-ratelimit-reset-requests",
"x-ratelimit-limit-tokens",
"x-ratelimit-remaining-tokens",
"x-ratelimit-reset-tokens",
),
}
# Present in some configurations only, so their absence is reported and never
# counted as a loss: the priority triples require a Priority Tier commitment,
# the project triples appear when a project ceiling applies, and retry-after
# itself is a 429 header that a healthy probe must not expect to see.
OPTIONAL = {
"anthropic": ("retry-after", "request-id", "anthropic-workspace-id",
"anthropic-priority-input-tokens-limit",
"anthropic-priority-output-tokens-limit"),
"openai": ("retry-after", "x-request-id",
"x-ratelimit-limit-project-tokens",
"x-ratelimit-remaining-project-tokens",
"x-ratelimit-reset-project-tokens"),
}
SKEW_SECONDS = 5.0
FINDINGS = ("headers-stripped", "headers-rewritten", "headers-absent",
"reset-in-the-past", "clock-skew")
DURATION = re.compile(r"(\d+(?:\.\d+)?)(ms|h|m|s)")
UNIT = {"ms": 0.001, "s": 1.0, "m": 60.0, "h": 3600.0}
def lower_headers(headers):
"""{lowercase name: value}. Pure.
HTTP header names are case-insensitive and middleboxes rewrite their casing
freely, so a comparison that does not normalise reports a stripped header
every time a proxy prefers title case.
"""
out = {}
for key, value in dict(headers or {}).items():
out[str(key).strip().lower()] = str(value)
return out
def missing(headers, provider):
"""Required header names absent from this response. Pure. Sorted."""
present = lower_headers(headers)
return sorted(n for n in REQUIRED.get(provider, ()) if n not in present)
def compare(direct, gateway, provider):
"""{header: (direct, gateway, state)} across two paths. Pure.
States: intact, stripped, added, rewritten, absent-both. Only -limit- values
are compared for equality. remaining and reset are supposed to differ
between two calls made a second apart, and comparing them would make every
healthy path look rewritten.
"""
left = lower_headers(direct)
right = lower_headers(gateway)
names = set(REQUIRED.get(provider, ())) | set(OPTIONAL.get(provider, ()))
names |= {n for n in list(left) + list(right)
if "ratelimit" in n or n == "retry-after"}
out = {}
for name in sorted(names):
a, b = left.get(name), right.get(name)
if a is None and b is None:
state = "absent-both"
elif a is not None and b is None:
state = "stripped"
elif a is None and b is not None:
state = "added"
elif "-limit" in name and a != b:
state = "rewritten"
else:
state = "intact"
out[name] = (a, b, state)
return out
def parse_reset(value):
"""(kind, seconds) for a reset header. Pure.
kind is "absolute" with a POSIX timestamp, "duration" with a count of
seconds, or "unknown" with None. Saying which it got matters: a duration
needs no clock and an instant needs two clocks to agree.
"""
text = str(value or "").strip()
if not text:
return ("unknown", None)
try:
stamp = dt.datetime.fromisoformat(text.replace("Z", "+00:00"))
if stamp.tzinfo is None:
stamp = stamp.replace(tzinfo=dt.timezone.utc)
return ("absolute", stamp.timestamp())
except ValueError:
pass
parts = DURATION.findall(text)
if parts and re.fullmatch(r"(?:\d+(?:\.\d+)?(?:ms|h|m|s))+", text):
return ("duration", sum(float(n) * UNIT[u] for n, u in parts))
try:
return ("duration", float(text))
except ValueError:
return ("unknown", None)
def clock_skew(date_header, local_epoch):
"""local clock minus the server's date header, in seconds. Pure.
None when the header is missing or unparseable. Compared against the
server's own clock rather than a third source, because the only agreement
that matters is between this client and the API answering it.
"""
text = str(date_header or "").strip()
if not text:
return None
try:
stamp = email.utils.parsedate_to_datetime(text)
except (TypeError, ValueError):
return None
if stamp is None:
return None
if stamp.tzinfo is None:
stamp = stamp.replace(tzinfo=dt.timezone.utc)
return float(local_epoch) - stamp.timestamp()
def stale_resets(headers, provider, server_epoch):
"""[(header, seconds_in_the_past)] for absolute resets already elapsed. Pure."""
present = lower_headers(headers)
out = []
for name in REQUIRED.get(provider, ()):
if not name.endswith("-reset"):
continue
kind, value = parse_reset(present.get(name))
if kind == "absolute" and value is not None and value < server_epoch:
out.append((name, server_epoch - value))
out.sort(key=lambda r: (-r[1], r[0]))
return out
def verdict(comparison, direct_missing, gateway_used, skew, stale):
"""Classify one provider's probe. Pure. Returns (state, detail).
Ordered so a transport failure is reported before a clock one: a stripped
header makes the clock question moot, since there is nothing to compute a
sleep from in the first place.
"""
comparison = comparison or {}
states = [s for _, _, s in comparison.values()]
stripped = [n for n, (_, _, s) in comparison.items() if s == "stripped"]
rewritten = [n for n, (_, _, s) in comparison.items() if s == "rewritten"]
total = len(REQUIRED_ANY(comparison))
if direct_missing and not gateway_used:
return ("headers-absent",
"%d required rate limit header(s) did not arrive at all, and "
"there is no gateway configured to blame for it"
% len(direct_missing))
if stripped:
return ("headers-stripped",
"%d of %d rate limit header(s) do not survive the gateway"
% (len(stripped), max(total, len(stripped))))
if rewritten:
return ("headers-rewritten",
"%d limit value(s) differ between the two paths, so something "
"is generating headers rather than forwarding them"
% len(rewritten))
if direct_missing:
return ("headers-absent",
"%d required rate limit header(s) are absent on both paths"
% len(direct_missing))
if stale:
return ("reset-in-the-past",
"%s is already %.0fs in the past by the server's own clock"
% (stale[0][0], stale[0][1]))
if skew is not None and abs(skew) > SKEW_SECONDS:
return ("clock-skew",
"local clock is %.0fs %s the server's date header"
% (abs(skew), "behind" if skew < 0 else "ahead of"))
intact = states.count("intact")
return ("headers-intact",
"%d rate limit header(s) present and consistent across every path "
"checked" % intact)
def REQUIRED_ANY(comparison):
"""The header names in a comparison that are required somewhere. Pure."""
names = set()
for provider, required in REQUIRED.items():
names |= {n for n in required if n in (comparison or {})}
return names
def repair_lines(state, provider="", names=()):
"""The repair for one verdict. Pure. Printed, never performed."""
names = list(names or [])
if state == "headers-stripped":
return ["retry-after travels with these. A path that drops them on a 200 "
"drops the wait instruction on a 429, and your backoff falls "
"back to a constant that retries into an empty bucket.",
"add these names to the response header allowlist on the "
"gateway: " + (", ".join(names[:6]) or "(none recorded)")
+ (" ..." if len(names) > 6 else "")]
if state == "headers-rewritten":
return ["a limit value that differs between two paths a second apart is "
"not a live number. Find the layer caching or synthesising "
"responses and make it forward the origin's headers unchanged.",
"this state is more dangerous than stripping, because the client "
"believes the numbers it is given and has no way to tell."]
if state == "headers-absent":
return ["nothing arrived on any path checked, so this is not attributable "
"yet. Re-run with the gateway base URL set, and confirm the "
"credential and endpoint are the ones production uses."]
if state == "reset-in-the-past":
return ["a reset instant already in the past makes any sleep computed "
"from it a no-op, so the client retries immediately and 429s "
"again. Prefer retry-after, which is relative."]
if state == "clock-skew":
if provider == "anthropic":
return ["anthropic reset values are RFC 3339 instants, so a sleep "
"computed from one is only as good as clock agreement. Fix "
"time sync on this host, or use retry-after instead, which "
"is relative and immune to skew.",
"the same skew affects any log correlation you do against "
"these timestamps, which is usually how it is finally noticed."]
return ["this provider returns reset values as durations, so backoff is "
"unaffected, but the skew will still misalign every log line you "
"correlate against the API's timestamps."]
return []
def probe(url, headers, timeout=30):
"""One GET. Returns (status, headers, note). Never retried, never repeated."""
try:
r = requests.get(url, headers=headers, timeout=timeout)
except requests.RequestException as exc:
return (None, {}, "request failed: %s" % exc)
note = ""
if r.status_code == 429:
# Not provoked, and not retried. If one happens to arrive it is the
# direct observation this script cannot go looking for.
note = ("a 429 arrived on its own. retry-after came back as %r"
% r.headers.get("retry-after"))
elif r.status_code in (401, 403):
note = "%d: the credential cannot read this path" % r.status_code
return (r.status_code, dict(r.headers), note)
def audit(provider, key, base_url):
direct_base = DIRECT[provider]
auth = ({"x-api-key": key, "anthropic-version": "2023-06-01"}
if provider == "anthropic" else {"Authorization": "Bearer " + key})
auth["User-Agent"] = "retry-after-header-probe/1.0"
log.info("%s: direct %s, %s", provider,
direct_base.split("//")[-1].split("/")[0],
"gateway " + base_url.split("//")[-1].split("/")[0]
if base_url else "no gateway configured")
status, direct_headers, note = probe(direct_base + "/models", auth)
if note:
log.info(" direct: %s", note)
gateway_headers = {}
if base_url:
time.sleep(1)
_, gateway_headers, gnote = probe(base_url.rstrip("/") + "/models", auth)
if gnote:
log.info(" gateway: %s", gnote)
comparison = compare(direct_headers, gateway_headers or direct_headers, provider)
direct_missing = missing(direct_headers, provider)
skew = clock_skew(lower_headers(direct_headers).get("date"), time.time())
server_epoch = time.time() - (skew or 0.0)
stale = stale_resets(direct_headers, provider, server_epoch)
state, detail = verdict(comparison, direct_missing, bool(base_url), skew, stale)
emit = log.warning if state in FINDINGS else log.info
emit("%-21s %s", state, detail)
stripped = [n for n, (_, _, s) in comparison.items() if s == "stripped"]
for name in stripped[:6]:
emit(" stripped %s", name)
for name, (a, _b, s) in sorted(comparison.items()):
if s == "intact" and name.endswith("-limit") and a:
emit(" intact %-42s %s", name, a)
for name, seconds in stale[:3]:
emit(" stale %s, %.0fs in the past", name, seconds)
for line in repair_lines(state, provider, stripped):
emit(" repair: %s", line)
return 1 if state in FINDINGS else 0
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--anthropic-base-url", default=os.environ.get("ANTHROPIC_BASE_URL"))
ap.add_argument("--openai-base-url", default=os.environ.get("OPENAI_BASE_URL"))
args = ap.parse_args()
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
openai_key = os.environ.get("OPENAI_API_KEY")
if not anthropic_key and not openai_key:
log.error("set ANTHROPIC_API_KEY, OPENAI_API_KEY, or both, and set the "
"matching base URL if production reaches the API through a "
"gateway")
return 2
findings = 0
if anthropic_key:
findings += audit("anthropic", anthropic_key, args.anthropic_base_url)
if openai_key:
findings += audit("openai", openai_key, args.openai_base_url)
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Prove that retry-after can reach your client before you need it.
*
* Read only, and deliberately small: one GET /v1/models per path. This script
* will not drive traffic into a 429 in order to photograph one.
*
* retry-after appears only on a 429, so its class is probed instead: the rate
* limit triples arrive on every response and are forwarded or dropped by the
* same middlebox rules. Two paths, because one cannot attribute a loss. Only
* the -limit- values are compared, because remaining and reset are supposed to
* move between two calls a second apart.
*/
const DIRECT = {
anthropic: 'https://api.anthropic.com/v1',
openai: 'https://api.openai.com/v1',
};
export const REQUIRED = {
anthropic: [
'anthropic-ratelimit-requests-limit',
'anthropic-ratelimit-requests-remaining',
'anthropic-ratelimit-requests-reset',
'anthropic-ratelimit-input-tokens-limit',
'anthropic-ratelimit-input-tokens-remaining',
'anthropic-ratelimit-input-tokens-reset',
'anthropic-ratelimit-output-tokens-limit',
'anthropic-ratelimit-output-tokens-remaining',
'anthropic-ratelimit-output-tokens-reset',
'anthropic-ratelimit-tokens-limit',
'anthropic-ratelimit-tokens-remaining',
'anthropic-ratelimit-tokens-reset',
],
openai: [
'x-ratelimit-limit-requests',
'x-ratelimit-remaining-requests',
'x-ratelimit-reset-requests',
'x-ratelimit-limit-tokens',
'x-ratelimit-remaining-tokens',
'x-ratelimit-reset-tokens',
],
};
export const OPTIONAL = {
anthropic: ['retry-after', 'request-id', 'anthropic-workspace-id',
'anthropic-priority-input-tokens-limit',
'anthropic-priority-output-tokens-limit'],
openai: ['retry-after', 'x-request-id', 'x-ratelimit-limit-project-tokens',
'x-ratelimit-remaining-project-tokens', 'x-ratelimit-reset-project-tokens'],
};
const SKEW_SECONDS = 5;
const FINDINGS = new Set(['headers-stripped', 'headers-rewritten', 'headers-absent',
'reset-in-the-past', 'clock-skew']);
const UNIT = { ms: 0.001, s: 1, m: 60, h: 3600 };
/** {lowercase name: value}. Pure. Middleboxes rewrite casing freely. */
export function lowerHeaders(headers) {
const out = {};
for (const [key, value] of Object.entries(headers ?? {})) {
out[String(key).trim().toLowerCase()] = String(value);
}
return out;
}
/** Required header names absent from this response. Pure. Sorted. */
export function missing(headers, provider) {
const present = lowerHeaders(headers);
return (REQUIRED[provider] ?? []).filter((n) => !(n in present)).sort();
}
/** {header: [direct, gateway, state]} across two paths. Pure. */
export function compare(direct, gateway, provider) {
const left = lowerHeaders(direct);
const right = lowerHeaders(gateway);
const names = new Set([...(REQUIRED[provider] ?? []), ...(OPTIONAL[provider] ?? [])]);
for (const n of [...Object.keys(left), ...Object.keys(right)]) {
if (n.includes('ratelimit') || n === 'retry-after') names.add(n);
}
const out = {};
for (const name of [...names].sort()) {
const a = left[name];
const b = right[name];
let state;
if (a === undefined && b === undefined) state = 'absent-both';
else if (a !== undefined && b === undefined) state = 'stripped';
else if (a === undefined && b !== undefined) state = 'added';
else if (name.includes('-limit') && a !== b) state = 'rewritten';
else state = 'intact';
out[name] = [a, b, state];
}
return out;
}
/** [kind, seconds] for a reset header. Pure. absolute | duration | unknown. */
export function parseReset(value) {
const text = String(value ?? '').trim();
if (!text) return ['unknown', null];
if (/^\d{4}-\d{2}-\d{2}[T ]/.test(text)) {
const ms = Date.parse(text);
if (Number.isFinite(ms)) return ['absolute', ms / 1000];
}
if (/^(?:\d+(?:\.\d+)?(?:ms|h|m|s))+$/.test(text)) {
let total = 0;
for (const [, n, u] of text.matchAll(/(\d+(?:\.\d+)?)(ms|h|m|s)/g)) {
total += Number(n) * UNIT[u];
}
return ['duration', total];
}
const plain = Number(text);
return Number.isFinite(plain) ? ['duration', plain] : ['unknown', null];
}
/** local clock minus the server's date header, in seconds. Pure. null if unknown. */
export function clockSkew(dateHeader, localEpoch) {
const text = String(dateHeader ?? '').trim();
if (!text) return null;
const ms = Date.parse(text);
if (!Number.isFinite(ms)) return null;
return Number(localEpoch) - ms / 1000;
}
/** [[header, secondsInThePast]] for absolute resets already elapsed. Pure. */
export function staleResets(headers, provider, serverEpoch) {
const present = lowerHeaders(headers);
const out = [];
for (const name of REQUIRED[provider] ?? []) {
if (!name.endsWith('-reset')) continue;
const [kind, value] = parseReset(present[name]);
if (kind === 'absolute' && value !== null && value < serverEpoch) {
out.push([name, serverEpoch - value]);
}
}
out.sort((a, b) => (b[1] - a[1]) || a[0].localeCompare(b[0]));
return out;
}
const requiredAny = (comparison) => {
const names = new Set();
for (const required of Object.values(REQUIRED)) {
for (const n of required) if (n in (comparison ?? {})) names.add(n);
}
return names;
};
/** Classify one provider's probe. Pure. Returns [state, detail]. */
export function verdict(comparison, directMissing, gatewayUsed, skew, stale) {
const rows = Object.entries(comparison ?? {});
const stripped = rows.filter(([, v]) => v[2] === 'stripped').map(([n]) => n);
const rewritten = rows.filter(([, v]) => v[2] === 'rewritten').map(([n]) => n);
const total = requiredAny(comparison).size;
const absent = (directMissing ?? []).length;
if (absent && !gatewayUsed) {
return ['headers-absent',
`${absent} required rate limit header(s) did not arrive at all, and `
+ 'there is no gateway configured to blame for it'];
}
if (stripped.length) {
return ['headers-stripped',
`${stripped.length} of ${Math.max(total, stripped.length)} rate limit `
+ 'header(s) do not survive the gateway'];
}
if (rewritten.length) {
return ['headers-rewritten',
`${rewritten.length} limit value(s) differ between the two paths, so `
+ 'something is generating headers rather than forwarding them'];
}
if (absent) {
return ['headers-absent',
`${absent} required rate limit header(s) are absent on both paths`];
}
if ((stale ?? []).length) {
return ['reset-in-the-past',
`${stale[0][0]} is already ${Math.round(stale[0][1])}s in the past by `
+ "the server's own clock"];
}
if (skew !== null && skew !== undefined && Math.abs(skew) > SKEW_SECONDS) {
return ['clock-skew',
`local clock is ${Math.round(Math.abs(skew))}s `
+ `${skew < 0 ? 'behind' : 'ahead of'} the server's date header`];
}
const intact = rows.filter(([, v]) => v[2] === 'intact').length;
return ['headers-intact',
`${intact} rate limit header(s) present and consistent across every path checked`];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, provider = '', names = []) {
const list = names ?? [];
if (state === 'headers-stripped') {
return ['retry-after travels with these. A path that drops them on a 200 drops '
+ 'the wait instruction on a 429, and your backoff falls back to a constant '
+ 'that retries into an empty bucket.',
`add these names to the response header allowlist on the gateway: ${
list.slice(0, 6).join(', ') || '(none recorded)'}${list.length > 6 ? ' ...' : ''}`];
}
if (state === 'headers-rewritten') {
return ['a limit value that differs between two paths a second apart is not a '
+ 'live number. Find the layer caching or synthesising responses and make it '
+ "forward the origin's headers unchanged.",
'this state is more dangerous than stripping, because the client believes the '
+ 'numbers it is given and has no way to tell.'];
}
if (state === 'headers-absent') {
return ['nothing arrived on any path checked, so this is not attributable yet. '
+ 'Re-run with the gateway base URL set, and confirm the credential and '
+ 'endpoint are the ones production uses.'];
}
if (state === 'reset-in-the-past') {
return ['a reset instant already in the past makes any sleep computed from it a '
+ 'no-op, so the client retries immediately and 429s again. Prefer '
+ 'retry-after, which is relative.'];
}
if (state === 'clock-skew') {
if (provider === 'anthropic') {
return ['anthropic reset values are RFC 3339 instants, so a sleep computed '
+ 'from one is only as good as clock agreement. Fix time sync on this host, '
+ 'or use retry-after instead, which is relative and immune to skew.',
'the same skew affects any log correlation you do against these timestamps, '
+ 'which is usually how it is finally noticed.'];
}
return ['this provider returns reset values as durations, so backoff is '
+ 'unaffected, but the skew will still misalign every log line you correlate '
+ "against the API's timestamps."];
}
return [];
}
async function probeOnce(url, headers) {
try {
const r = await fetch(url, { headers });
let note = '';
if (r.status === 429) {
note = 'a 429 arrived on its own. retry-after came back as '
+ `${JSON.stringify(r.headers.get('retry-after'))}`;
} else if (r.status === 401 || r.status === 403) {
note = `${r.status}: the credential cannot read this path`;
}
return [r.status, Object.fromEntries(r.headers.entries()), note];
} catch (err) {
return [null, {}, `request failed: ${err.message}`];
}
}
const host = (url) => String(url).split('//').pop().split('/')[0];
async function audit(provider, key, baseUrl) {
const directBase = DIRECT[provider];
const auth = provider === 'anthropic'
? { 'x-api-key': key, 'anthropic-version': '2023-06-01' }
: { Authorization: `Bearer ${key}` };
auth['User-Agent'] = 'retry-after-header-probe/1.0';
console.log(`${provider}: direct ${host(directBase)}, `
+ `${baseUrl ? `gateway ${host(baseUrl)}` : 'no gateway configured'}`);
const [, directHeaders, note] = await probeOnce(`${directBase}/models`, auth);
if (note) console.log(` direct: ${note}`);
let gatewayHeaders = {};
if (baseUrl) {
await new Promise((r) => { setTimeout(r, 1000); });
const [, gh, gnote] = await probeOnce(`${baseUrl.replace(/\/$/, '')}/models`, auth);
gatewayHeaders = gh;
if (gnote) console.log(` gateway: ${gnote}`);
}
const comparison = compare(directHeaders,
baseUrl ? gatewayHeaders : directHeaders, provider);
const directMissing = missing(directHeaders, provider);
const now = Date.now() / 1000;
const skew = clockSkew(lowerHeaders(directHeaders).date, now);
const serverEpoch = now - (skew ?? 0);
const stale = staleResets(directHeaders, provider, serverEpoch);
const [state, detail] = verdict(comparison, directMissing, Boolean(baseUrl),
skew, stale);
console.log(`${state.padEnd(21)} ${detail}`);
const stripped = Object.entries(comparison)
.filter(([, v]) => v[2] === 'stripped').map(([n]) => n);
for (const name of stripped.slice(0, 6)) console.log(` stripped ${name}`);
for (const [name, v] of Object.entries(comparison).sort()) {
if (v[2] === 'intact' && name.endsWith('-limit') && v[0]) {
console.log(` intact ${name.padEnd(42)} ${v[0]}`);
}
}
for (const [name, seconds] of stale.slice(0, 3)) {
console.log(` stale ${name}, ${Math.round(seconds)}s in the past`);
}
for (const line of repairLines(state, provider, stripped)) {
console.log(` repair: ${line}`);
}
return FINDINGS.has(state) ? 1 : 0;
}
async function main() {
const anthropicKey = process.env.ANTHROPIC_API_KEY;
const openaiKey = process.env.OPENAI_API_KEY;
if (!anthropicKey && !openaiKey) {
console.error('set ANTHROPIC_API_KEY, OPENAI_API_KEY, or both, and set the '
+ 'matching base URL if production reaches the API through a gateway');
process.exitCode = 2;
return;
}
let findings = 0;
if (anthropicKey) {
findings += await audit('anthropic', anthropicKey, process.env.ANTHROPIC_BASE_URL);
}
if (openaiKey) {
findings += await audit('openai', openaiKey, process.env.OPENAI_BASE_URL);
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the note: the same headers on both paths except that the gateway response has lost the input-token triple, which must come back as headers-stripped with the names printed for an allowlist. The second is the false positive this design has to avoid — two probes a second apart legitimately disagree on -remaining and -reset, and that must grade as intact, while a -limit that differs must grade as rewritten. Then the two reset formats, one an RFC 3339 instant and one a Go-style duration, told apart rather than guessed at; the clock read against the server's own date header; a reset already elapsed; and the ordering rule, which reports a stripped header before a skewed clock because there is nothing left to compute a sleep from.
from retry_after_header_probe import (clock_skew, compare, lower_headers,
missing, parse_reset, repair_lines,
stale_resets, verdict)
ANTHROPIC_OK = {
"Anthropic-Ratelimit-Requests-Limit": "1000",
"anthropic-ratelimit-requests-remaining": "998",
"anthropic-ratelimit-requests-reset": "2026-08-31T09:12:00Z",
"anthropic-ratelimit-input-tokens-limit": "10000000",
"anthropic-ratelimit-input-tokens-remaining": "9998000",
"anthropic-ratelimit-input-tokens-reset": "2026-08-31T09:12:00Z",
"anthropic-ratelimit-output-tokens-limit": "2000000",
"anthropic-ratelimit-output-tokens-remaining": "1999000",
"anthropic-ratelimit-output-tokens-reset": "2026-08-31T09:12:00Z",
"anthropic-ratelimit-tokens-limit": "12000000",
"anthropic-ratelimit-tokens-remaining": "11997000",
"anthropic-ratelimit-tokens-reset": "2026-08-31T09:12:00Z",
"date": "Mon, 31 Aug 2026 09:11:00 GMT",
}
def without(headers, prefix):
return {k: v for k, v in headers.items()
if not k.lower().startswith(prefix)}
def test_a_gateway_that_drops_the_triples_is_the_finding():
# The note. Header casing differs between the two paths on purpose: a
# comparison that does not normalise reports every proxy as stripping.
gateway = without(ANTHROPIC_OK, "anthropic-ratelimit-input")
rows = compare(ANTHROPIC_OK, gateway, "anthropic")
stripped = [n for n, (_, _, s) in rows.items() if s == "stripped"]
assert stripped == ["anthropic-ratelimit-input-tokens-limit",
"anthropic-ratelimit-input-tokens-remaining",
"anthropic-ratelimit-input-tokens-reset"]
state, detail = verdict(rows, [], True, 0.0, [])
assert state == "headers-stripped"
assert "do not survive the gateway" in detail
lines = repair_lines(state, "anthropic", stripped)
assert any("retry-after travels with these" in line for line in lines)
assert any("allowlist" in line for line in lines)
def test_remaining_may_differ_across_paths_but_a_limit_may_not():
# Two calls a second apart. remaining and reset are supposed to move, so
# comparing them would make every healthy path look rewritten.
later = dict(ANTHROPIC_OK)
later["anthropic-ratelimit-requests-remaining"] = "997"
later["anthropic-ratelimit-requests-reset"] = "2026-08-31T09:12:01Z"
rows = compare(ANTHROPIC_OK, later, "anthropic")
assert rows["anthropic-ratelimit-requests-remaining"][2] == "intact"
assert rows["anthropic-ratelimit-requests-reset"][2] == "intact"
assert verdict(rows, [], True, 0.0, [])[0] == "headers-intact"
faked = dict(ANTHROPIC_OK)
faked["anthropic-ratelimit-requests-limit"] = "50"
rows = compare(ANTHROPIC_OK, faked, "anthropic")
assert rows["anthropic-ratelimit-requests-limit"][2] == "rewritten"
state, detail = verdict(rows, [], True, 0.0, [])
assert state == "headers-rewritten"
assert "generating headers rather than forwarding" in detail
assert any("more dangerous than stripping" in line
for line in repair_lines(state))
def test_the_two_reset_formats_are_told_apart_rather_than_guessed():
kind, value = parse_reset("2026-08-31T09:12:00Z")
assert kind == "absolute" and value == 1788167520.0
assert parse_reset("6m0s") == ("duration", 360.0)
assert parse_reset("30s") == ("duration", 30.0)
assert parse_reset("1h2m3s") == ("duration", 3723.0)
assert parse_reset("500ms") == ("duration", 0.5)
assert parse_reset("12") == ("duration", 12.0)
assert parse_reset("") == ("unknown", None)
assert parse_reset("soon") == ("unknown", None)
def test_the_clock_is_read_against_the_server_and_a_stale_reset_is_its_own_state():
# 09:11:00 on the server, 09:11:42 locally: 42 seconds ahead.
skew = clock_skew("Mon, 31 Aug 2026 09:11:00 GMT", 1788167502.0)
assert round(skew) == 42
assert clock_skew("", 0) is None and clock_skew("not a date", 0) is None
state, detail = verdict(compare(ANTHROPIC_OK, ANTHROPIC_OK, "anthropic"),
[], True, skew, [])
assert state == "clock-skew"
assert "ahead of" in detail
assert any("RFC 3339 instants" in line
for line in repair_lines(state, "anthropic"))
# Reset instants already elapsed on the server's own clock.
stale = stale_resets(ANTHROPIC_OK, "anthropic", 1788167600.0)
assert len(stale) == 4 and stale[0][1] == 80.0
assert verdict(compare(ANTHROPIC_OK, ANTHROPIC_OK, "anthropic"),
[], True, 0.0, stale)[0] == "reset-in-the-past"
def test_a_transport_failure_is_reported_before_a_clock_one():
gateway = without(ANTHROPIC_OK, "anthropic-ratelimit-input")
rows = compare(ANTHROPIC_OK, gateway, "anthropic")
stale = stale_resets(ANTHROPIC_OK, "anthropic", 1788167600.0)
# Stripped headers, a stale reset and a large skew all at once. There is
# nothing to compute a sleep from, so the transport answer comes first.
assert verdict(rows, [], True, 300.0, stale)[0] == "headers-stripped"
def test_openai_headers_and_the_no_gateway_case():
openai = {"x-ratelimit-limit-requests": "10000",
"x-ratelimit-remaining-requests": "9999",
"x-ratelimit-reset-requests": "6m0s",
"x-ratelimit-limit-tokens": "2000000",
"x-ratelimit-remaining-tokens": "1999000",
"x-ratelimit-reset-tokens": "6m0s"}
assert missing(openai, "openai") == []
assert stale_resets(openai, "openai", 1788167600.0) == []
assert verdict(compare(openai, openai, "openai"), [], False, 0.0, [])[0] \
== "headers-intact"
bare = missing({}, "openai")
assert len(bare) == 6
state, detail = verdict(compare({}, {}, "openai"), bare, False, None, [])
assert state == "headers-absent"
assert "no gateway configured to blame" in detail
assert any("not attributable yet" in line for line in repair_lines(state))
assert lower_headers(None) == {} and repair_lines("headers-intact") == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { clockSkew, compare, lowerHeaders, missing, parseReset, repairLines,
staleResets, verdict } from './retry-after-header-probe.mjs';
const ANTHROPIC_OK = {
'Anthropic-Ratelimit-Requests-Limit': '1000',
'anthropic-ratelimit-requests-remaining': '998',
'anthropic-ratelimit-requests-reset': '2026-08-31T09:12:00Z',
'anthropic-ratelimit-input-tokens-limit': '10000000',
'anthropic-ratelimit-input-tokens-remaining': '9998000',
'anthropic-ratelimit-input-tokens-reset': '2026-08-31T09:12:00Z',
'anthropic-ratelimit-output-tokens-limit': '2000000',
'anthropic-ratelimit-output-tokens-remaining': '1999000',
'anthropic-ratelimit-output-tokens-reset': '2026-08-31T09:12:00Z',
'anthropic-ratelimit-tokens-limit': '12000000',
'anthropic-ratelimit-tokens-remaining': '11997000',
'anthropic-ratelimit-tokens-reset': '2026-08-31T09:12:00Z',
date: 'Mon, 31 Aug 2026 09:11:00 GMT',
};
const without = (headers, prefix) => Object.fromEntries(
Object.entries(headers).filter(([k]) => !k.toLowerCase().startsWith(prefix)));
test('a gateway that drops the triples is the finding', () => {
const gateway = without(ANTHROPIC_OK, 'anthropic-ratelimit-input');
const rows = compare(ANTHROPIC_OK, gateway, 'anthropic');
const stripped = Object.entries(rows).filter(([, v]) => v[2] === 'stripped')
.map(([n]) => n);
assert.deepEqual(stripped, ['anthropic-ratelimit-input-tokens-limit',
'anthropic-ratelimit-input-tokens-remaining',
'anthropic-ratelimit-input-tokens-reset']);
const [state, detail] = verdict(rows, [], true, 0, []);
assert.equal(state, 'headers-stripped');
assert.match(detail, /do not survive the gateway/);
const lines = repairLines(state, 'anthropic', stripped);
assert.ok(lines.some((l) => l.includes('retry-after travels with these')));
assert.ok(lines.some((l) => l.includes('allowlist')));
});
test('remaining may differ across paths but a limit may not', () => {
const later = { ...ANTHROPIC_OK,
'anthropic-ratelimit-requests-remaining': '997',
'anthropic-ratelimit-requests-reset': '2026-08-31T09:12:01Z' };
let rows = compare(ANTHROPIC_OK, later, 'anthropic');
assert.equal(rows['anthropic-ratelimit-requests-remaining'][2], 'intact');
assert.equal(rows['anthropic-ratelimit-requests-reset'][2], 'intact');
assert.equal(verdict(rows, [], true, 0, [])[0], 'headers-intact');
const faked = { ...ANTHROPIC_OK, 'anthropic-ratelimit-requests-limit': '50' };
rows = compare(ANTHROPIC_OK, faked, 'anthropic');
assert.equal(rows['anthropic-ratelimit-requests-limit'][2], 'rewritten');
const [state, detail] = verdict(rows, [], true, 0, []);
assert.equal(state, 'headers-rewritten');
assert.match(detail, /generating headers rather than forwarding/);
assert.ok(repairLines(state).some((l) => l.includes('more dangerous than stripping')));
});
test('the two reset formats are told apart rather than guessed', () => {
assert.deepEqual(parseReset('2026-08-31T09:12:00Z'), ['absolute', 1788167520]);
assert.deepEqual(parseReset('6m0s'), ['duration', 360]);
assert.deepEqual(parseReset('30s'), ['duration', 30]);
assert.deepEqual(parseReset('1h2m3s'), ['duration', 3723]);
assert.deepEqual(parseReset('500ms'), ['duration', 0.5]);
assert.deepEqual(parseReset('12'), ['duration', 12]);
assert.deepEqual(parseReset(''), ['unknown', null]);
assert.deepEqual(parseReset('soon'), ['unknown', null]);
});
test('the clock is read against the server and a stale reset is its own state', () => {
const skew = clockSkew('Mon, 31 Aug 2026 09:11:00 GMT', 1788167502);
assert.equal(Math.round(skew), 42);
assert.equal(clockSkew('', 0), null);
assert.equal(clockSkew('not a date', 0), null);
const [state, detail] = verdict(compare(ANTHROPIC_OK, ANTHROPIC_OK, 'anthropic'),
[], true, skew, []);
assert.equal(state, 'clock-skew');
assert.match(detail, /ahead of/);
assert.ok(repairLines(state, 'anthropic').some((l) => l.includes('RFC 3339 instants')));
const stale = staleResets(ANTHROPIC_OK, 'anthropic', 1788167600);
assert.equal(stale.length, 4);
assert.equal(stale[0][1], 80);
assert.equal(verdict(compare(ANTHROPIC_OK, ANTHROPIC_OK, 'anthropic'),
[], true, 0, stale)[0], 'reset-in-the-past');
});
test('a transport failure is reported before a clock one', () => {
const gateway = without(ANTHROPIC_OK, 'anthropic-ratelimit-input');
const rows = compare(ANTHROPIC_OK, gateway, 'anthropic');
const stale = staleResets(ANTHROPIC_OK, 'anthropic', 1788167600);
assert.equal(verdict(rows, [], true, 300, stale)[0], 'headers-stripped');
});
test('openai headers and the no gateway case', () => {
const openai = {
'x-ratelimit-limit-requests': '10000',
'x-ratelimit-remaining-requests': '9999',
'x-ratelimit-reset-requests': '6m0s',
'x-ratelimit-limit-tokens': '2000000',
'x-ratelimit-remaining-tokens': '1999000',
'x-ratelimit-reset-tokens': '6m0s',
};
assert.deepEqual(missing(openai, 'openai'), []);
assert.deepEqual(staleResets(openai, 'openai', 1788167600), []);
assert.equal(verdict(compare(openai, openai, 'openai'), [], false, 0, [])[0],
'headers-intact');
const bare = missing({}, 'openai');
assert.equal(bare.length, 6);
const [state, detail] = verdict(compare({}, {}, 'openai'), bare, false, null, []);
assert.equal(state, 'headers-absent');
assert.match(detail, /no gateway configured to blame/);
assert.ok(repairLines(state).some((l) => l.includes('not attributable yet')));
assert.deepEqual(lowerHeaders(null), {});
assert.deepEqual(repairLines('headers-intact'), []);
});
FAQ
Why not just trigger a 429 and read the header directly?
Because provoking the failure you are investigating is not a diagnostic. On an organization that is already saturated it is a second outage, and on a healthy one it burns capacity that belongs to production in order to learn something a single successful call already proves. The rate-limit triples and retry-after are the same class of header, added by the same layer and forwarded or dropped by the same middlebox rules, so the healthy call is a sound canary. If a 429 does arrive on its own while the script is probing, it records the retry-after that came with it and does not retry.
How is this different from the note about which limiter hit?
That note reads the header values to tell you which of the three buckets emptied, and the headroom note reads them to tell you how much room is left. Both assume the headers arrived. This one asks the prior question and never grades a value: does the header class survive the path your application actually uses, do the two paths agree about the limits, and is the reset timestamp usable on this host's clock. Different question, different output, and the three are usually read in that order once something has gone wrong.
What if there is no gateway between my code and the API?
Then set no base URL and the script checks one path: whether the required headers arrive at all, whether any absolute reset has already elapsed, and whether this host's clock agrees with the server's. That is still worth running, but it cannot attribute a loss. A single probe that comes back with nothing is genuinely ambiguous between the provider, the endpoint and your own infrastructure, and only the second path turns that into an answer.
A 429 arrived with no retry-after at all. Is that a stripped header?
Not necessarily, and this is the case worth knowing about. The 429 you get after an organization crosses its monthly spend cap deliberately carries no retry-after, because there is nothing to wait for: access resumes at the start of the next month or when the limit is raised. It is documented to be distinguishable by its error code rather than its headers. A client that treats every missing retry-after as a proxy problem will retry that one forever, which is the billing wall note rather than this one.
Which header names should the allowlist contain?
The script prints them for the provider it probed. On Anthropic that is the requests, input-token and output-token triples plus the aggregate tokens triple, and retry-after itself; on OpenAI, the request and token limit, remaining and reset headers, plus retry-after and the project-scoped triple if a project ceiling applies. Add retry-after explicitly even though a healthy probe never sees it, because it is the one that matters on the day the rest of this stops being theoretical.
Related field notes
- Which of the three limiters actually emptied
- The 429 that is a billing wall and never clears on retry
- 429s while every single minute sat under the configured limit
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits -- Claude platform docs
- Rate limits -- OpenAI platform docs
- Errors -- Claude platform docs
- Rate Limits API -- Claude platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.