Diagnostic LLM APIs
Cache hits fall away exactly when the fleet scales out
The cached share was fine in staging and fine in the first week of the rollout. Then autoscaling started doing its job, and the discount went the wrong way. At three in the morning the prompt caches at seventy per cent; at two in the afternoon, on the same template, the same model and the same code, it caches at sixteen. Everybody's first instinct is that something about the busy path is different. Nothing about the busy path is different. There are simply more machines in it, and none of them has seen this prefix before.
With an organization admin key: GET /v1/organization/usage/completions?start_time={T-7d}&bucket_width=1h&limit=168&group_by[]=project_id&group_by[]=model. Per hour, input_cached_tokens / input_tokens against num_model_requests.
Then correlate the two. A rank correlation at or below about -0.4, with the quiet third of hours caching well and the busy third caching at well under half that, is routing scatter. Prefix instability does not produce a slope; it produces a flat low line at every load.
Drop the hours that follow a gap in traffic before correlating. Those start cold no matter how the requests were routed, and if the batch that opens a burst is also its busiest hour — which is normal — leaving them in manufactures precisely the negative correlation you are testing for. Those hours belong to the retention note.
The repair is one parameter: prompt_cache_key, set to something coarse and stable per template. It influences routing. It does not pin a request to a machine and does not guarantee a hit.
The problem in plain words
OpenAI's prompt caching is automatic, which is usually described as a convenience and is occasionally a problem: there is no flag to switch on, so there is no flag whose absence explains a miss. Lookup is prefix-based, and it is also routing-sensitive. A cached prefix lives on the machine that served the request that created it. Send the next identical request to a different backend and it is a first call again, at full price, and nothing tells you that is what happened.
The consequence is genuinely counter-intuitive, which is why it survives so long. Every other cache in your stack gets more effective as traffic rises, because volume is what keeps entries warm. This one gets less effective, because volume is what spreads the traffic over more caches. A team scaling a service horizontally watches its per-request cost climb and reaches for the explanations that usually fit — a longer prompt, more retrieved context, a model change — and none of them fit, because the prompt is byte-identical and always was.
Why it happens
The slope is the finding, not the level. Three published notes and one sibling all read a cached share, so a low share is not evidence of anything by itself. What no other note here computes is the relationship between that share and the hour's request count. A rank correlation is used rather than a straight one because request counts are heavy tailed: one incident hour at ten times normal volume would otherwise decide the answer on its own.
Post-gap hours are excluded, and that exclusion is the whole boundary with the retention note. An hour that resumes traffic after idle time starts from an evicted cache regardless of routing, so it can say nothing about routing. Worse, scheduled work usually opens at full tilt, which makes the coldest hour also the busiest one and forges a strong negative correlation out of an eviction problem. The script computes the correlation over linked hours only, reports how many hours it threw away, and names the note those hours belong to.
Two degenerate cases get two different answers on purpose. A request rate that never varies returns no correlation at all, because nothing can be said about concurrency from a flat load. A cached share that never varies returns exactly zero, because "no relationship" is itself a finding and it is the one that points at a prefix changing between calls. Collapsing those two into one number would hide the difference between "cannot tell" and "definitely not this".
A share that rises with load is reported and is not a fault. Density keeps entries warm, so on a well-keyed fleet the busy hours should cache better than the quiet ones. Seeing that is the confirmation that the prefix is stable and the routing hint is doing its job, and the script prints it rather than staying silent, because a check that only ever speaks up when it is unhappy teaches you nothing about the healthy shape.
The quiet hours have to actually cache before any of this means anything. The finding requires a quiet-third share above a floor. If the prompt caches badly even at four in the morning with two workers running, the problem is not where the requests landed, and no routing hint will fix it. That branch is handed to the prefix note and to the eligibility check on the mean input size.
The fix, as a flow
This is the only cache fault in the section that gets worse when you add capacity, which is why it survives so long: every instinct says volume keeps a cache warm. Here volume spreads the same prompt over more machines, none of which has seen it before. The evidence is a slope rather than a level, and the hours that follow a gap are thrown away first, because those run cold whatever the routing did.
How to fix it
Pull a week of hourly buckets with the request count
bucket_width=1h with group_by[]=project_id and group_by[]=model. num_model_requests is the field that makes this note possible and it is the one Anthropic's equivalent report does not have, which is why this check is an OpenAI check.
Drop every hour that follows a gap
Keep only hours whose preceding hour also carried traffic. This is not tidying: post-gap hours run cold for a reason that has nothing to do with concurrency, and they cluster at the start of scheduled work, which is exactly when load spikes.
Pool the share, do not average it
Cached tokens over input tokens across the set, not the mean of the per-hour ratios. An hour with nine requests must not outvote an hour with nine thousand, and the whole quantity under test is what happens in the big hours.
Rank-correlate the share against the request rate
Spearman across the linked hours. At or below about -0.4, with the quiet third caching above a floor and the busy third at under sixty per cent of it, the share is degrading with load. Report both the correlation and the two shares: the number that persuades anyone is "seventy per cent at night, sixteen at noon".
Set a coarse cache key and re-measure the busy end
prompt_cache_key="rag-answer-v3", or the template name plus a tenant. Coarse enough that traffic concentrates; a per-request id scatters the fleet as thoroughly as no key at all. It is a routing hint, not content, so keep it out of the prompt. What moves first afterwards is the gap between the quiet and busy shares, not the average.
How to check it worked
Re-run over the same seven days after the key ships. The quiet share should barely move and the busy share should climb toward it; the correlation is the number to watch, because it goes to roughly zero long before the average looks impressive.
python3 openai_cache_key_routing_scatter.py --days 7
# load-correlated-misses proj_abc123 / gpt-5.6 cached share 68% in the quietest hours (187 req/h) and 16% in the busiest (3405 req/h), rank correlation -1.00 against request rate. The prefix is cacheable; the requests are not landing where it is cached.
# 21 hour(s) that follow a gap in traffic were excluded before correlating
# repair: set prompt_cache_key on the route
# repair: make it coarse. The template name, or the template plus tenant
# 5 project/model series checked, 1 finding(s)The full code
One paginated GET and no writes. Ten pure functions, and the two that carry the note are the exclusion and the correlation. continuation_rows keeps only hours whose predecessor also had traffic, which is what stops an eviction problem being read as a routing one; resumption_rows is its complement, kept so the count of discarded hours can be printed rather than quietly vanishing. spearman ranks with ties shared and returns two different degenerate answers on purpose: nothing at all when the load is flat, and exactly zero when the share is. load_split turns the slope into the pair of numbers anyone will actually quote.
"""Find OpenAI traffic whose cached share falls as concurrency rises.
Read only. One paginated GET against the Usage API, which needs an admin key
(sk-admin-...); a project key is rejected by every /v1/organization/ path, and
an admin key can be provisioned with the read scopes only.
Cache lookup is prefix-based and routing-sensitive. Without prompt_cache_key a
fleet sprays byte-identical prompts across many backends and each one sees a
cold prefix, so the hit rate gets *worse* as you scale out. That is the whole
signature: a cached share that is negatively correlated with the hour's request
count. A prefix that is simply unstable produces a flat low share at every load
and belongs to a different note.
Hours that follow a gap in traffic are dropped before anything is correlated.
Those hours run cold because the entry was evicted while nobody was calling,
which is a third note again, and leaving them in would manufacture exactly the
negative correlation this one is looking for.
The repair is printed, never performed.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_cache_key_routing_scatter")
API = "https://api.openai.com/v1"
FINDINGS = ("load-correlated-misses",)
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def hour_index(stamp):
"""Hours since the epoch. Pure. None if unreadable.
The usage buckets carry start_time as a unix integer, but the same code has
to survive an ISO string, and adjacency has to be integer arithmetic: gap
detection on formatted timestamps gets 23:00 and 00:00 wrong every night.
"""
if isinstance(stamp, bool) or stamp is None:
return None
if isinstance(stamp, (int, float)):
return int(stamp) // 3600
text = str(stamp).strip().replace(" ", "T")
if len(text) < 13:
return None
head = text[:13]
if head[4] != "-" or head[7] != "-" or head[10] != "T":
return None
for part in (head[0:4], head[5:7], head[8:10], head[11:13]):
if not part.isdigit():
return None
try:
when = dt.datetime(int(head[0:4]), int(head[5:7]), int(head[8:10]),
int(head[11:13]), tzinfo=dt.timezone.utc)
except ValueError:
return None
return int(when.timestamp()) // 3600
def hour_label(index):
"""Render an hour index back as a UTC stamp. Pure."""
if index is None:
return "unknown"
when = dt.datetime.fromtimestamp(int(index) * 3600, dt.timezone.utc)
return when.strftime("%Y-%m-%dT%H:00Z")
def rows_by_series(buckets):
"""Per (project_id, model), one row per active hour, sorted. Pure."""
merged = {}
for bucket in buckets or []:
index = hour_index(bucket.get("start_time"))
if index is None:
continue
for result in bucket.get("results") or []:
if not isinstance(result, dict):
continue
ident = (str(result.get("project_id") or "unknown"),
str(result.get("model") or "unknown"))
row = merged.setdefault((ident, index),
{"index": index, "hour": hour_label(index),
"requests": 0, "input": 0, "cached": 0})
row["requests"] += _int(result.get("num_model_requests"))
row["input"] += _int(result.get("input_tokens"))
row["cached"] += _int(result.get("input_cached_tokens"))
out = {}
for (ident, _index), row in merged.items():
if row["requests"] > 0 or row["input"] > 0:
out.setdefault(ident, []).append(row)
for rows in out.values():
rows.sort(key=lambda r: r["index"])
return out
def cached_share(rows):
"""Pooled cached share over a set of hours. Pure. None when nothing ran.
Pooled rather than averaged: an hour with nine requests must not carry the
same weight as an hour with nine thousand, which is the entire quantity
under test here.
"""
total = sum(_int(r.get("input")) for r in rows or [])
if total <= 0:
return None
return sum(_int(r.get("cached")) for r in rows or []) / float(total)
def continuation_rows(rows):
"""Hours whose previous hour also carried traffic. Pure.
The exclusion that keeps this note off someone else's ground. An hour that
follows idle time starts from an evicted cache no matter how the requests
were routed, so it cannot be evidence about routing.
"""
active = {_int(r.get("index")) for r in rows or []}
return [r for r in rows or [] if _int(r.get("index")) - 1 in active]
def resumption_rows(rows):
"""The hours the correlation deliberately threw away. Pure."""
active = {_int(r.get("index")) for r in rows or []}
return [r for r in rows or [] if _int(r.get("index")) - 1 not in active]
def _ranks(values):
"""Average ranks, ties shared. Pure."""
order = sorted(range(len(values)), key=lambda i: values[i])
ranks = [0.0] * len(values)
i = 0
while i < len(order):
j = i
while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]:
j += 1
shared = (i + j) / 2.0 + 1.0
for k in range(i, j + 1):
ranks[order[k]] = shared
i = j + 1
return ranks
def spearman(xs, ys):
"""Rank correlation between two equal-length series. Pure. None if flat.
Rank rather than Pearson because request counts are heavy tailed and one
incident hour would otherwise decide the answer.
The two degenerate cases are deliberately different answers. A load that
never varies returns None, because nothing at all can be said about
concurrency from a flat request rate. A share that never varies returns
0.0, because "no relationship" is a real finding here and it is the one
that sends the reader to the prefix-instability note.
"""
xs = list(xs or [])
ys = list(ys or [])
if len(xs) != len(ys) or len(xs) < 8:
return None
rx, ry = _ranks(xs), _ranks(ys)
n = float(len(xs))
mx, my = sum(rx) / n, sum(ry) / n
sxy = sum((a - mx) * (b - my) for a, b in zip(rx, ry))
sxx = sum((a - mx) ** 2 for a in rx)
syy = sum((b - my) ** 2 for b in ry)
if sxx <= 0:
return None
if syy <= 0:
return 0.0
return sxy / ((sxx ** 0.5) * (syy ** 0.5))
def load_split(rows, fraction=0.33):
"""Pooled cached share in the quietest and busiest hours. Pure.
Returns (quiet_share, busy_share, quiet_rate, busy_rate) or Nones. The
correlation says the relationship is monotone; this says how big it is in
the only units anyone will act on, which is the discount you are not getting
at peak.
"""
active = [r for r in rows or [] if _int(r.get("requests")) > 0]
if len(active) < 6:
return (None, None, None, None)
ordered = sorted(active, key=lambda r: _int(r.get("requests")))
size = max(2, int(len(ordered) * fraction))
quiet, busy = ordered[:size], ordered[-size:]
def rate(part):
return sum(_int(r.get("requests")) for r in part) / float(len(part))
return (cached_share(quiet), cached_share(busy), rate(quiet), rate(busy))
def handoff(state):
"""Which note owns this shape, when it is not this one. Pure."""
if state == "no-cached-tokens":
return ("not one cached token at any load, so the traffic never becomes "
"eligible rather than being routed away from its cache. Read the "
"prompt-below-model-cache-minimum note and check the mean input "
"per request against the model's floor first.")
if state == "flat-low-share":
return ("the share is low and stays low whatever the load, which is a "
"prefix that differs between calls rather than requests landing "
"on different machines. Read the "
"cache-invalidated-by-changing-prefix note.")
if state == "cold-only-after-idle":
return ("the cold hours are the ones that follow gaps in traffic, and "
"the busy hours are fine. That is eviction during idle time: "
"read the prompt-cache-retention-left-at-default note.")
return ""
def classify(rows, rho_floor=-0.4, ratio_floor=0.6, quiet_floor=0.15,
min_hours=24):
"""Classify one project and model series. Pure. Returns (state, detail).
Only a series whose cached share falls monotonically as the hour gets busier
belongs to this note, and only after the post-gap hours have been removed.
"""
rows = rows or []
linked = continuation_rows(rows)
if len(linked) < min_hours:
return ("too-few-linked-hours",
"%d hour(s) with traffic in the hour before them, under the "
"floor of %d. Correlating against load needs a run of busy "
"hours." % (len(linked), min_hours))
overall = cached_share(linked)
if overall is not None and overall <= 0.0:
return ("no-cached-tokens",
"%d input token(s) across %d linked hour(s) and not one cached"
% (sum(_int(r.get("input")) for r in linked), len(linked)))
quiet, busy, quiet_rate, busy_rate = load_split(linked)
rho = spearman([_int(r.get("requests")) for r in linked],
[cached_share([r]) or 0.0 for r in linked])
if rho is None or quiet is None or busy is None:
return ("load-does-not-vary",
"the request rate barely moves across the window, so nothing "
"here can be attributed to concurrency")
if rho <= rho_floor and quiet >= quiet_floor and busy <= quiet * ratio_floor:
return ("load-correlated-misses",
"cached share %.0f%% in the quietest hours (%.0f req/h) and "
"%.0f%% in the busiest (%.0f req/h), rank correlation %.2f "
"against request rate. The prefix is cacheable; the requests "
"are not landing where it is cached."
% (quiet * 100, quiet_rate, busy * 100, busy_rate, rho))
if rho >= -rho_floor:
return ("share-rises-with-load",
"cached share climbs with the request rate (%.2f): density is "
"keeping entries warm, which is the opposite of scatter" % rho)
cold = resumption_rows(rows)
cold_share = cached_share(cold)
if (overall is not None and cold_share is not None and cold_share <= 0.02
and overall >= quiet_floor and len(cold) >= 3):
return ("cold-only-after-idle",
"%.0f%% cached in linked hours against %.0f%% in the %d hour(s) "
"that follow a gap" % (overall * 100, cold_share * 100, len(cold)))
if overall is not None and overall < quiet_floor:
return ("flat-low-share",
"cached share %.0f%% overall with rank correlation %.2f against "
"load: low everywhere rather than low under load"
% (overall * 100, rho))
return ("healthy",
"cached share %.0f%% quiet and %.0f%% busy, correlation %.2f"
% (quiet * 100, busy * 100, rho))
def repair_lines():
"""The routing hint, and what makes a good one. Pure."""
return [
"set prompt_cache_key on the route: "
"client.responses.create(..., prompt_cache_key=\"rag-answer-v3\").",
"make it coarse. The template name, or the template plus tenant, so "
"traffic concentrates on a few caches. A per-request id scatters the "
"fleet exactly as badly as no key at all.",
"keep it out of the prompt. It is a routing hint, not content, and it "
"does not pin a request to a machine or guarantee a hit.",
"then re-read these same hourly buckets. What should move first is the "
"busy end: the gap between the quiet and busy shares closes before the "
"average does.",
]
def window_start(days):
"""Floor to the hour so start_time lands on a bucket boundary."""
now = dt.datetime.now(dt.timezone.utc).replace(minute=0, second=0, microsecond=0)
return int((now - dt.timedelta(days=days)).timestamp())
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/ needs an admin key "
"(sk-admin-...), not a project key" % r.status_code)
r.raise_for_status()
return r.json()
def read_buckets(session, path, params):
"""Walk the paginated usage endpoint."""
params = dict(params)
while True:
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=7,
help="days of hourly buckets to read (max 30)")
ap.add_argument("--rho-floor", type=float, default=-0.4,
help="rank correlation at or below which the share is "
"treated as load-correlated")
ap.add_argument("--show-all", action="store_true",
help="also print series that are behaving")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key "
"(sk-admin-...); a project key cannot read /v1/organization/")
return 2
days = max(2, min(int(args.days), 30))
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
buckets = read_buckets(session, "/organization/usage/completions", {
"start_time": window_start(days),
"bucket_width": "1h",
"limit": 168,
"group_by[]": ["project_id", "model"],
})
series = rows_by_series(buckets)
if not series:
log.info("no completions usage in the last %d day(s)", days)
return 0
checked = 0
bad = 0
for ident in sorted(series):
rows = series[ident]
state, detail = classify(rows, args.rho_floor)
checked += 1
line = "%-24s %s / %s %s" % (state, ident[0], ident[1], detail)
if state in FINDINGS:
bad += 1
log.warning(line)
log.warning(" %d hour(s) that follow a gap in traffic were excluded "
"before correlating; those run cold for a different "
"reason.", len(resumption_rows(rows)))
for repair in repair_lines():
log.warning(" repair: %s", repair)
else:
note = handoff(state)
if note:
log.info(line)
log.info(" %s", note)
elif args.show_all:
log.info(line)
log.info("%d project/model series checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI traffic whose cached share falls as concurrency rises.
*
* Read only. One paginated GET against the Usage API, which needs an admin key
* (sk-admin-...). A project key is rejected by /v1/organization/.
*
* Cache lookup is prefix-based and routing-sensitive. Without prompt_cache_key
* a fleet sprays byte-identical prompts across many backends and each one sees
* a cold prefix, so the hit rate gets worse as you scale out. The signature is
* a cached share negatively correlated with the hour's request count. A prefix
* that is simply unstable gives a flat low share at every load, which is a
* different note.
*
* Hours that follow a gap in traffic are dropped before anything is
* correlated: they run cold because the entry was evicted while nobody was
* calling, and leaving them in manufactures the very correlation being tested.
*/
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set(['load-correlated-misses']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Hours since the epoch. Pure. Null if unreadable.
* The buckets carry start_time as a unix integer, but the same code has to
* survive an ISO string, and gap detection has to be integer arithmetic:
* comparing formatted stamps gets 23:00 and 00:00 wrong every night.
*/
export function hourIndex(stamp) {
if (typeof stamp === 'boolean' || stamp === null || stamp === undefined) return null;
if (typeof stamp === 'number' && Number.isFinite(stamp)) {
return Math.floor(Math.trunc(stamp) / 3600);
}
const text = String(stamp).trim().replace(' ', 'T');
if (text.length < 13) return null;
const head = text.slice(0, 13);
if (!/^\d{4}-\d{2}-\d{2}T\d{2}$/.test(head)) return null;
const when = Date.parse(`${head}:00:00Z`);
if (Number.isNaN(when)) return null;
return Math.floor(when / 3600000);
}
/** Render an hour index back as a UTC stamp. Pure. */
export function hourLabel(index) {
if (index === null || index === undefined) return 'unknown';
return `${new Date(Math.trunc(index) * 3600000).toISOString().slice(0, 13)}:00Z`;
}
/** Per project_id and model, one row per active hour, sorted. Pure. */
export function rowsBySeries(buckets) {
const merged = new Map();
for (const bucket of buckets ?? []) {
const index = hourIndex(bucket?.start_time);
if (index === null) continue;
for (const result of bucket?.results ?? []) {
if (!result || typeof result !== 'object') continue;
const ident = `${result.project_id ?? 'unknown'}\t${result.model ?? 'unknown'}`;
const cell = `${ident}\t${index}`;
if (!merged.has(cell)) {
merged.set(cell, { ident, index, hour: hourLabel(index),
requests: 0, input: 0, cached: 0 });
}
const row = merged.get(cell);
row.requests += readInt(result.num_model_requests);
row.input += readInt(result.input_tokens);
row.cached += readInt(result.input_cached_tokens);
}
}
const out = new Map();
for (const row of merged.values()) {
if (row.requests <= 0 && row.input <= 0) continue;
if (!out.has(row.ident)) out.set(row.ident, []);
out.get(row.ident).push(row);
}
for (const rows of out.values()) rows.sort((a, b) => a.index - b.index);
return out;
}
/**
* Pooled cached share over a set of hours. Pure. Null when nothing ran.
* Pooled rather than averaged: an hour with nine requests must not carry the
* same weight as an hour with nine thousand.
*/
export function cachedShare(rows) {
let input = 0;
let cached = 0;
for (const row of rows ?? []) {
input += readInt(row?.input);
cached += readInt(row?.cached);
}
if (input <= 0) return null;
return cached / input;
}
/**
* Hours whose previous hour also carried traffic. Pure.
* The exclusion that keeps this note off someone else's ground.
*/
export function continuationRows(rows) {
const active = new Set((rows ?? []).map((r) => readInt(r?.index)));
return (rows ?? []).filter((r) => active.has(readInt(r?.index) - 1));
}
/** The hours the correlation deliberately threw away. Pure. */
export function resumptionRows(rows) {
const active = new Set((rows ?? []).map((r) => readInt(r?.index)));
return (rows ?? []).filter((r) => !active.has(readInt(r?.index) - 1));
}
/** Average ranks, ties shared. Pure. */
function ranks(values) {
const order = values.map((_v, i) => i).sort((a, b) => values[a] - values[b]);
const out = new Array(values.length).fill(0);
let i = 0;
while (i < order.length) {
let j = i;
while (j + 1 < order.length && values[order[j + 1]] === values[order[i]]) j += 1;
const shared = (i + j) / 2 + 1;
for (let k = i; k <= j; k += 1) out[order[k]] = shared;
i = j + 1;
}
return out;
}
/**
* Rank correlation between two equal-length series. Pure. Null if flat.
*
* Rank rather than Pearson because request counts are heavy tailed and one
* incident hour would otherwise decide the answer. The two degenerate cases
* are deliberately different answers: a load that never varies returns null,
* because nothing can be said about concurrency from a flat request rate,
* while a share that never varies returns 0, because "no relationship" is a
* real finding and it is the one that points at prefix instability.
*/
export function spearman(xs, ys) {
const a = [...(xs ?? [])];
const b = [...(ys ?? [])];
if (a.length !== b.length || a.length < 8) return null;
const rx = ranks(a);
const ry = ranks(b);
const n = a.length;
const mx = rx.reduce((s, v) => s + v, 0) / n;
const my = ry.reduce((s, v) => s + v, 0) / n;
let sxy = 0;
let sxx = 0;
let syy = 0;
for (let i = 0; i < n; i += 1) {
sxy += (rx[i] - mx) * (ry[i] - my);
sxx += (rx[i] - mx) ** 2;
syy += (ry[i] - my) ** 2;
}
if (sxx <= 0) return null;
if (syy <= 0) return 0;
return sxy / (Math.sqrt(sxx) * Math.sqrt(syy));
}
/**
* Pooled cached share in the quietest and busiest hours. Pure.
* Returns [quietShare, busyShare, quietRate, busyRate], or nulls.
*/
export function loadSplit(rows, fraction = 0.33) {
const active = (rows ?? []).filter((r) => readInt(r?.requests) > 0);
if (active.length < 6) return [null, null, null, null];
const ordered = [...active].sort((a, b) => readInt(a?.requests) - readInt(b?.requests));
const size = Math.max(2, Math.trunc(ordered.length * fraction));
const quiet = ordered.slice(0, size);
const busy = ordered.slice(-size);
const rate = (part) => part.reduce((s, r) => s + readInt(r?.requests), 0) / part.length;
return [cachedShare(quiet), cachedShare(busy), rate(quiet), rate(busy)];
}
/** Which note owns this shape, when it is not this one. Pure. */
export function handoff(state) {
if (state === 'no-cached-tokens') {
return 'not one cached token at any load, so the traffic never becomes '
+ 'eligible rather than being routed away from its cache. Read the '
+ "prompt-below-model-cache-minimum note and check the mean input per "
+ "request against the model's floor first.";
}
if (state === 'flat-low-share') {
return 'the share is low and stays low whatever the load, which is a prefix '
+ 'that differs between calls rather than requests landing on different '
+ 'machines. Read the cache-invalidated-by-changing-prefix note.';
}
if (state === 'cold-only-after-idle') {
return 'the cold hours are the ones that follow gaps in traffic, and the '
+ 'busy hours are fine. That is eviction during idle time: read the '
+ 'prompt-cache-retention-left-at-default note.';
}
return '';
}
/** Classify one project and model series. Pure. Returns [state, detail]. */
export function classify(rows, rhoFloor = -0.4, ratioFloor = 0.6,
quietFloor = 0.15, minHours = 24) {
const all = rows ?? [];
const linked = continuationRows(all);
if (linked.length < minHours) {
return ['too-few-linked-hours',
`${linked.length} hour(s) with traffic in the hour before them, under the `
+ `floor of ${minHours}. Correlating against load needs a run of busy hours.`];
}
const overall = cachedShare(linked);
if (overall !== null && overall <= 0) {
const input = linked.reduce((s, r) => s + readInt(r?.input), 0);
return ['no-cached-tokens',
`${input} input token(s) across ${linked.length} linked hour(s) and not one cached`];
}
const [quiet, busy, quietRate, busyRate] = loadSplit(linked);
const rho = spearman(linked.map((r) => readInt(r?.requests)),
linked.map((r) => cachedShare([r]) ?? 0));
if (rho === null || quiet === null || busy === null) {
return ['load-does-not-vary',
'the request rate barely moves across the window, so nothing here can be '
+ 'attributed to concurrency'];
}
if (rho <= rhoFloor && quiet >= quietFloor && busy <= quiet * ratioFloor) {
return ['load-correlated-misses',
`cached share ${(quiet * 100).toFixed(0)}% in the quietest hours `
+ `(${quietRate.toFixed(0)} req/h) and ${(busy * 100).toFixed(0)}% in the `
+ `busiest (${busyRate.toFixed(0)} req/h), rank correlation `
+ `${rho.toFixed(2)} against request rate. The prefix is cacheable; the `
+ 'requests are not landing where it is cached.'];
}
if (rho >= -rhoFloor) {
return ['share-rises-with-load',
`cached share climbs with the request rate (${rho.toFixed(2)}): density is `
+ 'keeping entries warm, which is the opposite of scatter'];
}
const cold = resumptionRows(all);
const coldShare = cachedShare(cold);
if (overall !== null && coldShare !== null && coldShare <= 0.02
&& overall >= quietFloor && cold.length >= 3) {
return ['cold-only-after-idle',
`${(overall * 100).toFixed(0)}% cached in linked hours against `
+ `${(coldShare * 100).toFixed(0)}% in the ${cold.length} hour(s) that follow a gap`];
}
if (overall !== null && overall < quietFloor) {
return ['flat-low-share',
`cached share ${(overall * 100).toFixed(0)}% overall with rank correlation `
+ `${rho.toFixed(2)} against load: low everywhere rather than low under load`];
}
return ['healthy',
`cached share ${(quiet * 100).toFixed(0)}% quiet and ${(busy * 100).toFixed(0)}% `
+ `busy, correlation ${rho.toFixed(2)}`];
}
/** The routing hint, and what makes a good one. Pure. */
export function repairLines() {
return [
'set prompt_cache_key on the route: '
+ 'client.responses.create(..., prompt_cache_key="rag-answer-v3").',
'make it coarse. The template name, or the template plus tenant, so traffic '
+ 'concentrates on a few caches. A per-request id scatters the fleet exactly '
+ 'as badly as no key at all.',
'keep it out of the prompt. It is a routing hint, not content, and it does '
+ 'not pin a request to a machine or guarantee a hit.',
'then re-read these same hourly buckets. What should move first is the busy '
+ 'end: the gap between the quiet and busy shares closes before the average does.',
];
}
function windowStart(days) {
const now = new Date();
now.setUTCMinutes(0, 0, 0);
return Math.floor((now.getTime() - days * 86400000) / 1000);
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization/ needs an admin `
+ 'key (sk-admin-...), not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* readBuckets(key, path, params) {
let query = { ...params };
for (;;) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key '
+ '(sk-admin-...); a project key cannot read /v1/organization/');
process.exitCode = 2;
return;
}
const days = Math.max(2, Math.min(Number(process.env.DAYS ?? 7), 30));
const rhoFloor = Number(process.env.RHO_FLOOR ?? -0.4);
const showAll = process.env.SHOW_ALL === '1';
const buckets = [];
for await (const bucket of readBuckets(admin, '/organization/usage/completions', {
start_time: windowStart(days),
bucket_width: '1h',
limit: 168,
'group_by[]': ['project_id', 'model'],
})) buckets.push(bucket);
const series = rowsBySeries(buckets);
if (series.size === 0) {
console.log(`no completions usage in the last ${days} day(s)`);
return;
}
let checked = 0;
let bad = 0;
for (const ident of [...series.keys()].sort()) {
const rows = series.get(ident);
const [state, detail] = classify(rows, rhoFloor);
checked += 1;
const line = `${state.padEnd(24)} ${ident.replace('\t', ' / ')} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
console.warn(` ${resumptionRows(rows).length} hour(s) that follow a gap in `
+ 'traffic were excluded before correlating; those run cold '
+ 'for a different reason.');
for (const repair of repairLines()) console.warn(` repair: ${repair}`);
} else {
const note = handoff(state);
if (note) {
console.log(line);
console.log(` ${note}`);
} else if (showAll) {
console.log(line);
}
}
}
console.log(`${checked} project/model series checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Three fixtures share one traffic profile — a week of a normal daily load curve, quiet overnight and a broad afternoon peak — and differ only in how the cached share responds to it. One falls with load, one ignores it, one rises with it, and the three land on three different verdicts from byte-identical request counts. The fourth fixture is the trap: bursts that open at full tilt after a five-hour gap, where the busiest hour of every burst is also the coldest. Correlate that raw and it reads as a textbook scatter signature at -0.87. Drop the post-gap hours first and the linked hours are uniformly warm, and the script hands it to the retention note by name.
from openai_cache_key_routing_scatter import (cached_share, classify,
continuation_rows, handoff,
hour_index, hour_label,
load_split, resumption_rows,
rows_by_series, spearman)
BASE = hour_index("2026-08-17T00:00Z")
# A daily traffic shape: quiet overnight, a broad afternoon peak.
LOAD = [200, 150, 120, 100, 120, 200, 400, 900, 1800, 3000, 3600, 3800,
3900, 3800, 3600, 3000, 2400, 1800, 1200, 800, 600, 450, 350, 260]
def hour(offset, requests, share):
tokens = requests * 2000
return {"index": BASE + offset, "hour": hour_label(BASE + offset),
"requests": requests, "input": tokens,
"cached": int(round(tokens * share))}
def contiguous(share_of_load):
"""Seven contiguous days on the daily load shape, no gaps anywhere."""
return [hour(i, LOAD[i % 24], share_of_load(LOAD[i % 24])) for i in range(168)]
# The note. The prefix is provably cacheable, because the quiet hours cache
# beautifully; the share falls away as the fleet fans out.
SCATTER = contiguous(lambda load: 0.72 - 0.00016 * load)
# Identical traffic, a share that ignores the load entirely. Prefix instability.
FLAT = contiguous(lambda _load: 0.10)
# Identical traffic, a share that improves with density. Not a fault.
RISING = contiguous(lambda load: 0.10 + 0.00016 * load)
def bursty():
"""Three-hour bursts five hours apart, each opening at full tilt.
The busiest hour of every burst is also the coldest one, because it is the
hour that follows the idle stretch. That is the trap: correlate the raw
series and the load looks guilty.
"""
shape = [(3000, 0.0), (1000, 0.7), (400, 0.7)]
rows = []
for burst in range(21):
for step, (requests, share) in enumerate(shape):
rows.append(hour(burst * 8 + step, requests, share))
return rows
BURSTY = bursty()
def test_the_cached_share_falling_with_load_is_the_finding():
# The note in one assertion: same prefix, same code, and the discount
# evaporates exactly when the fleet is widest.
quiet, busy, quiet_rate, busy_rate = load_split(SCATTER)
assert round(quiet, 2) == 0.69 and round(busy, 2) == 0.16
assert quiet_rate < 300 and busy_rate > 3400
rho = spearman([r["requests"] for r in SCATTER],
[cached_share([r]) for r in SCATTER])
assert round(rho, 3) == -1.0
state, detail = classify(SCATTER)
assert state == "load-correlated-misses"
assert "68% in the quietest hours" in detail
assert "16% in the busiest" in detail
assert "rank correlation -1.00" in detail
assert handoff(state) == ""
def test_the_same_traffic_with_a_flat_share_is_the_prefix_note():
# The control that makes the finding mean something. Byte-identical load,
# a share that does not move with it, and a different note owns it.
assert [r["requests"] for r in FLAT] == [r["requests"] for r in SCATTER]
assert spearman([r["requests"] for r in FLAT],
[cached_share([r]) for r in FLAT]) == 0.0
state, detail = classify(FLAT)
assert state == "flat-low-share"
assert "low everywhere rather than low under load" in detail
assert "cache-invalidated-by-changing-prefix" in handoff(state)
def test_a_share_that_climbs_with_load_is_not_scatter():
state, detail = classify(RISING)
assert state == "share-rises-with-load"
assert "the opposite of scatter" in detail
assert handoff(state) == ""
def test_hours_after_a_gap_are_excluded_before_correlating():
# The exclusion, tested by the case it exists for. Leave the post-gap hours
# in and the correlation goes sharply negative, because the cold hours are
# also the ones that open a burst as it scales up. Take them out and the
# linked hours are uniformly warm.
everything = spearman([r["requests"] for r in BURSTY],
[cached_share([r]) for r in BURSTY])
assert everything is not None and round(everything, 2) == -0.87
assert len(continuation_rows(BURSTY)) == 42
assert len(resumption_rows(BURSTY)) == 21
assert cached_share(continuation_rows(BURSTY)) == 0.7
assert cached_share(resumption_rows(BURSTY)) == 0.0
state, detail = classify(BURSTY)
assert state == "cold-only-after-idle"
assert "70% cached in linked hours against 0% in the 21 hour(s)" in detail
assert "prompt-cache-retention-left-at-default" in handoff(state)
def test_no_cached_tokens_at_any_load_is_an_eligibility_question():
state, detail = classify(contiguous(lambda _load: 0.0))
assert state == "no-cached-tokens"
assert "not one cached" in detail
assert "prompt-below-model-cache-minimum" in handoff(state)
def test_a_flat_request_rate_supports_no_verdict():
# Nothing can be said about concurrency when the concurrency never moves,
# and returning 0.0 there would read as "no relationship" rather than
# "no evidence".
steady = [hour(i, 1000, 0.6 if i % 2 else 0.2) for i in range(168)]
assert spearman([r["requests"] for r in steady],
[cached_share([r]) for r in steady]) is None
assert classify(steady)[0] == "load-does-not-vary"
def test_pooled_share_is_weighted_by_traffic():
# An hour with nine requests must not outvote an hour with nine thousand.
mixed = [hour(0, 10, 1.0), hour(1, 10_000, 0.1)]
assert round(cached_share(mixed), 4) == 0.1009
assert cached_share([]) is None
assert cached_share([{"input": 0, "cached": 0}]) is None
def test_the_hour_index_survives_both_shapes_and_midnight():
assert hour_index(1_755_388_800) == 1_755_388_800 // 3600
assert hour_index("2026-08-17T23:00Z") + 1 == hour_index("2026-08-18T00:00Z")
assert hour_label(hour_index("2026-08-17T09:00Z")) == "2026-08-17T09:00Z"
assert hour_index("nonsense") is None
assert hour_index(None) is None
def test_buckets_are_folded_into_project_and_model_series():
buckets = [{"start_time": (BASE + i) * 3600,
"results": [{"project_id": "proj_abc123", "model": "gpt-5.6",
"num_model_requests": LOAD[i % 24],
"input_tokens": LOAD[i % 24] * 2000,
"input_cached_tokens":
int(LOAD[i % 24] * 2000
* (0.72 - 0.00016 * LOAD[i % 24]))}]}
for i in range(168)]
series = rows_by_series(buckets)
rows = series[("proj_abc123", "gpt-5.6")]
assert len(rows) == 168
assert [r["index"] for r in rows] == sorted(r["index"] for r in rows)
assert classify(rows)[0] == "load-correlated-misses"
def test_thin_and_unreadable_windows_produce_no_verdict():
assert classify([hour(i, 500, 0.5) for i in range(10)])[0] == "too-few-linked-hours"
assert classify([])[0] == "too-few-linked-hours"
assert classify(None)[0] == "too-few-linked-hours"
assert spearman([1, 2], [1, 2]) is None
assert spearman([1, 2, 3], None) is None
assert load_split([]) == (None, None, None, None)
assert rows_by_series([{"start_time": "bad", "results": []}]) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cachedShare, classify, continuationRows, handoff, hourIndex, hourLabel,
loadSplit, resumptionRows, rowsBySeries, spearman }
from './openai-cache-key-routing-scatter.mjs';
const BASE = hourIndex('2026-08-17T00:00Z');
const LOAD = [200, 150, 120, 100, 120, 200, 400, 900, 1800, 3000, 3600, 3800,
3900, 3800, 3600, 3000, 2400, 1800, 1200, 800, 600, 450, 350, 260];
const hour = (offset, requests, share) => {
const tokens = requests * 2000;
return { index: BASE + offset, hour: hourLabel(BASE + offset), requests,
input: tokens, cached: Math.round(tokens * share) };
};
const contiguous = (shareOfLoad) => Array.from({ length: 168 },
(_, i) => hour(i, LOAD[i % 24], shareOfLoad(LOAD[i % 24])));
const SCATTER = contiguous((load) => 0.72 - 0.00016 * load);
const FLAT = contiguous(() => 0.10);
const RISING = contiguous((load) => 0.10 + 0.00016 * load);
// Three-hour bursts five hours apart, each opening at full tilt: the busiest
// hour of every burst is also the coldest, because it follows the idle stretch.
const BURSTY = (() => {
const shape = [[3000, 0.0], [1000, 0.7], [400, 0.7]];
const rows = [];
for (let burst = 0; burst < 21; burst += 1) {
shape.forEach(([requests, share], step) => {
rows.push(hour(burst * 8 + step, requests, share));
});
}
return rows;
})();
test('the cached share falling with load is the finding', () => {
const [quiet, busy, quietRate, busyRate] = loadSplit(SCATTER);
assert.equal(Number(quiet.toFixed(2)), 0.69);
assert.equal(Number(busy.toFixed(2)), 0.16);
assert.ok(quietRate < 300 && busyRate > 3400);
const rho = spearman(SCATTER.map((r) => r.requests),
SCATTER.map((r) => cachedShare([r])));
assert.equal(Number(rho.toFixed(3)), -1);
const [state, detail] = classify(SCATTER);
assert.equal(state, 'load-correlated-misses');
assert.match(detail, /68% in the quietest hours/);
assert.match(detail, /16% in the busiest/);
assert.match(detail, /rank correlation -1\.00/);
assert.equal(handoff(state), '');
});
test('the same traffic with a flat share is the prefix note', () => {
assert.deepEqual(FLAT.map((r) => r.requests), SCATTER.map((r) => r.requests));
assert.equal(spearman(FLAT.map((r) => r.requests),
FLAT.map((r) => cachedShare([r]))), 0);
const [state, detail] = classify(FLAT);
assert.equal(state, 'flat-low-share');
assert.match(detail, /low everywhere rather than low under load/);
assert.match(handoff(state), /cache-invalidated-by-changing-prefix/);
});
test('a share that climbs with load is not scatter', () => {
const [state, detail] = classify(RISING);
assert.equal(state, 'share-rises-with-load');
assert.match(detail, /the opposite of scatter/);
assert.equal(handoff(state), '');
});
test('hours after a gap are excluded before correlating', () => {
const everything = spearman(BURSTY.map((r) => r.requests),
BURSTY.map((r) => cachedShare([r])));
assert.equal(Number(everything.toFixed(2)), -0.87);
assert.equal(continuationRows(BURSTY).length, 42);
assert.equal(resumptionRows(BURSTY).length, 21);
assert.equal(cachedShare(continuationRows(BURSTY)), 0.7);
assert.equal(cachedShare(resumptionRows(BURSTY)), 0);
const [state, detail] = classify(BURSTY);
assert.equal(state, 'cold-only-after-idle');
assert.match(detail, /70% cached in linked hours against 0% in the 21 hour\(s\)/);
assert.match(handoff(state), /prompt-cache-retention-left-at-default/);
});
test('no cached tokens at any load is an eligibility question', () => {
const [state, detail] = classify(contiguous(() => 0));
assert.equal(state, 'no-cached-tokens');
assert.match(detail, /not one cached/);
assert.match(handoff(state), /prompt-below-model-cache-minimum/);
});
test('a flat request rate supports no verdict', () => {
const steady = Array.from({ length: 168 },
(_, i) => hour(i, 1000, i % 2 ? 0.6 : 0.2));
assert.equal(spearman(steady.map((r) => r.requests),
steady.map((r) => cachedShare([r]))), null);
assert.equal(classify(steady)[0], 'load-does-not-vary');
});
test('pooled share is weighted by traffic', () => {
const mixed = [hour(0, 10, 1.0), hour(1, 10000, 0.1)];
assert.equal(Number(cachedShare(mixed).toFixed(4)), 0.1009);
assert.equal(cachedShare([]), null);
assert.equal(cachedShare([{ input: 0, cached: 0 }]), null);
});
test('the hour index survives both shapes and midnight', () => {
assert.equal(hourIndex(1755388800), Math.floor(1755388800 / 3600));
assert.equal(hourIndex('2026-08-17T23:00Z') + 1, hourIndex('2026-08-18T00:00Z'));
assert.equal(hourLabel(hourIndex('2026-08-17T09:00Z')), '2026-08-17T09:00Z');
assert.equal(hourIndex('nonsense'), null);
assert.equal(hourIndex(null), null);
});
test('buckets are folded into project and model series', () => {
const buckets = Array.from({ length: 168 }, (_, i) => ({
start_time: (BASE + i) * 3600,
results: [{ project_id: 'proj_abc123', model: 'gpt-5.6',
num_model_requests: LOAD[i % 24],
input_tokens: LOAD[i % 24] * 2000,
input_cached_tokens: Math.trunc(LOAD[i % 24] * 2000
* (0.72 - 0.00016 * LOAD[i % 24])) }],
}));
const rows = rowsBySeries(buckets).get('proj_abc123\tgpt-5.6');
assert.equal(rows.length, 168);
assert.deepEqual(rows.map((r) => r.index), [...rows.map((r) => r.index)].sort((a, b) => a - b));
assert.equal(classify(rows)[0], 'load-correlated-misses');
});
test('thin and unreadable windows produce no verdict', () => {
const thin = Array.from({ length: 10 }, (_, i) => hour(i, 500, 0.5));
assert.equal(classify(thin)[0], 'too-few-linked-hours');
assert.equal(classify([])[0], 'too-few-linked-hours');
assert.equal(classify(null)[0], 'too-few-linked-hours');
assert.equal(spearman([1, 2], [1, 2]), null);
assert.equal(spearman([1, 2, 3], null), null);
assert.deepEqual(loadSplit([]), [null, null, null, null]);
assert.equal(rowsBySeries([{ start_time: 'bad', results: [] }]).size, 0);
});
FAQ
Why is a low cached share not enough on its own?
Because four different faults produce one. A prompt under the model's floor never caches at all. A prefix that changes between calls caches and never matches. An entry evicted during idle time is gone before the next run. And requests scattered across a fleet miss a cache that exists and is warm. Every one of those shows up as the same low number in the same field, and each has a different repair. The slope against load is what separates the fourth from the other three: it is the only one of the four that gets worse as you add capacity.
What does prompt_cache_key actually do?
It influences routing so that requests carrying the same key are more likely to reach the same cache. The documentation is careful about this and so should you be: it does not pin a request to a machine and it does not guarantee a cache read. It is a hint that raises the odds, and the effect is statistical and visible in aggregate over hours rather than deterministic per call. It is also not part of the prompt, so it does not change the prefix and cannot invalidate anything by being set.
How coarse should the key be?
Coarse enough to concentrate traffic and no coarser. The template name, or the template name plus a tenant id, is usually right. A per-request id is the classic mistake: it looks like a cache key, it satisfies whatever review asked for one, and it scatters the fleet exactly as badly as having no key at all, because no two requests ever share it. If you find yourself putting anything that varies per call into it, you have built a request id.
Why exclude the hours that follow a gap?
Because they run cold for a reason that has nothing to do with routing, and because scheduled work tends to open at its busiest. A nightly batch that fires hard at 02:00 after six idle hours produces a cold hour and a high request count at the same moment, which is a textbook negative correlation with no routing problem anywhere in it. The tests build exactly that series and check the script does not claim it. The count of dropped hours is printed so the exclusion is visible rather than assumed.
Should the cached share climb as traffic grows?
On a well-keyed fleet, yes, and the script reports that case rather than staying quiet about it. Density is what keeps entries warm, so busy hours ought to cache better than quiet ones. That is the shape to aim for after the repair, and it is a useful thing to be able to point at: a positive correlation is evidence that the prefix is stable and the routing hint is landing, which is a stronger statement than an average that happens to look acceptable this week.
Related field notes
- Every scheduled run starting on an entry that expired overnight
- A prefix that changes every call, so nothing is ever read back
- A prefix bracketed between the floors of the models it runs on
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Prompt caching — OpenAI developer docs
- Completions usage — OpenAI API reference
- Admin APIs — OpenAI developer docs
- Responses — OpenAI API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.