Diagnostic LLM APIs
Every scheduled run starts on a cache that was evicted
The nightly enrichment job has run at 02:00 for a year and its prompt has not changed since June. Its cached share is zero. Not low, and not erratic — zero, on the first hour, every single night, while the two hours that follow it cache at seventy-five per cent off the same prefix. Nothing is misconfigured. The entry written at 02:00 last night was evicted somewhere around 02:40, and by the time the job came back twenty-three hours later there was nothing left to match.
With an organization admin key: GET /v1/organization/usage/completions?start_time={T-14d}&bucket_width=1h&limit=168&group_by[]=project_id&group_by[]=model.
Do not zero-fill the idle hours. Keep only the hours that carried traffic, and label each one with how many idle hours sat immediately before it. Then compute the cached share separately for the hours that resume after a gap and the hours that simply continue from a busy hour.
The finding is a healthy continuation share next to a resumption share of roughly zero. Report it as the shortest gap length at which the share has already collapsed: one hour, two to five, six to twenty-three, or a day and over. That number is the same number as the retention setting, and it is what tells you whether the repair is a parameter or a schedule.
If the continuously busy hours are cold too, the entry is being lost while it is certainly still alive, which is a prefix problem and not this. If there are no gaps at all, read the routing note instead.
The problem in plain words
Cached prefixes do not live forever, and the default retention is measured in minutes rather than hours. On GPT-5.6 and later the window is controlled by prompt_cache_options.ttl; on earlier models by prompt_cache_retention, which accepts "in_memory" — the short default — or "24h". Neither is difficult to set. Both are almost always left alone, because the default is invisible and nothing ever fails.
The workloads this ruins are the ones that look most cacheable on paper. A nightly batch sends the same four-thousand-token prefix a hundred thousand times, so the discount ought to be enormous, and the first call of the run pays full price for a prefix that has not changed in months. A cron job every three hours does the same thing eight times a day. A tenant in a quiet time zone does it whenever they wake up. In each case the prompt is perfect and the schedule is the fault, and the usage report shows a cached share tracking request density rather than prefix stability — which is the wrong thing for it to track, and the sentence nobody says out loud.
Why it happens
The finding is positional, and nothing else in the section is. The other caching notes read a ratio against totals, against time, or against load. This one reads it against how long the traffic had been away. Two hours with identical request counts, identical input tokens and identical prompts get different verdicts here purely because of what happened in the hours before them, and that is the only signal that separates eviction from every other cause of a zero.
The idle hours must not become rows. A gap is the distance between two rows that carried traffic. Zero-filling the empty hours makes the series contiguous, which removes every gap in it and turns the check into an expensive way of computing the average. It also invents zero-share hours that would be the largest anomaly in most windows.
The shortest collapsed band is reported, not the worst one. Everything caches badly after a week; that tells you nothing. What decides the repair is whether a single idle hour was already enough, which means no retention option on offer covers the gap and the schedule has to change, or whether it takes a full day, which is precisely the case the 24-hour setting exists for. Reporting the worst band would send every reader to the same answer.
The warm baseline is the workload's own best hour. The comparison is against the same prefix on the same model at continuous traffic, not against a target borrowed from a docs page. That is what makes the foregone-token number defensible: it is the discount this exact traffic is already proving it can get, applied to the hours that did not get it. If the continuous hours are themselves weak, the script refuses the finding and says the gaps are not the main story.
Hourly buckets cannot see an idle stretch shorter than an hour, and the output says so. A gap band of "1h" is a ceiling on how quickly the entry went, not a measurement of it: the entry may well have expired thirty minutes in. The first row of the window is also dropped rather than guessed, because nothing is visible before the window opens and calling it a continuation would flatter the baseline the whole finding rests on.
The fix, as a flow
The workloads this ruins are the ones that look most cacheable on paper. A nightly batch sends the same long prefix a hundred thousand times, so the discount should be enormous, and the first call pays full price for a prompt that has not changed in months. The signal is positional: two hours with identical traffic get opposite verdicts because of what happened in the hours before them.
How to fix it
Pull a fortnight of hourly buckets per project and model
Fourteen days rather than seven, because a nightly job produces one resumption hour per day and a check that rests on thirteen of them wants more than seven. Group by project_id and model: a continuously busy service in the same project will otherwise supply a warm average that hides the batch entirely.
Keep only the hours with traffic, and measure the gaps between them
The gap before an hour is the number of empty hours immediately preceding it, computed on integer hour indices so a gap that crosses midnight is not counted wrongly. Drop the first row of the window; its gap is unknowable and guessing it either way biases the baseline.
Split into continuation hours and resumption hours
Gap of zero is a continuation, and those hours are the warm baseline. Anything above zero is a resumption. If there are almost no resumptions, this series has no idle periods and the note does not apply; if there are almost no continuations, there is no baseline to compare against and the script says so.
Bin the resumptions by gap length and find the shortest cold band
1h, 2–5h, 6–23h, 24h and over. Each band maps onto a different repair, and the shortest one already at zero is the answer. A band with only one or two hours in it decides nothing and is skipped.
Set the retention explicitly, then reshape the schedule
prompt_cache_retention="24h" on pre-GPT-5.6 routes; prompt_cache_options={"ttl": "30m"} on GPT-5.6 and later so the intent is visible in the code rather than inherited. Then the part no parameter fixes: run intermittent work in one contiguous window instead of scattering it across the day, so the first call warms an entry the rest of the batch reads.
How to check it worked
Re-read the same fortnight after the change. The band that was at zero should come up toward the continuation share; if it does not, the gap is longer than the retention you set and the remaining repair is the schedule.
python3 openai_cache_cold_after_idle.py --days 14
# cold-after-idle proj_abc123 / gpt-5.6 75% cached in continuously busy hours and 0% in the 13 hour(s) that resume after a gap of 6-23h. The prefix is fine; the entry is evicted while nobody is calling.
# after a gap of 6-23h 13 hour(s), 0% cached
# repair: the cache survives a busy hour and not a gap of 6-23h
# repair: on models before GPT-5.6, set prompt_cache_retention="24h" on this route
# repair: about 15600000 input token(s) in this window would have been cached at this workload's own continuous rate
# 4 project/model series checked, 1 finding(s)The full code
One paginated GET and no writes. The three functions that carry the note are with_gaps, bin_shares and collapse_bin. The first annotates every active hour with the idle hours before it and deliberately discards the first row of the window, because its gap is unknowable and a guess in either direction biases the baseline. The second is the shape nothing else in this section computes: a cached share per gap length. The third picks the shortest band that has already collapsed rather than the worst, because that is the number that decides between a parameter and a schedule. foregone_tokens prices the loss against this workload's own continuous rate rather than a borrowed target.
"""Find OpenAI traffic that runs cold in the hours that follow a gap.
Read only. One paginated GET against the Usage API, which needs an admin key
(sk-admin-...); a project key is rejected by every /v1/organization/ path, and
an admin key can be provisioned with the read scopes only.
Cached prefixes are evicted after an idle period, and the default window is
short. A nightly batch, a low-traffic tenant or a cron job that fires every few
hours therefore starts cold every single time, on a prefix that has not changed
in months. The signature is positional rather than arithmetic: the cold hours
are the ones that resume traffic after a gap, and the hours that follow a busy
hour are fine. Nothing about the prompt is wrong.
The finding is reported as the shortest gap length at which the share has
already collapsed, because that number and the retention setting are the same
number, and it is what tells you whether the repair is a parameter or a
schedule.
The repair is printed, never performed.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_cache_cold_after_idle")
API = "https://api.openai.com/v1"
# Gap lengths in hours, coarse enough that each bin maps onto a different
# repair. Ordered shortest first: the finding is the first one that is already
# cold, not the worst one.
BIN_ORDER = ("1h", "2-5h", "6-23h", "24h+")
FINDINGS = ("cold-after-idle",)
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def hour_index(stamp):
"""Hours since the epoch. Pure. None if unreadable.
Gaps have to be integer arithmetic. Counting idle hours by comparing
formatted stamps gets 23:00 and 00:00 wrong every night, and a nightly job
is exactly the workload this note is about.
"""
if isinstance(stamp, bool) or stamp is None:
return None
if isinstance(stamp, (int, float)):
return int(stamp) // 3600
text = str(stamp).strip().replace(" ", "T")
if len(text) < 13:
return None
head = text[:13]
if head[4] != "-" or head[7] != "-" or head[10] != "T":
return None
for part in (head[0:4], head[5:7], head[8:10], head[11:13]):
if not part.isdigit():
return None
try:
when = dt.datetime(int(head[0:4]), int(head[5:7]), int(head[8:10]),
int(head[11:13]), tzinfo=dt.timezone.utc)
except ValueError:
return None
return int(when.timestamp()) // 3600
def hour_label(index):
"""Render an hour index back as a UTC stamp. Pure."""
if index is None:
return "unknown"
when = dt.datetime.fromtimestamp(int(index) * 3600, dt.timezone.utc)
return when.strftime("%Y-%m-%dT%H:00Z")
def rows_by_series(buckets):
"""Per (project_id, model), one row per active hour, sorted. Pure.
Only hours that actually carried traffic become rows. The idle hours are
not rows and must not be: the gap is the distance between two rows, and a
zero-filled series would have no gaps in it at all.
"""
merged = {}
for bucket in buckets or []:
index = hour_index(bucket.get("start_time"))
if index is None:
continue
for result in bucket.get("results") or []:
if not isinstance(result, dict):
continue
ident = (str(result.get("project_id") or "unknown"),
str(result.get("model") or "unknown"))
row = merged.setdefault((ident, index),
{"index": index, "hour": hour_label(index),
"requests": 0, "input": 0, "cached": 0})
row["requests"] += _int(result.get("num_model_requests"))
row["input"] += _int(result.get("input_tokens"))
row["cached"] += _int(result.get("input_cached_tokens"))
out = {}
for (ident, _index), row in merged.items():
if row["requests"] > 0 or row["input"] > 0:
out.setdefault(ident, []).append(row)
for rows in out.values():
rows.sort(key=lambda r: r["index"])
return out
def cached_share(rows):
"""Pooled cached share over a set of hours. Pure. None when nothing ran."""
total = sum(_int(r.get("input")) for r in rows or [])
if total <= 0:
return None
return sum(_int(r.get("cached")) for r in rows or []) / float(total)
def with_gaps(rows):
"""Annotate each hour with the idle hours immediately before it. Pure.
The first row in the window is dropped rather than given a gap of zero.
Nothing is visible before the window starts, so whether it resumed after
idle time or continued from a busy hour is unknowable, and guessing either
way biases the very comparison this note rests on.
"""
ordered = sorted(rows or [], key=lambda r: _int(r.get("index")))
out = []
previous = None
for row in ordered:
index = _int(row.get("index"))
if previous is not None:
annotated = dict(row)
annotated["gap"] = index - previous - 1
out.append(annotated)
previous = index
return out
def gap_bin(gap):
"""Bucket a gap length into the band its repair belongs to. Pure."""
gap = _int(gap)
if gap <= 0:
return "continuous"
if gap == 1:
return "1h"
if gap <= 5:
return "2-5h"
if gap <= 23:
return "6-23h"
return "24h+"
def bin_shares(annotated):
"""Cached share per gap band. Pure. Returns {band: {hours, input, share}}.
This is the finding's shape. Everything else in the section reads a ratio
against time or against load; this reads it against how long the traffic
had been away.
"""
out = {}
for row in annotated or []:
band = gap_bin(row.get("gap"))
cell = out.setdefault(band, {"hours": 0, "input": 0, "cached": 0})
cell["hours"] += 1
cell["input"] += _int(row.get("input"))
cell["cached"] += _int(row.get("cached"))
for cell in out.values():
cell["share"] = (cell["cached"] / float(cell["input"])
if cell["input"] > 0 else None)
return out
def collapse_bin(bands, cold_ceiling=0.05, min_hours=3):
"""The shortest gap at which the share has already gone. Pure. None if none.
Shortest rather than worst on purpose. Everything caches badly after a
week; what decides the repair is whether one idle hour was already enough,
which is a retention default, or whether it takes a day, which is a
schedule.
"""
for band in BIN_ORDER:
cell = (bands or {}).get(band)
if not cell or cell.get("share") is None:
continue
if cell["hours"] >= min_hours and cell["share"] <= cold_ceiling:
return band
return None
def foregone_tokens(bands, warm_share):
"""Tokens that would have been cached at the warm rate. Pure.
The money. Uncached input in the resumption hours priced against the share
the same prefix achieves when traffic is continuous, which is the only
honest benchmark available: it is this workload's own best hour, not a
target borrowed from somewhere else.
"""
if warm_share is None:
return 0
total = 0
for band in BIN_ORDER:
cell = (bands or {}).get(band)
if not cell or cell.get("share") is None:
continue
total += int(max(0.0, warm_share - cell["share"]) * cell["input"])
return total
def handoff(state):
"""Which note owns this shape, when it is not this one. Pure."""
if state == "never-idle":
return ("this series has no gaps at all, so eviction between runs "
"cannot be the story. If the share is still low, read the "
"prompt-cache-key-not-set note and check whether it degrades "
"at peak instead.")
if state == "cold-everywhere":
return ("the continuously busy hours are cold too, so the prefix is "
"not being matched even when the entry is certainly alive. "
"Read cache-invalidated-by-changing-prefix, and "
"prompt-below-model-cache-minimum if nothing caches at all.")
return ""
def classify(rows, cold_ceiling=0.05, warm_floor=0.20, min_hours=24,
min_band_hours=3):
"""Classify one project and model series. Pure. Returns (state, detail)."""
annotated = with_gaps(rows)
if len(annotated) < min_hours:
return ("too-few-hours",
"%d usable hour(s) after dropping the first, under the floor of "
"%d" % (len(annotated), min_hours))
bands = bin_shares(annotated)
warm = bands.get("continuous") or {}
warm_share = warm.get("share")
idle_hours = sum(cell["hours"] for band, cell in bands.items()
if band != "continuous")
if idle_hours < min_band_hours:
return ("never-idle",
"%d hour(s) of traffic and only %d of them resume after a gap"
% (len(annotated), idle_hours))
if warm_share is None or warm["hours"] < min_band_hours:
return ("no-continuous-hours",
"traffic never runs two hours back to back, so there is no warm "
"baseline to compare a resumption against")
if warm_share <= cold_ceiling:
return ("cold-everywhere",
"%.0f%% cached even in continuously busy hours" % (warm_share * 100))
if warm_share < warm_floor:
return ("warm-baseline-too-weak",
"%.0f%% cached in continuously busy hours, under the floor of "
"%.0f%%. The prefix is barely caching at the best of times, so "
"the gaps are not the main story" % (warm_share * 100, warm_floor * 100))
band = collapse_bin(bands, cold_ceiling, min_band_hours)
if band is None:
return ("warm-after-idle",
"%.0f%% cached when continuous and no gap band has collapsed"
% (warm_share * 100))
cell = bands[band]
return ("cold-after-idle",
"%.0f%% cached in continuously busy hours and %.0f%% in the %d "
"hour(s) that resume after a gap of %s. The prefix is fine; the "
"entry is evicted while nobody is calling."
% (warm_share * 100, cell["share"] * 100, cell["hours"], band))
def repair_lines(band, foregone):
"""The repair, keyed to how short a gap already loses the cache. Pure."""
lines = []
if band == "1h":
lines.append("a single idle hour is already enough, so no retention "
"setting on offer covers it on its own: the 30m ttl "
"expires inside the gap.")
elif band in ("2-5h", "6-23h"):
lines.append("the cache survives a busy hour and not a gap of %s, which "
"is the default retention window doing exactly what it "
"says." % band)
elif band == "24h+":
lines.append("gaps of a day or more, which is the one case the 24h "
"retention option was added for.")
lines.extend([
"on models before GPT-5.6, set prompt_cache_retention=\"24h\" on this "
"route. It is opt-in and costs nothing extra to set.",
"on GPT-5.6 and later, set prompt_cache_options={\"ttl\": \"30m\"} "
"explicitly so the retention is visible in the code rather than "
"inherited, then check it against your actual gap length.",
"reshape the schedule. Run intermittent work in one contiguous window "
"instead of scattering it across the day, so the first call warms an "
"entry the rest of the batch reads.",
"about %d input token(s) in this window would have been cached at this "
"workload's own continuous rate." % foregone,
])
return lines
def window_start(days):
"""Floor to the hour so start_time lands on a bucket boundary."""
now = dt.datetime.now(dt.timezone.utc).replace(minute=0, second=0, microsecond=0)
return int((now - dt.timedelta(days=days)).timestamp())
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/ needs an admin key "
"(sk-admin-...), not a project key" % r.status_code)
r.raise_for_status()
return r.json()
def read_buckets(session, path, params):
"""Walk the paginated usage endpoint."""
params = dict(params)
while True:
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=14,
help="days of hourly buckets to read (max 30)")
ap.add_argument("--cold-ceiling", type=float, default=0.05,
help="cached share at or below which a band counts as cold")
ap.add_argument("--show-all", action="store_true",
help="also print series that are behaving")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key "
"(sk-admin-...); a project key cannot read /v1/organization/")
return 2
days = max(2, min(int(args.days), 30))
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
buckets = read_buckets(session, "/organization/usage/completions", {
"start_time": window_start(days),
"bucket_width": "1h",
"limit": 168,
"group_by[]": ["project_id", "model"],
})
series = rows_by_series(buckets)
if not series:
log.info("no completions usage in the last %d day(s)", days)
return 0
checked = 0
bad = 0
for ident in sorted(series):
rows = series[ident]
state, detail = classify(rows, args.cold_ceiling)
checked += 1
line = "%-24s %s / %s %s" % (state, ident[0], ident[1], detail)
if state in FINDINGS:
bad += 1
log.warning(line)
bands = bin_shares(with_gaps(rows))
warm = (bands.get("continuous") or {}).get("share")
band = collapse_bin(bands, args.cold_ceiling)
for name in BIN_ORDER:
cell = bands.get(name)
if cell and cell.get("share") is not None:
log.warning(" after a gap of %-5s %d hour(s), %.0f%% cached",
name, cell["hours"], cell["share"] * 100)
for repair in repair_lines(band, foregone_tokens(bands, warm)):
log.warning(" repair: %s", repair)
log.warning(" note: hourly buckets cannot see an idle stretch "
"shorter than an hour, so a gap band of 1h is a ceiling "
"on how quickly the entry actually went.")
else:
note = handoff(state)
if note:
log.info(line)
log.info(" %s", note)
elif args.show_all:
log.info(line)
log.info("%d project/model series checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI traffic that runs cold in the hours that follow a gap.
*
* Read only. One paginated GET against the Usage API, which needs an admin key
* (sk-admin-...). A project key is rejected by /v1/organization/.
*
* Cached prefixes are evicted after an idle period and the default window is
* short, so a nightly batch or a cron job starts cold every time on a prefix
* that has not changed in months. The signature is positional rather than
* arithmetic: the cold hours are the ones that resume after a gap, and the
* hours that follow a busy hour are fine.
*
* The finding is the shortest gap length at which the share has already
* collapsed, because that number and the retention setting are the same
* number, and it says whether the repair is a parameter or a schedule.
*/
const API = 'https://api.openai.com/v1';
/**
* Gap lengths in hours, coarse enough that each band maps onto a different
* repair. Shortest first: the finding is the first band that is already cold.
*/
export const BIN_ORDER = ['1h', '2-5h', '6-23h', '24h+'];
const FINDINGS = new Set(['cold-after-idle']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Hours since the epoch. Pure. Null if unreadable.
* Gaps have to be integer arithmetic: counting idle hours by comparing
* formatted stamps gets 23:00 and 00:00 wrong every night, and a nightly job
* is exactly the workload this note is about.
*/
export function hourIndex(stamp) {
if (typeof stamp === 'boolean' || stamp === null || stamp === undefined) return null;
if (typeof stamp === 'number' && Number.isFinite(stamp)) {
return Math.floor(Math.trunc(stamp) / 3600);
}
const text = String(stamp).trim().replace(' ', 'T');
if (text.length < 13) return null;
const head = text.slice(0, 13);
if (!/^\d{4}-\d{2}-\d{2}T\d{2}$/.test(head)) return null;
const when = Date.parse(`${head}:00:00Z`);
if (Number.isNaN(when)) return null;
return Math.floor(when / 3600000);
}
/** Render an hour index back as a UTC stamp. Pure. */
export function hourLabel(index) {
if (index === null || index === undefined) return 'unknown';
return `${new Date(Math.trunc(index) * 3600000).toISOString().slice(0, 13)}:00Z`;
}
/**
* Per project_id and model, one row per active hour, sorted. Pure.
* Only hours that carried traffic become rows, and they must not be
* zero-filled: the gap is the distance between two rows.
*/
export function rowsBySeries(buckets) {
const merged = new Map();
for (const bucket of buckets ?? []) {
const index = hourIndex(bucket?.start_time);
if (index === null) continue;
for (const result of bucket?.results ?? []) {
if (!result || typeof result !== 'object') continue;
const ident = `${result.project_id ?? 'unknown'}\t${result.model ?? 'unknown'}`;
const cell = `${ident}\t${index}`;
if (!merged.has(cell)) {
merged.set(cell, { ident, index, hour: hourLabel(index),
requests: 0, input: 0, cached: 0 });
}
const row = merged.get(cell);
row.requests += readInt(result.num_model_requests);
row.input += readInt(result.input_tokens);
row.cached += readInt(result.input_cached_tokens);
}
}
const out = new Map();
for (const row of merged.values()) {
if (row.requests <= 0 && row.input <= 0) continue;
if (!out.has(row.ident)) out.set(row.ident, []);
out.get(row.ident).push(row);
}
for (const rows of out.values()) rows.sort((a, b) => a.index - b.index);
return out;
}
/** Pooled cached share over a set of hours. Pure. Null when nothing ran. */
export function cachedShare(rows) {
let input = 0;
let cached = 0;
for (const row of rows ?? []) {
input += readInt(row?.input);
cached += readInt(row?.cached);
}
if (input <= 0) return null;
return cached / input;
}
/**
* Annotate each hour with the idle hours immediately before it. Pure.
* The first row is dropped rather than given a gap of zero: nothing is visible
* before the window starts, and guessing either way biases the comparison this
* note rests on.
*/
export function withGaps(rows) {
const ordered = [...(rows ?? [])].sort((a, b) => readInt(a?.index) - readInt(b?.index));
const out = [];
let previous = null;
for (const row of ordered) {
const index = readInt(row?.index);
if (previous !== null) out.push({ ...row, gap: index - previous - 1 });
previous = index;
}
return out;
}
/** Bucket a gap length into the band its repair belongs to. Pure. */
export function gapBin(gap) {
const n = readInt(gap);
if (n <= 0) return 'continuous';
if (n === 1) return '1h';
if (n <= 5) return '2-5h';
if (n <= 23) return '6-23h';
return '24h+';
}
/**
* Cached share per gap band. Pure. The finding's shape: everything else in the
* section reads a ratio against time or against load, this reads it against
* how long the traffic had been away.
*/
export function binShares(annotated) {
const out = {};
for (const row of annotated ?? []) {
const band = gapBin(row?.gap);
if (!out[band]) out[band] = { hours: 0, input: 0, cached: 0, share: null };
out[band].hours += 1;
out[band].input += readInt(row?.input);
out[band].cached += readInt(row?.cached);
}
for (const cell of Object.values(out)) {
cell.share = cell.input > 0 ? cell.cached / cell.input : null;
}
return out;
}
/**
* The shortest gap at which the share has already gone. Pure. Null if none.
* Shortest rather than worst on purpose: what decides the repair is whether
* one idle hour was already enough, or whether it takes a day.
*/
export function collapseBin(bands, coldCeiling = 0.05, minHours = 3) {
for (const band of BIN_ORDER) {
const cell = (bands ?? {})[band];
if (!cell || cell.share === null || cell.share === undefined) continue;
if (cell.hours >= minHours && cell.share <= coldCeiling) return band;
}
return null;
}
/**
* Tokens that would have been cached at the warm rate. Pure.
* Priced against the share this same prefix achieves when traffic is
* continuous, which is this workload's own best hour rather than a borrowed
* target.
*/
export function foregoneTokens(bands, warmShare) {
if (warmShare === null || warmShare === undefined) return 0;
let total = 0;
for (const band of BIN_ORDER) {
const cell = (bands ?? {})[band];
if (!cell || cell.share === null || cell.share === undefined) continue;
total += Math.trunc(Math.max(0, warmShare - cell.share) * cell.input);
}
return total;
}
/** Which note owns this shape, when it is not this one. Pure. */
export function handoff(state) {
if (state === 'never-idle') {
return 'this series has no gaps at all, so eviction between runs cannot be '
+ 'the story. If the share is still low, read the prompt-cache-key-not-set '
+ 'note and check whether it degrades at peak instead.';
}
if (state === 'cold-everywhere') {
return 'the continuously busy hours are cold too, so the prefix is not being '
+ 'matched even when the entry is certainly alive. Read '
+ 'cache-invalidated-by-changing-prefix, and '
+ 'prompt-below-model-cache-minimum if nothing caches at all.';
}
return '';
}
/** Classify one project and model series. Pure. Returns [state, detail]. */
export function classify(rows, coldCeiling = 0.05, warmFloor = 0.20,
minHours = 24, minBandHours = 3) {
const annotated = withGaps(rows);
if (annotated.length < minHours) {
return ['too-few-hours',
`${annotated.length} usable hour(s) after dropping the first, under the `
+ `floor of ${minHours}`];
}
const bands = binShares(annotated);
const warm = bands.continuous ?? {};
const warmShare = warm.share ?? null;
let idleHours = 0;
for (const [band, cell] of Object.entries(bands)) {
if (band !== 'continuous') idleHours += cell.hours;
}
if (idleHours < minBandHours) {
return ['never-idle',
`${annotated.length} hour(s) of traffic and only ${idleHours} of them `
+ 'resume after a gap'];
}
if (warmShare === null || (warm.hours ?? 0) < minBandHours) {
return ['no-continuous-hours',
'traffic never runs two hours back to back, so there is no warm baseline '
+ 'to compare a resumption against'];
}
if (warmShare <= coldCeiling) {
return ['cold-everywhere',
`${(warmShare * 100).toFixed(0)}% cached even in continuously busy hours`];
}
if (warmShare < warmFloor) {
return ['warm-baseline-too-weak',
`${(warmShare * 100).toFixed(0)}% cached in continuously busy hours, under `
+ `the floor of ${(warmFloor * 100).toFixed(0)}%. The prefix is barely `
+ 'caching at the best of times, so the gaps are not the main story'];
}
const band = collapseBin(bands, coldCeiling, minBandHours);
if (band === null) {
return ['warm-after-idle',
`${(warmShare * 100).toFixed(0)}% cached when continuous and no gap band has collapsed`];
}
const cell = bands[band];
return ['cold-after-idle',
`${(warmShare * 100).toFixed(0)}% cached in continuously busy hours and `
+ `${(cell.share * 100).toFixed(0)}% in the ${cell.hours} hour(s) that resume `
+ `after a gap of ${band}. The prefix is fine; the entry is evicted while `
+ 'nobody is calling.'];
}
/** The repair, keyed to how short a gap already loses the cache. Pure. */
export function repairLines(band, foregone) {
const lines = [];
if (band === '1h') {
lines.push('a single idle hour is already enough, so no retention setting '
+ 'on offer covers it on its own: the 30m ttl expires inside the gap.');
} else if (band === '2-5h' || band === '6-23h') {
lines.push(`the cache survives a busy hour and not a gap of ${band}, which `
+ 'is the default retention window doing exactly what it says.');
} else if (band === '24h+') {
lines.push('gaps of a day or more, which is the one case the 24h retention '
+ 'option was added for.');
}
lines.push(
'on models before GPT-5.6, set prompt_cache_retention="24h" on this route. '
+ 'It is opt-in and costs nothing extra to set.',
'on GPT-5.6 and later, set prompt_cache_options={"ttl": "30m"} explicitly so '
+ 'the retention is visible in the code rather than inherited, then check it '
+ 'against your actual gap length.',
'reshape the schedule. Run intermittent work in one contiguous window '
+ 'instead of scattering it across the day, so the first call warms an entry '
+ 'the rest of the batch reads.',
`about ${foregone} input token(s) in this window would have been cached at `
+ "this workload's own continuous rate.",
);
return lines;
}
function windowStart(days) {
const now = new Date();
now.setUTCMinutes(0, 0, 0);
return Math.floor((now.getTime() - days * 86400000) / 1000);
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization/ needs an admin `
+ 'key (sk-admin-...), not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* readBuckets(key, path, params) {
let query = { ...params };
for (;;) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key '
+ '(sk-admin-...); a project key cannot read /v1/organization/');
process.exitCode = 2;
return;
}
const days = Math.max(2, Math.min(Number(process.env.DAYS ?? 14), 30));
const coldCeiling = Number(process.env.COLD_CEILING ?? 0.05);
const showAll = process.env.SHOW_ALL === '1';
const buckets = [];
for await (const bucket of readBuckets(admin, '/organization/usage/completions', {
start_time: windowStart(days),
bucket_width: '1h',
limit: 168,
'group_by[]': ['project_id', 'model'],
})) buckets.push(bucket);
const series = rowsBySeries(buckets);
if (series.size === 0) {
console.log(`no completions usage in the last ${days} day(s)`);
return;
}
let checked = 0;
let bad = 0;
for (const ident of [...series.keys()].sort()) {
const rows = series.get(ident);
const [state, detail] = classify(rows, coldCeiling);
checked += 1;
const line = `${state.padEnd(24)} ${ident.replace('\t', ' / ')} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
const bands = binShares(withGaps(rows));
const warm = bands.continuous?.share ?? null;
const band = collapseBin(bands, coldCeiling);
for (const name of BIN_ORDER) {
const cell = bands[name];
if (cell && cell.share !== null) {
console.warn(` after a gap of ${name.padEnd(5)} ${cell.hours} hour(s), `
+ `${(cell.share * 100).toFixed(0)}% cached`);
}
}
for (const repair of repairLines(band, foregoneTokens(bands, warm))) {
console.warn(` repair: ${repair}`);
}
console.warn(' note: hourly buckets cannot see an idle stretch shorter '
+ 'than an hour, so a gap band of 1h is a ceiling on how '
+ 'quickly the entry actually went.');
} else {
const note = handoff(state);
if (note) {
console.log(line);
console.log(` ${note}`);
} else if (showAll) {
console.log(line);
}
}
}
console.log(`${checked} project/model series checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The main fixture is a batch that runs 02:00 to 05:00 and then sleeps for twenty-one hours, and its load-bearing detail is that every hour sends exactly the same number of requests. Nothing about volume separates the cold hour from the warm ones; only its position after the gap does, which is the claim the note makes and the reason it is not the routing note wearing a different hat. Beside it sits a two-hours-on, one-hour-off series whose collapse is already total after a single idle hour, which is a different repair and a different printed line. Then the refusals: a batch whose continuous hours are cold too, a series with no gaps at all, and a warm baseline too weak to compare against.
from openai_cache_cold_after_idle import (bin_shares, cached_share, classify,
collapse_bin, foregone_tokens,
gap_bin, handoff, hour_index,
hour_label, repair_lines,
rows_by_series, with_gaps)
BASE = hour_index("2026-08-17T00:00Z")
def hour(offset, share, requests=800):
tokens = requests * 2000
return {"index": BASE + offset, "hour": hour_label(BASE + offset),
"requests": requests, "input": tokens,
"cached": int(round(tokens * share))}
def nightly(resume_share=0.0, warm_share=0.75):
"""A batch that runs 02:00 to 05:00 and then sleeps for twenty-one hours.
Every hour sends the same number of requests, so nothing about load can
explain the difference between them.
"""
rows = []
for day in range(14):
for step in range(3):
rows.append(hour(day * 24 + 2 + step,
resume_share if step == 0 else warm_share))
return rows
def two_on_one_off(resume_share=0.0, warm_share=0.70):
"""Two hours on, one hour off, for a fortnight. Every gap is a single hour."""
rows = []
for pair in range(112):
rows.append(hour(pair * 3, resume_share))
rows.append(hour(pair * 3 + 1, warm_share))
return rows
NIGHTLY = nightly()
HOURLY_GAPS = two_on_one_off()
CONTINUOUS = [hour(i, 0.60) for i in range(336)]
def test_the_share_against_gap_length_is_the_finding():
# The note in one assertion. Same prefix, same request rate, and the only
# thing that separates a cold hour from a warm one is what happened in the
# twenty-one hours before it.
annotated = with_gaps(NIGHTLY)
assert len(annotated) == 41
assert {r["requests"] for r in annotated} == {800}
bands = bin_shares(annotated)
assert bands["continuous"]["hours"] == 28
assert bands["continuous"]["share"] == 0.75
assert bands["6-23h"]["hours"] == 13
assert bands["6-23h"]["share"] == 0.0
assert collapse_bin(bands) == "6-23h"
state, detail = classify(NIGHTLY)
assert state == "cold-after-idle"
assert "75% cached in continuously busy hours" in detail
assert "0% in the 13 hour(s) that resume after a gap of 6-23h" in detail
assert handoff(state) == ""
def test_the_shortest_collapsed_band_is_the_one_reported():
# Shortest, not worst. A single idle hour losing the entry is a retention
# default; a day losing it is a schedule, and the repair differs.
bands = bin_shares(with_gaps(HOURLY_GAPS))
assert bands["1h"]["hours"] == 111
assert bands["1h"]["share"] == 0.0
assert collapse_bin(bands) == "1h"
state, detail = classify(HOURLY_GAPS)
assert state == "cold-after-idle"
assert "gap of 1h" in detail
assert "a single idle hour is already enough" in repair_lines("1h", 0)[0]
assert "24h retention option" in repair_lines("24h+", 0)[0]
def test_a_series_with_no_gaps_is_someone_elses_note():
state, detail = classify(CONTINUOUS)
assert state == "never-idle"
assert "only 0 of them resume after a gap" in detail
assert "prompt-cache-key-not-set" in handoff(state)
def test_cold_in_the_busy_hours_too_is_not_eviction():
# If the entry is cold when it is certainly still alive, the gap is not
# what is losing it.
state, detail = classify(nightly(resume_share=0.0, warm_share=0.0))
assert state == "cold-everywhere"
assert "0% cached even in continuously busy hours" in detail
assert "cache-invalidated-by-changing-prefix" in handoff(state)
assert "prompt-below-model-cache-minimum" in handoff(state)
def test_a_weak_warm_baseline_refuses_the_finding():
state, detail = classify(nightly(resume_share=0.0, warm_share=0.10))
assert state == "warm-baseline-too-weak"
assert "barely caching at the best of times" in detail
def test_a_batch_that_resumes_warm_is_not_a_finding():
state, detail = classify(nightly(resume_share=0.55))
assert state == "warm-after-idle"
assert "no gap band has collapsed" in detail
def test_the_first_hour_of_the_window_is_dropped_not_guessed():
# Nothing is visible before the window starts, so the first row's gap is
# unknowable and counting it as continuous would flatter the baseline.
rows = [hour(0, 0.0), hour(1, 0.9), hour(9, 0.0), hour(10, 0.9)]
annotated = with_gaps(rows)
assert [r["gap"] for r in annotated] == [0, 7, 0]
assert len(annotated) == len(rows) - 1
assert with_gaps([hour(0, 0.5)]) == []
assert with_gaps([]) == []
def test_the_gap_bands_line_up_with_the_repairs():
assert gap_bin(0) == "continuous"
assert gap_bin(1) == "1h"
assert gap_bin(2) == "2-5h" and gap_bin(5) == "2-5h"
assert gap_bin(6) == "6-23h" and gap_bin(23) == "6-23h"
assert gap_bin(24) == "24h+" and gap_bin(500) == "24h+"
# A band with almost no hours in it cannot decide anything.
thin = {"1h": {"hours": 1, "input": 100, "cached": 0, "share": 0.0},
"24h+": {"hours": 40, "input": 100, "cached": 0, "share": 0.0}}
assert collapse_bin(thin) == "24h+"
def test_the_foregone_tokens_are_priced_at_the_workloads_own_warm_rate():
bands = bin_shares(with_gaps(NIGHTLY))
# 13 resumption hours of 1.6M input tokens, 75% of which would have been
# cached had the entry survived the night.
assert bands["6-23h"]["input"] == 13 * 800 * 2000
assert foregone_tokens(bands, 0.75) == 15_600_000
assert foregone_tokens(bands, None) == 0
assert foregone_tokens({}, 0.75) == 0
def test_buckets_are_folded_and_idle_hours_never_become_rows():
buckets = [{"start_time": (BASE + day * 24 + 2 + step) * 3600,
"results": [{"project_id": "proj_abc123", "model": "gpt-5.6",
"num_model_requests": 800,
"input_tokens": 1_600_000,
"input_cached_tokens": 0 if step == 0 else 1_200_000}]}
for day in range(14) for step in range(3)]
series = rows_by_series(buckets)
rows = series[("proj_abc123", "gpt-5.6")]
assert len(rows) == 42
assert cached_share(rows) == 0.5
assert classify(rows)[0] == "cold-after-idle"
def test_thin_and_unreadable_windows_produce_no_verdict():
assert classify([hour(i, 0.5) for i in range(10)])[0] == "too-few-hours"
assert classify([])[0] == "too-few-hours"
assert classify(None)[0] == "too-few-hours"
assert cached_share([]) is None
assert bin_shares([]) == {}
assert collapse_bin({}) is None
assert hour_index("nonsense") is None
assert rows_by_series([{"start_time": "bad", "results": []}]) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { binShares, cachedShare, classify, collapseBin, foregoneTokens, gapBin,
handoff, hourIndex, hourLabel, repairLines, rowsBySeries, withGaps }
from './openai-cache-cold-after-idle.mjs';
const BASE = hourIndex('2026-08-17T00:00Z');
const hour = (offset, share, requests = 800) => {
const tokens = requests * 2000;
return { index: BASE + offset, hour: hourLabel(BASE + offset), requests,
input: tokens, cached: Math.round(tokens * share) };
};
/** A batch that runs 02:00 to 05:00 and then sleeps for twenty-one hours. */
const nightly = (resumeShare = 0.0, warmShare = 0.75) => {
const rows = [];
for (let day = 0; day < 14; day += 1) {
for (let step = 0; step < 3; step += 1) {
rows.push(hour(day * 24 + 2 + step, step === 0 ? resumeShare : warmShare));
}
}
return rows;
};
/** Two hours on, one hour off, for a fortnight. Every gap is a single hour. */
const twoOnOneOff = (resumeShare = 0.0, warmShare = 0.70) => {
const rows = [];
for (let pair = 0; pair < 112; pair += 1) {
rows.push(hour(pair * 3, resumeShare));
rows.push(hour(pair * 3 + 1, warmShare));
}
return rows;
};
const NIGHTLY = nightly();
const HOURLY_GAPS = twoOnOneOff();
const CONTINUOUS = Array.from({ length: 336 }, (_, i) => hour(i, 0.60));
test('the share against gap length is the finding', () => {
const annotated = withGaps(NIGHTLY);
assert.equal(annotated.length, 41);
assert.deepEqual([...new Set(annotated.map((r) => r.requests))], [800]);
const bands = binShares(annotated);
assert.equal(bands.continuous.hours, 28);
assert.equal(bands.continuous.share, 0.75);
assert.equal(bands['6-23h'].hours, 13);
assert.equal(bands['6-23h'].share, 0);
assert.equal(collapseBin(bands), '6-23h');
const [state, detail] = classify(NIGHTLY);
assert.equal(state, 'cold-after-idle');
assert.match(detail, /75% cached in continuously busy hours/);
assert.match(detail, /0% in the 13 hour\(s\) that resume after a gap of 6-23h/);
assert.equal(handoff(state), '');
});
test('the shortest collapsed band is the one reported', () => {
const bands = binShares(withGaps(HOURLY_GAPS));
assert.equal(bands['1h'].hours, 111);
assert.equal(bands['1h'].share, 0);
assert.equal(collapseBin(bands), '1h');
const [state, detail] = classify(HOURLY_GAPS);
assert.equal(state, 'cold-after-idle');
assert.match(detail, /gap of 1h/);
assert.match(repairLines('1h', 0)[0], /a single idle hour is already enough/);
assert.match(repairLines('24h+', 0)[0], /24h retention option/);
});
test('a series with no gaps is someone elses note', () => {
const [state, detail] = classify(CONTINUOUS);
assert.equal(state, 'never-idle');
assert.match(detail, /only 0 of them resume after a gap/);
assert.match(handoff(state), /prompt-cache-key-not-set/);
});
test('cold in the busy hours too is not eviction', () => {
const [state, detail] = classify(nightly(0.0, 0.0));
assert.equal(state, 'cold-everywhere');
assert.match(detail, /0% cached even in continuously busy hours/);
assert.match(handoff(state), /cache-invalidated-by-changing-prefix/);
assert.match(handoff(state), /prompt-below-model-cache-minimum/);
});
test('a weak warm baseline refuses the finding', () => {
const [state, detail] = classify(nightly(0.0, 0.10));
assert.equal(state, 'warm-baseline-too-weak');
assert.match(detail, /barely caching at the best of times/);
});
test('a batch that resumes warm is not a finding', () => {
const [state, detail] = classify(nightly(0.55));
assert.equal(state, 'warm-after-idle');
assert.match(detail, /no gap band has collapsed/);
});
test('the first hour of the window is dropped not guessed', () => {
const rows = [hour(0, 0.0), hour(1, 0.9), hour(9, 0.0), hour(10, 0.9)];
const annotated = withGaps(rows);
assert.deepEqual(annotated.map((r) => r.gap), [0, 7, 0]);
assert.equal(annotated.length, rows.length - 1);
assert.deepEqual(withGaps([hour(0, 0.5)]), []);
assert.deepEqual(withGaps([]), []);
});
test('the gap bands line up with the repairs', () => {
assert.equal(gapBin(0), 'continuous');
assert.equal(gapBin(1), '1h');
assert.equal(gapBin(2), '2-5h');
assert.equal(gapBin(5), '2-5h');
assert.equal(gapBin(6), '6-23h');
assert.equal(gapBin(23), '6-23h');
assert.equal(gapBin(24), '24h+');
assert.equal(gapBin(500), '24h+');
const thin = { '1h': { hours: 1, input: 100, cached: 0, share: 0 },
'24h+': { hours: 40, input: 100, cached: 0, share: 0 } };
assert.equal(collapseBin(thin), '24h+');
});
test('the foregone tokens are priced at the workloads own warm rate', () => {
const bands = binShares(withGaps(NIGHTLY));
assert.equal(bands['6-23h'].input, 13 * 800 * 2000);
assert.equal(foregoneTokens(bands, 0.75), 15600000);
assert.equal(foregoneTokens(bands, null), 0);
assert.equal(foregoneTokens({}, 0.75), 0);
});
test('buckets are folded and idle hours never become rows', () => {
const buckets = [];
for (let day = 0; day < 14; day += 1) {
for (let step = 0; step < 3; step += 1) {
buckets.push({
start_time: (BASE + day * 24 + 2 + step) * 3600,
results: [{ project_id: 'proj_abc123', model: 'gpt-5.6',
num_model_requests: 800, input_tokens: 1600000,
input_cached_tokens: step === 0 ? 0 : 1200000 }],
});
}
}
const rows = rowsBySeries(buckets).get('proj_abc123\tgpt-5.6');
assert.equal(rows.length, 42);
assert.equal(cachedShare(rows), 0.5);
assert.equal(classify(rows)[0], 'cold-after-idle');
});
test('thin and unreadable windows produce no verdict', () => {
assert.equal(classify(Array.from({ length: 10 }, (_, i) => hour(i, 0.5)))[0],
'too-few-hours');
assert.equal(classify([])[0], 'too-few-hours');
assert.equal(classify(null)[0], 'too-few-hours');
assert.equal(cachedShare([]), null);
assert.deepEqual(binShares([]), {});
assert.equal(collapseBin({}), null);
assert.equal(hourIndex('nonsense'), null);
assert.equal(rowsBySeries([{ start_time: 'bad', results: [] }]).size, 0);
});
FAQ
How is this different from the routing note?
The routing note asks whether the cached share moves with the request rate; this one asks whether it moves with the time since the last request. They are deliberately built to hand each other the cases they cannot claim. The routing check drops every post-gap hour before it correlates anything, precisely so an eviction problem cannot be read as scatter. This check requires gaps to exist at all, and when a series has none it says the eviction story is unavailable and names the routing note. A workload can have both problems, and then both scripts fire.
Why bin by gap length instead of just reporting the resumption share?
Because the resumption share tells you there is a problem and the bins tell you which repair. If the share is already zero after one idle hour, no retention option on offer covers it, the 30-minute ttl expires inside the gap, and the only real fix is to stop scattering the work. If it survives five hours and dies at twenty-four, the 24-hour retention setting is exactly the thing that was built for you. Same symptom, opposite conclusions, and the only difference between them is the length of the gap.
Why not zero-fill the idle hours to get a regular series?
Because that deletes the finding. A gap is the distance between two hours that carried traffic; zero-filling makes every series contiguous and there are no gaps left to measure. It also invents hours with zero requests and zero cached tokens, which then look like the most dramatic collapse in the window and have to be explained away by every branch downstream. The idle hours are absent from the report for a good reason and they stay absent here.
Can hourly buckets really see this?
They can see the shape, not the moment. A collapse in the one-hour band means the entry was gone within an hour, which is a ceiling on how long it survived and not a measurement: it may well have expired thirty minutes in. That is stated in the output rather than glossed over. For the repairs this check leads to it does not matter much, because the choice between a retention parameter and a schedule change turns on whether the gap is hours or a day, and hourly resolution answers that cleanly.
How is the wasted spend calculated?
Uncached input in the resumption hours, priced at the cached share the same prefix achieves in continuously busy hours. That benchmark is deliberately this workload's own best hour rather than a figure from a pricing page, so it cannot be argued with: the traffic has already demonstrated it can hit that share on this exact prompt and model. If the continuous hours are weak, the script refuses the finding entirely rather than producing a number nobody can defend.
Related field notes
- A cached share that degrades exactly when the fleet is widest
- Cache writes paid for at a premium and never read back
- A prefix that changes every call, so nothing is ever read back
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Prompt caching — OpenAI developer docs
- Usage — OpenAI API reference
- Completions usage API — OpenAI Cookbook
- Admin APIs — OpenAI developer docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.