Diagnostic LLM APIs
Cache read share stepped down the day the model changed
The migration was a one-line change and it went perfectly: latency improved, quality held, the per-token rate went down. Three weeks later the input bill was up by a third and nobody could see why, because the thing that changed is not on the invoice as a line item. The cache-read share was sixty-eight per cent for a fortnight, dropped to eleven on the day the new model id first appears in the usage report, and has been eleven ever since. The prompt was never edited.
The short answer
With an Admin API key: GET /v1/organizations/usage_report/messages?starting_at={T-31d}&bucket_width=1d&limit=32&group_by[]=model. Per day, cache_read_input_tokens / (cache_read_input_tokens + uncached_input_tokens).
Find the day each model id first appears. Then do the part that makes the claim falsifiable: compute the largest downward step anywhere in the series, and require it to land on the arrival day. A collapse that is biggest three weeks either side of the switch was not caused by the switch, and the script says so.
Throw the switch day itself away. Caches are keyed per model, so the first day on a new id is cold by definition and every byte of it is expected. The comparison is the days before against the days after, with the day between them excluded. A dip that recovers is a cache warming up; only a floor that never comes back is structural.
The usual mechanism is the minimum cacheable length: Opus 4.6, Opus 4.5 and Haiku 4.5 need 4,096 tokens where Opus 5 needs 512, so a prompt that never changed can stop qualifying overnight. That is the floor note, and this one names it.
The problem in plain words
Model migrations are reviewed for output quality and for the per-token rate, and both of those are visible before you ship. What is not visible is that a cache is keyed per model, so the switch throws away every warm entry you had and starts again. That part is fine and everybody expects it: one cold day, then back to normal.
The failure is when it does not come back. Three things can hold it down and none of them is in the diff. The new model's minimum cacheable prompt length may be higher, so a prefix that used to qualify no longer does — and the jump is not small, from 512 tokens on Opus 5 to 4,096 on Haiku 4.5. Thinking or effort defaults differ per model and sit inside the cached prefix, so a different default is a different prefix. And a newer tokenizer can produce materially more tokens for the same text, which moves a prefix that used to sit just above a boundary. In all three cases the prompt is untouched, the code is untouched, and the caching outcome changed anyway.
Why it happens
The switch day is excluded, and that single decision is most of the note. A cold cache on day one of a new model is correct behaviour, so a check that averages that day into either side turns a successful migration into an alarm. Every organization that upgrades a model would fire it. The comparison is strictly the days before against the days after, and the dip on the day itself is reported as context rather than evidence.
Alignment is checked against the whole window, not assumed. Any thirty-one day window containing both a new model and a decline can be told as a causal story if you only look where you expect. So the script computes the largest downward step at every possible split point and requires the winner to be the arrival. If the biggest fall is somewhere else, the verdict is that something else changed on that day, and the reader is sent to look at deploys with a date in hand.
A model nobody uses is never blamed. A canary taking three per cent of input cannot move an organization-wide ratio, but it will happily appear as a new id on exactly the day someone else broke the prefix. The check requires the new model to carry at least a fifth of input since it arrived before it is allowed to be the explanation.
Sustained means every day after sits below every day before. Not a lower mean, which a single bad week can produce; a floor that never recovers. A window where some days since the switch have climbed back above the pre-switch minimum is reported as suggestive and not conclusive, with the advice to widen the window rather than a verdict.
This is an organization-wide ratio and the note says so out loud. Grouped by model, a second workload that changed on the same day is folded into the same number, and the report carries no request count to normalise against. The output is a date and a magnitude, which is enough to line up against a deploy, and it is deliberately not enough to skip that step.
The fix, as a flow
A model switch starts from a cold cache by definition, so the first day is expected to be bad and a check that fires on it is simply wrong. What matters is the days after. The claim that the collapse belongs to the switch then has to survive a search of the whole window: the largest step down anywhere in it must land on the day the new id first appears, or the switch is not the explanation.
How to fix it
Pull thirty-one daily buckets grouped by model
Daily rather than hourly, because the thing being detected is a permanent shift and not a spacing. Grouping by model is what supplies the arrival dates; the share itself is computed across all models, because the question is what happened to the organization's caching, not to one id's.
Skip the days with no traffic instead of zero-filling them
A day with no input has no share. Inventing a zero for a quiet weekend creates the largest step in most windows, and every branch afterwards then has to explain it away.
Find the arrival day of each model id, and ignore day one
A model present on the first day of the window may have been running for a year; its "arrival" is an artefact of where you started reading. Only ids that first appear after the window opens count. Among those, take the one that carries the most input afterwards.
Compute the step across the arrival, with that day removed
Mean share before against mean share after, excluding the arrival day itself. A drop of about fifteen share points, with the after-share under sixty per cent of the before-share, is a step worth explaining. Then check it is sustained: every day since below every day before.
Prove the alignment, then read the floors
Compute the largest step at every split point and require it at the arrival. Once it holds, compare the two models' minimum cacheable lengths and their thinking and effort defaults, move the breakpoint, and re-measure over three days rather than one — the day after any breakpoint change is cold for the same reason the switch day was.
How to check it worked
Re-run after the breakpoint moves. Give it three days: the first is cold by construction, and a one-day read on this check is how a successful repair gets rolled back.
python3 anthropic_cache_step_after_model_switch.py --days 31
# collapsed-after-model-change cache-read share 70% before claude-haiku-4-5-20251001 arrived on 2026-08-16 and 10% after, with the switch day itself excluded. claude-haiku-4-5-20251001 now carries 100% of input and the largest step in the window is exactly there.
# repair: claude-haiku-4-5-20251001 needs 4096 tokens before a prefix is cacheable and claude-opus-5 needed 512
# repair: compare their thinking and effort defaults
# note: this is an organization-wide ratio.
# 31 day(s) checked, 1 finding(s)The full code
One GET and eleven pure functions, three of which exist to try to falsify the finding rather than to make it. step_at measures the change across the arrival with that day deleted, because a cold first day is what a model switch is supposed to cost. best_split computes the largest downward step at every possible split point, so the claim "the collapse aligns with the switch" has to survive a search of the whole window rather than being asserted where it was expected. input_share_after refuses to blame a model that carries three per cent of traffic. floor_note reaches for the mechanism only once the step is confirmed, and names the sibling note that owns it.
Get this script on GitHub
Follow @allanninal
Python and Node.js, with tests. Read only, it never writes. One of 97 LLM API fixes, free and open source.
anthropic_cache_step_after_model_switch.py
"""Align a collapse in cache-read share with the day a new model id appeared.
Read only. One GET against the Admin API, which needs an Admin API key
(sk-ant-admin...); a workspace key is rejected by every /v1/organizations/
path, and an Admin key can be provisioned read-only.
Caches are keyed per model, so the first day on a new model is cold by
definition and a note that fires on it is wrong. What matters is what happens
on the days after. This finds the single largest step down in the daily
cache-read share anywhere in the window, and then asks whether that step sits
where the new model id first appears. A collapse that lines up with the switch
is the switch; a collapse three weeks either side of it is something else, and
this says so rather than taking the credit.
The repair is printed, never performed.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_cache_step_after_model_switch")
API = "https://api.anthropic.com/v1"
VERSION = "2023-06-01"
# Published minimum cacheable prompt length per model family, in tokens. Only
# used to explain a confirmed step, never to make one: a migration from a 512
# floor to a 4,096 floor is the most common reason the share never comes back.
CACHE_MINIMUMS = {
"claude-opus-5": 512, "claude-fable-5": 512, "claude-mythos-5": 512,
"claude-mythos-preview": 2048, "claude-opus-4-8": 1024,
"claude-opus-4-7": 2048, "claude-opus-4-6": 4096, "claude-opus-4-5": 4096,
"claude-opus-4-1": 1024, "claude-opus-4": 1024, "claude-sonnet-5": 1024,
"claude-sonnet-4-6": 1024, "claude-sonnet-4-5": 1024, "claude-sonnet-4": 1024,
"claude-haiku-4-5": 4096, "claude-haiku-3-5": 2048,
}
FINDINGS = ("collapsed-after-model-change",)
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def cache_minimum(model):
"""The model's minimum cacheable prompt length. Pure. None if unrecognised."""
name = str(model or "").strip().lower()
if not name:
return None
best = None
for family, floor in CACHE_MINIMUMS.items():
if name == family or name.startswith(family + "-"):
if best is None or len(family) > len(best[0]):
best = (family, floor)
return best[1] if best else None
def day_key(stamp):
"""Normalise a timestamp to a UTC day. Pure. None if unreadable."""
if isinstance(stamp, bool):
return None
if isinstance(stamp, (int, float)):
try:
when = dt.datetime.fromtimestamp(int(stamp), dt.timezone.utc)
except (ValueError, OSError, OverflowError):
return None
return when.strftime("%Y-%m-%d")
text = str(stamp or "").strip().replace(" ", "T")
if len(text) < 10:
return None
head = text[:10]
if head[4] != "-" or head[7] != "-":
return None
for part in (head[0:4], head[5:7], head[8:10]):
if not part.isdigit():
return None
return head
def daily_rows(buckets):
"""One row per day that carried input, sorted. Pure.
Days with no traffic are left out rather than zero-filled. A zero-share day
invented for a weekend would be the largest step in most windows and would
then have to be explained away by every branch below it.
"""
merged = {}
for bucket in buckets or []:
day = day_key(bucket.get("starting_at") or bucket.get("start_time"))
if day is None:
continue
for result in bucket.get("results") or []:
if not isinstance(result, dict):
continue
model = str(result.get("model") or "unknown")
creation = result.get("cache_creation") or {}
row = merged.setdefault(day, {"day": day, "uncached": 0, "reads": 0,
"writes": 0, "by_model": {}})
uncached = _int(result.get("uncached_input_tokens"))
reads = _int(result.get("cache_read_input_tokens"))
writes = (_int(creation.get("ephemeral_5m_input_tokens"))
+ _int(creation.get("ephemeral_1h_input_tokens")))
row["uncached"] += uncached
row["reads"] += reads
row["writes"] += writes
row["by_model"][model] = (row["by_model"].get(model, 0)
+ uncached + reads + writes)
rows = [r for r in merged.values() if r["uncached"] + r["reads"] > 0]
rows.sort(key=lambda r: r["day"])
for position, row in enumerate(rows):
row["position"] = position
row["share"] = row["reads"] / float(row["reads"] + row["uncached"])
return rows
def arrival_positions(rows):
"""Models that first appear after the window opens. Pure.
A model present on day one might have been running for a year, so its
"arrival" is an artefact of where the window starts and it is excluded.
"""
first = {}
for row in rows or []:
for model in (row.get("by_model") or {}):
first.setdefault(model, _int(row.get("position")))
return {model: position for model, position in first.items() if position > 0}
def input_share_after(rows, model, position):
"""Fraction of input on one model from a position onward. Pure. None if idle.
The guard against blaming a model nobody uses. A canary taking one percent
of traffic cannot move an organization-wide ratio, and a note that lets it
take the blame will point at the wrong deploy every time.
"""
total = 0
mine = 0
for row in rows or []:
if _int(row.get("position")) < position:
continue
for name, tokens in (row.get("by_model") or {}).items():
total += _int(tokens)
if name == model:
mine += _int(tokens)
if total <= 0:
return None
return mine / float(total)
def step_at(shares, position, min_side=3):
"""The step across one position, with that day itself left out. Pure.
Returns (before, after, delta) or Nones. Excluding the day is the whole
care in this function: a new model's first day is cold because the cache is
empty, which is correct behaviour, and averaging it into either side turns
an expected cold start into a finding.
"""
shares = list(shares or [])
before = shares[:position]
after = shares[position + 1:]
if len(before) < min_side or len(after) < min_side:
return (None, None, None)
b = sum(before) / float(len(before))
a = sum(after) / float(len(after))
return (b, a, b - a)
def best_split(shares, min_side=3):
"""The largest downward step anywhere in the series. Pure.
Returns (position, delta) or (None, None). This is what makes the alignment
claim falsifiable: without it, any window containing both a new model and a
decline reads as causation. With it, the decline has to be biggest at the
switch and nowhere else.
"""
shares = list(shares or [])
n = len(shares)
if n < min_side * 2:
return (None, None)
best_position, best_delta = None, None
for position in range(min_side, n - min_side + 1):
b = sum(shares[:position]) / float(position)
a = sum(shares[position:]) / float(n - position)
delta = b - a
if best_delta is None or delta > best_delta:
best_position, best_delta = position, delta
return (best_position, best_delta)
def sustained(shares, position, min_side=3):
"""True when every day after the switch sits below every day before. Pure.
A dip that recovers is a deploy that was rolled back, or a cache warming up
over a few days. Only a floor that never comes back is structural.
"""
shares = list(shares or [])
before = shares[:position]
after = shares[position + 1:]
if len(before) < min_side or len(after) < min_side:
return False
return max(after) < min(before)
def floor_note(old_model, new_model):
"""Why the share might not come back, when the floors explain it. Pure."""
old_floor = cache_minimum(old_model)
new_floor = cache_minimum(new_model)
if old_floor is None or new_floor is None:
return ""
if new_floor > old_floor:
return ("%s needs %d tokens before a prefix is cacheable and %s needed "
"%d, so a prompt that has not changed can have stopped "
"qualifying. That is the prompt-below-model-cache-minimum note, "
"and it is the most likely mechanism here."
% (new_model, new_floor, old_model, old_floor))
return ("%s has the same or a lower cache minimum (%d) as %s (%d), so the "
"floor does not explain this. Look at thinking or effort defaults "
"and at the tokenizer instead."
% (new_model, new_floor, old_model, old_floor))
def handoff(state):
"""Which note owns this shape, when it is not this one. Pure."""
if state == "no-new-model":
return ("no model id appeared for the first time in this window, so "
"nothing here can be attributed to a switch. If the share is "
"low, read cache-invalidated-by-changing-prefix and "
"prompt-caching-never-used.")
if state == "step-elsewhere":
return ("the largest step in the series is not where the new model "
"arrived, so something else changed on that day. Read the "
"cache-invalidated-by-changing-prefix note and line the step up "
"against your deploys.")
if state == "expected-cold-start":
return ("the share dropped on the switch day and came back. That is a "
"cold cache filling up, which is what a model change is "
"supposed to cost, and it is not a finding.")
return ""
def classify(rows, min_days=14, min_drop=0.15, ratio_floor=0.6,
min_migration=0.20, min_side=3):
"""Classify one window. Pure. Returns (state, detail)."""
rows = rows or []
if len(rows) < min_days:
return ("too-few-days",
"%d day(s) with input in the window, under the floor of %d"
% (len(rows), min_days))
shares = [r["share"] for r in rows]
arrivals = arrival_positions(rows)
if not arrivals:
return ("no-new-model",
"every model id in this window was already present on day one")
ranked = sorted(arrivals.items(),
key=lambda item: input_share_after(rows, item[0], item[1]) or 0.0,
reverse=True)
model, position = ranked[0]
migration = input_share_after(rows, model, position) or 0.0
if migration < min_migration:
return ("new-model-marginal",
"%s arrived on %s but carries only %.0f%% of input since, under "
"the floor of %.0f%%. Too small to move the ratio."
% (model, rows[position]["day"], migration * 100,
min_migration * 100))
before, after, delta = step_at(shares, position, min_side)
if delta is None:
return ("window-too-short-around-the-switch",
"%s arrived on %s with fewer than %d day(s) either side of it"
% (model, rows[position]["day"], min_side))
# Alignment is checked before magnitude, and only against a step that is
# material on its own. A big fall somewhere else in the window disqualifies
# the switch outright, however the numbers either side of the switch read.
peak, peak_delta = best_split(shares, min_side)
if (peak is not None and peak_delta is not None and peak_delta >= min_drop
and abs(peak - position) > 1):
return ("step-elsewhere",
"the share falls hardest at %s, not at the %s switch on %s"
% (rows[peak]["day"], model, rows[position]["day"]))
if delta < min_drop or after > before * ratio_floor:
if before - shares[position] >= min_drop:
return ("expected-cold-start",
"%s arrived on %s, the share dipped to %.0f%% that day and "
"settled back at %.0f%% against %.0f%% before"
% (model, rows[position]["day"], shares[position] * 100,
after * 100, before * 100))
return ("steady",
"%s arrived on %s and the share held at %.0f%% against %.0f%% "
"before" % (model, rows[position]["day"], after * 100,
before * 100))
if peak is None or abs(peak - position) > 1:
return ("step-elsewhere",
"the share falls hardest at %s, not at the %s switch on %s"
% (rows[peak]["day"] if peak is not None else "no single day",
model, rows[position]["day"]))
if not sustained(shares, position, min_side):
return ("partial-recovery",
"%.0f%% before the %s switch and %.0f%% after, but some days "
"since have recovered above the pre-switch floor. Suggestive "
"and not conclusive: widen the window."
% (before * 100, model, after * 100))
return ("collapsed-after-model-change",
"cache-read share %.0f%% before %s arrived on %s and %.0f%% after, "
"with the switch day itself excluded. %s now carries %.0f%% of "
"input and the largest step in the window is exactly there."
% (before * 100, model, rows[position]["day"], after * 100, model,
migration * 100))
def previous_model(rows, position):
"""The model carrying the most input before the switch. Pure."""
totals = {}
for row in rows or []:
if _int(row.get("position")) >= position:
continue
for name, tokens in (row.get("by_model") or {}).items():
totals[name] = totals.get(name, 0) + _int(tokens)
if not totals:
return None
return max(totals.items(), key=lambda item: item[1])[0]
def repair_lines(old_model, new_model):
"""What to check about the new model, in the order that pays. Pure."""
lines = []
note = floor_note(old_model, new_model)
if note:
lines.append(note)
lines.extend([
"compare the two models' minimum cacheable token counts and move the "
"cache_control breakpoint so the prefix clears the higher one.",
"compare their thinking and effort defaults. Those are model-specific "
"and they sit inside the cached prefix, so a different default is a "
"different prefix.",
"count the prefix again under the new model id. A newer tokenizer can "
"produce materially more tokens for the same text, which moves a prefix "
"that used to sit just above a boundary.",
"then re-measure the cache-read share over the following three days, "
"not the following one. The first day after any breakpoint change is "
"cold for the same reason the switch day was.",
])
return lines
def window_start(days):
"""Floor to the day: starting_at has to sit on a bucket boundary."""
now = dt.datetime.now(dt.timezone.utc).replace(hour=0, minute=0, second=0,
microsecond=0)
return (now - dt.timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from Anthropic: /v1/organizations/ needs an Admin "
"API key (sk-ant-admin...), not a workspace key"
% r.status_code)
r.raise_for_status()
return r.json()
def read_buckets(session, path, params):
"""Walk the paginated usage report."""
params = dict(params)
while True:
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=31,
help="days of daily buckets to read (max 90)")
ap.add_argument("--min-drop", type=float, default=0.15,
help="fall in cache-read share, in share points, that counts "
"as a step")
args = ap.parse_args()
admin = os.environ.get("ANTHROPIC_ADMIN_KEY")
if not admin:
log.error("set ANTHROPIC_ADMIN_KEY to an Admin API key "
"(sk-ant-admin...); a workspace key cannot read "
"/v1/organizations/")
return 2
days = max(14, min(int(args.days), 90))
session = requests.Session()
session.headers.update({"x-api-key": admin, "anthropic-version": VERSION})
buckets = read_buckets(session, "/organizations/usage_report/messages", {
"starting_at": window_start(days),
"bucket_width": "1d",
"limit": days + 1,
"group_by[]": ["model"],
})
rows = daily_rows(buckets)
if not rows:
log.info("no messages usage in the last %d day(s)", days)
return 0
state, detail = classify(rows, min_drop=args.min_drop)
line = "%-32s %s" % (state, detail)
if state in FINDINGS:
log.warning(line)
arrivals = arrival_positions(rows)
ranked = sorted(arrivals.items(),
key=lambda item: input_share_after(rows, item[0], item[1]) or 0.0,
reverse=True)
model, position = ranked[0]
for repair in repair_lines(previous_model(rows, position), model):
log.warning(" repair: %s", repair)
log.warning(" note: this is an organization-wide ratio. A second "
"workload that changed on the same day would be folded into "
"it, so line the date up against a deploy before acting.")
log.info("%d day(s) checked, 1 finding(s)", len(rows))
return 1
note = handoff(state)
log.info(line)
if note:
log.info(" %s", note)
log.info("%d day(s) checked, 0 finding(s)", len(rows))
return 0
if __name__ == "__main__":
sys.exit(main())
anthropic-cache-step-after-model-switch.mjs
/**
* Align a collapse in cache-read share with the day a new model id appeared.
*
* Read only. One GET against the Admin API, which needs an Admin API key
* (sk-ant-admin...). A workspace key is rejected by /v1/organizations/.
*
* Caches are keyed per model, so the first day on a new model is cold by
* definition and a note that fires on it is wrong. This finds the single
* largest step down in the daily cache-read share anywhere in the window and
* asks whether it sits where the new model id first appears. A collapse three
* weeks either side of the switch is something else, and this says so rather
* than taking the credit.
*/
const API = 'https://api.anthropic.com/v1';
const VERSION = '2023-06-01';
/**
* Published minimum cacheable prompt length per model family, in tokens. Used
* only to explain a confirmed step, never to make one.
*/
export const CACHE_MINIMUMS = {
'claude-opus-5': 512,
'claude-fable-5': 512,
'claude-mythos-5': 512,
'claude-mythos-preview': 2048,
'claude-opus-4-8': 1024,
'claude-opus-4-7': 2048,
'claude-opus-4-6': 4096,
'claude-opus-4-5': 4096,
'claude-opus-4-1': 1024,
'claude-opus-4': 1024,
'claude-sonnet-5': 1024,
'claude-sonnet-4-6': 1024,
'claude-sonnet-4-5': 1024,
'claude-sonnet-4': 1024,
'claude-haiku-4-5': 4096,
'claude-haiku-3-5': 2048,
};
const FINDINGS = new Set(['collapsed-after-model-change']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/** The model's minimum cacheable prompt length. Pure. Null if unrecognised. */
export function cacheMinimum(model) {
const name = String(model ?? '').trim().toLowerCase();
if (!name) return null;
let best = null;
for (const [family, floor] of Object.entries(CACHE_MINIMUMS)) {
if (name === family || name.startsWith(`${family}-`)) {
if (best === null || family.length > best[0].length) best = [family, floor];
}
}
return best ? best[1] : null;
}
/** Normalise a timestamp to a UTC day. Pure. Null if unreadable. */
export function dayKey(stamp) {
if (typeof stamp === 'boolean') return null;
if (typeof stamp === 'number' && Number.isFinite(stamp)) {
const when = new Date(Math.trunc(stamp) * 1000);
if (Number.isNaN(when.getTime())) return null;
return when.toISOString().slice(0, 10);
}
const text = String(stamp ?? '').trim().replace(' ', 'T');
if (text.length < 10) return null;
const head = text.slice(0, 10);
if (!/^\d{4}-\d{2}-\d{2}$/.test(head)) return null;
return head;
}
/**
* One row per day that carried input, sorted. Pure.
* Days with no traffic are left out rather than zero-filled: an invented
* zero-share weekend would be the largest step in most windows.
*/
export function dailyRows(buckets) {
const merged = new Map();
for (const bucket of buckets ?? []) {
const day = dayKey(bucket?.starting_at ?? bucket?.start_time);
if (day === null) continue;
for (const result of bucket?.results ?? []) {
if (!result || typeof result !== 'object') continue;
const model = String(result.model ?? 'unknown');
const creation = result.cache_creation ?? {};
if (!merged.has(day)) {
merged.set(day, { day, uncached: 0, reads: 0, writes: 0, byModel: {} });
}
const row = merged.get(day);
const uncached = readInt(result.uncached_input_tokens);
const reads = readInt(result.cache_read_input_tokens);
const writes = readInt(creation.ephemeral_5m_input_tokens)
+ readInt(creation.ephemeral_1h_input_tokens);
row.uncached += uncached;
row.reads += reads;
row.writes += writes;
row.byModel[model] = (row.byModel[model] ?? 0) + uncached + reads + writes;
}
}
const rows = [...merged.values()].filter((r) => r.uncached + r.reads > 0);
rows.sort((a, b) => a.day.localeCompare(b.day));
rows.forEach((row, position) => {
row.position = position;
row.share = row.reads / (row.reads + row.uncached);
});
return rows;
}
/**
* Models that first appear after the window opens. Pure.
* A model present on day one might have been running for a year, so its
* arrival is an artefact of where the window starts.
*/
export function arrivalPositions(rows) {
const first = new Map();
for (const row of rows ?? []) {
for (const model of Object.keys(row?.byModel ?? {})) {
if (!first.has(model)) first.set(model, readInt(row?.position));
}
}
const out = new Map();
for (const [model, position] of first) if (position > 0) out.set(model, position);
return out;
}
/**
* Fraction of input on one model from a position onward. Pure. Null if idle.
* The guard against blaming a model nobody uses: a canary on one percent of
* traffic cannot move an organization-wide ratio.
*/
export function inputShareAfter(rows, model, position) {
let total = 0;
let mine = 0;
for (const row of rows ?? []) {
if (readInt(row?.position) < position) continue;
for (const [name, tokens] of Object.entries(row?.byModel ?? {})) {
total += readInt(tokens);
if (name === model) mine += readInt(tokens);
}
}
if (total <= 0) return null;
return mine / total;
}
/**
* The step across one position, with that day itself left out. Pure.
* Excluding the day is the whole care here: a new model's first day is cold
* because the cache is empty, which is correct behaviour.
*/
export function stepAt(shares, position, minSide = 3) {
const all = [...(shares ?? [])];
const before = all.slice(0, position);
const after = all.slice(position + 1);
if (before.length < minSide || after.length < minSide) return [null, null, null];
const b = before.reduce((s, v) => s + v, 0) / before.length;
const a = after.reduce((s, v) => s + v, 0) / after.length;
return [b, a, b - a];
}
/**
* The largest downward step anywhere in the series. Pure.
* What makes the alignment claim falsifiable: without it, any window holding
* both a new model and a decline reads as causation.
*/
export function bestSplit(shares, minSide = 3) {
const all = [...(shares ?? [])];
const n = all.length;
if (n < minSide * 2) return [null, null];
let bestPosition = null;
let bestDelta = null;
for (let position = minSide; position <= n - minSide; position += 1) {
const b = all.slice(0, position).reduce((s, v) => s + v, 0) / position;
const a = all.slice(position).reduce((s, v) => s + v, 0) / (n - position);
const delta = b - a;
if (bestDelta === null || delta > bestDelta) {
bestPosition = position;
bestDelta = delta;
}
}
return [bestPosition, bestDelta];
}
/** True when every day after the switch sits below every day before. Pure. */
export function sustained(shares, position, minSide = 3) {
const all = [...(shares ?? [])];
const before = all.slice(0, position);
const after = all.slice(position + 1);
if (before.length < minSide || after.length < minSide) return false;
return Math.max(...after) < Math.min(...before);
}
/** Why the share might not come back, when the floors explain it. Pure. */
export function floorNote(oldModel, newModel) {
const oldFloor = cacheMinimum(oldModel);
const newFloor = cacheMinimum(newModel);
if (oldFloor === null || newFloor === null) return '';
if (newFloor > oldFloor) {
return `${newModel} needs ${newFloor} tokens before a prefix is cacheable `
+ `and ${oldModel} needed ${oldFloor}, so a prompt that has not changed `
+ 'can have stopped qualifying. That is the '
+ 'prompt-below-model-cache-minimum note, and it is the most likely '
+ 'mechanism here.';
}
return `${newModel} has the same or a lower cache minimum (${newFloor}) as `
+ `${oldModel} (${oldFloor}), so the floor does not explain this. Look at `
+ 'thinking or effort defaults and at the tokenizer instead.';
}
/** Which note owns this shape, when it is not this one. Pure. */
export function handoff(state) {
if (state === 'no-new-model') {
return 'no model id appeared for the first time in this window, so nothing '
+ 'here can be attributed to a switch. If the share is low, read '
+ 'cache-invalidated-by-changing-prefix and prompt-caching-never-used.';
}
if (state === 'step-elsewhere') {
return 'the largest step in the series is not where the new model arrived, '
+ 'so something else changed on that day. Read the '
+ 'cache-invalidated-by-changing-prefix note and line the step up against '
+ 'your deploys.';
}
if (state === 'expected-cold-start') {
return 'the share dropped on the switch day and came back. That is a cold '
+ 'cache filling up, which is what a model change is supposed to cost, '
+ 'and it is not a finding.';
}
return '';
}
/** Classify one window. Pure. Returns [state, detail]. */
export function classify(rows, minDays = 14, minDrop = 0.15, ratioFloor = 0.6,
minMigration = 0.20, minSide = 3) {
const all = rows ?? [];
if (all.length < minDays) {
return ['too-few-days',
`${all.length} day(s) with input in the window, under the floor of ${minDays}`];
}
const shares = all.map((r) => r.share);
const arrivals = arrivalPositions(all);
if (arrivals.size === 0) {
return ['no-new-model', 'every model id in this window was already present on day one'];
}
const ranked = [...arrivals.entries()].sort(
(a, b) => (inputShareAfter(all, b[0], b[1]) ?? 0) - (inputShareAfter(all, a[0], a[1]) ?? 0));
const [model, position] = ranked[0];
const migration = inputShareAfter(all, model, position) ?? 0;
if (migration < minMigration) {
return ['new-model-marginal',
`${model} arrived on ${all[position].day} but carries only `
+ `${(migration * 100).toFixed(0)}% of input since, under the floor of `
+ `${(minMigration * 100).toFixed(0)}%. Too small to move the ratio.`];
}
const [before, after, delta] = stepAt(shares, position, minSide);
if (delta === null) {
return ['window-too-short-around-the-switch',
`${model} arrived on ${all[position].day} with fewer than ${minSide} day(s) `
+ 'either side of it'];
}
// Alignment before magnitude, and only against a step that is material on
// its own: a big fall elsewhere disqualifies the switch outright.
const [peak, peakDelta] = bestSplit(shares, minSide);
if (peak !== null && peakDelta !== null && peakDelta >= minDrop
&& Math.abs(peak - position) > 1) {
return ['step-elsewhere',
`the share falls hardest at ${all[peak].day}, not at the ${model} switch `
+ `on ${all[position].day}`];
}
if (delta < minDrop || after > before * ratioFloor) {
if (before - shares[position] >= minDrop) {
return ['expected-cold-start',
`${model} arrived on ${all[position].day}, the share dipped to `
+ `${(shares[position] * 100).toFixed(0)}% that day and settled back at `
+ `${(after * 100).toFixed(0)}% against ${(before * 100).toFixed(0)}% before`];
}
return ['steady',
`${model} arrived on ${all[position].day} and the share held at `
+ `${(after * 100).toFixed(0)}% against ${(before * 100).toFixed(0)}% before`];
}
if (peak === null || Math.abs(peak - position) > 1) {
return ['step-elsewhere',
`the share falls hardest at ${peak === null ? 'no single day' : all[peak].day}, `
+ `not at the ${model} switch on ${all[position].day}`];
}
if (!sustained(shares, position, minSide)) {
return ['partial-recovery',
`${(before * 100).toFixed(0)}% before the ${model} switch and `
+ `${(after * 100).toFixed(0)}% after, but some days since have recovered `
+ 'above the pre-switch floor. Suggestive and not conclusive: widen the window.'];
}
return ['collapsed-after-model-change',
`cache-read share ${(before * 100).toFixed(0)}% before ${model} arrived on `
+ `${all[position].day} and ${(after * 100).toFixed(0)}% after, with the `
+ `switch day itself excluded. ${model} now carries `
+ `${(migration * 100).toFixed(0)}% of input and the largest step in the `
+ 'window is exactly there.'];
}
/** The model carrying the most input before the switch. Pure. */
export function previousModel(rows, position) {
const totals = new Map();
for (const row of rows ?? []) {
if (readInt(row?.position) >= position) continue;
for (const [name, tokens] of Object.entries(row?.byModel ?? {})) {
totals.set(name, (totals.get(name) ?? 0) + readInt(tokens));
}
}
if (totals.size === 0) return null;
return [...totals.entries()].reduce((a, b) => (a[1] >= b[1] ? a : b))[0];
}
/** What to check about the new model, in the order that pays. Pure. */
export function repairLines(oldModel, newModel) {
const lines = [];
const note = floorNote(oldModel, newModel);
if (note) lines.push(note);
lines.push(
"compare the two models' minimum cacheable token counts and move the "
+ 'cache_control breakpoint so the prefix clears the higher one.',
'compare their thinking and effort defaults. Those are model-specific and '
+ 'they sit inside the cached prefix, so a different default is a different prefix.',
'count the prefix again under the new model id. A newer tokenizer can '
+ 'produce materially more tokens for the same text, which moves a prefix '
+ 'that used to sit just above a boundary.',
'then re-measure the cache-read share over the following three days, not '
+ 'the following one. The first day after any breakpoint change is cold for '
+ 'the same reason the switch day was.',
);
return lines;
}
function windowStart(days) {
const now = new Date();
now.setUTCHours(0, 0, 0, 0);
return `${new Date(now.getTime() - days * 86400000).toISOString().slice(0, 19)}Z`;
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, {
headers: { 'x-api-key': key, 'anthropic-version': VERSION },
});
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Anthropic: /v1/organizations/ needs an `
+ 'Admin API key (sk-ant-admin...), not a workspace key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* readBuckets(key, path, params) {
let query = { ...params };
for (;;) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function main() {
const admin = process.env.ANTHROPIC_ADMIN_KEY;
if (!admin) {
console.error('set ANTHROPIC_ADMIN_KEY to an Admin API key (sk-ant-admin...); '
+ 'a workspace key cannot read /v1/organizations/');
process.exitCode = 2;
return;
}
const days = Math.max(14, Math.min(Number(process.env.DAYS ?? 31), 90));
const minDrop = Number(process.env.MIN_DROP ?? 0.15);
const buckets = [];
for await (const bucket of readBuckets(admin, '/organizations/usage_report/messages', {
starting_at: windowStart(days),
bucket_width: '1d',
limit: days + 1,
'group_by[]': ['model'],
})) buckets.push(bucket);
const rows = dailyRows(buckets);
if (rows.length === 0) {
console.log(`no messages usage in the last ${days} day(s)`);
return;
}
const [state, detail] = classify(rows, 14, minDrop);
const line = `${state.padEnd(32)} ${detail}`;
if (FINDINGS.has(state)) {
console.warn(line);
const arrivals = arrivalPositions(rows);
const ranked = [...arrivals.entries()].sort(
(a, b) => (inputShareAfter(rows, b[0], b[1]) ?? 0)
- (inputShareAfter(rows, a[0], a[1]) ?? 0));
const [model, position] = ranked[0];
for (const repair of repairLines(previousModel(rows, position), model)) {
console.warn(` repair: ${repair}`);
}
console.warn(' note: this is an organization-wide ratio. A second workload '
+ 'that changed on the same day would be folded into it, so '
+ 'line the date up against a deploy before acting.');
console.log(`${rows.length} day(s) checked, 1 finding(s)`);
process.exitCode = 1;
return;
}
const note = handoff(state);
console.log(line);
if (note) console.log(` ${note}`);
console.log(`${rows.length} day(s) checked, 0 finding(s)`);
process.exitCode = 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Two fixtures are identical for the first sixteen days and differ only in what happens on day seventeen. Both run Opus 5 at a seventy per cent cache-read share for a fortnight, both switch to Haiku 4.5, and both dip to twenty per cent on the switch day. One stays down and one comes back, and they get opposite verdicts — which is the entire argument for deleting the switch day from the comparison. Around them: a collapse five days after an arrival but fifteen days before its own changepoint, which fails the alignment test outright; a canary model on three per cent of input, which is never allowed to be the cause; and a window whose every model id was already present on day one, where no claim is available at all.
test_anthropic_cache_step_after_model_switch.py
from anthropic_cache_step_after_model_switch import (arrival_positions,
best_split, cache_minimum,
classify, daily_rows,
day_key, floor_note,
handoff, input_share_after,
previous_model,
repair_lines, step_at,
sustained)
OLD = "claude-opus-5"
NEW = "claude-haiku-4-5-20251001"
def day(position, share, models):
"""One day of the org-wide report. models maps id to input tokens."""
total = sum(models.values())
reads = int(round(total * share))
return {"day": "2026-08-%02d" % (position + 1), "position": position,
"share": share, "reads": reads, "uncached": total - reads,
"writes": 0, "by_model": dict(models)}
def switched(before=0.70, cold=0.20, after=0.10, at=15, new_share=1.0,
days=31):
"""A migration on day `at`, with the new model taking `new_share` of input."""
rows = []
for position in range(days):
if position < at:
rows.append(day(position, before, {OLD: 40_000_000}))
else:
mix = {NEW: int(40_000_000 * new_share)}
if new_share < 1.0:
mix[OLD] = 40_000_000 - mix[NEW]
rows.append(day(position, cold if position == at else after, mix))
return rows
STEP = switched()
def test_the_step_aligned_with_the_arrival_is_the_finding():
# The note in one assertion. The switch day itself is thrown away, because
# a cold cache on day one of a new model is correct behaviour.
assert arrival_positions(STEP) == {NEW: 15}
assert round(input_share_after(STEP, NEW, 15), 3) == 1.0
shares = [r["share"] for r in STEP]
before, after, delta = step_at(shares, 15)
assert round(before, 2) == 0.70 and round(after, 2) == 0.10
assert round(delta, 2) == 0.60
assert best_split(shares)[0] == 15
assert sustained(shares, 15) is True
state, detail = classify(STEP)
assert state == "collapsed-after-model-change"
assert "70% before claude-haiku-4-5-20251001 arrived on 2026-08-16" in detail
assert "10% after, with the switch day itself excluded" in detail
assert "largest step in the window is exactly there" in detail
assert handoff(state) == ""
def test_a_dip_that_recovers_is_the_cold_cache_doing_its_job():
# Identical arrival, identical switch-day dip, opposite verdict. This is
# the case that excluding the switch day exists for.
recovered = switched(before=0.70, cold=0.20, after=0.70)
assert [r["share"] for r in recovered][15] == 0.20
state, detail = classify(recovered)
assert state == "expected-cold-start"
assert "dipped to 20% that day and settled back at 70%" in detail
assert "not a finding" in handoff(state)
def test_a_collapse_somewhere_else_is_not_the_switch():
# The new model arrives on day 5 and the share holds for another fortnight
# before falling off a cliff. Alignment is what makes the claim falsifiable.
rows = []
for position in range(31):
models = {OLD: 40_000_000} if position < 5 else {NEW: 40_000_000}
rows.append(day(position, 0.70 if position < 20 else 0.10, models))
shares = [r["share"] for r in rows]
assert arrival_positions(rows) == {NEW: 5}
assert best_split(shares)[0] == 20
state, detail = classify(rows)
assert state == "step-elsewhere"
assert "falls hardest at 2026-08-21" in detail
assert "cache-invalidated-by-changing-prefix" in handoff(state)
def test_a_canary_model_is_never_blamed():
# A new id carrying three percent of input cannot move an org-wide ratio,
# and letting it take the blame points at the wrong deploy.
rows = switched(new_share=0.03)
assert round(input_share_after(rows, NEW, 15), 2) == 0.03
state, detail = classify(rows)
assert state == "new-model-marginal"
assert "carries only 3% of input since" in detail
def test_a_window_with_no_new_model_makes_no_claim():
rows = [day(p, 0.70 if p < 15 else 0.10, {OLD: 40_000_000}) for p in range(31)]
assert arrival_positions(rows) == {}
state, detail = classify(rows)
assert state == "no-new-model"
assert "already present on day one" in detail
assert "cache-invalidated-by-changing-prefix" in handoff(state)
def test_a_share_that_holds_across_the_switch_is_steady():
state, detail = classify(switched(before=0.70, cold=0.70, after=0.70))
assert state == "steady"
assert "held at 70% against 70% before" in detail
def test_a_recovery_after_the_step_is_only_suggestive():
rows = switched()
rows[28]["share"] = 0.90
state, detail = classify(rows)
assert state == "partial-recovery"
assert "recovered above the pre-switch floor" in detail
assert sustained([r["share"] for r in rows], 15) is False
def test_the_floors_explain_the_step_without_making_it():
assert cache_minimum(NEW) == 4096
assert cache_minimum(OLD) == 512
note = floor_note(OLD, NEW)
assert "needs 4096 tokens" in note
assert "prompt-below-model-cache-minimum" in note
# A move to a lower floor gets the opposite sentence, not silence.
other = floor_note("claude-haiku-4-5", "claude-opus-5")
assert "does not explain this" in other
assert floor_note(OLD, "gpt-5.6") == ""
assert any("thinking" in line for line in repair_lines(OLD, NEW))
def test_the_report_is_folded_into_days_and_models():
buckets = []
for position in range(31):
model = OLD if position < 15 else NEW
share = 0.70 if position < 15 else (0.20 if position == 15 else 0.10)
total = 40_000_000
reads = int(total * share)
buckets.append({"starting_at": "2026-08-%02dT00:00:00Z" % (position + 1),
"results": [{"model": model,
"uncached_input_tokens": total - reads,
"cache_read_input_tokens": reads,
"cache_creation": {
"ephemeral_5m_input_tokens": 0,
"ephemeral_1h_input_tokens": 0}}]})
rows = daily_rows(buckets)
assert len(rows) == 31
assert [r["position"] for r in rows] == list(range(31))
assert round(rows[0]["share"], 2) == 0.70
assert previous_model(rows, 15) == OLD
assert classify(rows)[0] == "collapsed-after-model-change"
def test_thin_and_unreadable_windows_produce_no_verdict():
assert classify([day(p, 0.5, {OLD: 1000}) for p in range(5)])[0] == "too-few-days"
assert classify([])[0] == "too-few-days"
assert classify(None)[0] == "too-few-days"
assert step_at([0.1, 0.2], 1) == (None, None, None)
assert best_split([0.1, 0.2]) == (None, None)
assert sustained([], 3) is False
assert input_share_after([], OLD, 0) is None
assert previous_model([], 3) is None
assert day_key("nonsense") is None
assert daily_rows([{"starting_at": "bad", "results": []}]) == []
anthropic-cache-step-after-model-switch.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { arrivalPositions, bestSplit, cacheMinimum, classify, dailyRows, dayKey,
floorNote, handoff, inputShareAfter, previousModel, repairLines,
stepAt, sustained }
from './anthropic-cache-step-after-model-switch.mjs';
const OLD = 'claude-opus-5';
const NEW = 'claude-haiku-4-5-20251001';
const day = (position, share, models) => {
const total = Object.values(models).reduce((s, v) => s + v, 0);
const reads = Math.round(total * share);
return { day: `2026-08-${String(position + 1).padStart(2, '0')}`, position,
share, reads, uncached: total - reads, writes: 0,
byModel: { ...models } };
};
const switched = ({ before = 0.70, cold = 0.20, after = 0.10, at = 15,
newShare = 1.0, days = 31 } = {}) => {
const rows = [];
for (let position = 0; position < days; position += 1) {
if (position < at) {
rows.push(day(position, before, { [OLD]: 40000000 }));
} else {
const mix = { [NEW]: Math.trunc(40000000 * newShare) };
if (newShare < 1.0) mix[OLD] = 40000000 - mix[NEW];
rows.push(day(position, position === at ? cold : after, mix));
}
}
return rows;
};
const STEP = switched();
test('the step aligned with the arrival is the finding', () => {
assert.deepEqual([...arrivalPositions(STEP)], [[NEW, 15]]);
assert.equal(Number(inputShareAfter(STEP, NEW, 15).toFixed(3)), 1);
const shares = STEP.map((r) => r.share);
const [before, after, delta] = stepAt(shares, 15);
assert.equal(Number(before.toFixed(2)), 0.70);
assert.equal(Number(after.toFixed(2)), 0.10);
assert.equal(Number(delta.toFixed(2)), 0.60);
assert.equal(bestSplit(shares)[0], 15);
assert.equal(sustained(shares, 15), true);
const [state, detail] = classify(STEP);
assert.equal(state, 'collapsed-after-model-change');
assert.match(detail, /70% before claude-haiku-4-5-20251001 arrived on 2026-08-16/);
assert.match(detail, /10% after, with the switch day itself excluded/);
assert.match(detail, /largest step in the window is exactly there/);
assert.equal(handoff(state), '');
});
test('a dip that recovers is the cold cache doing its job', () => {
const recovered = switched({ after: 0.70 });
assert.equal(recovered.map((r) => r.share)[15], 0.20);
const [state, detail] = classify(recovered);
assert.equal(state, 'expected-cold-start');
assert.match(detail, /dipped to 20% that day and settled back at 70%/);
assert.match(handoff(state), /not a finding/);
});
test('a collapse somewhere else is not the switch', () => {
const rows = [];
for (let position = 0; position < 31; position += 1) {
const models = position < 5 ? { [OLD]: 40000000 } : { [NEW]: 40000000 };
rows.push(day(position, position < 20 ? 0.70 : 0.10, models));
}
const shares = rows.map((r) => r.share);
assert.deepEqual([...arrivalPositions(rows)], [[NEW, 5]]);
assert.equal(bestSplit(shares)[0], 20);
const [state, detail] = classify(rows);
assert.equal(state, 'step-elsewhere');
assert.match(detail, /falls hardest at 2026-08-21/);
assert.match(handoff(state), /cache-invalidated-by-changing-prefix/);
});
test('a canary model is never blamed', () => {
const rows = switched({ newShare: 0.03 });
assert.equal(Number(inputShareAfter(rows, NEW, 15).toFixed(2)), 0.03);
const [state, detail] = classify(rows);
assert.equal(state, 'new-model-marginal');
assert.match(detail, /carries only 3% of input since/);
});
test('a window with no new model makes no claim', () => {
const rows = Array.from({ length: 31 },
(_, p) => day(p, p < 15 ? 0.70 : 0.10, { [OLD]: 40000000 }));
assert.equal(arrivalPositions(rows).size, 0);
const [state, detail] = classify(rows);
assert.equal(state, 'no-new-model');
assert.match(detail, /already present on day one/);
assert.match(handoff(state), /cache-invalidated-by-changing-prefix/);
});
test('a share that holds across the switch is steady', () => {
const [state, detail] = classify(switched({ cold: 0.70, after: 0.70 }));
assert.equal(state, 'steady');
assert.match(detail, /held at 70% against 70% before/);
});
test('a recovery after the step is only suggestive', () => {
const rows = switched();
rows[28].share = 0.90;
const [state, detail] = classify(rows);
assert.equal(state, 'partial-recovery');
assert.match(detail, /recovered above the pre-switch floor/);
assert.equal(sustained(rows.map((r) => r.share), 15), false);
});
test('the floors explain the step without making it', () => {
assert.equal(cacheMinimum(NEW), 4096);
assert.equal(cacheMinimum(OLD), 512);
const note = floorNote(OLD, NEW);
assert.match(note, /needs 4096 tokens/);
assert.match(note, /prompt-below-model-cache-minimum/);
assert.match(floorNote('claude-haiku-4-5', 'claude-opus-5'), /does not explain this/);
assert.equal(floorNote(OLD, 'gpt-5.6'), '');
assert.ok(repairLines(OLD, NEW).some((l) => l.includes('thinking')));
});
test('the report is folded into days and models', () => {
const buckets = Array.from({ length: 31 }, (_, position) => {
const model = position < 15 ? OLD : NEW;
const share = position < 15 ? 0.70 : (position === 15 ? 0.20 : 0.10);
const total = 40000000;
const reads = Math.trunc(total * share);
return { starting_at: `2026-08-${String(position + 1).padStart(2, '0')}T00:00:00Z`,
results: [{ model, uncached_input_tokens: total - reads,
cache_read_input_tokens: reads,
cache_creation: { ephemeral_5m_input_tokens: 0,
ephemeral_1h_input_tokens: 0 } }] };
});
const rows = dailyRows(buckets);
assert.equal(rows.length, 31);
assert.deepEqual(rows.map((r) => r.position), Array.from({ length: 31 }, (_, i) => i));
assert.equal(Number(rows[0].share.toFixed(2)), 0.70);
assert.equal(previousModel(rows, 15), OLD);
assert.equal(classify(rows)[0], 'collapsed-after-model-change');
});
test('thin and unreadable windows produce no verdict', () => {
const thin = Array.from({ length: 5 }, (_, p) => day(p, 0.5, { [OLD]: 1000 }));
assert.equal(classify(thin)[0], 'too-few-days');
assert.equal(classify([])[0], 'too-few-days');
assert.equal(classify(null)[0], 'too-few-days');
assert.deepEqual(stepAt([0.1, 0.2], 1), [null, null, null]);
assert.deepEqual(bestSplit([0.1, 0.2]), [null, null]);
assert.equal(sustained([], 3), false);
assert.equal(inputShareAfter([], OLD, 0), null);
assert.equal(previousModel([], 3), null);
assert.equal(dayKey('nonsense'), null);
assert.deepEqual(dailyRows([{ starting_at: 'bad', results: [] }]), []);
});
FAQ
Is a drop on the day of the switch a problem?
No, and treating it as one is the mistake this note is built around. Caches are keyed per model, so the day you switch you have no warm entries and every call writes. That is the documented cost of a migration and it should recover within a day or two. The script removes the switch day from both sides of the comparison entirely and reports it separately as context. A dip that recovers gets its own verdict that says, in as many words, this is not a finding.
Why check the largest step in the whole window?
Because otherwise the check cannot be wrong. Take any month containing a model migration and a rise in cost, look only at the migration date, and you will find a story. Computing the biggest downward step at every split point and requiring it to land on the arrival makes the claim refutable: if the share actually fell off a cliff two weeks later, the script names that date instead and tells you to line it up against your deploys. Most of the value of this note is in the cases where it declines to blame the model.
What actually causes the share not to come back?
Three things, in rough order of likelihood. The new model's minimum cacheable prompt length may be higher, and the jump is not subtle: 512 tokens on Opus 5, 4,096 on Opus 4.6, Opus 4.5 and Haiku 4.5. Thinking and effort parameters have model-specific defaults and they sit inside the cached prefix, so a default that differs is a prefix that differs. And a newer tokenizer can produce materially more tokens for the same text, which can push a prefix across a boundary in either direction.
How does this differ from the cache minimum note?
That note is the mechanism and this one is the event. It looks across models on one key at a single moment and brackets the prefix between two floors; it will find a route that has never cached on a given model regardless of when that started. This one looks along time at an organization-wide ratio and finds the day it changed. When the finding here is confirmed and the new model's floor is higher, the script points straight at that note by name, because the repair lives there.
Can this fire on something other than a model change?
It can be confounded, and the output warns about it. The ratio is organization-wide and the report carries no request count, so a second workload that changed its prompt on the same day is folded into the same number. Two guards limit the damage: the new model must carry at least a fifth of input, and the largest step in the window must be at the arrival rather than somewhere else. What comes out is a date and a magnitude, which is exactly enough to check against a deploy log and deliberately not enough to skip doing so.
Related field notes
- A prefix bracketed between the floors of the models it runs on
- A model id that moves under you without a deploy
- Prompt caching that was never switched on at all
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Prompt caching — Claude Docs
- Get messages usage report — Claude Admin API
- Usage and Cost API — Claude Docs
- Models overview — Claude Docs
Stuck on a tricky one?
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.