Diagnostic LLM APIs
a frontier model is answering twenty-token questions
Somebody built the intent router in an afternoon eighteen months ago. They pasted the model name out of the quickstart, because on that afternoon the question was whether the thing worked at all and the answer was worth whatever it cost. It works. It has worked every day since, four hundred thousand times a month, and every one of those calls returns a single word from a list of nine. The model that returns it is the most expensive one the organization can buy.
One call with an organization admin key: GET /v1/organization/usage/completions?start_time={now-14d}&bucket_width=1d&limit=14&group_by=model&group_by=project_id. Every result carries input_tokens, output_tokens and num_model_requests. Fold them per model and divide.
The finding is a ratio, not a total. output_tokens / num_model_requests is the mean length of what the model actually said. A premium model with a high request count and a mean answer under about fifty tokens is answering questions a -mini sibling would answer identically, at roughly a tenth of the price.
Two shapes have to be held apart from that one. A premium model with long answers is doing the work it was chosen for. A premium model with short answers and enormous prompts is not a model-size problem at all — the bill there is input, and the lever is caching.
The problem in plain words
Nothing is broken. There is no error, no latency complaint, no failing eval, and no dashboard row that looks unusual: the model with the largest spend is the model doing the most work, which is what you would expect. The workload that is mis-sized is buried inside that row, and the only thing that distinguishes it is that its answers are short.
What keeps it alive is that model selection is a string literal. It is chosen once, during prototyping, at the exact moment when correctness matters and cost does not, and then it is inherited — copied into the next service, pulled from the shared config, defaulted in a wrapper library. Nothing in the API distinguishes a model that is necessary from a model that is habitual, so nothing ever prompts the question. Twelve months later the classifier, the title generator, the tag extractor and the yes/no guardrail are all running on the frontier model, and each of them is one line away from costing a tenth as much.
Why it happens
Shape is visible where quality is not. The API will never tell you whether a model was needed. It will tell you how many requests were made and how many tokens came back, and the quotient of those two is the closest thing to a description of the task that the platform holds. Twenty tokens is a label. Two thousand is an argument. The first does not need a frontier model and the second might.
Volume and shape are independent, and only one of them is a finding. A model with a small share of spend can still be mis-sized, and a model with most of the spend can be perfectly chosen. Sorting by cost finds the biggest line; sorting by mean output length finds the wrong one. They are different questions and this script asks the second.
A short answer over a huge prompt is a different problem. Retrieval and summarisation produce exactly the signature this check looks for — many requests, tiny outputs — and swapping the model there saves far less than it looks like, because the money is on the input side. Mean input tokens per request separates the two, and the script reports them as different states rather than one.
The durable fix is a permission, not a config change. GET /v1/organization/projects/{project_id}/model_permissions returns a mode of allow_list or deny_list and a list of model_ids. A project that is unconstrained will drift back to the expensive model the next time somebody copies a snippet. A project restricted to the cheap models cannot.
This check cannot be done on the Claude side. GET /v1/organizations/usage_report/messages returns token sums per bucket and carries no request-count field at all, so there is no denominator and no mean answer length to compute. Model right-sizing on Anthropic has to be argued from token volume and from your own client-side call counts, not from the usage report.
The fix, as a flow
The usage endpoint already returns both numbers. Requests is one column and output tokens is another, and nobody divides them, so the shape of the work stays invisible behind its volume. The fix reads the quotient and then refuses to give the same answer to three different shapes.
How to fix it
Pull fourteen days grouped by model and project
GET /v1/organization/usage/completions with start_time set to fourteen days ago, bucket_width=1d, limit=14, and both group_by=model and group_by=project_id. Fourteen days is long enough to average out a quiet weekend and short enough that a model changed last month is not still in the numbers.
Fold the buckets before you divide
Each daily bucket holds one result per model and project combination. Sum num_model_requests, input_tokens and output_tokens across the whole window first, then take the quotient. Dividing per bucket and averaging the quotients weights a quiet Sunday the same as a Tuesday.
Put a floor under the request count
A model with forty calls in a fortnight has a mean output length that means nothing, and reporting it wastes the reader's attention on noise. The script reports anything under the floor as low-volume and moves on rather than pretending to have a verdict.
Read mean input as well as mean output
Short answers plus small prompts is a mis-sized model. Short answers plus twenty-thousand-token prompts is a retrieval workload, and the saving there is in the prefix, not the model tier. The script says which one it found, and points the second at the caching note instead of at a cheaper model.
Price it, then print the permission body
GET /v1/organization/costs?start_time=…&group_by=line_item gives the model's real thirty-day spend, which turns "use the mini one" into a number. Then print the model_permissions body that would stop the project reaching the expensive model at all. Printing it is the whole point: an audit script holding an admin key should not be what decides which model serves your traffic.
How to check it worked
Re-run a fortnight after the router is moved. The model should have dropped out of the findings entirely, not merely shrunk.
python3 openai_model_rightsizing_audit.py
# oversized gpt-5 412,880 request(s), mean output 19 token(s)
# repair: gpt-5-mini answers this shape of question; 30d spend on gpt-5 was $3411.20
# 6 model(s) checked, 1 finding(s)The full code
One usage call, one costs call, and one permissions call per project that appeared, all GET. It wants OPENAI_ADMIN_KEY, an organization admin key with read scopes, because a project key is rejected by every /v1/organization endpoint. Five pure functions carry the judgement: which tier a model id belongs to, which cheaper sibling replaces it, how the daily buckets fold, what the folded numbers mean, and whether the project is constrained from reaching the expensive model in the first place.
"""Report OpenAI models that are larger than the work they are doing.
Read only. GET requests and nothing else: OPENAI_ADMIN_KEY must be an
organization admin key (sk-admin-...) with read scopes, because every
/v1/organization endpoint rejects a project key outright.
The repair is printed, never performed. Which model serves production traffic
is a deploy, and restricting a project's model permissions changes what your
colleagues are allowed to call. Neither belongs to an audit script.
"""
import argparse
import datetime as dt
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_model_rightsizing_audit")
API = "https://api.openai.com/v1"
# Substrings that mean "this is already the small sibling". Matched on the model
# id because there is no field on the usage result that says how big a model is.
SMALL_MARKERS = ("mini", "nano", "small", "lite", "embedding", "moderation")
# The families worth right-sizing, in the order they are tested. Each maps to the
# cheaper sibling that answers the same shape of question. Kept as a table rather
# than a string rule because "gpt-5" -> "gpt-5-mini" is a naming convention, not
# a guarantee, and a wrong suggestion here is worse than none.
SIBLINGS = (
("gpt-5", "gpt-5-mini"),
("gpt-4.1", "gpt-4.1-mini"),
("gpt-4o", "gpt-4o-mini"),
("o3", "o4-mini"),
("o1", "o4-mini"),
)
FINDINGS = ("oversized",)
def tier(model):
"""Classify a model id. Pure, and deliberately conservative.
Returns "custom" for a fine-tune, "small" for a model that is already the
cheap sibling, "premium" for a family with a cheaper sibling to move to, and
"unknown" for everything else. Unknown is not a finding: a model this table
has never heard of is a model this script has no business advising on.
"""
name = str(model or "").strip().lower()
if not name:
return "unknown"
if name.startswith("ft:"):
return "custom"
if any(marker in name for marker in SMALL_MARKERS):
return "small"
for family, _cheaper in SIBLINGS:
if name.startswith(family):
return "premium"
return "unknown"
def sibling(model):
"""The cheaper model that answers the same shape of question, or None. Pure."""
name = str(model or "").strip().lower()
if tier(name) != "premium":
return None
for family, cheaper in SIBLINGS:
if name.startswith(family):
return cheaper
return None
def fold(pages):
"""Sum the daily buckets into one row per model. Pure.
Folding before dividing matters: a mean taken per bucket and then averaged
weights a quiet Sunday exactly as heavily as a Tuesday, which is how a model
that is busy on weekdays acquires a flattering output-per-request number.
project_ids are collected as a sorted list so the caller knows which projects
to ask about model permissions, and so two runs print the same order.
"""
out = {}
for page in pages:
for bucket in page.get("data") or []:
for result in bucket.get("results") or []:
model = str(result.get("model") or "").strip()
if not model:
continue
row = out.setdefault(model, {"requests": 0, "input": 0,
"output": 0, "projects": set()})
for field, key in (("num_model_requests", "requests"),
("input_tokens", "input"),
("output_tokens", "output")):
try:
row[key] += int(result.get(field) or 0)
except (TypeError, ValueError):
pass
project = result.get("project_id")
if project:
row["projects"].add(str(project))
return {m: {**row, "projects": sorted(row["projects"])} for m, row in out.items()}
def verdict(model, row, min_requests=500, trivial_output=50, long_input=20000):
"""Classify one folded model row. Pure. Returns (state, detail).
The order is the argument. A model with too few calls has no shape to read.
A model that is already small, or that this script does not recognise, is
not advised on at all. Only then does the ratio decide, and short answers
over enormous prompts are separated out because the money there is on the
input side and swapping the model saves almost none of it.
"""
try:
requests_made = int(row.get("requests") or 0)
except (TypeError, ValueError):
return ("unreadable",
"num_model_requests did not sum to an integer, so there is no "
"denominator and no ratio to read")
if requests_made <= 0:
return ("unreadable",
"0 request(s) in the window, so there is nothing to divide by")
if requests_made < min_requests:
return ("low-volume",
"%d request(s) in the window, under the floor of %d. A mean "
"taken over this few calls is noise, not a shape."
% (requests_made, min_requests))
out_per = (row.get("output") or 0) / float(requests_made)
in_per = (row.get("input") or 0) / float(requests_made)
shape = ("%d request(s), mean output %.0f token(s), mean input %.0f token(s)"
% (requests_made, out_per, in_per))
kind = tier(model)
if kind == "custom":
return ("custom-model",
"%s. This is a fine-tune, and its size is inherited from the "
"base model rather than chosen here." % shape)
if kind == "small":
return ("right-sized",
"%s. Already the cheap sibling for its family." % shape)
if kind != "premium":
return ("unknown-model",
"%s. No cheaper sibling is known for this model id, so this "
"script has no recommendation to make about it." % shape)
if out_per >= trivial_output:
return ("deliberative",
"%s. The answers are long enough that the model is plausibly "
"doing the work it was chosen for." % shape)
if in_per >= long_input:
return ("input-bound",
"%s. Short answers over very large prompts. The bill here is "
"input, not model tier, so caching the prefix will save more "
"than downgrading the model." % shape)
return ("oversized",
"%s. A premium model returning answers this short is answering "
"questions a cheaper sibling would answer identically." % shape)
def permissions_state(perms, model):
"""Can this project still reach this model? Pure. Returns a state string.
GET /v1/organization/projects/{id}/model_permissions returns a mode of
allow_list or deny_list with a model_ids array. An unconstrained project is
the durable half of the finding: without a restriction the expensive model
comes back the next time somebody copies a snippet from the quickstart.
"""
if not isinstance(perms, dict):
return "unreadable"
mode = str(perms.get("mode") or "").strip().lower()
ids = perms.get("model_ids")
if not isinstance(ids, list):
ids = []
ids = [str(i).strip().lower() for i in ids]
name = str(model or "").strip().lower()
if mode == "allow_list":
if not ids:
return "blocked"
return "allowed" if name in ids else "blocked"
if mode == "deny_list":
if not ids:
return "unconstrained"
return "blocked" if name in ids else "allowed"
return "unreadable"
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code == 401:
raise SystemExit("401 from OpenAI: OPENAI_ADMIN_KEY must be an "
"organization admin key, not a project key")
if r.status_code == 403:
raise SystemExit("403 from OpenAI: the key is not authorised for "
"/v1/organization. A project key cannot read usage.")
r.raise_for_status()
return r.json()
def usage_pages(session, start_time, days, max_pages=20):
"""Walk the usage endpoint, which paginates on next_page."""
params = {"start_time": start_time, "bucket_width": "1d", "limit": days,
"group_by": ["model", "project_id"]}
for _ in range(max_pages):
page = get(session, "/organization/usage/completions", params)
yield page
cursor = page.get("next_page")
if not cursor:
return
params = dict(params, page=cursor)
def spend_by_line_item(session, start_time):
"""Thirty days of spend, keyed by the cost report's line_item string."""
out = {}
page = get(session, "/organization/costs",
{"start_time": start_time, "limit": 31, "group_by": "line_item"})
for bucket in page.get("data") or []:
for result in bucket.get("results") or []:
item = str(result.get("line_item") or "")
amount = (result.get("amount") or {}).get("value") or 0
try:
out[item] = out.get(item, 0.0) + float(amount)
except (TypeError, ValueError):
pass
return out
def spend_for(model, spend):
"""Spend on exactly this model, from the cost report's line items. Pure.
Substring matching is not good enough here. "gpt-5" occurs inside
"gpt-5-mini, input tokens" and inside a fine-tune id built on it, and
quoting either as the premium model's spend overstates the saving in the
one line a reader is actually going to act on. So the match has to sit
between boundaries: no letter, digit, dot, dash or colon on either side.
"""
name = str(model or "").strip().lower()
if not name:
return 0.0
pattern = re.compile(r"(?<![-a-z0-9.:])" + re.escape(name) + r"(?![-a-z0-9.])")
total = 0.0
for item, amount in (spend or {}).items():
if pattern.search(str(item).lower()):
try:
total += float(amount)
except (TypeError, ValueError):
pass
return total
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=14,
help="days of usage to fold (default 14)")
ap.add_argument("--min-requests", type=int, default=500,
help="ignore models with fewer calls than this (default 500)")
ap.add_argument("--trivial-output", type=int, default=50,
help="mean output tokens under which work is trivial (default 50)")
ap.add_argument("--show-all", action="store_true",
help="also print models that are the right size")
args = ap.parse_args()
key = os.environ.get("OPENAI_ADMIN_KEY")
if not key:
log.error("set OPENAI_ADMIN_KEY (an organization admin key with read scopes)")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
now = dt.datetime.now(dt.timezone.utc)
usage_start = int((now - dt.timedelta(days=args.days)).timestamp())
cost_start = int((now - dt.timedelta(days=30)).timestamp())
rows = fold(usage_pages(session, usage_start, args.days))
spend = spend_by_line_item(session, cost_start)
checked = 0
bad = 0
for model in sorted(rows):
row = rows[model]
state, detail = verdict(model, row, args.min_requests, args.trivial_output)
checked += 1
line = "%-14s %-16s %s" % (state, model, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
cheaper = sibling(model)
money = spend_for(model, spend)
log.warning(" repair: %s answers this shape of question; 30d spend "
"on %s was $%.2f", cheaper, model, money)
for project in row["projects"]:
perms = get(session,
"/organization/projects/%s/model_permissions" % project)
where = permissions_state(perms, model)
if where == "unconstrained":
log.warning(" repair: project %s is unconstrained. To make "
"the change durable, set model_permissions to "
"mode allow_list with model_ids [%r] so the "
"expensive model cannot come back.",
project, cheaper)
else:
log.warning(" note: project %s model_permissions say %s",
project, where)
elif state == "input-bound":
log.warning(line)
log.warning(" repair: read the prompt-caching note before changing "
"the model. A stable prefix at this size is the bill.")
elif state in ("unreadable",):
log.warning(line)
elif args.show_all:
log.info(line)
log.info("%d model(s) checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report OpenAI models that are larger than the work they are doing.
*
* Read only. GET requests and nothing else: OPENAI_ADMIN_KEY must be an
* organization admin key with read scopes, because every /v1/organization
* endpoint rejects a project key outright. The repair is printed, never
* performed.
*/
const API = 'https://api.openai.com/v1';
// Substrings that mean "this is already the small sibling".
const SMALL_MARKERS = ['mini', 'nano', 'small', 'lite', 'embedding', 'moderation'];
// The families worth right-sizing, each mapped to the cheaper sibling that
// answers the same shape of question. A table rather than a string rule,
// because a wrong suggestion here is worse than no suggestion.
const SIBLINGS = [
['gpt-5', 'gpt-5-mini'],
['gpt-4.1', 'gpt-4.1-mini'],
['gpt-4o', 'gpt-4o-mini'],
['o3', 'o4-mini'],
['o1', 'o4-mini'],
];
const FINDINGS = ['oversized'];
/**
* Classify a model id. Pure, and deliberately conservative: "unknown" is not a
* finding, because a model this table has never heard of is one this script has
* no business advising on.
*/
export function tier(model) {
const name = String(model ?? '').trim().toLowerCase();
if (!name) return 'unknown';
if (name.startsWith('ft:')) return 'custom';
if (SMALL_MARKERS.some((m) => name.includes(m))) return 'small';
for (const [family] of SIBLINGS) if (name.startsWith(family)) return 'premium';
return 'unknown';
}
/** The cheaper model answering the same shape of question, or null. Pure. */
export function sibling(model) {
const name = String(model ?? '').trim().toLowerCase();
if (tier(name) !== 'premium') return null;
for (const [family, cheaper] of SIBLINGS) {
if (name.startsWith(family)) return cheaper;
}
return null;
}
/**
* Sum the daily buckets into one row per model. Pure.
*
* Folding before dividing matters: a mean taken per bucket and then averaged
* weights a quiet Sunday as heavily as a Tuesday.
*/
export function fold(pages) {
const out = new Map();
for (const page of pages) {
for (const bucket of page.data ?? []) {
for (const result of bucket.results ?? []) {
const model = String(result.model ?? '').trim();
if (!model) continue;
if (!out.has(model)) {
out.set(model, { requests: 0, input: 0, output: 0, projects: new Set() });
}
const row = out.get(model);
for (const [field, key] of [['num_model_requests', 'requests'],
['input_tokens', 'input'],
['output_tokens', 'output']]) {
const n = Number(result[field] ?? 0);
if (Number.isFinite(n)) row[key] += Math.trunc(n);
}
if (result.project_id) row.projects.add(String(result.project_id));
}
}
}
const folded = {};
for (const [model, row] of out) {
folded[model] = { ...row, projects: [...row.projects].sort() };
}
return folded;
}
/**
* Classify one folded model row. Pure. Returns [state, detail].
* Short answers over enormous prompts are separated out, because the money
* there is on the input side and swapping the model saves almost none of it.
*/
export function verdict(model, row, minRequests = 500, trivialOutput = 50,
longInput = 20000) {
const requestsMade = Number(row.requests ?? 0);
if (!Number.isFinite(requestsMade)) {
return ['unreadable',
'num_model_requests did not sum to a number, so there is no denominator ' +
'and no ratio to read'];
}
if (requestsMade <= 0) {
return ['unreadable', '0 request(s) in the window, so there is nothing to divide by'];
}
if (requestsMade < minRequests) {
return ['low-volume',
`${requestsMade} request(s) in the window, under the floor of ` +
`${minRequests}. A mean taken over this few calls is noise, not a shape.`];
}
const outPer = Number(row.output ?? 0) / requestsMade;
const inPer = Number(row.input ?? 0) / requestsMade;
const shape = `${requestsMade} request(s), mean output ${outPer.toFixed(0)} ` +
`token(s), mean input ${inPer.toFixed(0)} token(s)`;
const kind = tier(model);
if (kind === 'custom') {
return ['custom-model',
`${shape}. This is a fine-tune, and its size is inherited from the base ` +
'model rather than chosen here.'];
}
if (kind === 'small') {
return ['right-sized', `${shape}. Already the cheap sibling for its family.`];
}
if (kind !== 'premium') {
return ['unknown-model',
`${shape}. No cheaper sibling is known for this model id, so this script ` +
'has no recommendation to make about it.'];
}
if (outPer >= trivialOutput) {
return ['deliberative',
`${shape}. The answers are long enough that the model is plausibly doing ` +
'the work it was chosen for.'];
}
if (inPer >= longInput) {
return ['input-bound',
`${shape}. Short answers over very large prompts. The bill here is input, ` +
'not model tier, so caching the prefix will save more than downgrading ' +
'the model.'];
}
return ['oversized',
`${shape}. A premium model returning answers this short is answering ` +
'questions a cheaper sibling would answer identically.'];
}
/**
* Can this project still reach this model? Pure. An unconstrained project is
* the durable half of the finding: without a restriction the expensive model
* comes back the next time somebody copies a snippet from the quickstart.
*/
export function permissionsState(perms, model) {
if (perms === null || typeof perms !== 'object' || Array.isArray(perms)) {
return 'unreadable';
}
const mode = String(perms.mode ?? '').trim().toLowerCase();
const ids = (Array.isArray(perms.model_ids) ? perms.model_ids : [])
.map((i) => String(i).trim().toLowerCase());
const name = String(model ?? '').trim().toLowerCase();
if (mode === 'allow_list') {
if (ids.length === 0) return 'blocked';
return ids.includes(name) ? 'allowed' : 'blocked';
}
if (mode === 'deny_list') {
if (ids.length === 0) return 'unconstrained';
return ids.includes(name) ? 'blocked' : 'allowed';
}
return 'unreadable';
}
async function get(key, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) {
if (Array.isArray(v)) for (const item of v) url.searchParams.append(k, String(item));
else if (v !== undefined && v !== null) url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401) {
throw new Error('401 from OpenAI: OPENAI_ADMIN_KEY must be an organization ' +
'admin key, not a project key');
}
if (res.status === 403) {
throw new Error('403 from OpenAI: the key is not authorised for ' +
'/v1/organization. A project key cannot read usage.');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function usagePages(key, startTime, days, maxPages = 20) {
const pages = [];
let params = {
start_time: startTime, bucket_width: '1d', limit: days,
group_by: ['model', 'project_id'],
};
for (let i = 0; i < maxPages; i += 1) {
const page = await get(key, '/organization/usage/completions', params);
pages.push(page);
if (!page.next_page) break;
params = { ...params, page: page.next_page };
}
return pages;
}
async function spendByLineItem(key, startTime) {
const out = {};
const page = await get(key, '/organization/costs',
{ start_time: startTime, limit: 31, group_by: 'line_item' });
for (const bucket of page.data ?? []) {
for (const result of bucket.results ?? []) {
const item = String(result.line_item ?? '');
const amount = Number(result.amount?.value ?? 0);
if (Number.isFinite(amount)) out[item] = (out[item] ?? 0) + amount;
}
}
return out;
}
/**
* Spend on exactly this model, from the cost report's line items. Pure.
* Substring matching is not good enough: "gpt-5" occurs inside "gpt-5-mini,
* input tokens" and inside a fine-tune id built on it, and quoting either as
* the premium model's spend overstates the saving in the one line a reader is
* going to act on. Model ids only contain letters, digits, dots and dashes, so
* the escape is a pair of character classes rather than a backslash dance.
*/
export function spendFor(model, spend) {
const name = String(model ?? '').trim().toLowerCase();
if (!name) return 0;
const escaped = name.replace(/[.-]/g, (c) => `[${c}]`);
const pattern = new RegExp(`(?<![-a-z0-9.:])${escaped}(?![-a-z0-9.])`);
let total = 0;
for (const [item, amount] of Object.entries(spend ?? {})) {
if (pattern.test(String(item).toLowerCase())) {
const value = Number(amount);
if (Number.isFinite(value)) total += value;
}
}
return total;
}
async function main() {
const key = process.env.OPENAI_ADMIN_KEY;
if (!key) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key with read scopes)');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 14);
const minRequests = Number(process.env.MIN_REQUESTS ?? 500);
const trivialOutput = Number(process.env.TRIVIAL_OUTPUT ?? 50);
const showAll = process.argv.includes('--show-all');
const now = Math.floor(Date.now() / 1000);
const rows = fold(await usagePages(key, now - days * 86400, days));
const spend = await spendByLineItem(key, now - 30 * 86400);
let checked = 0;
let bad = 0;
for (const model of Object.keys(rows).sort()) {
const row = rows[model];
const [state, detail] = verdict(model, row, minRequests, trivialOutput);
checked += 1;
const line = `${state.padEnd(14)} ${model.padEnd(16)} ${detail}`;
if (FINDINGS.includes(state)) {
bad += 1;
console.warn(line);
const cheaper = sibling(model);
console.warn(` repair: ${cheaper} answers this shape of question; 30d ` +
`spend on ${model} was $${spendFor(model, spend).toFixed(2)}`);
for (const project of row.projects) {
const perms = await get(key,
`/organization/projects/${project}/model_permissions`);
const where = permissionsState(perms, model);
if (where === 'unconstrained') {
console.warn(` repair: project ${project} is unconstrained. To make ` +
`the change durable, set model_permissions to mode allow_list with ` +
`model_ids ['${cheaper}'] so the expensive model cannot come back.`);
} else {
console.warn(` note: project ${project} model_permissions say ${where}`);
}
}
} else if (state === 'input-bound') {
console.warn(line);
console.warn(' repair: read the prompt-caching note before changing the ' +
'model. A stable prefix at this size is the bill.');
} else if (state === 'unreadable') {
console.warn(line);
} else if (showAll) {
console.log(line);
}
}
console.log(`${checked} model(s) checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is that a high-volume premium model with a mean output of nineteen tokens is a finding, and that the same ratio on the mini sibling is not — the whole note is that shape and tier have to be read together. The rest hold the near misses apart: long answers on a premium model are the model doing its job, short answers over a huge prompt are a caching problem wearing this problem's signature, and a model too quiet to have a shape gets no verdict at all.
from openai_model_rightsizing_audit import (fold, permissions_state, sibling,
spend_for, tier, verdict)
def row(requests=10000, output=190000, input_=900000, projects=("proj_a",)):
"""A folded row shaped like fold() returns them."""
return {"requests": requests, "output": output, "input": input_,
"projects": list(projects)}
def bucket(**results):
"""One daily bucket from GET /v1/organization/usage/completions."""
return {"data": [{"start_time": 0, "results": [
{"model": m, "num_model_requests": r, "input_tokens": i,
"output_tokens": o, "project_id": p}
for m, (r, i, o, p) in results.items()]}]}
def test_a_premium_model_with_tiny_answers_is_the_finding():
# The whole note: 412,880 calls, mean answer 19 tokens, on the frontier model.
state, detail = verdict("gpt-5", row(requests=412880, output=7844720,
input_=170000000))
assert state == "oversized"
assert "mean output 19 token(s)" in detail
assert sibling("gpt-5") == "gpt-5-mini"
def test_the_same_shape_on_the_mini_sibling_is_not_a_finding():
state, _ = verdict("gpt-5-mini", row(requests=412880, output=7844720,
input_=170000000))
assert state == "right-sized"
def test_long_answers_are_the_model_doing_its_job():
state, detail = verdict("gpt-5", row(requests=9000, output=18000000,
input_=9000000))
assert state == "deliberative"
assert "mean output 2000 token(s)" in detail
def test_short_answers_over_huge_prompts_are_a_caching_problem():
# Same ratio as the finding on the output side, 40k tokens of prompt on the
# input side. Downgrading the model here saves almost nothing.
state, detail = verdict("gpt-4.1", row(requests=5000, output=95000,
input_=200000000))
assert state == "input-bound"
assert "caching the prefix" in detail
def test_a_model_too_quiet_to_have_a_shape_gets_no_verdict():
assert verdict("gpt-5", row(requests=40, output=760))[0] == "low-volume"
assert verdict("gpt-5", row(requests=0, output=0))[0] == "unreadable"
def test_tiers_are_conservative_about_what_they_claim_to_know():
assert tier("ft:gpt-4o-mini-2024-07-18:acme::AbC123") == "custom"
assert tier("text-embedding-3-large") == "small"
assert tier("some-model-we-have-never-heard-of") == "unknown"
assert sibling("some-model-we-have-never-heard-of") is None
assert verdict("ft:gpt-4o-2024-08-06:acme::X", row())[0] == "custom-model"
assert verdict("some-model-we-have-never-heard-of", row())[0] == "unknown-model"
def test_buckets_are_folded_before_the_division():
pages = [bucket(**{"gpt-5": (100, 50000, 1000, "proj_a")}),
bucket(**{"gpt-5": (900, 450000, 9000, "proj_b")})]
folded = fold(pages)
assert folded["gpt-5"]["requests"] == 1000
assert folded["gpt-5"]["output"] == 10000
assert folded["gpt-5"]["projects"] == ["proj_a", "proj_b"]
# 10000/1000 = 10 tokens a call. Averaging the two buckets' quotients would
# have given (10 + 10) / 2 by luck here and something wrong on real data.
assert "mean output 10 token(s)" in verdict("gpt-5", folded["gpt-5"],
min_requests=100)[1]
def test_permissions_say_whether_the_expensive_model_can_come_back():
assert permissions_state({"mode": "deny_list", "model_ids": []},
"gpt-5") == "unconstrained"
assert permissions_state({"mode": "deny_list", "model_ids": ["gpt-5"]},
"gpt-5") == "blocked"
assert permissions_state({"mode": "allow_list", "model_ids": ["gpt-5-mini"]},
"gpt-5") == "blocked"
assert permissions_state({"mode": "allow_list", "model_ids": ["gpt-5"]},
"gpt-5") == "allowed"
assert permissions_state({}, "gpt-5") == "unreadable"
assert permissions_state(None, "gpt-5") == "unreadable"
def test_spend_is_matched_to_the_model_and_not_to_its_siblings():
# The repair line quotes a dollar figure, so a substring match that swept in
# the mini model would overstate exactly the number a reader acts on.
spend = {"gpt-5, input tokens": 3000.00,
"gpt-5, output tokens": 411.20,
"gpt-5-mini, input tokens": 90.00,
"ft:gpt-5:acme::x, input tokens": 12.00}
assert spend_for("gpt-5", spend) == 3411.20
assert spend_for("gpt-5-mini", spend) == 90.00
assert spend_for("", spend) == 0.0
assert spend_for("gpt-5", {}) == 0.0
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { fold, permissionsState, sibling, spendFor, tier, verdict }
from './openai-model-rightsizing-audit.mjs';
/** A folded row shaped like fold() returns them. */
function row({ requests = 10000, output = 190000, input = 900000,
projects = ['proj_a'] } = {}) {
return { requests, output, input, projects };
}
/** One daily bucket from GET /v1/organization/usage/completions. */
function bucket(results) {
return {
data: [{
start_time: 0,
results: Object.entries(results).map(([model, [r, i, o, p]]) => ({
model, num_model_requests: r, input_tokens: i, output_tokens: o,
project_id: p,
})),
}],
};
}
test('a premium model with tiny answers is the finding', () => {
const [state, detail] = verdict('gpt-5',
row({ requests: 412880, output: 7844720, input: 170000000 }));
assert.equal(state, 'oversized');
assert.match(detail, /mean output 19 token/);
assert.equal(sibling('gpt-5'), 'gpt-5-mini');
});
test('the same shape on the mini sibling is not a finding', () => {
const [state] = verdict('gpt-5-mini',
row({ requests: 412880, output: 7844720, input: 170000000 }));
assert.equal(state, 'right-sized');
});
test('long answers are the model doing its job', () => {
const [state, detail] = verdict('gpt-5',
row({ requests: 9000, output: 18000000, input: 9000000 }));
assert.equal(state, 'deliberative');
assert.match(detail, /mean output 2000 token/);
});
test('short answers over huge prompts are a caching problem', () => {
const [state, detail] = verdict('gpt-4.1',
row({ requests: 5000, output: 95000, input: 200000000 }));
assert.equal(state, 'input-bound');
assert.match(detail, /caching the prefix/);
});
test('a model too quiet to have a shape gets no verdict', () => {
assert.equal(verdict('gpt-5', row({ requests: 40, output: 760 }))[0], 'low-volume');
assert.equal(verdict('gpt-5', row({ requests: 0, output: 0 }))[0], 'unreadable');
});
test('tiers are conservative about what they claim to know', () => {
assert.equal(tier('ft:gpt-4o-mini-2024-07-18:acme::AbC123'), 'custom');
assert.equal(tier('text-embedding-3-large'), 'small');
assert.equal(tier('some-model-we-have-never-heard-of'), 'unknown');
assert.equal(sibling('some-model-we-have-never-heard-of'), null);
assert.equal(verdict('ft:gpt-4o-2024-08-06:acme::X', row())[0], 'custom-model');
assert.equal(verdict('some-model-we-have-never-heard-of', row())[0], 'unknown-model');
});
test('buckets are folded before the division', () => {
const pages = [bucket({ 'gpt-5': [100, 50000, 1000, 'proj_a'] }),
bucket({ 'gpt-5': [900, 450000, 9000, 'proj_b'] })];
const folded = fold(pages);
assert.equal(folded['gpt-5'].requests, 1000);
assert.equal(folded['gpt-5'].output, 10000);
assert.deepEqual(folded['gpt-5'].projects, ['proj_a', 'proj_b']);
assert.match(verdict('gpt-5', folded['gpt-5'], 100)[1], /mean output 10 token/);
});
test('permissions say whether the expensive model can come back', () => {
assert.equal(permissionsState({ mode: 'deny_list', model_ids: [] }, 'gpt-5'),
'unconstrained');
assert.equal(permissionsState({ mode: 'deny_list', model_ids: ['gpt-5'] }, 'gpt-5'),
'blocked');
assert.equal(permissionsState({ mode: 'allow_list', model_ids: ['gpt-5-mini'] }, 'gpt-5'),
'blocked');
assert.equal(permissionsState({ mode: 'allow_list', model_ids: ['gpt-5'] }, 'gpt-5'),
'allowed');
assert.equal(permissionsState({}, 'gpt-5'), 'unreadable');
assert.equal(permissionsState(null, 'gpt-5'), 'unreadable');
});
test('spend is matched to the model and not to its siblings', () => {
const spend = {
'gpt-5, input tokens': 3000.00,
'gpt-5, output tokens': 411.20,
'gpt-5-mini, input tokens': 90.00,
'ft:gpt-5:acme::x, input tokens': 12.00,
};
assert.equal(spendFor('gpt-5', spend), 3411.20);
assert.equal(spendFor('gpt-5-mini', spend), 90.00);
assert.equal(spendFor('', spend), 0);
assert.equal(spendFor('gpt-5', {}), 0);
});
FAQ
What counts as a trivial answer?
The default floor in the script is fifty output tokens on average, which is roughly a sentence. Classifiers, routers, tag extractors, yes/no guardrails and title generators all land far below it, usually under twenty. Anything above a couple of hundred tokens is prose the model had to compose, and the check should not be firing on it. Move the threshold to fit your workloads rather than arguing with the default.
Will a mini model actually give the same answer?
For classification into a fixed set of labels, extraction against a schema, and routing, usually yes, and you can find out cheaply. Run the same thousand production inputs through both, diff the outputs, and look at the disagreements. That is a day's work against a spend difference of roughly an order of magnitude, and it is the only evidence anyone should accept for a model swap.
Why does the script report short answers over long prompts separately?
Because it is a different bill. A retrieval or summarisation step sends twenty thousand tokens of context and gets three hundred back, so almost all the money is on the input side. Downgrading the model there saves a fraction of what caching the prefix saves, and reporting the two findings with the same sentence sends people to the wrong lever.
Can I do the same check on the Claude API?
Not this way. GET /v1/organizations/usage_report/messages returns token sums per bucket with no request-count field, so there is no denominator and no mean answer length to compute. On that side you can compare token volume between models and workspaces, but the per-request shape has to come from your own client-side metrics. The right-sizing move is the same shape though: Claude Haiku 4.5 for the trivial workloads, an Opus or Sonnet 5 model where the reasoning is worth paying for.
Why print the model_permissions body instead of just changing the config?
Because a config change lasts until the next person copies a snippet from the quickstart, and a project restricted to an allow_list does not. The permission is the durable half of the repair, which is also exactly why an audit script should not apply it: restricting which models your colleagues can call is a decision with an owner, and that owner is not a cron job holding an admin key.
Related field notes
- Why the Usage API cannot segment by your customers
- Output tokens, not input, are what the bill is made of
- A stable prefix reprocessed at full price on every call
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Usage: completions — OpenAI API reference
- Administration and the Admin APIs — OpenAI developer docs
- Projects — OpenAI API reference
- Costs — OpenAI API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.