Diagnostic LLM APIs
Requests billed, zero output tokens: max_tokens refused
The model constant changed in a one-line pull request, because the old id has a shutdown date and somebody diaried it properly. The deploy went out on Thursday. Nothing paged: the endpoint returns a 500 to the user and the retry wrapper swallows it, and the error-rate dashboard is scoped to the gateway rather than to this worker. What the organization usage report shows for Friday is eleven thousand requests against the new model, no input tokens, and no output tokens at all.
Read the buckets and look for the impossible row. With an organization admin key: GET /v1/organization/usage/completions?start_time={now-24h}&bucket_width=1h&group_by=model&group_by=project_id. A result with num_model_requests above zero, output_tokens at zero and input_tokens at zero is a set of calls that never reached the model: the request body was rejected on validation.
On the reasoning families that is almost always one field. Chat Completions replaced max_tokens with max_completion_tokens, because the cap now has to cover reasoning tokens as well as visible output, and the old name is refused outright with code: "unsupported_parameter". On the Responses API the field is max_output_tokens. The same rejection covers temperature, top_p, presence_penalty and frequency_penalty, which reasoning models replace with a reasoning effort setting.
This is not a number that is too large. No value of max_tokens works, so raising or lowering it changes nothing; the parameter is refused by name. Confirm with one GET /v1/models/{model} using a Read Only project key: a 200 proves the id is valid and reachable, which puts the fault in the request body rather than in access or retirement.
The problem in plain words
There is no request log to check. Neither API has an endpoint that lists calls with their status codes, so a fleet that is 400ing on every single request looks, from the provider's side, exactly like a fleet that is quietly succeeding — unless you notice that the successful ones would have produced tokens. Requests are counted. Tokens are not. That gap is the entire signal.
What makes it survive a week is the retry layer. A 400 is not retryable, but a wrapper that catches broadly retries it anyway, three times, and then raises something the caller has always treated as transient. Users see a slow failure. The queue drains, eventually, into a dead-letter table nobody reads. The invoice for the model goes to nearly nothing, which is the one visible symptom, and a bill that went down is not a bill anyone investigates.
Why it happens
The cap changed meaning, not just its name. A reasoning model generates tokens you never see before it generates the ones you do, and both are billed as output. max_completion_tokens caps the sum. Renaming the field and keeping the number is a second bug waiting behind the first: a budget that used to fit a four-hundred-word answer now has to fit the thinking as well, and a request that runs out mid-reasoning comes back with an empty message and a length finish reason. Rename it and raise it.
A rejected parameter and a parameter that is out of range are different findings. A max_tokens above the model's own ceiling is a number to lower, provable in advance against the model object, and it fails with a value error. This one fails with unsupported_parameter and no value of the field is acceptable. They read almost identically in an incident channel and they have nothing in common in the code.
Zero output with input tokens present is a different problem again. If the buckets show input tokens being read and nothing coming back, the prompt reached the model and generation was blocked: organization verification on a streaming path, a content filter, or a cap set to zero. This script keeps those apart rather than folding both into "the model is broken", because the second one sends you to the console and the first one sends you to a diff.
Partial silence means a partial deploy. A share of requests generating nothing, rather than all of them, is usually one replica set that did not restart, one Lambda alias still on the old package, or a canary. The finding is the same field and the repair is a rollout rather than a code change, so the script reports the share instead of rounding it to yes or no.
The sampling parameters go the same way. temperature=0 "for determinism" is one of the most common defaults in the ecosystem, and reasoning models reject it with unsupported_value because variance is controlled by effort rather than by sampling. A codebase that fixes only max_tokens ships, fails identically the same afternoon, and nobody believes the diagnosis the second time. Print the whole list at once.
The fix, as a flow
A model id changed, and the field that used to cap the answer is now refused by name rather than by value. Nothing in the aggregate says 400, because the aggregate has no status codes in it. What it has is a request count sitting on top of no tokens at all, which is the only shape a body rejected before generation can make.
How to fix it
Read a day of hourly buckets grouped by model and project
GET /v1/organization/usage/completions with bucket_width=1h, group_by=model and group_by=project_id. Both groupings matter: the model tells you which family refused the parameter and the project tells you whose deploy did it. An hour is the right grain because the fault starts at a deploy, and a daily bucket smears the before and the after together.
Find rows with requests and no tokens on either side
num_model_requests > 0 with output_tokens == 0. Then split on input: no input tokens means the body was rejected before the prompt was read, input tokens present means the prompt was read and generation did not happen. Only the first is this note.
Confirm the model id is reachable before blaming the model
One GET /v1/models/{model} with a Read Only project key. A 200 says the id exists and this key can use it, which leaves the request body as the only remaining suspect. A 404 says the opposite and hands you to the retirement and entitlement notes instead — same symptom in the log, unrelated repair.
Print the rename for the surface the code actually uses
Chat Completions wants max_completion_tokens. The Responses API wants max_output_tokens. They are not interchangeable, and a wrapper library that supports both surfaces needs the branch rather than one global replace. Print both lines and let the reader pick.
Fix the sampling parameters in the same change
temperature, top_p, presence_penalty, frequency_penalty and logprobs are refused by the same models for the same reason. Remove them, express the intent as a reasoning effort instead, and do not send temperature: 1 explicitly to be safe — omit the field. Two deploys for one incident is how a team stops trusting the diagnosis.
How to check it worked
Re-run an hour after the deploy. The row should keep its request count and grow output tokens; a row that keeps 100% silence has a second rejected field in it.
python3 openai_zero_output_buckets.py --hours 24
# parameter-rejected proj_api / gpt-5.1 11482 request(s) over 24 bucket(s), 0 input token(s) and 0 output token(s). Nothing was read and nothing was generated.
# the id resolves for this key, so the fault is in the request body and not in access
# repair: Chat Completions: send max_completion_tokens instead of max_tokens, and raise the number.
# 6 model/project row(s) checked, 1 finding(s)The full code
One GET for the buckets and, when a finding turns up, one cheap GET per model id to prove the id is reachable. Six pure functions: the fold, which keeps the silent buckets countable rather than summing them away; the family test, which has to say no to gpt-4o as confidently as it says yes to o3-mini; the share; the classifier, which splits a rejected body from blocked generation on whether any input tokens were read; the repair lines for each API surface; and the reading of the model lookup's status code, because a 404 there is a different note with the same symptom.
"""Find OpenAI usage buckets that counted requests and generated nothing.
Read only. One GET against the organization usage report, which needs an
organization admin key (sk-admin-) and can be provisioned read-only, plus an
optional GET /v1/models/{id} with a project key set to Read Only.
Neither API lists individual requests, so this is a shape in the aggregate
rather than an error log: num_model_requests above zero with output_tokens at
zero is a set of calls that never reached generation, and no input tokens with
it means the request body was rejected before the prompt was read.
The repair is printed, never performed. Renaming a request field is a deploy.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_zero_output_buckets")
API = "https://api.openai.com/v1"
# The families that replaced max_tokens with max_completion_tokens and refuse
# the sampling parameters outright. Matched as whole id prefixes, because a
# substring test for "o1" or "o3" also matches ids that have nothing to do with
# reasoning and a substring test for "o" matches gpt-4o.
REASONING_PREFIXES = ("o1", "o3", "o4", "gpt-5")
FINDINGS = ("parameter-rejected", "partial-rejection")
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def is_reasoning_model(model):
"""Is this id one of the families that refuse max_tokens? Pure.
Whole-prefix matching only. gpt-4o must come back False here or the script
prints a rename that does not apply and sends somebody to change a field
that was never the problem.
"""
name = str(model or "").strip().lower()
if not name:
return False
for prefix in REASONING_PREFIXES:
if name == prefix or name.startswith(prefix + "-") or name.startswith(prefix + "."):
return True
return False
def fold(buckets):
"""Fold usage buckets into one row per (project, model). Pure.
The silent buckets are counted rather than summed away. "Every bucket in
the window generated nothing" and "one bucket in twelve generated nothing"
are a broken deploy and a half-finished rollout, and a total cannot tell
them apart.
"""
rows = {}
for bucket in buckets or []:
for result in bucket.get("results") or []:
key = (str(result.get("project_id") or "unknown"),
str(result.get("model") or "unknown"))
row = rows.setdefault(key, {"requests": 0, "input": 0, "output": 0,
"buckets": 0, "silent_buckets": 0,
"silent_requests": 0, "silent_input": 0})
made = _int(result.get("num_model_requests"))
read = _int(result.get("input_tokens"))
wrote = _int(result.get("output_tokens"))
row["requests"] += made
row["input"] += read
row["output"] += wrote
row["buckets"] += 1
if made > 0 and wrote == 0:
row["silent_buckets"] += 1
row["silent_requests"] += made
row["silent_input"] += read
return rows
def silent_share(row):
"""Share of a row's requests that generated no output at all. Pure.
None when there were no requests, which is a different state from zero and
must not be rounded into one.
"""
requests_made = _int((row or {}).get("requests"))
if requests_made <= 0:
return None
return min(1.0, _int(row.get("silent_requests")) / float(requests_made))
def classify(model, row, min_requests=50, partial_floor=0.2, total_floor=0.99):
"""Classify one (project, model) row. Pure. Returns (state, detail).
The split that matters is on input tokens inside the silent buckets. No
input and no output means the request body was rejected on validation.
Input read with no output means the prompt reached the model and generation
was blocked, which is verification or a filter and a different repair.
"""
row = row or {}
requests_made = _int(row.get("requests"))
if requests_made < min_requests:
return ("too-few-requests",
"%d request(s) in the window, under the floor of %d. A silence "
"this small is not evidence of anything."
% (requests_made, min_requests))
share = silent_share(row) or 0.0
shape = ("%d request(s) over %d bucket(s), %d input token(s) and %d output "
"token(s)" % (requests_made, _int(row.get("buckets")),
_int(row.get("input")), _int(row.get("output"))))
if share >= total_floor:
if _int(row.get("silent_input")) == 0:
return ("parameter-rejected",
shape + ". Nothing was read and nothing was generated, so "
"these calls were rejected on the request body before the "
"prompt was processed.")
return ("generation-blocked",
shape + ". The prompt was read and nothing came back, which is "
"not a refused parameter name: look at organization "
"verification, a content filter, or an output cap of zero.")
if share >= partial_floor:
return ("partial-rejection",
"%s, and %.0f%% of those requests generated nothing. Part of "
"the fleet is still sending the old field."
% (shape, share * 100))
return ("generating", shape + ".")
def repair_lines(model):
"""The exact request-body repair for one model id. Pure.
Both API surfaces, because they are not interchangeable and a wrapper that
supports both needs the branch rather than one global replace.
"""
if is_reasoning_model(model):
return [
"Chat Completions: send max_completion_tokens instead of "
"max_tokens, and raise the number. The cap now has to absorb "
"reasoning tokens as well as the visible answer.",
"Responses API: the same field is called max_output_tokens.",
"Remove temperature, top_p, presence_penalty, frequency_penalty "
"and logprobs for this model and express the intent as a reasoning "
"effort setting. Do not send temperature 1 explicitly; omit it.",
]
return [
"This id is not one of the reasoning families, so a refused parameter "
"name is the less likely cause here. Read one 400 body for its code "
"and param fields before changing anything.",
]
def model_verdict(status):
"""What the model lookup says about whose fault the failure is. Pure."""
if status is None:
return ("unchecked",
"no project key was supplied, so the model id itself was not "
"checked")
if status == 200:
return ("id-resolves",
"the id resolves for this key, so the fault is in the request "
"body and not in access")
if status == 404:
return ("id-unreachable",
"the id does not resolve for this key. That is retirement or "
"entitlement rather than a parameter name, and it is a "
"different repair")
if status in (401, 403):
return ("check-refused",
"the project key could not read the model list, so the id was "
"not confirmed either way")
return ("check-inconclusive", "the model lookup returned %d" % int(status))
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key (sk-admin-), not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def pages(session, path, params, max_pages=40):
"""Walk the usage report, which paginates on an opaque page cursor."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params = dict(params)
params["page"] = page["next_page"]
def check_model(key, model):
"""One cheap GET to prove the id is reachable. Returns a status code."""
if not key:
return None
try:
r = requests.get(API + "/models/" + str(model),
headers={"Authorization": "Bearer " + key}, timeout=30)
except requests.RequestException:
return None
return r.status_code
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--hours", type=int, default=24,
help="hours of hourly buckets to read (default 24)")
ap.add_argument("--min-requests", type=int, default=50,
help="ignore rows below this many requests (default 50)")
ap.add_argument("--show-all", action="store_true",
help="also print rows that are generating normally")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY (an organization admin key; read-only "
"scopes are enough)")
return 2
project_key = os.environ.get("OPENAI_API_KEY")
hours = max(1, min(int(args.hours), 168))
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
buckets = pages(session, "/organization/usage/completions", {
"start_time": int(time.time()) - hours * 3600,
"bucket_width": "1h",
"limit": hours,
"group_by": ["model", "project_id"],
})
rows = fold(buckets)
if not rows:
log.info("no completions usage in the last %d hour(s)", hours)
return 0
checked = 0
bad = 0
for project, model in sorted(rows, key=lambda k: -rows[k]["requests"]):
row = rows[(project, model)]
state, detail = classify(model, row, args.min_requests)
checked += 1
line = "%-19s %s / %s %s" % (state, project, model, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
_, note = model_verdict(check_model(project_key, model))
log.warning(" %s", note)
for repair in repair_lines(model):
log.warning(" repair: %s", repair)
elif state == "generation-blocked":
log.warning(line)
log.warning(" repair: this is not the parameter rename. Check "
"organization verification for the streaming path and "
"the project's model permissions before touching the "
"request body.")
elif args.show_all:
log.info(line)
log.info("%d model/project row(s) checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI usage buckets that counted requests and generated nothing.
*
* Read only. One GET against the organization usage report, which needs an
* organization admin key (sk-admin-), plus an optional GET /v1/models/{id}
* with a project key set to Read Only.
*
* num_model_requests above zero with output_tokens at zero is a set of calls
* that never reached generation; no input tokens with it means the body was
* rejected before the prompt was read. The repair is printed, never performed.
*/
const API = 'https://api.openai.com/v1';
// Whole-id prefixes. A substring test for "o" would match gpt-4o.
const REASONING_PREFIXES = ['o1', 'o3', 'o4', 'gpt-5'];
const FINDINGS = new Set(['parameter-rejected', 'partial-rejection']);
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Is this id one of the families that refuse max_tokens? Pure.
* gpt-4o must be false here, or the script prints a rename that does not apply.
*/
export function isReasoningModel(model) {
const name = String(model ?? '').trim().toLowerCase();
if (!name) return false;
return REASONING_PREFIXES.some(
(p) => name === p || name.startsWith(`${p}-`) || name.startsWith(`${p}.`));
}
/**
* Fold usage buckets into one row per (project, model). Pure.
* Silent buckets are counted rather than summed away: all of them and one in
* twelve are a broken deploy and a half-finished rollout.
*/
export function fold(buckets) {
const rows = new Map();
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const key = `${result?.project_id ?? 'unknown'}\u0000${result?.model ?? 'unknown'}`;
if (!rows.has(key)) {
rows.set(key, { project: String(result?.project_id ?? 'unknown'),
model: String(result?.model ?? 'unknown'),
requests: 0, input: 0, output: 0, buckets: 0,
silentBuckets: 0, silentRequests: 0, silentInput: 0 });
}
const row = rows.get(key);
const made = readInt(result?.num_model_requests);
const read = readInt(result?.input_tokens);
const wrote = readInt(result?.output_tokens);
row.requests += made;
row.input += read;
row.output += wrote;
row.buckets += 1;
if (made > 0 && wrote === 0) {
row.silentBuckets += 1;
row.silentRequests += made;
row.silentInput += read;
}
}
}
return rows;
}
/** Share of a row's requests that generated no output. Pure. Null when none. */
export function silentShare(row) {
const made = readInt(row?.requests);
if (made <= 0) return null;
return Math.min(1, readInt(row?.silentRequests) / made);
}
/**
* Classify one (project, model) row. Pure. Returns [state, detail].
* The split is on input tokens inside the silent buckets: none means the body
* was rejected on validation, some means generation was blocked instead.
*/
export function classify(model, row, minRequests = 50, partialFloor = 0.2,
totalFloor = 0.99) {
const made = readInt(row?.requests);
if (made < minRequests) {
return ['too-few-requests',
`${made} request(s) in the window, under the floor of ${minRequests}. ` +
'A silence this small is not evidence of anything.'];
}
const share = silentShare(row) ?? 0;
const shape = `${made} request(s) over ${readInt(row?.buckets)} bucket(s), ` +
`${readInt(row?.input)} input token(s) and ${readInt(row?.output)} output token(s)`;
if (share >= totalFloor) {
if (readInt(row?.silentInput) === 0) {
return ['parameter-rejected',
`${shape}. Nothing was read and nothing was generated, so these calls ` +
'were rejected on the request body before the prompt was processed.'];
}
return ['generation-blocked',
`${shape}. The prompt was read and nothing came back, which is not a ` +
'refused parameter name: look at organization verification, a content ' +
'filter, or an output cap of zero.'];
}
if (share >= partialFloor) {
return ['partial-rejection',
`${shape}, and ${(share * 100).toFixed(0)}% of those requests generated ` +
'nothing. Part of the fleet is still sending the old field.'];
}
return ['generating', `${shape}.`];
}
/** The exact request-body repair for one model id. Pure. */
export function repairLines(model) {
if (isReasoningModel(model)) {
return [
'Chat Completions: send max_completion_tokens instead of max_tokens, and ' +
'raise the number. The cap now has to absorb reasoning tokens as well as ' +
'the visible answer.',
'Responses API: the same field is called max_output_tokens.',
'Remove temperature, top_p, presence_penalty, frequency_penalty and ' +
'logprobs for this model and express the intent as a reasoning effort ' +
'setting. Do not send temperature 1 explicitly; omit it.',
];
}
return [
'This id is not one of the reasoning families, so a refused parameter name ' +
'is the less likely cause here. Read one 400 body for its code and param ' +
'fields before changing anything.',
];
}
/** What the model lookup says about whose fault the failure is. Pure. */
export function modelVerdict(status) {
if (status === null || status === undefined) {
return ['unchecked',
'no project key was supplied, so the model id itself was not checked'];
}
if (status === 200) {
return ['id-resolves',
'the id resolves for this key, so the fault is in the request body and ' +
'not in access'];
}
if (status === 404) {
return ['id-unreachable',
'the id does not resolve for this key. That is retirement or entitlement ' +
'rather than a parameter name, and it is a different repair'];
}
if (status === 401 || status === 403) {
return ['check-refused',
'the project key could not read the model list, so the id was not ' +
'confirmed either way'];
}
return ['check-inconclusive', `the model lookup returned ${status}`];
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization/* needs an ` +
'organization admin key (sk-admin-), not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* pages(key, path, params, maxPages = 40) {
let query = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function checkModel(key, model) {
if (!key) return null;
try {
const res = await fetch(`${API}/models/${model}`,
{ headers: { Authorization: `Bearer ${key}` } });
return res.status;
} catch {
return null;
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key; read-only ' +
'scopes are enough)');
process.exitCode = 2;
return;
}
const projectKey = process.env.OPENAI_API_KEY;
const hours = Math.max(1, Math.min(Number(process.env.HOURS ?? 24), 168));
const minRequests = Number(process.env.MIN_REQUESTS ?? 50);
const showAll = process.env.SHOW_ALL === '1';
const buckets = [];
for await (const bucket of pages(admin, '/organization/usage/completions', {
start_time: Math.floor(Date.now() / 1000) - hours * 3600,
bucket_width: '1h',
limit: hours,
group_by: ['model', 'project_id'],
})) buckets.push(bucket);
const rows = fold(buckets);
if (rows.size === 0) {
console.log(`no completions usage in the last ${hours} hour(s)`);
return;
}
let checked = 0;
let bad = 0;
const ordered = [...rows.values()].sort((a, b) => b.requests - a.requests);
for (const row of ordered) {
const [state, detail] = classify(row.model, row, minRequests);
checked += 1;
const line = `${state.padEnd(19)} ${row.project} / ${row.model} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
const [, note] = modelVerdict(await checkModel(projectKey, row.model));
console.warn(` ${note}`);
for (const repair of repairLines(row.model)) console.warn(` repair: ${repair}`);
} else if (state === 'generation-blocked') {
console.warn(line);
console.warn(' repair: this is not the parameter rename. Check organization ' +
'verification for the streaming path and the project model ' +
'permissions before touching the request body.');
} else if (showAll) {
console.log(line);
}
}
console.log(`${checked} model/project row(s) checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is the row that has requests and nothing else: eleven thousand calls, no input tokens, no output tokens, classified as a rejected body rather than as a quiet model. Beside it sits the row this note is most often confused with — same request count, same zero output, but input tokens were read — and it has to come back as a different state with a different repair. The family test earns its place by saying no to gpt-4o, which a substring match on o would happily call a reasoning model, and the partial case pins the share so a half-finished rollout is not rounded up into a total outage.
from openai_zero_output_buckets import (classify, fold, is_reasoning_model,
model_verdict, repair_lines,
silent_share)
def bucket(project, model, requests_made, input_tokens, output_tokens):
return {"results": [{"project_id": project, "model": model,
"num_model_requests": requests_made,
"input_tokens": input_tokens,
"output_tokens": output_tokens}]}
def test_requests_with_no_tokens_either_side_is_a_rejected_body():
# The note in one assertion. Every call counted, nothing read, nothing
# written: the body never got past validation.
rows = fold([bucket("proj_api", "gpt-5.1", 500, 0, 0) for _ in range(24)])
row = rows[("proj_api", "gpt-5.1")]
assert row["requests"] == 12000
assert row["buckets"] == 24 and row["silent_buckets"] == 24
assert silent_share(row) == 1.0
state, detail = classify("gpt-5.1", row)
assert state == "parameter-rejected"
assert "0 input token(s) and 0 output token(s)" in detail
assert "max_completion_tokens" in repair_lines("gpt-5.1")[0]
assert "max_output_tokens" in repair_lines("gpt-5.1")[1]
def test_input_read_and_nothing_generated_is_a_different_finding():
# Same request count, same zero output, and not this note: the prompt
# reached the model, so the body was accepted and generation was blocked.
rows = fold([bucket("proj_api", "gpt-5.1", 500, 900000, 0) for _ in range(24)])
state, detail = classify("gpt-5.1", rows[("proj_api", "gpt-5.1")])
assert state == "generation-blocked"
assert "verification" in detail
def test_a_partial_rollout_is_not_rounded_up_to_a_total_outage():
silent = [bucket("proj_api", "o3-mini", 100, 0, 0) for _ in range(6)]
healthy = [bucket("proj_api", "o3-mini", 100, 200000, 40000) for _ in range(18)]
row = fold(silent + healthy)[("proj_api", "o3-mini")]
assert silent_share(row) == 0.25
state, detail = classify("o3-mini", row)
assert state == "partial-rejection"
assert "25%" in detail
def test_the_reasoning_families_are_matched_as_whole_prefixes():
for model in ("o1", "o3-mini", "o4-mini", "gpt-5", "gpt-5.1-mini",
"gpt-5-2026-01-15"):
assert is_reasoning_model(model) is True
# gpt-4o is the one a careless substring match gets wrong.
for model in ("gpt-4o", "gpt-4o-mini", "gpt-4.1", "claude-sonnet-5", "", None):
assert is_reasoning_model(model) is False
assert "reasoning families" in repair_lines("gpt-4o")[0]
def test_a_quiet_row_is_not_a_silent_one():
assert silent_share({"requests": 0, "silent_requests": 0}) is None
assert silent_share(None) is None
state, _ = classify("gpt-5.1", {"requests": 4, "silent_requests": 4})
assert state == "too-few-requests"
healthy = fold([bucket("p", "gpt-5.1", 500, 200000, 60000)])
assert classify("gpt-5.1", healthy[("p", "gpt-5.1")])[0] == "generating"
def test_a_404_on_the_model_lookup_is_a_different_note_entirely():
assert model_verdict(200)[0] == "id-resolves"
assert model_verdict(404)[0] == "id-unreachable"
assert "retirement or entitlement" in model_verdict(404)[1]
assert model_verdict(403)[0] == "check-refused"
assert model_verdict(None)[0] == "unchecked"
def test_unreadable_usage_fields_do_not_become_phantom_requests():
rows = fold([{"results": [{"project_id": "p", "model": "gpt-5.1",
"num_model_requests": None,
"input_tokens": "nonsense",
"output_tokens": None}]}])
assert rows[("p", "gpt-5.1")]["requests"] == 0
assert rows[("p", "gpt-5.1")]["silent_buckets"] == 0
assert fold([]) == {}
assert fold(None) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, fold, isReasoningModel, modelVerdict, repairLines, silentShare }
from './openai-zero-output-buckets.mjs';
const bucket = (project, model, made, input, output) => ({
results: [{ project_id: project, model, num_model_requests: made,
input_tokens: input, output_tokens: output }],
});
const rowFor = (buckets, project, model) =>
[...fold(buckets).values()].find((r) => r.project === project && r.model === model);
test('requests with no tokens either side is a rejected body', () => {
const buckets = Array.from({ length: 24 },
() => bucket('proj_api', 'gpt-5.1', 500, 0, 0));
const row = rowFor(buckets, 'proj_api', 'gpt-5.1');
assert.equal(row.requests, 12000);
assert.equal(row.buckets, 24);
assert.equal(row.silentBuckets, 24);
assert.equal(silentShare(row), 1);
const [state, detail] = classify('gpt-5.1', row);
assert.equal(state, 'parameter-rejected');
assert.match(detail, /0 input token\(s\) and 0 output token\(s\)/);
assert.match(repairLines('gpt-5.1')[0], /max_completion_tokens/);
assert.match(repairLines('gpt-5.1')[1], /max_output_tokens/);
});
test('input read and nothing generated is a different finding', () => {
const buckets = Array.from({ length: 24 },
() => bucket('proj_api', 'gpt-5.1', 500, 900000, 0));
const [state, detail] = classify('gpt-5.1', rowFor(buckets, 'proj_api', 'gpt-5.1'));
assert.equal(state, 'generation-blocked');
assert.match(detail, /verification/);
});
test('a partial rollout is not rounded up to a total outage', () => {
const silent = Array.from({ length: 6 }, () => bucket('proj_api', 'o3-mini', 100, 0, 0));
const healthy = Array.from({ length: 18 },
() => bucket('proj_api', 'o3-mini', 100, 200000, 40000));
const row = rowFor([...silent, ...healthy], 'proj_api', 'o3-mini');
assert.equal(silentShare(row), 0.25);
const [state, detail] = classify('o3-mini', row);
assert.equal(state, 'partial-rejection');
assert.match(detail, /25%/);
});
test('the reasoning families are matched as whole prefixes', () => {
for (const model of ['o1', 'o3-mini', 'o4-mini', 'gpt-5', 'gpt-5.1-mini',
'gpt-5-2026-01-15']) {
assert.equal(isReasoningModel(model), true, model);
}
for (const model of ['gpt-4o', 'gpt-4o-mini', 'gpt-4.1', 'claude-sonnet-5', '', null]) {
assert.equal(isReasoningModel(model), false, String(model));
}
assert.match(repairLines('gpt-4o')[0], /reasoning families/);
});
test('a quiet row is not a silent one', () => {
assert.equal(silentShare({ requests: 0, silentRequests: 0 }), null);
assert.equal(silentShare(null), null);
assert.equal(classify('gpt-5.1', { requests: 4, silentRequests: 4 })[0], 'too-few-requests');
const healthy = rowFor([bucket('p', 'gpt-5.1', 500, 200000, 60000)], 'p', 'gpt-5.1');
assert.equal(classify('gpt-5.1', healthy)[0], 'generating');
});
test('a 404 on the model lookup is a different note entirely', () => {
assert.equal(modelVerdict(200)[0], 'id-resolves');
assert.equal(modelVerdict(404)[0], 'id-unreachable');
assert.match(modelVerdict(404)[1], /retirement or entitlement/);
assert.equal(modelVerdict(403)[0], 'check-refused');
assert.equal(modelVerdict(null)[0], 'unchecked');
});
test('unreadable usage fields do not become phantom requests', () => {
const row = rowFor([{ results: [{ project_id: 'p', model: 'gpt-5.1',
num_model_requests: null,
input_tokens: 'nonsense',
output_tokens: null }] }], 'p', 'gpt-5.1');
assert.equal(row.requests, 0);
assert.equal(row.silentBuckets, 0);
assert.equal(fold([]).size, 0);
assert.equal(fold(null).size, 0);
});
FAQ
Why not just read the 400 body instead of the usage report?
Because you cannot, from the API. Neither OpenAI nor Anthropic exposes an endpoint that lists individual requests with their status codes and error bodies, so the only place a fleet-wide 400 shows up in the platform's own data is as requests that consumed nothing. If your application logs the response bodies then read those first, obviously; this script exists for the case where the errors were swallowed by a retry wrapper and the logs say nothing useful.
Is this the same as max_tokens being above the model's cap?
No, and they are worth keeping apart because they read the same in an incident channel. A value above the model's ceiling is a number to lower and it can be checked in advance against the model object's own max output tokens. This one is a field refused by name, with code unsupported_parameter, and no value of it is accepted. One is arithmetic; the other is a rename.
The buckets show input tokens and no output. Same bug?
Different bug. Input tokens mean the prompt reached the model, so the request body was accepted. Generation was blocked afterwards: organization verification on a streaming path, a content filter, or an output cap of zero. The script reports that as its own state precisely so nobody spends an afternoon renaming a field that was never rejected.
Do the failed requests cost anything?
A request rejected on validation generates nothing and there is nothing to bill for, which is exactly why the row looks the way it does. The cost is elsewhere: the work is not being done, the retries are consuming rate-limit budget, and the spend against that model has quietly gone to near zero, which is the one visible symptom and the one nobody investigates.
Which parameters do reasoning models refuse, other than max_tokens?
temperature, top_p, presence_penalty, frequency_penalty and logprobs, all with unsupported_value rather than unsupported_parameter. Variance is controlled by the reasoning effort setting instead, which is why the sampling knobs are refused rather than ignored. Fix them in the same change: shipping the max_tokens rename alone means failing again the same afternoon, and nobody believes the second diagnosis.
Related field notes
- Output tokens you are billed for and never see
- The shutdown date that forces the migration in the first place
- A model id in the config that the API no longer knows
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Reasoning models — OpenAI developer docs
- Completions usage — OpenAI API reference
- Models — OpenAI API reference
- Administration and the Admin APIs — OpenAI developer docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.