Diagnostic LLM APIs
Model visible, streaming refused: the org is unverified
The nightly summarisation job is fine. The evaluation suite is fine. CI is fine. The chat panel in the product returns nothing at all, and has since the release that switched it to the newer model, and the only thing anyone can say about it is that it used to work. The model id is right — you checked, it resolves. The key is right — it is the same key the job uses. The difference between the route that works and the route that does not is one field in the request body, "stream": true, and the reason it fails has nothing to do with your code at all.
Compare two keys on one model, not one model against itself. With an organization admin key: GET /v1/organization/usage/completions?start_time={now-24h}&bucket_width=1h&group_by[]=model&group_by[]=api_key_id. Fold the buckets per model and look for the shape where one key shows num_model_requests above zero with no tokens either side, while another key on the same model in the same window produces output normally.
That contrast is the whole note. Requests billed with nothing generated means calls rejected on validation before they reached the model, and there are several reasons for that — but a reason that lives in the model, like a parameter the model refuses by name, refuses it for every key. A gate on the streaming path does not: the batch key buffers a whole response and is served, the UI key sets stream and is not.
Confirm the model is reachable with a Read Only project key: GET /v1/models/{model} returning 200 proves the id is valid and entitled for that key, which is what separates this from access. If it 404s instead, this is not the note — that reading belongs to the model-list diff, and the script hands it over by name rather than grading it.
Now the honest part. No endpoint on the OpenAI API reports whether your organization is verified. That state lives in Console, the 400 body that would name it is not replayable, and there is no field to read. What this script proves is that a set of requests on one key was rejected before generation while a sibling key on the same model was not. What it infers, and says it is inferring, is the most common cause of exactly that shape: verification is required for streaming and for reasoning summaries on the advanced models, and a non-streaming path is unaffected.
The repair is a link and a stopgap. Verify the organization in Console and allow up to fifteen minutes for it to propagate; meanwhile buffer the affected route with stream unset and drop reasoning.summary, which is gated the same way.
The problem in plain words
Organization verification is a gate on capabilities, not on models. The id is in your model list, GET /v1/models/{id} returns it, a plain request against it works, and the platform is in every respect willing to serve you. Then you add "stream": true and get a 400 whose message is a link to a settings page. Reasoning summaries are gated the same way, which means the two features most associated with a responsive product surface are exactly the two that are unavailable.
Because the gate is on the feature and not the credential, it splits your traffic along the least convenient line. Batch jobs buffer. Evaluations buffer. Tests buffer. The only code path that streams is the one a person is sitting in front of, and it is often the only one that was not covered when the model was rolled out. So the failure is discovered by a user, in production, on a route that never appeared in a test run.
Then it disappears into the aggregate. Neither provider exposes a request log, so there is no endpoint that will tell you which requests failed yesterday or why. All that survives of a rejected call is arithmetic: the usage report counts the request and reports no tokens against it, because nothing was generated. One such row among thousands of healthy ones is not something anybody notices, and it looks identical to several other faults that also reject before generation.
The variant worth knowing about is the 404. Some models refuse an unverified organization at the model lookup itself, with a message about verification rather than about the id. That is a different symptom with a different first move, and it is indistinguishable from a retired or unentitled id without reading the message body — which is why the script routes it elsewhere rather than guessing.
Why it happens
The row shape belongs to more than one fault, so the row alone is not a finding. Requests counted with zero tokens either side is the signature of every rejection that happens before generation. The published note on reasoning models reaching for max_tokens reads exactly this shape and reaches a completely different conclusion. What separates them is not the row, it is the comparison: a parameter a model refuses is refused on every key, and a gate on one route is not. So this script folds by api_key_id within a model and looks for disagreement between keys, which the other one has no reason to do.
When the keys agree, this note is wrong, and it says so by name. If every key on the model is mute, the fault is a property of the model or of the request body that every caller shares, and the script prints the other note rather than reporting a verification problem it has no evidence for. A diagnostic that only ever confirms itself is worse than no diagnostic.
One key using the model is not evidence of anything and must not be graded as if it were. Plenty of models are called from exactly one place. With nothing to compare against, the contrast this note depends on does not exist, so the script names the state, says what would resolve it — route a canary through a second key, or read the setting in Console — and declines to guess.
Zero output with non-zero input is a different animal and gets its own state. A request that was rejected before generation bills nothing at all. A request that consumed input and produced no output ran, and stopped, which is a content or a truncation question rather than a rejection. Folding the two together would put a real class of problem behind the wrong repair.
Verification status is not an API field, and pretending otherwise would be the whole failure of this note. There is no endpoint that returns it. The error body that names it cannot be fetched after the fact. So the output separates what was measured from what is being inferred, in those words, and the inference is offered with its most likely alternative attached rather than as a verdict.
The model lookup is one cheap call and it removes the largest competing explanation. A 200 from GET /v1/models/{id} with a project key says the id exists and is reachable for that credential, which rules out retirement and entitlement in a single request. A 404 says the opposite loudly enough that the script stops and hands the reading to the note that owns the model list.
The fix, as a flow
Neither API reports whether an organization is verified, and no endpoint replays the 400 that a streaming call received yesterday. What survives is a row in the usage report, and the row on its own is ambiguous: requests billed with no tokens either side is also what a refused parameter looks like. The separation is the comparison. One model, one hour, two keys, and only one of them producing output.
How to fix it
Use an admin-read key for usage and a project key for the lookup
OPENAI_ADMIN_KEY is an organization admin key; /v1/organization/* rejects a project key outright. OPENAI_API_KEY is a Read Only project key and is used for nothing except GET /v1/models/{id}. Both are GETs, and the script never constructs a request body.
Read 24 hours of hourly buckets grouped by model and key
group_by[]=model&group_by[]=api_key_id. The key dimension is the point: without it the report answers a question about models, and this note is a question about two keys that disagree.
Classify each key on each model
Requests above zero with no tokens either side is mute: rejected before generation. Requests with input and no output is ran and produced nothing, which is a different problem. Anything with output is producing. No requests is idle and is not evidence.
Look for disagreement, not for a threshold
One mute key beside one producing key on the same model is the finding. Every key mute is the parameter note. One key in total is unresolvable, and the script says which of the three it is instead of printing a percentage.
Confirm reachability, then print the repair and the caveat
GET /v1/models/{model}. On 200, print the verification repair with the fifteen-minute propagation note and the buffering stopgap. On 404, print the other note's name. Either way, print the sentence that verification state is not readable through any endpoint.
How to check it worked
After verifying, re-run with the same window and give it fifteen minutes plus an hour, because the buckets are hourly and the propagation is not instant. The mute key should start producing output on the same model. What should not change is the shape of the output for the models where every key was mute: those were never this note, and a run that quietly starts reporting them as verification problems is a run to distrust.
python3 openai_streaming_verification_probe.py --hours 24
# 3 model(s) with traffic in the last 24h
# verification-suspected gpt-5.6: key_9fA2 billed 1,204 request(s) with no tokens
# either side while key_3bQ7 produced 812,004 output
# token(s) on the same model in the same window
# model lookup: 200, so the id resolves for this key and access is not the fault
# measured: requests on one key were rejected before generation, on a model
# another key is generating with normally
# inferred: organization verification, which gates streaming and reasoning
# summaries. No endpoint reports verification state, so this is the
# most likely cause and not a reading
# repair: verify the organization in Console, then allow up to 15 minutes to
# propagate. One government ID verifies one organization per 90 days.
# repair: as a stopgap on the affected route only, unset stream and buffer the
# whole response, and remove reasoning summary requests.
# model-wide-mute o4-mini: all 3 key(s) with traffic are mute, so this is
# a property of the model or the body every caller sends
# repair: not this note. Read the reasoning-model parameter note: max_tokens,
# temperature and top_p are refused by name on those families.
# healthy gpt-5.1: 4 key(s) with traffic, all producing output
# 1 finding(s)The full code
One paged GET for usage, one GET per model id, and six pure functions. flatten, which walks buckets into rows and coerces every count so a missing field cannot become a string in a comparison; by_model, which sums per model and per key because the same key appears once per hourly bucket; key_state, which separates rejected before generation from ran and produced nothing rather than folding them together; contrast, which is the note itself and looks only for disagreement between keys; verdict, which lets the model lookup override the contrast and hands a 404 to another note by name; and repair_lines, which always prints what was measured next to what is being inferred.
"""Find a model that one key can list and another key cannot generate with.
Read only. Two GET endpoints: /v1/organization/usage/completions with an admin
read key, and /v1/models/{id} with a Read Only project key. No request body is
ever constructed, and nothing here sends a completion of any kind.
The subject is a contrast, not a row. Requests counted with no tokens either
side means calls rejected before generation, which is the signature of several
different faults. A fault that lives in the model refuses every key; a gate on
one route does not. So this folds usage by api_key_id inside a model and looks
for two keys that disagree.
What cannot be read is stated rather than guessed: no OpenAI endpoint reports
whether an organization is verified, and the 400 body that would name it is not
retrievable after the fact. The script separates the measurement from the
inference in its own output.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_streaming_verification_probe")
API = "https://api.openai.com/v1"
# Only one state here is this note's finding. The others are real states that
# belong to other notes, and they are printed with the other note's name rather
# than folded into a verdict this script has no evidence for.
FINDINGS = ("verification-suspected",)
MEASURED = ("requests on one key were rejected before generation, on a model "
"another key is generating with normally")
INFERRED = ("organization verification, which gates streaming and reasoning "
"summaries. No endpoint reports verification state, so this is the "
"most likely cause and not a reading")
def flatten(buckets):
"""[(model, api_key_id, requests, input_tokens, output_tokens)]. Pure.
Every count is coerced. A missing field becomes 0 rather than None, because
the whole note is a comparison between two numbers and a None propagating
into it would read as silence from a key that was merely unreported.
"""
rows = []
for bucket in buckets or []:
for entry in (bucket or {}).get("results") or []:
row = entry or {}
counts = []
for field in ("num_model_requests", "input_tokens", "output_tokens"):
try:
counts.append(int(row.get(field) or 0))
except (TypeError, ValueError):
counts.append(0)
rows.append((str(row.get("model") or "(unattributed)"),
str(row.get("api_key_id") or "(unattributed)"),
counts[0], counts[1], counts[2]))
return rows
def by_model(rows):
"""{model: {api_key_id: {requests, input, output}}}. Pure.
Summed, because a key appears once per hourly bucket and the question is
about a whole window rather than about any one hour in it.
"""
out = {}
for model, key_id, requests_n, input_n, output_n in rows or []:
slot = out.setdefault(model, {}).setdefault(
key_id, {"requests": 0, "input": 0, "output": 0})
slot["requests"] += requests_n
slot["input"] += input_n
slot["output"] += output_n
return out
def key_state(row, min_requests=1):
"""What one key did on one model. Pure. One of four words.
"mute" and "no-output" are deliberately not the same word. A request that
was rejected before generation bills nothing at all; a request that read
input and produced nothing ran and stopped, which is a truncation or
content question with an entirely different repair.
"""
row = row or {}
requests_n = int(row.get("requests") or 0)
if requests_n < max(1, int(min_requests)):
return "idle"
if int(row.get("output") or 0) > 0:
return "producing"
if int(row.get("input") or 0) > 0:
return "no-output"
return "mute"
def contrast(per_key, min_requests=1):
"""The note itself. Pure. Returns (state, detail).
Looks for disagreement between keys and nothing else. No threshold, no
ratio: one mute key beside one producing key on the same model in the same
window is the entire claim, and any other combination is somebody else's.
"""
per_key = dict(per_key or {})
states = {k: key_state(v, min_requests) for k, v in per_key.items()}
mute = sorted(k for k, s in states.items() if s == "mute")
producing = sorted(k for k, s in states.items() if s == "producing")
silent = sorted(k for k, s in states.items() if s == "no-output")
active = mute + producing + silent
if not active:
return ("no-traffic", "no key sent enough requests to grade")
if mute and producing:
first_mute = per_key[mute[0]]
first_prod = per_key[producing[0]]
return ("verification-suspected",
"%s billed %s request(s) with no tokens either side while %s "
"produced %s output token(s) on the same model in the same "
"window" % (mute[0], format(first_mute["requests"], ","),
producing[0], format(first_prod["output"], ",")))
if mute and len(active) == 1:
return ("single-key-model",
"%s is the only key with traffic on this model, so there is "
"nothing to compare it against" % mute[0])
if mute:
return ("model-wide-mute",
"all %d key(s) with traffic are mute, so this is a property of "
"the model or the body every caller sends" % len(active))
if silent and not producing:
return ("input-without-output",
"%d key(s) consumed input and produced no output, which is a "
"request that ran rather than one that was refused"
% len(silent))
return ("healthy",
"%d key(s) with traffic, all producing output" % len(producing))
def verdict(model_status, per_key, min_requests=1):
"""Combine reachability with the contrast. Pure. Returns (state, detail).
The lookup can veto. A model that does not resolve for a project key is a
question about the model list, which is another note entirely, and grading
a usage contrast on top of it would attach the wrong repair to it.
"""
state, detail = contrast(per_key, min_requests)
if model_status is None:
return (state, detail + " (the model id itself was not checked, so "
"supply a project key to rule out access)")
status = int(model_status)
if status == 404:
return ("model-not-visible",
"the id does not resolve for the project key. That is "
"retirement or entitlement rather than a gated feature, and it "
"belongs to the model-list note")
if status in (401, 403):
return (state, detail + " (the model lookup was refused, so access was "
"not confirmed either way)")
if status != 200:
return (state, detail + " (the model lookup returned %d)" % status)
return (state, detail)
def repair_lines(state):
"""The repair for one verdict. Pure. Printed, never performed.
The measured line and the inferred line are always printed together for the
finding, because the difference between them is the difference between what
this script saw and what it thinks it means.
"""
if state == "verification-suspected":
return [
"measured: " + MEASURED,
"inferred: " + INFERRED,
"verify the organization in Console, then allow up to 15 minutes "
"to propagate. One government ID verifies one organization per 90 "
"days, which matters if several organizations share an owner.",
"as a stopgap on the affected route only, unset stream and buffer "
"the whole response, and remove reasoning summary requests. Leave "
"the batch and evaluation routes alone; they are already working.",
"if the organization is already verified, the next candidate is a "
"parameter that route sends and the working key does not. Diff the "
"two request builders before changing anything in Console.",
]
if state == "model-wide-mute":
return ["not this note. Read the reasoning-model parameter note: "
"max_tokens, temperature and top_p are refused by name on "
"those families, and a refusal by name hits every key."]
if state == "single-key-model":
return ["route a canary through a second key on the same model, or "
"read the verification setting in Console. With one key there "
"is no contrast, and this script will not invent one.",
"measured: requests were rejected before generation on the "
"only key that uses this model. Nothing more than that."]
if state == "model-not-visible":
return ["check the id against GET /v1/models first. A model that does "
"not resolve is a retirement or entitlement question, and it "
"has a different repair from a gated capability."]
if state == "input-without-output":
return ["these requests reached the model and returned nothing, which "
"is truncation or a refusal rather than a rejected body. Read "
"the structured-output and refusal notes instead."]
return []
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key (sk-admin-), not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def pages(session, path, params, max_pages=40):
"""Walk the usage report, which paginates on an opaque page cursor."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params = dict(params)
params["page"] = page["next_page"]
def check_model(key, model):
"""One cheap GET to prove the id is reachable. Returns a status code."""
if not key:
return None
try:
r = requests.get(API + "/models/" + str(model),
headers={"Authorization": "Bearer " + key}, timeout=30)
except requests.RequestException:
return None
return r.status_code
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--hours", type=int, default=24,
help="hours of hourly buckets to read (default 24)")
ap.add_argument("--min-requests", type=int, default=20,
help="requests below which a key is treated as idle")
ap.add_argument("--model", action="append", default=[],
help="restrict to these model ids (repeatable)")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY (an organization admin key; read-only "
"scopes are enough)")
return 2
project_key = os.environ.get("OPENAI_API_KEY")
hours = max(1, min(int(args.hours), 168))
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
buckets = pages(session, "/organization/usage/completions", {
"start_time": int(time.time()) - hours * 3600,
"bucket_width": "1h",
"limit": hours,
"group_by": ["model", "api_key_id"],
})
grouped = by_model(flatten(buckets))
wanted = set(args.model or [])
if wanted:
grouped = {m: v for m, v in grouped.items() if m in wanted}
if not grouped:
log.info("no completions usage in the last %d hour(s)", hours)
return 0
log.info("%d model(s) with traffic in the last %dh", len(grouped), hours)
findings = 0
for model in sorted(grouped, key=lambda m: -sum(
r["requests"] for r in grouped[m].values())):
per_key = grouped[model]
preliminary, _ = contrast(per_key, args.min_requests)
status = (check_model(project_key, model)
if preliminary in ("verification-suspected",
"single-key-model", "model-wide-mute")
else None)
state, detail = verdict(status, per_key, args.min_requests)
emit = log.warning if state in FINDINGS or state != "healthy" else log.info
emit("%-23s %s: %s", state, model, detail)
if status is not None:
emit(" model lookup: %d", int(status))
for line in repair_lines(state):
emit(" repair: %s" if not line.startswith(("measured:", "inferred:"))
else " %s", line)
if state in FINDINGS:
findings += 1
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find a model that one key can list and another key cannot generate with.
*
* Read only. Two GET endpoints: the organization usage report with an admin
* read key, and /v1/models/{id} with a Read Only project key. No request body
* is ever constructed and nothing here sends a completion.
*
* The subject is a contrast, not a row. A fault that lives in the model
* refuses every key; a gate on one route does not. What cannot be read is
* stated rather than guessed: no endpoint reports verification state.
*/
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set(['verification-suspected']);
export const MEASURED =
'requests on one key were rejected before generation, on a model another key '
+ 'is generating with normally';
export const INFERRED =
'organization verification, which gates streaming and reasoning summaries. No '
+ 'endpoint reports verification state, so this is the most likely cause and '
+ 'not a reading';
const int = (v) => {
const n = Number(v ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
};
/** [[model, apiKeyId, requests, input, output]]. Pure. */
export function flatten(buckets) {
const rows = [];
for (const bucket of buckets ?? []) {
for (const entry of (bucket ?? {}).results ?? []) {
const row = entry ?? {};
rows.push([String(row.model ?? '(unattributed)'),
String(row.api_key_id ?? '(unattributed)'),
int(row.num_model_requests), int(row.input_tokens),
int(row.output_tokens)]);
}
}
return rows;
}
/** {model: {apiKeyId: {requests, input, output}}}. Pure. Summed. */
export function byModel(rows) {
const out = {};
for (const [model, keyId, requests, input, output] of rows ?? []) {
const perModel = (out[model] ??= {});
const slot = (perModel[keyId] ??= { requests: 0, input: 0, output: 0 });
slot.requests += requests;
slot.input += input;
slot.output += output;
}
return out;
}
/** What one key did on one model. Pure. One of four words. */
export function keyState(row, minRequests = 1) {
const r = row ?? {};
if (int(r.requests) < Math.max(1, int(minRequests))) return 'idle';
if (int(r.output) > 0) return 'producing';
if (int(r.input) > 0) return 'no-output';
return 'mute';
}
/** The note itself. Pure. Returns [state, detail]. */
export function contrast(perKey, minRequests = 1) {
const rows = { ...(perKey ?? {}) };
const states = Object.fromEntries(
Object.entries(rows).map(([k, v]) => [k, keyState(v, minRequests)]));
const pick = (want) => Object.keys(states).filter((k) => states[k] === want).sort();
const mute = pick('mute');
const producing = pick('producing');
const silent = pick('no-output');
const active = [...mute, ...producing, ...silent];
if (!active.length) return ['no-traffic', 'no key sent enough requests to grade'];
if (mute.length && producing.length) {
const n = (v) => v.toLocaleString('en-US');
return ['verification-suspected',
`${mute[0]} billed ${n(rows[mute[0]].requests)} request(s) with no tokens `
+ `either side while ${producing[0]} produced ${n(rows[producing[0]].output)} `
+ 'output token(s) on the same model in the same window'];
}
if (mute.length && active.length === 1) {
return ['single-key-model',
`${mute[0]} is the only key with traffic on this model, so there is `
+ 'nothing to compare it against'];
}
if (mute.length) {
return ['model-wide-mute',
`all ${active.length} key(s) with traffic are mute, so this is a property `
+ 'of the model or the body every caller sends'];
}
if (silent.length && !producing.length) {
return ['input-without-output',
`${silent.length} key(s) consumed input and produced no output, which is a `
+ 'request that ran rather than one that was refused'];
}
return ['healthy', `${producing.length} key(s) with traffic, all producing output`];
}
/** Combine reachability with the contrast. Pure. Returns [state, detail]. */
export function verdict(modelStatus, perKey, minRequests = 1) {
const [state, detail] = contrast(perKey, minRequests);
if (modelStatus === null || modelStatus === undefined) {
return [state, detail + ' (the model id itself was not checked, so supply a '
+ 'project key to rule out access)'];
}
const status = int(modelStatus);
if (status === 404) {
return ['model-not-visible',
'the id does not resolve for the project key. That is retirement or '
+ 'entitlement rather than a gated feature, and it belongs to the '
+ 'model-list note'];
}
if (status === 401 || status === 403) {
return [state, detail + ' (the model lookup was refused, so access was not '
+ 'confirmed either way)'];
}
if (status !== 200) return [state, detail + ` (the model lookup returned ${status})`];
return [state, detail];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state) {
if (state === 'verification-suspected') {
return [
'measured: ' + MEASURED,
'inferred: ' + INFERRED,
'verify the organization in Console, then allow up to 15 minutes to '
+ 'propagate. One government ID verifies one organization per 90 days, '
+ 'which matters if several organizations share an owner.',
'as a stopgap on the affected route only, unset stream and buffer the '
+ 'whole response, and remove reasoning summary requests. Leave the batch '
+ 'and evaluation routes alone; they are already working.',
'if the organization is already verified, the next candidate is a '
+ 'parameter that route sends and the working key does not. Diff the two '
+ 'request builders before changing anything in Console.',
];
}
if (state === 'model-wide-mute') {
return ['not this note. Read the reasoning-model parameter note: max_tokens, '
+ 'temperature and top_p are refused by name on those families, and a '
+ 'refusal by name hits every key.'];
}
if (state === 'single-key-model') {
return ['route a canary through a second key on the same model, or read the '
+ 'verification setting in Console. With one key there is no contrast, and '
+ 'this script will not invent one.',
'measured: requests were rejected before generation on the only key that '
+ 'uses this model. Nothing more than that.'];
}
if (state === 'model-not-visible') {
return ['check the id against GET /v1/models first. A model that does not '
+ 'resolve is a retirement or entitlement question, and it has a different '
+ 'repair from a gated capability.'];
}
if (state === 'input-without-output') {
return ['these requests reached the model and returned nothing, which is '
+ 'truncation or a refusal rather than a rejected body. Read the '
+ 'structured-output and refusal notes instead.'];
}
return [];
}
async function read(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) for (const item of v) url.searchParams.append(k, String(item));
else url.searchParams.set(k, String(v));
}
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/organization/* needs an `
+ 'organization admin key, not a project key');
}
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function* pages(key, path, params, maxPages = 40) {
let q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, path, q);
for (const bucket of page.data ?? []) yield bucket;
if (!page.has_more || !page.next_page) return;
q = { ...q, page: page.next_page };
}
}
async function checkModel(key, model) {
if (!key) return null;
try {
const r = await fetch(`${API}/models/${model}`,
{ headers: { Authorization: `Bearer ${key}` } });
return r.status;
} catch {
return null;
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key; read-only '
+ 'scopes are enough)');
process.exitCode = 2;
return;
}
const projectKey = process.env.OPENAI_API_KEY;
const hours = Math.max(1, Math.min(int(process.env.USAGE_HOURS ?? 24), 168));
const minRequests = Math.max(1, int(process.env.MIN_REQUESTS ?? 20));
const buckets = [];
for await (const bucket of pages(admin, '/organization/usage/completions', {
start_time: Math.floor(Date.now() / 1000) - hours * 3600,
bucket_width: '1h',
limit: hours,
group_by: ['model', 'api_key_id'],
})) buckets.push(bucket);
const grouped = byModel(flatten(buckets));
const models = Object.keys(grouped);
if (!models.length) {
console.log(`no completions usage in the last ${hours} hour(s)`);
return;
}
console.log(`${models.length} model(s) with traffic in the last ${hours}h`);
const total = (m) => Object.values(grouped[m]).reduce((a, r) => a + r.requests, 0);
let findings = 0;
for (const model of models.sort((a, b) => total(b) - total(a))) {
const perKey = grouped[model];
const [preliminary] = contrast(perKey, minRequests);
const status = ['verification-suspected', 'single-key-model', 'model-wide-mute']
.includes(preliminary) ? await checkModel(projectKey, model) : null;
const [state, detail] = verdict(status, perKey, minRequests);
console.log(`${state.padEnd(23)} ${model}: ${detail}`);
if (status !== null && status !== undefined) console.log(` model lookup: ${status}`);
for (const line of repairLines(state)) {
const prefix = (line.startsWith('measured:') || line.startsWith('inferred:'))
? ' ' : ' repair: ';
console.log(prefix + line);
}
if (FINDINGS.has(state)) findings += 1;
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the pairing the note is built on: one key with requests and no tokens beside one key producing output, on the same model, has to be the finding and has to name both keys. The second is the boundary that keeps this out of the published parameter note — every key mute must come back as model-wide-mute, must not be this note's finding, and must print the other note by name. The third is the case the script refuses to grade, a model used by exactly one key, where the contrast simply does not exist. Then the 404 handed to the model-list note, the separation of rejected before generation from ran and produced nothing, and last the assertion that the finding always prints what was measured next to what is only inferred.
from openai_streaming_verification_probe import (INFERRED, MEASURED, by_model,
contrast, flatten, key_state,
repair_lines, verdict)
def bucket(*results):
return {"start_time": 1_700_000_000, "results": list(results)}
def result(model, key_id, requests, input_tokens=0, output_tokens=0):
return {"model": model, "api_key_id": key_id,
"num_model_requests": requests, "input_tokens": input_tokens,
"output_tokens": output_tokens}
def test_two_keys_disagreeing_on_one_model_is_the_finding():
rows = flatten([bucket(result("gpt-5.6", "key_9fA2", 1204),
result("gpt-5.6", "key_3bQ7", 900, 400_000, 812_004))])
per_key = by_model(rows)["gpt-5.6"]
state, detail = verdict(200, per_key)
assert state == "verification-suspected"
assert "key_9fA2" in detail and "key_3bQ7" in detail
assert "1,204" in detail and "812,004" in detail
def test_every_key_mute_is_the_other_note_and_says_so():
# The boundary. A parameter a model refuses by name is refused for every
# key, so agreement between keys means this note has no evidence at all.
per_key = by_model(flatten([bucket(result("o4-mini", "key_a", 400),
result("o4-mini", "key_b", 900),
result("o4-mini", "key_c", 30))]))["o4-mini"]
state, detail = verdict(200, per_key)
assert state == "model-wide-mute"
assert "every caller sends" in detail
assert any("reasoning-model parameter note" in line
for line in repair_lines(state))
assert repair_lines(state) and state not in ("verification-suspected",)
def test_one_key_on_a_model_is_unresolvable_and_is_not_graded():
per_key = by_model(flatten([bucket(result("gpt-5.1", "key_only", 800))]))["gpt-5.1"]
state, detail = verdict(200, per_key)
assert state == "single-key-model"
assert "nothing to compare it against" in detail
lines = repair_lines(state)
assert any("canary" in line for line in lines)
assert any(line.startswith("measured:") for line in lines)
def test_a_model_that_does_not_resolve_belongs_to_the_model_list_note():
per_key = by_model(flatten([bucket(result("gpt-4-0613", "key_a", 500),
result("gpt-4-0613", "key_b", 500, 1, 9))]))["gpt-4-0613"]
state, detail = verdict(404, per_key)
assert state == "model-not-visible"
assert "model-list note" in detail
assert any("GET /v1/models" in line for line in repair_lines(state))
def test_rejected_before_generation_is_not_the_same_as_produced_nothing():
assert key_state({"requests": 100, "input": 0, "output": 0}) == "mute"
assert key_state({"requests": 100, "input": 900, "output": 0}) == "no-output"
assert key_state({"requests": 100, "input": 900, "output": 4}) == "producing"
assert key_state({"requests": 0, "input": 0, "output": 0}) == "idle"
assert key_state({"requests": 5, "input": 0, "output": 0}, 20) == "idle"
per_key = by_model(flatten([bucket(result("m", "key_a", 100, 900, 0))]))["m"]
state, _ = contrast(per_key)
assert state == "input-without-output"
assert any("truncation or a refusal" in line for line in repair_lines(state))
def test_the_finding_separates_what_was_measured_from_what_was_inferred():
lines = repair_lines("verification-suspected")
assert lines[0] == "measured: " + MEASURED
assert lines[1] == "inferred: " + INFERRED
assert "No endpoint reports verification state" in INFERRED
assert any("15 minutes" in line for line in lines)
assert any("unset stream" in line for line in lines)
assert any("already verified" in line for line in lines)
def test_counts_are_coerced_and_missing_fields_do_not_become_silence():
rows = flatten([bucket({"model": None, "api_key_id": None,
"num_model_requests": "not-a-number"})])
assert rows == [("(unattributed)", "(unattributed)", 0, 0, 0)]
assert flatten(None) == [] and by_model(None) == {}
assert contrast({})[0] == "no-traffic"
assert verdict(None, {})[1].endswith("to rule out access)")
assert repair_lines("healthy") == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { INFERRED, MEASURED, byModel, contrast, flatten, keyState, repairLines,
verdict } from './openai-streaming-verification-probe.mjs';
const bucket = (...results) => ({ start_time: 1700000000, results });
const result = (model, keyId, requests, inputTokens = 0, outputTokens = 0) =>
({ model, api_key_id: keyId, num_model_requests: requests,
input_tokens: inputTokens, output_tokens: outputTokens });
test('two keys disagreeing on one model is the finding', () => {
const rows = flatten([bucket(result('gpt-5.6', 'key_9fA2', 1204),
result('gpt-5.6', 'key_3bQ7', 900, 400000, 812004))]);
const perKey = byModel(rows)['gpt-5.6'];
const [state, detail] = verdict(200, perKey);
assert.equal(state, 'verification-suspected');
assert.ok(detail.includes('key_9fA2') && detail.includes('key_3bQ7'));
assert.ok(detail.includes('1,204') && detail.includes('812,004'));
});
test('every key mute is the other note and says so', () => {
const perKey = byModel(flatten([bucket(result('o4-mini', 'key_a', 400),
result('o4-mini', 'key_b', 900),
result('o4-mini', 'key_c', 30))]))['o4-mini'];
const [state, detail] = verdict(200, perKey);
assert.equal(state, 'model-wide-mute');
assert.ok(detail.includes('every caller sends'));
assert.ok(repairLines(state).some((l) => l.includes('reasoning-model parameter note')));
});
test('one key on a model is unresolvable and is not graded', () => {
const perKey = byModel(flatten([bucket(result('gpt-5.1', 'key_only', 800))]))['gpt-5.1'];
const [state, detail] = verdict(200, perKey);
assert.equal(state, 'single-key-model');
assert.ok(detail.includes('nothing to compare it against'));
const lines = repairLines(state);
assert.ok(lines.some((l) => l.includes('canary')));
assert.ok(lines.some((l) => l.startsWith('measured:')));
});
test('a model that does not resolve belongs to the model list note', () => {
const perKey = byModel(flatten([bucket(result('gpt-4-0613', 'key_a', 500),
result('gpt-4-0613', 'key_b', 500, 1, 9))]))['gpt-4-0613'];
const [state, detail] = verdict(404, perKey);
assert.equal(state, 'model-not-visible');
assert.ok(detail.includes('model-list note'));
assert.ok(repairLines(state).some((l) => l.includes('GET /v1/models')));
});
test('rejected before generation is not the same as produced nothing', () => {
assert.equal(keyState({ requests: 100, input: 0, output: 0 }), 'mute');
assert.equal(keyState({ requests: 100, input: 900, output: 0 }), 'no-output');
assert.equal(keyState({ requests: 100, input: 900, output: 4 }), 'producing');
assert.equal(keyState({ requests: 0, input: 0, output: 0 }), 'idle');
assert.equal(keyState({ requests: 5, input: 0, output: 0 }, 20), 'idle');
const perKey = byModel(flatten([bucket(result('m', 'key_a', 100, 900, 0))])).m;
assert.equal(contrast(perKey)[0], 'input-without-output');
assert.ok(repairLines('input-without-output').some((l) => l.includes('truncation or a refusal')));
});
test('the finding separates what was measured from what was inferred', () => {
const lines = repairLines('verification-suspected');
assert.equal(lines[0], 'measured: ' + MEASURED);
assert.equal(lines[1], 'inferred: ' + INFERRED);
assert.ok(INFERRED.includes('No endpoint reports verification state'));
assert.ok(lines.some((l) => l.includes('15 minutes')));
assert.ok(lines.some((l) => l.includes('unset stream')));
assert.ok(lines.some((l) => l.includes('already verified')));
});
test('counts are coerced and missing fields do not become silence', () => {
const rows = flatten([bucket({ model: null, api_key_id: null,
num_model_requests: 'not-a-number' })]);
assert.deepEqual(rows, [['(unattributed)', '(unattributed)', 0, 0, 0]]);
assert.deepEqual(flatten(null), []);
assert.deepEqual(byModel(null), {});
assert.equal(contrast({})[0], 'no-traffic');
assert.ok(verdict(null, {})[1].endsWith('to rule out access)'));
assert.deepEqual(repairLines('healthy'), []);
});
FAQ
Can the script actually tell me my organization is unverified?
No, and it says so in its own output. There is no endpoint on the OpenAI API that reports verification status; it is a Console setting. There is also no request log, so the 400 body that would have named it cannot be fetched after the fact. What the script proves is narrower and still useful: on this model, in this window, one key had requests billed with no tokens either side while another key produced output normally. That rules out the model, the id and the credential in one reading, and leaves verification as the most likely of a short list of causes. The output prints the measurement and the inference on separate lines for exactly that reason.
How is this different from the note about reasoning models refusing max_tokens?
They read the same row and separate on one thing. Both are looking at requests counted with zero tokens either side, which is what a rejection before generation looks like in an aggregate that has no error field. That note groups by model and project and concludes that a parameter is being refused by name; this one groups by api_key_id inside a model and looks for two keys that disagree. The logic is that a parameter a model refuses is refused for every caller, and a gate on the streaming path is not. When the keys agree, this script prints that note's name and stops.
Why does it matter that only one route is broken?
Because it decides where you look and it explains why nothing caught it. Verification gates streaming and reasoning summaries, not the model, so every code path that buffers a complete response keeps working: batch jobs, evaluations, tests, back-office scripts. The only path that streams is usually the one with a person waiting at the end of it, and it is often the least covered. That asymmetry is also what makes the two-key comparison possible, since in most organizations those routes are served by different keys.
What if the model returns 404 rather than serving the request?
Then this is not the note and the script hands it over. Some models refuse an unverified organization at the lookup itself with a message about verification, and a retired or unentitled id refuses with a message about the id. Both are a 404 and the script cannot read the message body of a request it did not make. Since the repair for a model that does not resolve starts by diffing your configured ids against the model list, that reading belongs to the retired-id note, and duplicating it here would put two different repairs behind one symptom.
The model is used by exactly one key. Can I still get an answer?
Not from this script, deliberately. The finding is a disagreement between two keys, and with one key there is nothing to disagree with. The state is named single-key-model, and the printed suggestion is to create the contrast rather than to guess: route a canary through a second key on the same model, or read the setting in Console directly. A script that graded one mute key as a verification problem would be right sometimes and would also flag every model whose only caller has a broken request body.
Related field notes
- The same usage row, when every key on the model is silent
- The model ids in your config, diffed against the model list
- When the same key works here and 403s where it is deployed
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- API organization verification — OpenAI help centre
- Usage — OpenAI API reference
- Models — OpenAI API reference
- Admin APIs — OpenAI platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.