Diagnostic LLM APIs
reasoning tokens are billed as output but never returned
Somebody changed one model constant. The answers coming back are the same length as before, the prompts are the same length as before, the request count is flat — and the line for that model on the cost report went up by a factor of four. The tokens you are paying for were generated, billed at the output rate, and then not returned to you.
Read GET /v1/organization/usage/completions?start_time={T-30d}&bucket_width=1d&group_by[]=model with an admin key and compute output_tokens / num_model_requests per model per day. A step change in that ratio, with input_tokens / num_model_requests flat across the same boundary, is reasoning tokens and nothing else. If both moved, your prompts grew; if requests moved and the ratios did not, you simply sent more traffic.
The tokens themselves are never in a response body, so no read call will ever show them to you directly. What you can see is their weight in the aggregate.
The problem in plain words
Reasoning tokens are generated, priced at the output rate, and consume the context window, but they are not returned in the API response. The visible answer is the tip; the bill is the iceberg. A team costing a migration by measuring the length of the text they actually receive will underestimate by the entire reasoning fraction, which on a deliberation-heavy task is most of it.
Because nothing errors and nothing looks different, the discovery is almost always the invoice. And by then the change that caused it is weeks back in the history — a model constant bumped, an effort raised from low to high, or a move to a model where deliberation is on by default and omitting the parameter no longer means off.
Why it happens
The number is real and it is invisible. usage.output_tokens in a response includes the reasoning tokens; usage.output_tokens_details.reasoning_tokens tells you how many of them there were. If your metrics count the characters you received instead of reading that block, your own dashboards disagree with the invoice by design.
One flag can multiply the bill. Reasoning effort is a parameter, and the higher settings do more model work for the same request. No error, no warning, no change to the shape of the response. A single line in a config file is enough.
Prompts and parameters are not readable. Nothing in either API returns what you sent, so a script cannot look at your reasoning setting and tell you it is too high. It can only look at what the setting cost. That is why the detection here is a ratio over aggregate buckets rather than a configuration check.
Anthropic's usage report has no request count. GET /v1/organizations/usage_report/messages returns token sums per bucket and nothing else, so a per-request ratio is not computable there. The fallback is output tokens per input token, which is a weaker signal because it also moves when prompts change — and the script should say which of the two it used rather than quietly presenting them as the same measurement.
The fix, as a flow
The script compares output tokens per request against input tokens per request across a boundary you choose, because a total tells you the bill moved and a ratio tells you which of three ordinary things moved it.
How to fix it
Pull daily buckets grouped by model
Admin key. GET /v1/organization/usage/completions?start_time={T-30d}&bucket_width=1d&group_by[]=model&group_by[]=project_id. Group by model or the step change disappears into an average: a cheap model absorbing most of the traffic will hide a fourfold jump on the expensive one completely.
Compute the ratios per bucket, not per month
output_tokens / num_model_requests and input_tokens / num_model_requests. Buckets with zero requests have no ratio and must be dropped rather than treated as zero; averaging a zero in is how a quiet weekend becomes a fictional improvement.
Compare a recent window against a prior one
Split the series in two — the last seven days against the seven before, or whatever bracket a deploy fell in — and compare the means. What you want is a factor, not a p-value: output per request up by half again or more, with input per request within twenty percent of where it was.
Rule out the two innocent explanations first
If input per request rose by a similar factor, the prompts grew and reasoning is not the story. If the request count rose and both ratios held, you sent more traffic and the unit economics are unchanged. Both are ordinary and both look like a cost spike on a chart with no denominator.
Print the repair; do not send it
Lower reasoning effort where the task does not need deliberation, and drop the higher modes unless an eval justifies the premium. Then log usage.output_tokens_details.reasoning_tokens per call, so the invisible half is visible in your own metrics next time instead of on an invoice. Cross-check the money with GET /v1/organization/costs?group_by[]=line_item.
How to check it worked
Re-run a week after the effort change. The output-per-request ratio should return toward its prior level with input per request unmoved.
python3 openai_reasoning_token_audit.py --days 30 --window 7
# gpt-5.6 steady 1,180 output/request, 940 input/request
# 2 model(s) over 30 day(s), 0 finding(s)The full code
Two GETs against /v1/organization/*, so OPENAI_ADMIN_KEY has to be an organization admin key. All three judgement calls are pure functions: the summing, the split of the series into a prior and a recent window against a clock you pass in, and the verdict that decides which of the four explanations for a cost jump the numbers actually support.
"""Find the cost jump that is reasoning tokens rather than traffic or prompts.
Read only. Two GET requests and nothing else: OPENAI_ADMIN_KEY must be an
organization admin key (sk-admin-...) with read scopes, because /v1/organization
endpoints reject project keys. The repair is printed, never performed, because
this script holds a credential that can spend money on inference.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_reasoning_token_audit")
API = "https://api.openai.com/v1"
def totals(buckets):
"""Sum a list of usage buckets into one row. Pure.
OpenAI's completions usage carries num_model_requests; Anthropic's messages
usage report does not carry any request count at all, so a caller working
against that side gets requests == 0 here and the verdict falls back to a
weaker ratio rather than dividing by nothing.
"""
row = {"requests": 0, "input": 0, "output": 0, "buckets": 0}
for b in buckets:
row["buckets"] += 1
for r in b.get("results", []) or []:
row["requests"] += int(r.get("num_model_requests") or 0)
row["input"] += int(r.get("input_tokens")
or r.get("uncached_input_tokens") or 0)
row["output"] += int(r.get("output_tokens") or 0)
return row
def split(buckets, now, window_days=7):
"""Cut a daily series into (prior, recent) around a boundary. Pure, clock
passed in, so the boundary in a test is a date you can read rather than a
function of when the suite happened to run.
Buckets older than twice the window are dropped: comparing last week against
a quarter ago answers a different question than the one being asked.
"""
edge = now.timestamp() - window_days * 86400
floor = now.timestamp() - 2 * window_days * 86400
prior, recent = [], []
for b in buckets:
start = b.get("start_time")
if not isinstance(start, (int, float)):
continue
if start >= edge:
recent.append(b)
elif start >= floor:
prior.append(b)
return prior, recent
def verdict(prior, recent, jump=1.5, flat=0.2):
"""Say which of the four explanations for a cost jump the numbers support.
Pure. `jump` is the factor that counts as a step change; `flat` is how far
the other ratio may move and still be called unchanged.
Returns (state, detail).
"""
a, b = totals(prior), totals(recent)
if not b["requests"] and not b["output"]:
return ("no-data", "no usage in the recent window")
if b["requests"] and not b["output"]:
return ("failing-before-generation",
"%d request(s) in the recent window generated zero output "
"tokens. Those calls were rejected before the model ran; that "
"is an error shape and not a reasoning one." % b["requests"])
if not a["requests"] or not b["requests"]:
# Anthropic's usage report has no request count, so this is the honest
# fallback rather than a per-request claim that cannot be made.
if a["input"] and b["input"]:
before = a["output"] / a["input"]
after = b["output"] / b["input"]
if before and after / before >= jump:
return ("unmeasurable-but-rising",
"no request count in these buckets, so this is output "
"per input token, not per request: %.2f to %.2f. "
"Consistent with reasoning, but prompt shrinkage looks "
"identical." % (before, after))
return ("unmeasurable",
"no request count in these buckets. Output per input token "
"is %.2f against %.2f before, which is the strongest claim "
"available without a request count." % (after, before))
return ("unmeasurable",
"no request count and no input tokens to fall back on")
in_before = a["input"] / a["requests"]
in_after = b["input"] / b["requests"]
out_before = a["output"] / a["requests"]
out_after = b["output"] / b["requests"]
numbers = ("%.0f to %.0f output tokens per request, %.0f to %.0f input"
% (out_before, out_after, in_before, in_after))
out_factor = (out_after / out_before) if out_before else 0.0
in_factor = (in_after / in_before) if in_before else 0.0
if out_factor >= jump and abs(in_factor - 1.0) <= flat:
return ("reasoning-tax",
"%s. Output per request rose %.1fx while input per request held "
"steady. Those tokens were generated and billed at the output "
"rate and never returned to you." % (numbers, out_factor))
if out_factor >= jump and in_factor >= jump:
return ("longer-prompts",
"%s. Both ratios rose together, so the prompts grew. Raising "
"reasoning effort does not move the input side."
% numbers)
if b["requests"] >= a["requests"] * jump:
return ("volume-only",
"%s. Requests rose from %d to %d with the ratios unchanged: the "
"bill grew because traffic grew, and unit economics did not "
"move." % (numbers, a["requests"], b["requests"]))
return ("steady", numbers)
def get(session, path, params):
r = session.get(API + path, params=params, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization endpoints need an "
"organization admin key, not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def usage_by_model(session, since, days):
"""Read daily completion usage grouped by model, following next_page."""
params = [("start_time", int(since.timestamp())), ("bucket_width", "1d"),
("limit", max(days, 1)), ("group_by[]", "model")]
out = {}
while True:
page = get(session, "/organization/usage/completions", params)
for b in page.get("data", []):
for r in b.get("results", []) or []:
model = r.get("model") or "unspecified"
out.setdefault(model, []).append(
{"start_time": b.get("start_time"), "results": [r]})
if not page.get("has_more") or not page.get("next_page"):
break
params = [p for p in params if p[0] != "page"] + [("page", page["next_page"])]
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30,
help="how far back to read daily usage buckets")
ap.add_argument("--window", type=int, default=7,
help="days in the recent window, compared against the days before it")
ap.add_argument("--jump", type=float, default=1.5,
help="factor that counts as a step change")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY (an organization admin key with read "
"scopes; project keys are rejected by /v1/organization/*)")
return 2
now = dt.datetime.now(dt.timezone.utc)
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + admin})
since = now - dt.timedelta(days=args.days)
by_model = usage_by_model(s, since, args.days)
if not by_model:
log.info("no completion usage in the last %d day(s)", args.days)
return 0
bad = 0
for model, buckets in sorted(by_model.items()):
prior, recent = split(buckets, now, args.window)
state, detail = verdict(prior, recent, args.jump)
line = "%-22s %-26s %s" % (model, state, detail)
if state in ("steady", "volume-only", "no-data", "unmeasurable"):
log.info(line)
continue
bad += 1
log.warning(line)
if state in ("reasoning-tax", "unmeasurable-but-rising"):
log.warning(" repair: lower the reasoning effort on this model for "
"tasks that do not need deliberation, and drop the "
"higher modes unless an eval justifies them. Log "
"usage.output_tokens_details.reasoning_tokens per call "
"so the invisible half shows up in your own metrics.")
log.warning(" cross-check the money: GET %s/organization/costs"
"?start_time=%d&bucket_width=1d&group_by[]=line_item",
API, int(since.timestamp()))
log.info("%d model(s) over %d day(s), %d finding(s)",
len(by_model), args.days, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find the cost jump that is reasoning tokens rather than traffic or prompts.
*
* Read only. Two GET requests and nothing else: OPENAI_ADMIN_KEY must be an
* organization admin key with read scopes, because /v1/organization endpoints
* reject project keys. The repair is printed, never performed.
*/
const API = 'https://api.openai.com/v1';
/**
* Sum a list of usage buckets into one row. Pure. Anthropic's messages usage
* report carries no request count, so requests comes back 0 there and the
* verdict falls back to a weaker ratio rather than dividing by nothing.
*/
export function totals(buckets) {
const row = { requests: 0, input: 0, output: 0, buckets: 0 };
for (const b of buckets) {
row.buckets += 1;
for (const r of b.results ?? []) {
row.requests += Number(r.num_model_requests ?? 0);
row.input += Number(r.input_tokens ?? r.uncached_input_tokens ?? 0);
row.output += Number(r.output_tokens ?? 0);
}
}
return row;
}
/**
* Cut a daily series into [prior, recent] around a boundary. Pure, clock passed
* in, so a test's boundary is a date you can read. Buckets older than twice the
* window are dropped: last week against a quarter ago is a different question.
*/
export function split(buckets, now, windowDays = 7) {
const edge = now.getTime() / 1000 - windowDays * 86400;
const floor = now.getTime() / 1000 - 2 * windowDays * 86400;
const prior = [];
const recent = [];
for (const b of buckets) {
if (typeof b.start_time !== 'number') continue;
if (b.start_time >= edge) recent.push(b);
else if (b.start_time >= floor) prior.push(b);
}
return [prior, recent];
}
/**
* Say which of the four explanations for a cost jump the numbers support. Pure.
* Returns [state, detail].
*/
export function verdict(prior, recent, jump = 1.5, flat = 0.2) {
const a = totals(prior);
const b = totals(recent);
if (!b.requests && !b.output) return ['no-data', 'no usage in the recent window'];
if (b.requests && !b.output) {
return ['failing-before-generation',
`${b.requests} request(s) in the recent window generated zero output ` +
'tokens. Those calls were rejected before the model ran; that is an error ' +
'shape and not a reasoning one.'];
}
if (!a.requests || !b.requests) {
if (a.input && b.input) {
const before = a.output / a.input;
const after = b.output / b.input;
if (before && after / before >= jump) {
return ['unmeasurable-but-rising',
'no request count in these buckets, so this is output per input token, ' +
`not per request: ${before.toFixed(2)} to ${after.toFixed(2)}. ` +
'Consistent with reasoning, but prompt shrinkage looks identical.'];
}
return ['unmeasurable',
`no request count in these buckets. Output per input token is ` +
`${after.toFixed(2)} against ${before.toFixed(2)} before, which is the ` +
'strongest claim available without a request count.'];
}
return ['unmeasurable', 'no request count and no input tokens to fall back on'];
}
const inBefore = a.input / a.requests;
const inAfter = b.input / b.requests;
const outBefore = a.output / a.requests;
const outAfter = b.output / b.requests;
const numbers = `${outBefore.toFixed(0)} to ${outAfter.toFixed(0)} output ` +
`tokens per request, ${inBefore.toFixed(0)} to ${inAfter.toFixed(0)} input`;
const outFactor = outBefore ? outAfter / outBefore : 0;
const inFactor = inBefore ? inAfter / inBefore : 0;
if (outFactor >= jump && Math.abs(inFactor - 1) <= flat) {
return ['reasoning-tax',
`${numbers}. Output per request rose ${outFactor.toFixed(1)}x while input ` +
'per request held steady. Those tokens were generated and billed at the ' +
'output rate and never returned to you.'];
}
if (outFactor >= jump && inFactor >= jump) {
return ['longer-prompts',
`${numbers}. Both ratios rose together, so the prompts grew. Raising ` +
'reasoning effort does not move the input side.'];
}
if (b.requests >= a.requests * jump) {
return ['volume-only',
`${numbers}. Requests rose from ${a.requests} to ${b.requests} with the ` +
'ratios unchanged: the bill grew because traffic grew, and unit economics ' +
'did not move.'];
}
return ['steady', numbers];
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of params) url.searchParams.append(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization endpoints need ` +
'an organization admin key, not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
export async function usageByModel(key, since, days) {
let params = [['start_time', String(since)], ['bucket_width', '1d'],
['limit', String(Math.max(days, 1))], ['group_by[]', 'model']];
const out = new Map();
for (;;) {
const page = await get(key, '/organization/usage/completions', params);
for (const b of page.data ?? []) {
for (const r of b.results ?? []) {
const model = r.model ?? 'unspecified';
if (!out.has(model)) out.set(model, []);
out.get(model).push({ start_time: b.start_time, results: [r] });
}
}
if (!page.has_more || !page.next_page) break;
params = params.filter((p) => p[0] !== 'page').concat([['page', page.next_page]]);
}
return out;
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key with read ' +
'scopes; project keys are rejected by /v1/organization/*)');
process.exitCode = 2;
return;
}
const argv = process.argv;
const days = Number(argv.includes('--days') ? argv[argv.indexOf('--days') + 1] : 30) || 30;
const win = Number(argv.includes('--window') ? argv[argv.indexOf('--window') + 1] : 7) || 7;
const now = new Date();
const since = Math.floor(now.getTime() / 1000 - days * 86400);
const byModel = await usageByModel(admin, since, days);
if (byModel.size === 0) {
console.log(`no completion usage in the last ${days} day(s)`);
return;
}
let bad = 0;
for (const [model, buckets] of [...byModel.entries()].sort()) {
const [prior, recent] = split(buckets, now, win);
const [state, detail] = verdict(prior, recent);
const line = `${model.padEnd(22)} ${state.padEnd(26)} ${detail}`;
if (['steady', 'volume-only', 'no-data', 'unmeasurable'].includes(state)) {
console.log(line);
continue;
}
bad += 1;
console.warn(line);
if (state === 'reasoning-tax' || state === 'unmeasurable-but-rising') {
console.warn(' repair: lower the reasoning effort on this model for tasks ' +
'that do not need deliberation, and drop the higher modes ' +
'unless an eval justifies them. Log ' +
'usage.output_tokens_details.reasoning_tokens per call so the ' +
'invisible half shows up in your own metrics.');
console.warn(` cross-check the money: GET ${API}/organization/costs` +
`?start_time=${since}&bucket_width=1d&group_by[]=line_item`);
}
}
console.log(`${byModel.size} model(s) over ${days} day(s), ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The interesting tests are the ones that refuse to cry wolf. Output per request tripling with input per request tripling is longer prompts, not reasoning, and has to come back as a different state. Traffic doubling at constant ratios is not a finding at all. And a series with no request count — which is every Anthropic bucket — must degrade to an explicitly weaker claim rather than dividing by zero or pretending it measured something it did not.
import datetime as dt
from openai_reasoning_token_audit import split, totals, verdict
NOW = dt.datetime(2026, 8, 30, 0, 0, tzinfo=dt.timezone.utc)
def days_ago(d):
return int(NOW.timestamp() - d * 86400)
def day(d, requests=100, inp=90000, out=100000, model="gpt-5.6"):
return {"start_time": days_ago(d),
"results": [{"model": model, "num_model_requests": requests,
"input_tokens": inp, "output_tokens": out}]}
def anthropic_day(d, inp=90000, out=100000):
"""No num_model_requests: that field does not exist on Anthropic's report."""
return {"start_time": days_ago(d),
"results": [{"uncached_input_tokens": inp, "output_tokens": out}]}
def test_totals_sums_and_tolerates_a_missing_request_count():
assert totals([day(1), day(2)]) == {"requests": 200, "input": 180000,
"output": 200000, "buckets": 2}
assert totals([anthropic_day(1)])["requests"] == 0
def test_split_cuts_the_series_at_the_clock_it_is_given():
prior, recent = split([day(1), day(3), day(9), day(30)], NOW, 7)
assert [b["start_time"] for b in recent] == [days_ago(1), days_ago(3)]
assert [b["start_time"] for b in prior] == [days_ago(9)]
# 30 days back is outside twice the window and is dropped, not compared.
def test_the_finding_output_per_request_rises_while_input_holds():
prior = [day(9, requests=100, inp=90000, out=100000)]
recent = [day(1, requests=100, inp=91000, out=400000)]
state, detail = verdict(prior, recent)
assert state == "reasoning-tax"
assert "4.0x" in detail
assert "never returned" in detail
def test_prompts_growing_is_not_the_same_finding():
prior = [day(9, requests=100, inp=90000, out=100000)]
recent = [day(1, requests=100, inp=360000, out=400000)]
assert verdict(prior, recent)[0] == "longer-prompts"
def test_more_traffic_at_the_same_ratios_is_not_a_finding_at_all():
prior = [day(9, requests=100, inp=90000, out=100000)]
recent = [day(1, requests=400, inp=360000, out=400000)]
state, detail = verdict(prior, recent)
assert state == "volume-only"
assert "unit economics" in detail
def test_flat_ratios_and_flat_traffic_are_steady():
prior = [day(9, requests=100, inp=90000, out=100000)]
recent = [day(1, requests=110, inp=99000, out=110000)]
assert verdict(prior, recent)[0] == "steady"
def test_no_request_count_degrades_to_a_weaker_claim_and_says_so():
prior = [anthropic_day(9, inp=90000, out=100000)]
recent = [anthropic_day(1, inp=90000, out=400000)]
state, detail = verdict(prior, recent)
assert state == "unmeasurable-but-rising"
assert "per input token, not per request" in detail
assert verdict([anthropic_day(9)], [anthropic_day(1)])[0] == "unmeasurable"
def test_requests_with_no_output_is_an_error_shape_not_a_reasoning_one():
prior = [day(9)]
recent = [day(1, requests=50, inp=45000, out=0)]
state, _ = verdict(prior, recent)
assert state == "failing-before-generation"
def test_an_empty_recent_window_claims_nothing():
assert verdict([day(9)], [])[0] == "no-data"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { split, totals, verdict } from './openai-reasoning-token-audit.mjs';
const NOW = new Date('2026-08-30T00:00:00Z');
const daysAgo = (d) => Math.floor(NOW.getTime() / 1000 - d * 86400);
const day = (d, requests = 100, inp = 90000, out = 100000, model = 'gpt-5.6') => ({
start_time: daysAgo(d),
results: [{ model, num_model_requests: requests, input_tokens: inp, output_tokens: out }],
});
// No num_model_requests: that field does not exist on Anthropic's report.
const anthropicDay = (d, inp = 90000, out = 100000) => ({
start_time: daysAgo(d),
results: [{ uncached_input_tokens: inp, output_tokens: out }],
});
test('totals sums and tolerates a missing request count', () => {
assert.deepEqual(totals([day(1), day(2)]),
{ requests: 200, input: 180000, output: 200000, buckets: 2 });
assert.equal(totals([anthropicDay(1)]).requests, 0);
});
test('split cuts the series at the clock it is given', () => {
const [prior, recent] = split([day(1), day(3), day(9), day(30)], NOW, 7);
assert.deepEqual(recent.map((b) => b.start_time), [daysAgo(1), daysAgo(3)]);
assert.deepEqual(prior.map((b) => b.start_time), [daysAgo(9)]);
});
test('the finding: output per request rises while input holds', () => {
const [state, detail] = verdict([day(9, 100, 90000, 100000)],
[day(1, 100, 91000, 400000)]);
assert.equal(state, 'reasoning-tax');
assert.match(detail, /4\.0x/);
assert.match(detail, /never returned/);
});
test('prompts growing is not the same finding', () => {
assert.equal(verdict([day(9, 100, 90000, 100000)],
[day(1, 100, 360000, 400000)])[0], 'longer-prompts');
});
test('more traffic at the same ratios is not a finding at all', () => {
const [state, detail] = verdict([day(9, 100, 90000, 100000)],
[day(1, 400, 360000, 400000)]);
assert.equal(state, 'volume-only');
assert.match(detail, /unit economics/);
});
test('flat ratios and flat traffic are steady', () => {
assert.equal(verdict([day(9, 100, 90000, 100000)],
[day(1, 110, 99000, 110000)])[0], 'steady');
});
test('no request count degrades to a weaker claim and says so', () => {
const [state, detail] = verdict([anthropicDay(9, 90000, 100000)],
[anthropicDay(1, 90000, 400000)]);
assert.equal(state, 'unmeasurable-but-rising');
assert.match(detail, /per input token, not per request/);
assert.equal(verdict([anthropicDay(9)], [anthropicDay(1)])[0], 'unmeasurable');
});
test('requests with no output is an error shape, not a reasoning one', () => {
assert.equal(verdict([day(9)], [day(1, 50, 45000, 0)])[0],
'failing-before-generation');
});
test('an empty recent window claims nothing', () => {
assert.equal(verdict([day(9)], [])[0], 'no-data');
});
FAQ
Can I see reasoning tokens in a response?
You can see how many there were, not what they said. usage.output_tokens_details.reasoning_tokens carries the count, and usage.output_tokens already includes them. The content is never returned. If your own cost metric counts the characters you received, it will disagree with the invoice by exactly that number.
Why does the script compare ratios instead of totals?
Because a total tells you the bill went up and nothing about why. Output tokens per request, against input tokens per request, separates the three ordinary explanations: more traffic moves the request count, longer prompts move the input ratio, and reasoning moves the output ratio on its own.
Do reasoning tokens use my context window?
Yes. They are generated, they occupy the window, and they are billed at the output rate. That is also why raising effort can start producing truncated answers on requests that used to fit comfortably: the deliberation is competing with the response for the same budget.
Can a read-only script tell me what reasoning effort I have configured?
No. Neither API returns what you sent, so prompts, parameters and client configuration are invisible to any read call. The script can only measure what the setting cost, which is why the finding is a step change in a ratio rather than a configuration warning.
Does this work against Anthropic?
Partly, and the script says so. The Claude messages usage report has no request-count field, so no per-request ratio can be computed there. The fallback is output tokens per input token, which also moves when prompts change, so it is reported as a weaker claim rather than presented as the same measurement.
Related field notes
- Output tokens are what the bill is made of
- No spend limit means no ceiling
- A 429 that is a wall, not a throttle
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Reasoning models — OpenAI API
- Usage — OpenAI API reference
- Administration APIs — OpenAI API
- Extended thinking — Claude Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.