Diagnostic LLM APIs
JSON cut off mid-object because the ceiling was reached
The extraction pipeline had been green for six weeks and then started dropping about one document in forty into the dead-letter queue with a JSONDecodeError. The traceback pointed at a worker three hops from any HTTP client, so the first day went on the queue and the second on the worker. What it turned out to be was a 200. The model had followed the schema exactly, the way strict mode promises, and had been cut off halfway through the fourth line item of an unusually long invoice. Everything about that request succeeded, including the bill.
Read the stored response before you read its text. With a project key set to Read Only: GET /v1/responses/{response_id}, then check status. A response with status: "incomplete" and incomplete_details.reason == "max_output_tokens" was cut off by the output ceiling; on Chat Completions the same event is finish_reason: "length".
Structured Outputs guarantees that the model follows the schema. It does not guarantee that the model finishes. When generation hits the ceiling the response is truncated exactly like any other completion, which leaves a half-written object: a string still open, a bracket still owed. That text is a valid prefix of valid JSON, and a prefix of valid JSON is not JSON.
On the Anthropic side the same finding is a whole corpus rather than one id. GET /v1/messages/batches/{batch_id}/results streams one JSON object per finished request, and every line with stop_reason: "max_tokens" is this note. Key by custom_id and never by position, because results come back in any order.
Then split the finding in two, because it has two repairs. If usage.output_tokens_details.reasoning_tokens accounts for most of the output, the ceiling was spent thinking and the answer never started; raising the cap and lowering the effort are both available. If it did not, the schema simply emits more than the cap allows.
The problem in plain words
Nothing raises. The transport layer saw a 200, the SDK saw a well-formed response object, the usage block is populated and the invoice is correct. Every retry policy in the stack is keyed on exceptions and status codes, and there is no exception and the status code is fine. The first thing that objects is a JSON parser somewhere downstream, and by then the API call is several frames away and often several services away.
What makes it so hard to date is that it is proportional to the input. A cap that fits ninety-eight documents in a hundred is invisible until the day somebody uploads a longer one, and then it is a trickle rather than an outage — a rate low enough to look like bad data, high enough to matter, and stable enough that nobody suspects a deploy. The failures also cluster on your largest customers, because their records are the long ones.
Why it happens
Following a schema and finishing an answer are different promises. Constrained decoding restricts which token can come next; it has no opinion about how many tokens there is room for. The two guarantees are independent and only one of them is on offer. A team that has just adopted strict mode is the most likely to be surprised by this, because strict mode has been working so well that the output stopped being checked at all.
The ceiling covers the thinking too. On a reasoning model the cap has to absorb tokens you never see before it absorbs the ones you do, so a cap sized for the previous model can be entirely consumed before the JSON begins. That produces an incomplete response with almost no text in it, which reads as an empty answer rather than a truncated one. The script names that case separately because the repair is different: raise the ceiling, or spend less on deliberation.
This is not the same as a cap above the model's limit. A configured max_tokens larger than the model allows is a 400 with a message, provable in advance against the model object, and no request is ever made. This is the opposite: the request was made, served and billed, and the number was too small rather than too large. They share a field name and nothing else.
It is not a rejected parameter either. A reasoning model that refuses max_tokens by name fails every call before generation and leaves requests with no tokens under them. Here the tokens are all there, right up to the cap, which is the tell: output that sits exactly on a round number is a ceiling and not a distribution.
A cut answer and a wrong answer are worth telling apart in code. Both make json.loads raise with the same class of exception. But a document that stops inside a string was interrupted, and a document that closes every bracket it opened and still fails to parse was written badly — which is a schema that was never enforced, not a ceiling. The script scans for that difference rather than reporting one parse error for both, because the two send you to different files.
The most expensive version ends in a tool call. When the ceiling lands mid-tool_use, the final block is a call whose arguments cannot be executed. In a single-turn extraction that is one lost record; in an agent loop it poisons the conversation, because the turn now contains a call with no result and every subsequent turn inherits it. The batch reader flags that case on its own line.
The fix, as a flow
Structured output guarantees the model follows the schema. It does not guarantee the model finishes. When the ceiling arrives mid-object the request has still completed successfully by every measure the transport layer has: 200, a body, a usage block, a bill. What comes back is a valid prefix of the answer you wanted, and a prefix of valid JSON is not JSON.
How to fix it
Collect the response ids you can actually read
There is no list endpoint for /v1/responses: stored responses are reachable only by an id you already hold, and only if the call was made with storage on. Export the ids from your own records into a file, one per line. If you do not store them, this is the first repair and it is worth making before anything else here is runnable.
Read status before you read text
GET /v1/responses/{response_id} and branch on status, then on incomplete_details.reason. Do this in the script and do it in the application: the rule that fixes this class of bug permanently is that no text goes to a JSON parser until the response has said it completed.
Ask where the document stops
A text that leaves a string open or a bracket owed is a cut answer. A text that balances and still fails to parse is a model that wrote the wrong thing, which is a different note with a different repair. The script scans for the difference rather than trusting the parse error, because the exception message is the same either way.
Check whether the ceiling was spent on reasoning
Compare usage.output_tokens_details.reasoning_tokens against usage.output_tokens. When the first is most of the second the visible answer never began, and the finding is reported under its own name so that nobody reshapes a schema to fix a thinking budget.
Sweep a batch results file for the same shape at scale
One stored response proves the mechanism; a batch results file measures it. Stream GET /v1/messages/batches/{batch_id}/results, count the lines with stop_reason: "max_tokens", and key everything by custom_id. Flag separately any line whose last content block is a tool_use, because those are the ones that break an agent loop rather than a record.
How to check it worked
After the ceiling is raised, re-read the same ids. The finding should disappear without the token counts changing much: the answers were nearly complete, which is why nobody noticed.
python3 openai_truncated_structured_output.py --ids response_ids.txt
# truncated-by-length resp_68c4a1 Stopped on the output ceiling mid-object: the text is a valid prefix that never closes. Output sat at 100% of the configured ceiling.
# repair: Check that the response completed before parsing anything: branch on status and on incomplete_details.reason.
# repair: This call was capped at 1024 output tokens and used 1024 of them. Raise the ceiling above the largest record the schema can emit, with room for reasoning.
# 240 response(s) checked, 6 cut shortThe full code
Two GETs, both of them reads of things that already exist: one stored response per id, and optionally one batch results file streamed a line at a time. Eight pure functions carry the judgement. json_state is the one that earns the note, because it separates a document that was interrupted from a document that was wrong and those are different bugs. Beside it: the text reader, which has to handle both API surfaces; the reason reader, which maps two vocabularies onto one word; the refusal test, which exists only to hand a response to another note; the two ratios, which return nothing rather than zero when the response does not carry the numbers; the classifier; the repair lines, which quote this call's own cap; and the batch line reader, keyed by custom_id because results arrive in any order.
"""Find stored OpenAI responses whose structured output was cut off mid-object.
Read only. GET /v1/responses/{response_id} for each id you supply, with a
project key set to Read Only, and optionally one GET against an Anthropic
Message Batches results file, which is a complete corpus of finished responses
and needs a workspace key.
There is no list endpoint for /v1/responses, so the ids have to come from your
own records: one id per line in a file. That is a limitation of the API and not
of this script.
The finding is a request that succeeded and stopped early: status "incomplete"
with an incomplete_details reason of max_output_tokens on the Responses API,
stop_reason "max_tokens" in an Anthropic batch result. The body is a valid
prefix of the answer, and a prefix of valid JSON is not JSON.
The repair is printed, never performed. Raising a ceiling or reshaping a schema
is a deploy with an owner.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_truncated_structured_output")
OPENAI_API = "https://api.openai.com/v1"
ANTHROPIC_API = "https://api.anthropic.com/v1"
ANTHROPIC_VERSION = "2023-06-01"
# The states this note owns. Everything else the classifier can return is a
# handoff to a sibling note and is reported without being counted as a finding.
FINDINGS = ("truncated-by-length", "ceiling-spent-on-reasoning", "cut-without-a-reason")
# Above this share of the output tokens, the ceiling was consumed by reasoning
# before the visible answer began. That is a different repair from a schema
# that is simply too large, so it gets its own state.
REASONING_DOMINANT = 0.6
def output_text(response):
"""Concatenate the visible text of a stored response. Pure.
Both surfaces, because a codebase that has half-migrated to the Responses
API stores both shapes and a checker that only reads one of them reports
every Chat Completions record as empty.
"""
parts = []
response = response or {}
for item in response.get("output") or []:
for content in item.get("content") or []:
if content.get("type") in ("output_text", "text"):
parts.append(str(content.get("text") or ""))
for choice in response.get("choices") or []:
text = (choice.get("message") or {}).get("content")
if isinstance(text, str):
parts.append(text)
return "".join(parts)
def json_state(text):
"""Where a JSON document stops. Pure. One of empty, parses, truncated,
malformed.
"truncated" means the text is a valid prefix that never closes: a string
still open, or a brace still owed. That is the difference between an answer
that was cut and an answer the model got wrong, and json.loads collapses
both into one exception with the same message.
"""
body = str(text or "").strip()
if not body:
return "empty"
try:
json.loads(body)
return "parses"
except ValueError:
pass
depth = 0
in_string = False
escaped = False
for ch in body:
if in_string:
if escaped:
escaped = False
elif ch == "\\":
escaped = True
elif ch == '"':
in_string = False
continue
if ch == '"':
in_string = True
elif ch in "{[":
depth += 1
elif ch in "}]":
depth -= 1
if depth < 0:
return "malformed"
if in_string or escaped or depth > 0:
return "truncated"
return "malformed"
def incomplete_reason(response):
"""Why a stored response stopped early, or None. Pure.
The Responses API says it in status plus incomplete_details.reason. Chat
Completions said it in finish_reason, and the two vocabularies are mapped
onto one here so the rest of the script has a single word to branch on.
"""
response = response or {}
if str(response.get("status") or "") == "incomplete":
details = response.get("incomplete_details") or {}
return str(details.get("reason") or "unknown")
for choice in response.get("choices") or []:
finish = str(choice.get("finish_reason") or "")
if finish == "length":
return "max_output_tokens"
if finish == "content_filter":
return "content_filter"
return None
def has_refusal(response):
"""Does this response carry a refusal rather than an answer? Pure."""
response = response or {}
for item in response.get("output") or []:
for content in item.get("content") or []:
if content.get("type") == "refusal":
return True
for choice in response.get("choices") or []:
if (choice.get("message") or {}).get("refusal"):
return True
return False
def ceiling_use(response):
"""Output tokens as a share of the configured ceiling. Pure.
None when the response does not carry a ceiling, which is a different state
from zero and must not be printed as one.
"""
response = response or {}
usage = response.get("usage") or {}
try:
cap = int(response.get("max_output_tokens"))
used = int(usage.get("output_tokens"))
except (TypeError, ValueError):
return None
if cap <= 0:
return None
return min(1.0, used / float(cap))
def reasoning_share(response):
"""Share of the output tokens that were never returned to you. Pure.
Reasoning tokens sit inside the same ceiling as the visible answer, so a
cap sized for the old model can be entirely consumed before generation of
the JSON starts. None when the response does not report them.
"""
usage = (response or {}).get("usage") or {}
details = usage.get("output_tokens_details") or {}
try:
total = int(usage.get("output_tokens"))
reasoning = int(details.get("reasoning_tokens"))
except (TypeError, ValueError):
return None
if total <= 0:
return None
return min(1.0, reasoning / float(total))
def classify(response):
"""Classify one stored response. Pure. Returns (state, detail).
Four of the states are handoffs. Two notes in this batch read the same
object and reach a different conclusion from it, and a script that folded
them in here would be telling a reader to raise a ceiling that was never
reached.
"""
response = response or {}
reason = incomplete_reason(response)
text = output_text(response)
shape = json_state(text)
used = ceiling_use(response)
at_cap = "" if used is None else " Output sat at %.0f%% of the configured ceiling." % (used * 100)
if reason == "max_output_tokens":
thinking = reasoning_share(response)
if thinking is not None and thinking >= REASONING_DOMINANT:
return ("ceiling-spent-on-reasoning",
"Stopped on the output ceiling with %.0f%% of the output "
"tokens spent on reasoning, so the visible answer barely "
"started.%s" % (thinking * 100, at_cap))
if shape == "truncated":
return ("truncated-by-length",
"Stopped on the output ceiling mid-object: the text is a "
"valid prefix that never closes.%s" % at_cap)
return ("truncated-by-length",
"Stopped on the output ceiling. The stored text is %s.%s"
% (shape, at_cap))
if reason == "content_filter":
return ("stopped-by-filter",
"Generation was halted by the content filter rather than by "
"the ceiling. That is the refusal note, not this one.")
if reason is not None:
return ("incomplete-other",
"Incomplete for reason %r, which is not an output ceiling." % reason)
if has_refusal(response):
return ("refused",
"The response completed and carries a refusal instead of an "
"answer. Nothing was cut. Read the refusal note.")
if shape == "parses":
return ("complete", "Completed and the stored text parses.")
if shape == "empty":
return ("empty-output",
"Completed with no text at all, which a ceiling reached during "
"reasoning can also produce without reporting one.")
if shape == "truncated":
return ("cut-without-a-reason",
"The text stops mid-object and the response reports no reason "
"for it. Read the raw record: a Chat Completions row stored "
"without its finish_reason looks exactly like this.")
return ("schema-not-followed",
"Completed, and the text is broken in a way truncation does not "
"explain. That is an advisory schema, not a ceiling.")
def repair_lines(state, response):
"""The repair for one state, with the numbers from this response. Pure."""
usage = (response or {}).get("usage") or {}
cap = (response or {}).get("max_output_tokens")
used = usage.get("output_tokens")
if state == "truncated-by-length":
lines = ["Check that the response completed before parsing anything: "
"branch on status and on incomplete_details.reason, and never "
"hand the text to a JSON parser until it says completed."]
if cap and used:
lines.append("This call was capped at %s output tokens and used %s "
"of them. Raise the ceiling above the largest record "
"the schema can emit, with room for reasoning."
% (cap, used))
else:
lines.append("Raise the output ceiling above the largest record "
"the schema can emit, with room for reasoning.")
lines.append("Or reshape the schema so one call emits fewer and "
"shorter fields, and paginate. A long free-text field or "
"an unbounded array inside the schema is the usual cause.")
return lines
if state == "ceiling-spent-on-reasoning":
return ["The ceiling covers reasoning tokens as well as the answer, and "
"here it was gone before the JSON began. Raise it, or lower the "
"reasoning effort for this call.",
"A structured-output call that needs no deliberation is the "
"cheapest place to spend less thinking."]
if state == "cut-without-a-reason":
return ["Store the whole response object, not just its text. Without "
"status, incomplete_details and usage there is no way to tell a "
"cut answer from a wrong one after the fact."]
if state == "stopped-by-filter":
return ["Not a ceiling. Handle the filter stop and the refusal channel "
"together, as a first-class branch before parsing."]
if state == "refused":
return ["Not a ceiling. Read the refusal text and surface it; a refusal "
"is an answer, not an error and not a truncation."]
if state == "schema-not-followed":
return ["Not a ceiling. Check whether strict was set on the schema at "
"all, because an advisory schema produces exactly this."]
return []
def batch_line_verdict(line):
"""Read one line of an Anthropic batch results file. Pure.
Returns (custom_id, state, detail). Results arrive in any order, so the
custom_id is the only safe key; position is meaningless.
"""
try:
record = json.loads(str(line or ""))
except ValueError:
return (None, "unreadable", "the line is not JSON")
if not isinstance(record, dict):
return (None, "unreadable", "the line is not an object")
custom_id = record.get("custom_id")
result = record.get("result") or {}
if str(result.get("type") or "") != "succeeded":
return (custom_id, "not-succeeded",
"result type %r, which is a different note"
% str(result.get("type") or "missing"))
message = result.get("message") or {}
stop = str(message.get("stop_reason") or "")
blocks = message.get("content") or []
last = (blocks[-1] or {}).get("type") if blocks else None
if stop == "max_tokens":
if last == "tool_use":
return (custom_id, "truncated-tool-use",
"cut on the ceiling and the final block is an incomplete "
"tool_use, so the arguments cannot be executed at all")
return (custom_id, "truncated-by-length",
"cut on the ceiling with %s output token(s)"
% ((message.get("usage") or {}).get("output_tokens", "an unknown number of")))
return (custom_id, "complete", "stop_reason %r" % (stop or "missing"))
def read_ids(path):
"""One response id per line, blanks and # comments ignored."""
ids = []
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if line and not line.startswith("#"):
ids.append(line)
return ids
def fetch_response(session, response_id):
r = session.get(OPENAI_API + "/responses/" + response_id, timeout=60)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: this needs a project key that can "
"read stored responses" % r.status_code)
r.raise_for_status()
return r.json()
def fetch_batch_results(key, batch_id):
"""Stream an Anthropic batch results file, one JSONL line at a time."""
url = ANTHROPIC_API + "/messages/batches/" + batch_id + "/results"
with requests.get(url, headers={"x-api-key": key,
"anthropic-version": ANTHROPIC_VERSION},
stream=True, timeout=300) as r:
r.raise_for_status()
for line in r.iter_lines(decode_unicode=True):
if line:
yield line
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ids", help="file of stored response ids, one per line")
ap.add_argument("--batch", help="Anthropic message batch id to read results for")
ap.add_argument("--show-all", action="store_true",
help="also print responses that completed cleanly")
args = ap.parse_args()
if not args.ids and not args.batch:
log.error("give --ids (a file of stored response ids) or --batch "
"(an Anthropic batch id), or both")
return 2
checked = 0
bad = 0
if args.ids:
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY, a project key set to Read Only")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
for response_id in read_ids(args.ids):
stored = fetch_response(session, response_id)
checked += 1
if stored is None:
log.warning("%-26s %s not found. Stored responses expire, and "
"a response created without storage was never "
"readable.", "unreadable", response_id)
continue
state, detail = classify(stored)
line = "%-26s %s %s" % (state, response_id, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
for repair in repair_lines(state, stored):
log.warning(" repair: %s", repair)
elif state in ("complete",):
if args.show_all:
log.info(line)
else:
log.info(line)
for repair in repair_lines(state, stored):
log.info(" note: %s", repair)
if args.batch:
key = os.environ.get("ANTHROPIC_API_KEY")
if not key:
log.error("set ANTHROPIC_API_KEY, a workspace key, to read batch results")
return 2
counts = {}
for line in fetch_batch_results(key, args.batch):
custom_id, state, detail = batch_line_verdict(line)
checked += 1
counts[state] = counts.get(state, 0) + 1
if state in ("truncated-by-length", "truncated-tool-use"):
bad += 1
log.warning("%-26s %s %s", state, custom_id, detail)
for state in sorted(counts):
log.info("batch %s: %d line(s) %s", args.batch, counts[state], state)
log.info("%d response(s) checked, %d cut short", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find stored OpenAI responses whose structured output was cut off mid-object.
*
* Read only. GET /v1/responses/{response_id} for each id you supply, with a
* project key set to Read Only, and optionally one GET against an Anthropic
* Message Batches results file, which needs a workspace key.
*
* There is no list endpoint for /v1/responses, so the ids come from your own
* records: one id per line in a file. The finding is a request that succeeded
* and stopped early, leaving a valid prefix of the answer behind. A prefix of
* valid JSON is not JSON. The repair is printed, never performed.
*/
import { readFile } from 'node:fs/promises';
const OPENAI_API = 'https://api.openai.com/v1';
const ANTHROPIC_API = 'https://api.anthropic.com/v1';
const ANTHROPIC_VERSION = '2023-06-01';
const FINDINGS = new Set([
'truncated-by-length', 'ceiling-spent-on-reasoning', 'cut-without-a-reason']);
const REASONING_DOMINANT = 0.6;
/** Concatenate the visible text of a stored response. Pure. Both surfaces. */
export function outputText(response) {
const parts = [];
for (const item of response?.output ?? []) {
for (const content of item?.content ?? []) {
if (content?.type === 'output_text' || content?.type === 'text') {
parts.push(String(content.text ?? ''));
}
}
}
for (const choice of response?.choices ?? []) {
const text = choice?.message?.content;
if (typeof text === 'string') parts.push(text);
}
return parts.join('');
}
/**
* Where a JSON document stops. Pure. empty | parses | truncated | malformed.
* "truncated" means a valid prefix that never closes, which is the difference
* between an answer that was cut and an answer the model got wrong.
*/
export function jsonState(text) {
const body = String(text ?? '').trim();
if (!body) return 'empty';
try {
JSON.parse(body);
return 'parses';
} catch { /* fall through to the scanner */ }
let depth = 0;
let inString = false;
let escaped = false;
for (const ch of body) {
if (inString) {
if (escaped) escaped = false;
else if (ch === '\\') escaped = true;
else if (ch === '"') inString = false;
continue;
}
if (ch === '"') inString = true;
else if (ch === '{' || ch === '[') depth += 1;
else if (ch === '}' || ch === ']') {
depth -= 1;
if (depth < 0) return 'malformed';
}
}
return (inString || escaped || depth > 0) ? 'truncated' : 'malformed';
}
/** Why a stored response stopped early, or null. Pure. Both vocabularies. */
export function incompleteReason(response) {
if (String(response?.status ?? '') === 'incomplete') {
return String(response?.incomplete_details?.reason ?? 'unknown');
}
for (const choice of response?.choices ?? []) {
const finish = String(choice?.finish_reason ?? '');
if (finish === 'length') return 'max_output_tokens';
if (finish === 'content_filter') return 'content_filter';
}
return null;
}
/** Does this response carry a refusal rather than an answer? Pure. */
export function hasRefusal(response) {
for (const item of response?.output ?? []) {
for (const content of item?.content ?? []) {
if (content?.type === 'refusal') return true;
}
}
for (const choice of response?.choices ?? []) {
if (choice?.message?.refusal) return true;
}
return false;
}
/** Output tokens as a share of the configured ceiling. Pure. Null, not zero. */
export function ceilingUse(response) {
const cap = Number(response?.max_output_tokens);
const used = Number(response?.usage?.output_tokens);
if (!Number.isFinite(cap) || !Number.isFinite(used) || cap <= 0) return null;
return Math.min(1, used / cap);
}
/** Share of the output tokens that were never returned to you. Pure. */
export function reasoningShare(response) {
const total = Number(response?.usage?.output_tokens);
const reasoning = Number(response?.usage?.output_tokens_details?.reasoning_tokens);
if (!Number.isFinite(total) || !Number.isFinite(reasoning) || total <= 0) return null;
return Math.min(1, reasoning / total);
}
/** Classify one stored response. Pure. Four of the states are handoffs. */
export function classify(response) {
const reason = incompleteReason(response);
const shape = jsonState(outputText(response));
const used = ceilingUse(response);
const atCap = used === null ? ''
: ` Output sat at ${(used * 100).toFixed(0)}% of the configured ceiling.`;
if (reason === 'max_output_tokens') {
const thinking = reasoningShare(response);
if (thinking !== null && thinking >= REASONING_DOMINANT) {
return ['ceiling-spent-on-reasoning',
`Stopped on the output ceiling with ${(thinking * 100).toFixed(0)}% of ` +
'the output tokens spent on reasoning, so the visible answer barely ' +
`started.${atCap}`];
}
if (shape === 'truncated') {
return ['truncated-by-length',
'Stopped on the output ceiling mid-object: the text is a valid prefix ' +
`that never closes.${atCap}`];
}
return ['truncated-by-length',
`Stopped on the output ceiling. The stored text is ${shape}.${atCap}`];
}
if (reason === 'content_filter') {
return ['stopped-by-filter',
'Generation was halted by the content filter rather than by the ceiling. ' +
'That is the refusal note, not this one.'];
}
if (reason !== null) {
return ['incomplete-other',
`Incomplete for reason '${reason}', which is not an output ceiling.`];
}
if (hasRefusal(response)) {
return ['refused',
'The response completed and carries a refusal instead of an answer. ' +
'Nothing was cut. Read the refusal note.'];
}
if (shape === 'parses') return ['complete', 'Completed and the stored text parses.'];
if (shape === 'empty') {
return ['empty-output',
'Completed with no text at all, which a ceiling reached during reasoning ' +
'can also produce without reporting one.'];
}
if (shape === 'truncated') {
return ['cut-without-a-reason',
'The text stops mid-object and the response reports no reason for it. ' +
'Read the raw record: a Chat Completions row stored without its ' +
'finish_reason looks exactly like this.'];
}
return ['schema-not-followed',
'Completed, and the text is broken in a way truncation does not explain. ' +
'That is an advisory schema, not a ceiling.'];
}
/** The repair for one state, with the numbers from this response. Pure. */
export function repairLines(state, response) {
const cap = response?.max_output_tokens;
const used = response?.usage?.output_tokens;
if (state === 'truncated-by-length') {
const lines = ['Check that the response completed before parsing anything: ' +
'branch on status and on incomplete_details.reason, and never hand the ' +
'text to a JSON parser until it says completed.'];
if (cap && used) {
lines.push(`This call was capped at ${cap} output tokens and used ${used} ` +
'of them. Raise the ceiling above the largest record the schema can ' +
'emit, with room for reasoning.');
} else {
lines.push('Raise the output ceiling above the largest record the schema ' +
'can emit, with room for reasoning.');
}
lines.push('Or reshape the schema so one call emits fewer and shorter ' +
'fields, and paginate. A long free-text field or an unbounded array ' +
'inside the schema is the usual cause.');
return lines;
}
if (state === 'ceiling-spent-on-reasoning') {
return ['The ceiling covers reasoning tokens as well as the answer, and here ' +
'it was gone before the JSON began. Raise it, or lower the reasoning ' +
'effort for this call.',
'A structured-output call that needs no deliberation is the cheapest place ' +
'to spend less thinking.'];
}
if (state === 'cut-without-a-reason') {
return ['Store the whole response object, not just its text. Without status, ' +
'incomplete_details and usage there is no way to tell a cut answer from a ' +
'wrong one after the fact.'];
}
if (state === 'stopped-by-filter') {
return ['Not a ceiling. Handle the filter stop and the refusal channel ' +
'together, as a first-class branch before parsing.'];
}
if (state === 'refused') {
return ['Not a ceiling. Read the refusal text and surface it; a refusal is ' +
'an answer, not an error and not a truncation.'];
}
if (state === 'schema-not-followed') {
return ['Not a ceiling. Check whether strict was set on the schema at all, ' +
'because an advisory schema produces exactly this.'];
}
return [];
}
/**
* Read one line of an Anthropic batch results file. Pure.
* Results arrive in any order, so custom_id is the only safe key.
*/
export function batchLineVerdict(line) {
let record;
try {
record = JSON.parse(String(line ?? ''));
} catch {
return [null, 'unreadable', 'the line is not JSON'];
}
if (!record || typeof record !== 'object' || Array.isArray(record)) {
return [null, 'unreadable', 'the line is not an object'];
}
const customId = record.custom_id ?? null;
const result = record.result ?? {};
if (String(result.type ?? '') !== 'succeeded') {
return [customId, 'not-succeeded',
`result type '${String(result.type ?? 'missing')}', which is a different note`];
}
const message = result.message ?? {};
const stop = String(message.stop_reason ?? '');
const blocks = message.content ?? [];
const last = blocks.length ? blocks[blocks.length - 1]?.type : null;
if (stop === 'max_tokens') {
if (last === 'tool_use') {
return [customId, 'truncated-tool-use',
'cut on the ceiling and the final block is an incomplete tool_use, so ' +
'the arguments cannot be executed at all'];
}
const tokens = message.usage?.output_tokens ?? 'an unknown number of';
return [customId, 'truncated-by-length',
`cut on the ceiling with ${tokens} output token(s)`];
}
return [customId, 'complete', `stop_reason '${stop || 'missing'}'`];
}
async function fetchResponse(key, responseId) {
const res = await fetch(`${OPENAI_API}/responses/${responseId}`,
{ headers: { Authorization: `Bearer ${key}` } });
if (res.status === 404) return null;
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: this needs a project key that ` +
'can read stored responses');
}
if (!res.ok) throw new Error(`${res.status} from /responses/${responseId}`);
return res.json();
}
async function* batchResults(key, batchId) {
const res = await fetch(`${ANTHROPIC_API}/messages/batches/${batchId}/results`,
{ headers: { 'x-api-key': key, 'anthropic-version': ANTHROPIC_VERSION } });
if (!res.ok) throw new Error(`${res.status} from the batch results file`);
const text = await res.text();
for (const line of text.split('\n')) if (line.trim()) yield line;
}
async function main() {
const idsFile = process.env.RESPONSE_IDS;
const batchId = process.env.BATCH_ID;
if (!idsFile && !batchId) {
console.error('set RESPONSE_IDS (a file of stored response ids) or BATCH_ID ' +
'(an Anthropic batch id), or both');
process.exitCode = 2;
return;
}
const showAll = process.env.SHOW_ALL === '1';
let checked = 0;
let bad = 0;
if (idsFile) {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY, a project key set to Read Only');
process.exitCode = 2;
return;
}
const ids = (await readFile(idsFile, 'utf8')).split('\n')
.map((l) => l.trim()).filter((l) => l && !l.startsWith('#'));
for (const responseId of ids) {
const stored = await fetchResponse(key, responseId);
checked += 1;
if (stored === null) {
console.warn(`${'unreadable'.padEnd(26)} ${responseId} not found. Stored ` +
'responses expire, and a response created without storage was never readable.');
continue;
}
const [state, detail] = classify(stored);
const line = `${state.padEnd(26)} ${responseId} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
for (const repair of repairLines(state, stored)) console.warn(` repair: ${repair}`);
} else if (state === 'complete') {
if (showAll) console.log(line);
} else {
console.log(line);
for (const repair of repairLines(state, stored)) console.log(` note: ${repair}`);
}
}
}
if (batchId) {
const key = process.env.ANTHROPIC_API_KEY;
if (!key) {
console.error('set ANTHROPIC_API_KEY, a workspace key, to read batch results');
process.exitCode = 2;
return;
}
const counts = new Map();
for await (const line of batchResults(key, batchId)) {
const [customId, state, detail] = batchLineVerdict(line);
checked += 1;
counts.set(state, (counts.get(state) ?? 0) + 1);
if (state === 'truncated-by-length' || state === 'truncated-tool-use') {
bad += 1;
console.warn(`${state.padEnd(26)} ${customId} ${detail}`);
}
}
for (const state of [...counts.keys()].sort()) {
console.log(`batch ${batchId}: ${counts.get(state)} line(s) ${state}`);
}
}
console.log(`${checked} response(s) checked, ${bad} cut short`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is one incomplete response holding a string that never closes: the reason field says the ceiling, the scanner says truncated, and the repair quotes the cap the call was made under. Beside it sit the three responses this note keeps having to disown — a refusal, a filter stop, and a completed response whose text balances and still will not parse — each of which has to come back as a different state pointing at a different note. The scanner is tested on a trailing comma specifically, because that closes every bracket it opened and is therefore not a truncation however much the exception looks like one.
import json
from openai_truncated_structured_output import (batch_line_verdict, ceiling_use,
classify, incomplete_reason,
json_state, output_text,
reasoning_share, repair_lines)
def stored(text, *, status="completed", reason=None, cap=None, used=None,
reasoning=None):
body = {"id": "resp_1", "status": status,
"output": [{"type": "message",
"content": [{"type": "output_text", "text": text}]}]}
if reason:
body["incomplete_details"] = {"reason": reason}
if cap is not None:
body["max_output_tokens"] = cap
if used is not None:
body["usage"] = {"output_tokens": used}
if reasoning is not None:
body["usage"]["output_tokens_details"] = {"reasoning_tokens": reasoning}
return body
def test_an_incomplete_response_holding_a_json_prefix_is_the_whole_note():
# 200, a body, a bill, and a record that stops inside a string.
half = '{"invoice_id": "INV-8817", "lines": [{"sku": "AB-1", "note": "part'
response = stored(half, status="incomplete", reason="max_output_tokens",
cap=1024, used=1024)
assert incomplete_reason(response) == "max_output_tokens"
assert json_state(half) == "truncated"
assert ceiling_use(response) == 1.0
state, detail = classify(response)
assert state == "truncated-by-length"
assert "valid prefix that never closes" in detail
repairs = repair_lines(state, response)
assert "incomplete_details.reason" in repairs[0]
assert "1024 output tokens" in repairs[1]
def test_a_ceiling_eaten_by_reasoning_gets_its_own_state():
# Same reason, same 200, and raising the cap is not the only repair on
# offer: the answer never started because the thinking used the budget.
response = stored("", status="incomplete", reason="max_output_tokens",
cap=2000, used=2000, reasoning=1900)
assert reasoning_share(response) == 0.95
state, detail = classify(response)
assert state == "ceiling-spent-on-reasoning"
assert "visible answer barely started" in detail
assert "reasoning effort" in " ".join(repair_lines(state, response))
def test_json_state_separates_a_cut_document_from_a_wrong_one():
assert json_state('{"a": 1}') == "parses"
assert json_state('{"a": [1, 2,') == "truncated"
assert json_state('{"a": "unter') == "truncated"
assert json_state('{"a": "esc\\\\') == "truncated"
# A trailing comma closes every bracket it opened, so nothing was cut:
# the model wrote invalid JSON and finished doing it.
assert json_state('{"a": 1,}') == "malformed"
assert json_state("Sorry, I cannot help with that.") == "malformed"
assert json_state(" ") == "empty"
assert json_state(None) == "empty"
def test_a_refusal_and_a_filter_stop_are_handed_to_the_other_note():
refusal = {"status": "completed",
"output": [{"type": "message",
"content": [{"type": "refusal",
"refusal": "I can't help with that."}]}]}
state, detail = classify(refusal)
assert state == "refused"
assert "Nothing was cut" in detail
filtered = stored("", status="incomplete", reason="content_filter")
assert classify(filtered)[0] == "stopped-by-filter"
assert "refusal note" in classify(filtered)[1]
def test_a_completed_response_that_still_fails_to_parse_is_not_this_note():
# Finished, and broken in a way a ceiling cannot explain. That is a schema
# that was never enforced, and it has its own note.
state, detail = classify(stored('{"total": 12,}'))
assert state == "schema-not-followed"
assert "advisory schema" in detail
assert classify(stored('{"total": 12}'))[0] == "complete"
assert classify(stored('{"total": 12,')) [0] == "cut-without-a-reason"
def test_chat_completions_rows_are_read_as_well_as_responses_rows():
legacy = {"choices": [{"finish_reason": "length",
"message": {"content": '{"rows": [{"id": 1'}}]}
assert output_text(legacy) == '{"rows": [{"id": 1'
assert incomplete_reason(legacy) == "max_output_tokens"
assert classify(legacy)[0] == "truncated-by-length"
def test_a_missing_ceiling_is_not_a_ceiling_of_zero():
assert ceiling_use(stored("{}")) is None
assert ceiling_use(stored("{}", cap=0, used=0)) is None
assert ceiling_use(None) is None
assert reasoning_share(stored("{}", cap=10, used=0)) is None
assert classify(None)[0] == "empty-output"
def test_batch_results_are_keyed_by_custom_id_and_read_line_by_line():
cut = json.dumps({"custom_id": "row-9", "result": {
"type": "succeeded",
"message": {"stop_reason": "max_tokens",
"usage": {"output_tokens": 4096},
"content": [{"type": "text", "text": '{"a": 1'}]}}})
assert batch_line_verdict(cut)[:2] == ("row-9", "truncated-by-length")
assert "4096" in batch_line_verdict(cut)[2]
tool = json.dumps({"custom_id": "row-10", "result": {
"type": "succeeded",
"message": {"stop_reason": "max_tokens",
"content": [{"type": "tool_use", "name": "charge",
"input": {}}]}}})
assert batch_line_verdict(tool)[1] == "truncated-tool-use"
assert "cannot be executed" in batch_line_verdict(tool)[2]
done = json.dumps({"custom_id": "row-11", "result": {
"type": "succeeded", "message": {"stop_reason": "end_turn",
"content": []}}})
assert batch_line_verdict(done)[1] == "complete"
errored = json.dumps({"custom_id": "row-12", "result": {"type": "errored"}})
assert batch_line_verdict(errored)[1] == "not-succeeded"
assert batch_line_verdict("{not json")[1] == "unreadable"
assert batch_line_verdict("")[1] == "unreadable"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
batchLineVerdict, ceilingUse, classify, incompleteReason, jsonState,
outputText, reasoningShare, repairLines,
} from './openai-truncated-structured-output.mjs';
const stored = (text, opts = {}) => {
const body = {
id: 'resp_1',
status: opts.status ?? 'completed',
output: [{ type: 'message', content: [{ type: 'output_text', text }] }],
};
if (opts.reason) body.incomplete_details = { reason: opts.reason };
if (opts.cap !== undefined) body.max_output_tokens = opts.cap;
if (opts.used !== undefined) {
body.usage = { output_tokens: opts.used };
if (opts.reasoning !== undefined) {
body.usage.output_tokens_details = { reasoning_tokens: opts.reasoning };
}
}
return body;
};
test('an incomplete response holding a json prefix is the whole note', () => {
const half = '{"invoice_id": "INV-8817", "lines": [{"sku": "AB-1", "note": "part';
const response = stored(half, {
status: 'incomplete', reason: 'max_output_tokens', cap: 1024, used: 1024 });
assert.equal(incompleteReason(response), 'max_output_tokens');
assert.equal(jsonState(half), 'truncated');
assert.equal(ceilingUse(response), 1);
const [state, detail] = classify(response);
assert.equal(state, 'truncated-by-length');
assert.match(detail, /valid prefix that never closes/);
const repairs = repairLines(state, response);
assert.match(repairs[0], /incomplete_details\.reason/);
assert.match(repairs[1], /1024 output tokens/);
});
test('a ceiling eaten by reasoning gets its own state', () => {
const response = stored('', {
status: 'incomplete', reason: 'max_output_tokens',
cap: 2000, used: 2000, reasoning: 1900 });
assert.equal(reasoningShare(response), 0.95);
const [state, detail] = classify(response);
assert.equal(state, 'ceiling-spent-on-reasoning');
assert.match(detail, /visible answer barely started/);
assert.match(repairLines(state, response).join(' '), /reasoning effort/);
});
test('jsonState separates a cut document from a wrong one', () => {
assert.equal(jsonState('{"a": 1}'), 'parses');
assert.equal(jsonState('{"a": [1, 2,'), 'truncated');
assert.equal(jsonState('{"a": "unter'), 'truncated');
assert.equal(jsonState('{"a": "esc\\\\'), 'truncated');
assert.equal(jsonState('{"a": 1,}'), 'malformed');
assert.equal(jsonState('Sorry, I cannot help with that.'), 'malformed');
assert.equal(jsonState(' '), 'empty');
assert.equal(jsonState(null), 'empty');
});
test('a refusal and a filter stop are handed to the other note', () => {
const refusal = { status: 'completed', output: [{ type: 'message',
content: [{ type: 'refusal', refusal: "I can't help with that." }] }] };
const [state, detail] = classify(refusal);
assert.equal(state, 'refused');
assert.match(detail, /Nothing was cut/);
const filtered = stored('', { status: 'incomplete', reason: 'content_filter' });
assert.equal(classify(filtered)[0], 'stopped-by-filter');
assert.match(classify(filtered)[1], /refusal note/);
});
test('a completed response that still fails to parse is not this note', () => {
const [state, detail] = classify(stored('{"total": 12,}'));
assert.equal(state, 'schema-not-followed');
assert.match(detail, /advisory schema/);
assert.equal(classify(stored('{"total": 12}'))[0], 'complete');
assert.equal(classify(stored('{"total": 12,'))[0], 'cut-without-a-reason');
});
test('chat completions rows are read as well as responses rows', () => {
const legacy = { choices: [{ finish_reason: 'length',
message: { content: '{"rows": [{"id": 1' } }] };
assert.equal(outputText(legacy), '{"rows": [{"id": 1');
assert.equal(incompleteReason(legacy), 'max_output_tokens');
assert.equal(classify(legacy)[0], 'truncated-by-length');
});
test('a missing ceiling is not a ceiling of zero', () => {
assert.equal(ceilingUse(stored('{}')), null);
assert.equal(ceilingUse(stored('{}', { cap: 0, used: 0 })), null);
assert.equal(ceilingUse(null), null);
assert.equal(reasoningShare(stored('{}', { cap: 10, used: 0 })), null);
assert.equal(classify(null)[0], 'empty-output');
});
test('batch results are keyed by custom_id and read line by line', () => {
const cut = JSON.stringify({ custom_id: 'row-9', result: {
type: 'succeeded',
message: { stop_reason: 'max_tokens', usage: { output_tokens: 4096 },
content: [{ type: 'text', text: '{"a": 1' }] } } });
assert.deepEqual(batchLineVerdict(cut).slice(0, 2), ['row-9', 'truncated-by-length']);
assert.match(batchLineVerdict(cut)[2], /4096/);
const tool = JSON.stringify({ custom_id: 'row-10', result: {
type: 'succeeded',
message: { stop_reason: 'max_tokens',
content: [{ type: 'tool_use', name: 'charge', input: {} }] } } });
assert.equal(batchLineVerdict(tool)[1], 'truncated-tool-use');
assert.match(batchLineVerdict(tool)[2], /cannot be executed/);
const done = JSON.stringify({ custom_id: 'row-11', result: {
type: 'succeeded', message: { stop_reason: 'end_turn', content: [] } } });
assert.equal(batchLineVerdict(done)[1], 'complete');
const errored = JSON.stringify({ custom_id: 'row-12', result: { type: 'errored' } });
assert.equal(batchLineVerdict(errored)[1], 'not-succeeded');
assert.equal(batchLineVerdict('{not json')[1], 'unreadable');
assert.equal(batchLineVerdict('')[1], 'unreadable');
});
FAQ
Why can the script not just list my responses and check them all?
Because /v1/responses has no list endpoint. Stored responses are reachable only by an id you already hold, and conversations are the same. That is a genuine gap in the API rather than an omission here, and it is why every script in this batch takes a file of ids. If you are not recording response ids, start there: it costs a column and it is the difference between being able to answer this question and not.
Is this the same as max_tokens being set above the model's cap?
No, and they are almost opposites. A cap above the model's own maximum is a 400 before anything is generated, catchable in advance against the model object, and the fix is a smaller number. This one is a 200 after everything was generated and billed, and the fix is a larger number. The only thing they share is the field.
The response is incomplete but there is no text at all. Same thing?
Usually yes, and the script says so under a different name. On a reasoning model the output ceiling covers the thinking as well as the answer, so a cap sized for an older model can be spent before the JSON starts. That comes back as ceiling-spent-on-reasoning, because raising the cap and lowering the reasoning effort are both real repairs and reshaping the schema is not.
Do I pay for a truncated response?
Fully, for everything that was generated, including reasoning tokens you never received. That is the sharpest part of this: the money is spent, the work is not done, and the record is thrown away by a parser several services downstream. A truncation rate is a waste rate, which is a better argument for fixing it than the error count is.
How do I catch this without a script, in the application?
One rule: never hand model output to a JSON parser until the response says it completed. Branch on status and incomplete_details.reason for the Responses API, on finish_reason for Chat Completions, and on stop_reason for Anthropic. Everything else here is archaeology for the responses you already stored under the old rule.
Related field notes
- Tool arguments your dispatcher cannot use
- The ceiling checked against the model's own maximum before you send
- Output tokens you are billed for and never see
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Structured Outputs — OpenAI developer docs
- Responses — OpenAI API reference
- Handling stop reasons — Claude API
- Batch processing — Claude API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.