Diagnostic LLM APIs
Tool-call arguments that parse and still break the schema
The agent had been running for a month when a run started dying in the middle of a turn, roughly once in three hundred calls, always on the same tool. The traceback was a KeyError inside the handler, four frames below anything that knew about HTTP. The arguments string had parsed without complaint. It was well-formed JSON, it was even plausible JSON, and it described a call to a function whose signature had changed in a pull request the tool schema had not followed.
Parse the arguments, and then keep going. With a project key set to Read Only: GET /v1/responses/{response_id}, take every output item with type: "function_call", and read its arguments field — which is a string containing JSON, not an object. The documentation is explicit that the string may be malformed.
The second half is the half nobody writes. The same response object carries the tools it was generated with, so the declared parameter schema is right there beside the emitted call. Validate one against the other: required parameters that are absent, keys that were never declared, values of the wrong type, enum values that are not in the enum. Every one of those parses perfectly and none of them is dispatchable.
Attribute a parse failure before you report it. If the response also stopped on the output ceiling, the argument string was cut mid-write and belongs to the truncation note; the tool schema is not the problem and tightening it will not help.
And treat an unknown tool name as its own state. A dispatcher that indexes a handler map by name raises there, before any argument is touched, which is a different bug with a different cause: a tool renamed on one side of the wire only.
The problem in plain words
Everything the API promised, it delivered. The response is a 200, the tool call is well-formed as a protocol object, and arguments contains exactly what the model produced. The contract that broke is between the model and your handler, and there is no layer in between that was ever asked to check it.
In a single-turn application that is a lost record. In an agent loop it is worse, because the turn dies with a tool call in the conversation and no result beside it. Retrying the turn replays the same broken call. Retrying the whole run costs every token spent so far. And the failure rate is low enough — a few calls in a thousand — that it reads as a transient until somebody reads a stored response carefully.
Why it happens
Arguments are a string, deliberately, and strings can be wrong. They arrive JSON-encoded rather than as a parsed object, and the docs say plainly that they may not parse. Without strict: true on the tool there is no grammar constraining generation at all: the model is writing JSON from memory, and mostly it is very good at that.
The failure this note is about survives a perfect parse. Wrapping json.loads in a try/except is the advice everyone has heard and it catches the smaller half of the problem. A missing required parameter, a string where an integer belongs, an extra key, an enum value that is close but not in the list — all of those are valid JSON. The exception they cause is raised by your dispatcher, sometimes several frames into a handler, and it is not a JSON error at all.
The schema is on the response, which makes this checkable without your source. Because the tool definitions are echoed back, the comparison can be made from stored data alone: this is the call the model made, and this is the schema it was given. No guessing about what the running deploy sent, and no need to keep a copy of the schema in the checker.
A cut argument string is a different note wearing this one's clothes. When the ceiling lands mid-write the arguments are a truncated prefix, and the parse error is downstream of a token budget rather than a schema. The script checks the response status first and says so, because tightening a tool schema to fix an output ceiling is a change that cannot possibly work.
An unknown tool name is a lookup failure, not a parse failure. It comes from a tool renamed on one side of the wire, a handler map that drifted from the tool list, or a model recalling a tool from earlier in the conversation. The arguments may be immaculate. The dispatcher still raises, and it raises earlier, which is why the state is separate.
Even a valid call deserves a note when the tool is not strict. This one matched the schema. Nothing promised the next one would. That is the advisory-schema note seen from the tool side, and the script reports it quietly rather than as a finding, because it is a risk rather than a defect.
The fix, as a flow
Function arguments come back JSON encoded, as a string, and the docs are explicit that the string may be malformed. Two quite different faults arrive through that one field. One is a string that will not parse. The other is a string that parses perfectly and describes a call your tool cannot accept, and that second one is never caught by any amount of care around json.loads.
How to fix it
Pull every function call out of the stored responses
Responses API: output items with type: "function_call", each carrying name, call_id and arguments. Chat Completions: message.tool_calls[] with the same content under function. Read both, in emission order, because a half-migrated codebase has both in its records and a turn with several calls needs each one judged separately.
Check whether the response was cut before blaming the arguments
status: "incomplete" with incomplete_details.reason == "max_output_tokens" means the argument string was truncated mid-write. Report that as its own state and send the reader to the ceiling. Doing this check second, after the parse, produces a report full of parse errors with one real cause hidden inside them.
Parse, and treat an empty string as an empty object
A tool that takes no parameters is legally called with an empty arguments string, and a bare json.loads raises on it. That single case accounts for a surprising share of the JSONDecodeErrors people attribute to model behaviour.
Validate the parsed object against the declared schema
Required parameters present, no undeclared keys where additionalProperties: false, types matching, enums respected, and the same rules applied to nested objects and array items. Report the path of each violation. This is the step that finds the calls a try/except will never see.
Print the repair as a loop change, not just a flag
Two things fix this permanently. Validate before dispatch and return the validation error to the model as the tool result, so it corrects itself instead of crashing the turn. And set strict: true on the tool, with additionalProperties: false and every parameter required, so the grammar is constrained in the first place.
How to check it worked
After validation moves in front of dispatch, re-run over a fresh window of ids. The violations should still appear in the report — the model has not changed — and none of them should be reaching a handler any more.
python3 openai_tool_call_arguments.py --ids response_ids.txt
# arguments-violate-schema resp_68f11c charge/call_1 the arguments parse cleanly and break the declared schema in 3 place(s): arguments.amount_cents: expected integer, got str; arguments.currency: 'gbp' is not one of the 2 declared value(s); arguments.idempotency_key: not declared, and the schema forbids extra keys
# repair: Validate arguments against the tool schema before dispatch, and feed the validation error back to the model as the tool result so it can correct itself.
# repair: Set strict: true on tool charge, with additionalProperties: false and every parameter listed in required.
# 512 tool call(s) checked, 7 your dispatcher cannot useThe full code
One GET per id and nothing else; the tool schemas travel with the response. Eight pure functions. function_calls and declared_tools read the two API surfaces into one shape. parse_arguments is small and opinionated: an empty string is an empty object, because a tool that takes nothing is called that way and a bare parse raises on it. schema_violations is the centre of the note — a recursive walk covering types, required keys, undeclared keys and enums, which is deliberately not a full JSON Schema implementation and is exactly the set of failures that reach a dispatcher. Then the truncation test that keeps another note's findings out of this one, the classifier whose ordering matters, and the repair lines.
"""Check every stored tool call against the tool schema declared beside it.
Read only. GET /v1/responses/{response_id} for each id you supply, with a
project key set to Read Only. There is no list endpoint for /v1/responses, so
the ids come from your own records: one id per line in a file.
Function arguments come back JSON encoded, as a string, and the documentation
is explicit that the string may be malformed. Two quite different faults arrive
through that one field:
* the string will not parse, which a careful try/except catches;
* the string parses perfectly and describes a call your handler cannot
accept, which nothing around json.loads will ever catch.
The second one is why this script exists. The response object carries the tool
definitions it was generated with, so the declared schema and the emitted call
can be compared without reading a line of your source, and the thing that
throws in production is your dispatcher rather than the API.
The repair is printed, never performed.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_tool_call_arguments")
API = "https://api.openai.com/v1"
FINDINGS = ("arguments-violate-schema", "arguments-unparseable", "unknown-tool")
# JSON Schema type names mapped onto what a parsed document can actually be.
TYPE_TESTS = {
"object": lambda v: isinstance(v, dict),
"array": lambda v: isinstance(v, list),
"string": lambda v: isinstance(v, str),
"boolean": lambda v: isinstance(v, bool),
"null": lambda v: v is None,
"number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
"integer": lambda v: isinstance(v, int) and not isinstance(v, bool),
}
def function_calls(response):
"""Every function call in a stored response. Pure.
Returns dicts with name, call_id and the raw arguments string, in the order
the model emitted them. Both surfaces: the Responses API puts a
function_call item in output[], Chat Completions puts tool_calls on the
message.
"""
calls = []
response = response or {}
for item in response.get("output") or []:
if not isinstance(item, dict) or item.get("type") != "function_call":
continue
calls.append({"name": str(item.get("name") or ""),
"call_id": str(item.get("call_id") or item.get("id") or ""),
"arguments": item.get("arguments")})
for choice in response.get("choices") or []:
for call in (choice.get("message") or {}).get("tool_calls") or []:
fn = call.get("function") or {}
calls.append({"name": str(fn.get("name") or ""),
"call_id": str(call.get("id") or ""),
"arguments": fn.get("arguments")})
return calls
def declared_tools(response):
"""The tool definitions the response was generated with. Pure.
Keyed by name, carrying the parameter schema and the strict flag. Taken
from the response rather than from your source tree, because the definition
that matters is the one that was actually sent.
"""
tools = {}
for tool in (response or {}).get("tools") or []:
if not isinstance(tool, dict):
continue
if str(tool.get("type") or "function") != "function":
continue
inner = tool.get("function") if isinstance(tool.get("function"), dict) else tool
name = str(inner.get("name") or "")
if name:
tools[name] = {"parameters": inner.get("parameters"),
"strict": inner.get("strict") is True}
return tools
def parse_arguments(text):
"""Parse one arguments string. Pure. Returns (value, error).
error is None on success. An empty string is a legal way for a model to
call a tool that takes nothing, so it parses to an empty object rather than
failing, which is a distinction a naive json.loads gets wrong on its first
day in production.
"""
if text is None:
return (None, "the arguments field is absent")
if isinstance(text, dict):
# Some SDKs hand back a parsed object. Nothing to do.
return (text, None)
body = str(text).strip()
if not body:
return ({}, None)
try:
value = json.loads(body)
except ValueError as exc:
return (None, str(exc))
if not isinstance(value, dict):
return (None, "arguments parsed to %s, not an object"
% type(value).__name__)
return (value, None)
def schema_violations(value, schema, path="arguments"):
"""Where a parsed argument object departs from its declared schema. Pure.
Deliberately small: types, required keys, unexpected keys and enums. Those
four cover the failures that actually reach a dispatcher, and a full JSON
Schema implementation in a field note would be a library nobody asked for.
"""
problems = []
if not isinstance(schema, dict) or not schema:
return problems
raw = schema.get("type")
kinds = raw if isinstance(raw, list) else ([raw] if raw else [])
kinds = [str(k) for k in kinds]
known = [k for k in kinds if k in TYPE_TESTS]
if known and not any(TYPE_TESTS[k](value) for k in known):
got = "null" if value is None else type(value).__name__
return ["%s: expected %s, got %s" % (path, " or ".join(known), got)]
choices = schema.get("enum")
if isinstance(choices, list) and choices and value not in choices:
problems.append("%s: %r is not one of the %d declared value(s)"
% (path, value, len(choices)))
if isinstance(value, dict):
properties = schema.get("properties")
properties = properties if isinstance(properties, dict) else {}
required = schema.get("required")
required = required if isinstance(required, list) else []
for name in required:
if name not in value:
problems.append("%s.%s: required and missing" % (path, name))
if schema.get("additionalProperties") is False:
for name in sorted(set(value) - set(properties)):
problems.append("%s.%s: not declared, and the schema forbids "
"extra keys" % (path, name))
for name in sorted(set(value) & set(properties)):
problems.extend(schema_violations(value[name], properties[name],
"%s.%s" % (path, name)))
if isinstance(value, list):
items = schema.get("items")
if isinstance(items, dict):
for index, entry in enumerate(value):
problems.extend(schema_violations(entry, items,
"%s[%d]" % (path, index)))
return problems
def classify(call, tools, truncated=False):
"""Classify one function call. Pure. Returns (state, detail).
The order matters. A call whose arguments were cut off belongs to the
truncation note, and saying so before reporting a parse error keeps the
reader from tuning a tool schema to fix an output ceiling.
"""
call = call or {}
name = str(call.get("name") or "")
tools = tools or {}
value, error = parse_arguments(call.get("arguments"))
if error is not None:
if truncated:
return ("arguments-truncated",
"the arguments string does not parse (%s) and the response "
"stopped on the output ceiling, so it was cut mid-write "
"rather than written wrongly" % error)
return ("arguments-unparseable",
"the arguments string does not parse (%s) and the response "
"completed, so nothing was constraining the grammar" % error)
if name not in tools:
return ("unknown-tool",
"the arguments parse cleanly and no tool named %r was declared "
"on this response. A dispatcher that indexes a handler map by "
"name raises here, not at the parse." % name)
schema = tools[name].get("parameters")
problems = schema_violations(value, schema)
if problems:
return ("arguments-violate-schema",
"the arguments parse cleanly and break the declared schema in "
"%d place(s): %s" % (len(problems), "; ".join(problems)))
if not tools[name].get("strict"):
return ("dispatchable-unconstrained",
"this call matches the schema, but the tool was declared "
"without strict: true, so nothing guaranteed that it would")
return ("dispatchable", "parses and matches the declared schema")
def repair_lines(state, name=None):
"""The repair for one state. Pure."""
if state == "arguments-violate-schema":
return ["Validate arguments against the tool schema before dispatch, "
"and feed the validation error back to the model as the tool "
"result so it can correct itself. A crashed turn teaches the "
"model nothing; a returned error usually fixes the next call.",
"Set strict: true on tool %s, with additionalProperties: false "
"and every parameter listed in required. Without it the schema "
"is a suggestion." % (name or "this tool")]
if state == "arguments-unparseable":
return ["Wrap every argument parse in try/except and return the parse "
"error to the model as the tool result rather than raising "
"through the turn.",
"Set strict: true on the tool so constrained decoding holds the "
"grammar in the first place."]
if state == "arguments-truncated":
return ["Not a schema problem. The output ceiling cut the argument "
"string mid-write, so raise it and check the response status "
"before touching any tool call."]
if state == "unknown-tool":
return ["Handle an unknown tool name explicitly: return a tool result "
"saying the tool does not exist. A KeyError out of the handler "
"map ends the turn and loses the conversation state.",
"Check that the tool list sent on this call matches the handler "
"map. A tool renamed on one side only produces exactly this."]
if state == "dispatchable-unconstrained":
return ["This call was fine. Set strict: true on the tool anyway, "
"because nothing about this response promised it would be."]
return []
def read_ids(path):
"""One response id per line, blanks and # comments ignored."""
ids = []
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if line and not line.startswith("#"):
ids.append(line)
return ids
def was_truncated(response):
"""Did this response stop on the output ceiling? Pure."""
response = response or {}
if str(response.get("status") or "") == "incomplete":
reason = (response.get("incomplete_details") or {}).get("reason")
return str(reason or "") == "max_output_tokens"
for choice in response.get("choices") or []:
if str(choice.get("finish_reason") or "") == "length":
return True
return False
def fetch_response(session, response_id):
r = session.get(API + "/responses/" + response_id, timeout=60)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: this needs a project key that can "
"read stored responses" % r.status_code)
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ids", required=True,
help="file of stored response ids, one per line")
ap.add_argument("--show-all", action="store_true",
help="also print calls that parse and validate")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY, a project key set to Read Only")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
checked = 0
bad = 0
for response_id in read_ids(args.ids):
stored = fetch_response(session, response_id)
if stored is None:
log.warning("%-27s %s not found. Stored responses expire, and a "
"response created without storage was never readable.",
"unreadable", response_id)
continue
tools = declared_tools(stored)
truncated = was_truncated(stored)
calls = function_calls(stored)
if not calls:
continue
if len(calls) > 1:
log.info("%-27s %s %d call(s) in one turn", "parallel-calls",
response_id, len(calls))
for call in calls:
checked += 1
state, detail = classify(call, tools, truncated)
line = "%-27s %s %s/%s %s" % (state, response_id, call["name"],
call["call_id"] or "-", detail)
if state in FINDINGS:
bad += 1
log.warning(line)
for repair in repair_lines(state, call["name"]):
log.warning(" repair: %s", repair)
elif state == "dispatchable":
if args.show_all:
log.info(line)
else:
log.info(line)
for repair in repair_lines(state, call["name"]):
log.info(" note: %s", repair)
log.info("%d tool call(s) checked, %d your dispatcher cannot use",
checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Check every stored tool call against the tool schema declared beside it.
*
* Read only. GET /v1/responses/{response_id} for each id you supply, with a
* project key set to Read Only. There is no list endpoint for /v1/responses,
* so the ids come from your own records: one id per line in a file.
*
* Function arguments come back JSON encoded, as a string, and the docs are
* explicit that the string may be malformed. Two different faults arrive
* through that one field: a string that will not parse, which a careful
* try/catch handles, and a string that parses perfectly and describes a call
* your handler cannot accept, which nothing around JSON.parse will catch.
*
* The response carries the tool definitions it was generated with, so the
* declared schema and the emitted call can be compared without reading a line
* of your source. The repair is printed, never performed.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set([
'arguments-violate-schema', 'arguments-unparseable', 'unknown-tool']);
/** What a parsed JSON value actually is, in schema vocabulary. Pure. */
export function typeName(value) {
if (value === null) return 'null';
if (Array.isArray(value)) return 'array';
if (typeof value === 'number') return Number.isInteger(value) ? 'integer' : 'number';
return typeof value;
}
const TYPE_TESTS = {
object: (v) => v !== null && typeof v === 'object' && !Array.isArray(v),
array: (v) => Array.isArray(v),
string: (v) => typeof v === 'string',
boolean: (v) => typeof v === 'boolean',
null: (v) => v === null,
number: (v) => typeof v === 'number' && Number.isFinite(v),
integer: (v) => typeof v === 'number' && Number.isInteger(v),
};
/** Every function call in a stored response, in emission order. Pure. */
export function functionCalls(response) {
const calls = [];
for (const item of response?.output ?? []) {
if (!item || typeof item !== 'object' || item.type !== 'function_call') continue;
calls.push({ name: String(item.name ?? ''),
callId: String(item.call_id ?? item.id ?? ''),
arguments: item.arguments });
}
for (const choice of response?.choices ?? []) {
for (const call of choice?.message?.tool_calls ?? []) {
const fn = call?.function ?? {};
calls.push({ name: String(fn.name ?? ''),
callId: String(call?.id ?? ''),
arguments: fn.arguments });
}
}
return calls;
}
/** The tool definitions the response was generated with, keyed by name. Pure. */
export function declaredTools(response) {
const tools = new Map();
for (const tool of response?.tools ?? []) {
if (!tool || typeof tool !== 'object') continue;
if (String(tool.type ?? 'function') !== 'function') continue;
const inner = (tool.function && typeof tool.function === 'object') ? tool.function : tool;
const name = String(inner.name ?? '');
if (name) tools.set(name, { parameters: inner.parameters, strict: inner.strict === true });
}
return tools;
}
/**
* Parse one arguments string. Pure. Returns [value, error].
* An empty string is a legal way to call a tool that takes nothing, so it
* parses to an empty object rather than failing.
*/
export function parseArguments(text) {
if (text === null || text === undefined) return [null, 'the arguments field is absent'];
if (typeof text === 'object' && !Array.isArray(text)) return [text, null];
const body = String(text).trim();
if (!body) return [{}, null];
let value;
try {
value = JSON.parse(body);
} catch (err) {
return [null, err.message];
}
if (value === null || typeof value !== 'object' || Array.isArray(value)) {
return [null, `arguments parsed to ${typeName(value)}, not an object`];
}
return [value, null];
}
/**
* Where a parsed argument object departs from its declared schema. Pure.
* Types, required keys, unexpected keys and enums: the four failures that
* actually reach a dispatcher.
*/
export function schemaViolations(value, schema, path = 'arguments') {
const problems = [];
if (!schema || typeof schema !== 'object' || Object.keys(schema).length === 0) {
return problems;
}
const raw = schema.type;
const kinds = (Array.isArray(raw) ? raw : (raw ? [raw] : [])).map(String);
const known = kinds.filter((k) => k in TYPE_TESTS);
if (known.length && !known.some((k) => TYPE_TESTS[k](value))) {
return [`${path}: expected ${known.join(' or ')}, got ${typeName(value)}`];
}
if (Array.isArray(schema.enum) && schema.enum.length && !schema.enum.includes(value)) {
problems.push(`${path}: ${JSON.stringify(value)} is not one of the ` +
`${schema.enum.length} declared value(s)`);
}
if (value !== null && typeof value === 'object' && !Array.isArray(value)) {
const properties = (schema.properties && typeof schema.properties === 'object')
? schema.properties : {};
for (const name of Array.isArray(schema.required) ? schema.required : []) {
if (!(name in value)) problems.push(`${path}.${name}: required and missing`);
}
if (schema.additionalProperties === false) {
for (const name of Object.keys(value).filter((k) => !(k in properties)).sort()) {
problems.push(`${path}.${name}: not declared, and the schema forbids extra keys`);
}
}
for (const name of Object.keys(value).filter((k) => k in properties).sort()) {
problems.push(...schemaViolations(value[name], properties[name], `${path}.${name}`));
}
}
if (Array.isArray(value) && schema.items && typeof schema.items === 'object') {
value.forEach((entry, index) => {
problems.push(...schemaViolations(entry, schema.items, `${path}[${index}]`));
});
}
return problems;
}
/**
* Classify one function call. Pure. Returns [state, detail].
* Order matters: a call whose arguments were cut off belongs to the truncation
* note, and saying so first keeps the reader from tuning a tool schema to fix
* an output ceiling.
*/
export function classify(call, tools, truncated = false) {
const name = String(call?.name ?? '');
const map = tools instanceof Map ? tools : new Map(Object.entries(tools ?? {}));
const [value, error] = parseArguments(call?.arguments);
if (error !== null) {
if (truncated) {
return ['arguments-truncated',
`the arguments string does not parse (${error}) and the response ` +
'stopped on the output ceiling, so it was cut mid-write rather than ' +
'written wrongly'];
}
return ['arguments-unparseable',
`the arguments string does not parse (${error}) and the response ` +
'completed, so nothing was constraining the grammar'];
}
if (!map.has(name)) {
return ['unknown-tool',
`the arguments parse cleanly and no tool named '${name}' was declared on ` +
'this response. A dispatcher that indexes a handler map by name raises ' +
'here, not at the parse.'];
}
const tool = map.get(name);
const problems = schemaViolations(value, tool.parameters);
if (problems.length) {
return ['arguments-violate-schema',
`the arguments parse cleanly and break the declared schema in ` +
`${problems.length} place(s): ${problems.join('; ')}`];
}
if (!tool.strict) {
return ['dispatchable-unconstrained',
'this call matches the schema, but the tool was declared without ' +
'strict: true, so nothing guaranteed that it would'];
}
return ['dispatchable', 'parses and matches the declared schema'];
}
/** The repair for one state. Pure. */
export function repairLines(state, name) {
if (state === 'arguments-violate-schema') {
return ['Validate arguments against the tool schema before dispatch, and feed ' +
'the validation error back to the model as the tool result so it can correct ' +
'itself. A crashed turn teaches the model nothing; a returned error usually ' +
'fixes the next call.',
`Set strict: true on tool ${name ?? 'this tool'}, with additionalProperties: ` +
'false and every parameter listed in required. Without it the schema is a ' +
'suggestion.'];
}
if (state === 'arguments-unparseable') {
return ['Wrap every argument parse in try/catch and return the parse error to ' +
'the model as the tool result rather than raising through the turn.',
'Set strict: true on the tool so constrained decoding holds the grammar in ' +
'the first place.'];
}
if (state === 'arguments-truncated') {
return ['Not a schema problem. The output ceiling cut the argument string ' +
'mid-write, so raise it and check the response status before touching any ' +
'tool call.'];
}
if (state === 'unknown-tool') {
return ['Handle an unknown tool name explicitly: return a tool result saying ' +
'the tool does not exist. A thrown lookup error out of the handler map ends ' +
'the turn and loses the conversation state.',
'Check that the tool list sent on this call matches the handler map. A tool ' +
'renamed on one side only produces exactly this.'];
}
if (state === 'dispatchable-unconstrained') {
return ['This call was fine. Set strict: true on the tool anyway, because ' +
'nothing about this response promised it would be.'];
}
return [];
}
/** Did this response stop on the output ceiling? Pure. */
export function wasTruncated(response) {
if (String(response?.status ?? '') === 'incomplete') {
return String(response?.incomplete_details?.reason ?? '') === 'max_output_tokens';
}
for (const choice of response?.choices ?? []) {
if (String(choice?.finish_reason ?? '') === 'length') return true;
}
return false;
}
async function fetchResponse(key, responseId) {
const res = await fetch(`${API}/responses/${responseId}`,
{ headers: { Authorization: `Bearer ${key}` } });
if (res.status === 404) return null;
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: this needs a project key that ` +
'can read stored responses');
}
if (!res.ok) throw new Error(`${res.status} from /responses/${responseId}`);
return res.json();
}
async function main() {
const key = process.env.OPENAI_API_KEY;
const idsFile = process.env.RESPONSE_IDS;
if (!key || !idsFile) {
console.error('set OPENAI_API_KEY (a project key set to Read Only) and ' +
'RESPONSE_IDS (a file of stored response ids, one per line)');
process.exitCode = 2;
return;
}
const showAll = process.env.SHOW_ALL === '1';
const ids = (await readFile(idsFile, 'utf8')).split('\n')
.map((l) => l.trim()).filter((l) => l && !l.startsWith('#'));
let checked = 0;
let bad = 0;
for (const responseId of ids) {
const stored = await fetchResponse(key, responseId);
if (stored === null) {
console.warn(`${'unreadable'.padEnd(27)} ${responseId} not found. Stored ` +
'responses expire, and a response created without storage was never readable.');
continue;
}
const tools = declaredTools(stored);
const truncated = wasTruncated(stored);
const calls = functionCalls(stored);
if (!calls.length) continue;
if (calls.length > 1) {
console.log(`${'parallel-calls'.padEnd(27)} ${responseId} ${calls.length} ` +
'call(s) in one turn');
}
for (const call of calls) {
checked += 1;
const [state, detail] = classify(call, tools, truncated);
const line = `${state.padEnd(27)} ${responseId} ${call.name}/` +
`${call.callId || '-'} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
for (const repair of repairLines(state, call.name)) console.warn(` repair: ${repair}`);
} else if (state === 'dispatchable') {
if (showAll) console.log(line);
} else {
console.log(line);
for (const repair of repairLines(state, call.name)) console.log(` note: ${repair}`);
}
}
}
console.log(`${checked} tool call(s) checked, ${bad} your dispatcher cannot use`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first test is the one that justifies the note: a call whose arguments parse without complaint and break the schema in three separate ways at once, each reported with its path. Nothing built around a try/except sees any of it. Then a missing required parameter, then the two attributions that keep this note honest — a truncated argument string handed to the ceiling note, and an unknown tool name named as a lookup failure rather than a parse failure. The last three cover a valid call, a valid call on a tool that never asked for strict, and the empty argument string that a bare parse raises on.
from openai_tool_call_arguments import (classify, declared_tools,
function_calls, parse_arguments,
repair_lines, schema_violations,
was_truncated)
CHARGE = {
"type": "object",
"additionalProperties": False,
"required": ["account_id", "amount_cents", "currency"],
"properties": {
"account_id": {"type": "string"},
"amount_cents": {"type": "integer"},
"currency": {"type": "string", "enum": ["usd", "eur"]},
},
}
def response(arguments, *, name="charge", strict=True, status="completed"):
return {"id": "resp_t", "status": status,
"tools": [{"type": "function", "name": "charge",
"parameters": CHARGE, "strict": strict}],
"output": [{"type": "function_call", "name": name,
"call_id": "call_1", "arguments": arguments}]}
def test_arguments_that_parse_and_still_break_the_contract():
# The centre of the note. json.loads is perfectly happy; the handler is
# not, and no amount of care around the parse would have caught it.
stored = response('{"account_id": "acct_9", "amount_cents": "1200", '
'"currency": "gbp", "idempotency_key": "k1"}')
call = function_calls(stored)[0]
value, error = parse_arguments(call["arguments"])
assert error is None and isinstance(value, dict)
state, detail = classify(call, declared_tools(stored))
assert state == "arguments-violate-schema"
assert "amount_cents: expected integer, got str" in detail
assert "currency: 'gbp' is not one of the 2 declared value(s)" in detail
assert "idempotency_key: not declared" in detail
assert "feed the validation error back to the model" in repair_lines(state)[0]
def test_a_missing_required_argument_is_found_before_the_handler_is_called():
stored = response('{"account_id": "acct_9", "currency": "usd"}')
state, detail = classify(function_calls(stored)[0], declared_tools(stored))
assert state == "arguments-violate-schema"
assert "arguments.amount_cents: required and missing" in detail
def test_a_cut_argument_string_belongs_to_the_truncation_note():
stored = response('{"account_id": "acct_9", "amount_cent',
status="incomplete")
stored["incomplete_details"] = {"reason": "max_output_tokens"}
assert was_truncated(stored) is True
state, detail = classify(function_calls(stored)[0], declared_tools(stored),
was_truncated(stored))
assert state == "arguments-truncated"
assert "cut mid-write rather than written wrongly" in detail
assert "Not a schema problem" in repair_lines(state)[0]
def test_a_broken_string_on_a_completed_response_is_the_models_own_work():
stored = response('{{"account_id": "acct_9"}}')
assert was_truncated(stored) is False
state, detail = classify(function_calls(stored)[0], declared_tools(stored))
assert state == "arguments-unparseable"
assert "nothing was constraining the grammar" in detail
def test_an_unknown_tool_name_is_a_lookup_error_not_a_parse_error():
stored = response('{"account_id": "acct_9"}', name="charge_v2")
state, detail = classify(function_calls(stored)[0], declared_tools(stored))
assert state == "unknown-tool"
assert "indexes a handler map by name raises here" in detail
assert "renamed on one side only" in repair_lines(state)[1]
def test_a_valid_call_is_dispatchable_and_an_unstrict_one_is_flagged_anyway():
good = '{"account_id": "acct_9", "amount_cents": 1200, "currency": "usd"}'
stored = response(good)
assert classify(function_calls(stored)[0], declared_tools(stored))[0] == "dispatchable"
loose = response(good, strict=False)
state, detail = classify(function_calls(loose)[0], declared_tools(loose))
assert state == "dispatchable-unconstrained"
assert "nothing guaranteed that it would" in detail
def test_the_chat_completions_shape_and_the_empty_argument_string():
legacy = {"choices": [{"finish_reason": "tool_calls", "message": {
"tool_calls": [{"id": "call_9", "type": "function",
"function": {"name": "ping", "arguments": ""}}]}}],
"tools": [{"type": "function",
"function": {"name": "ping", "strict": True,
"parameters": {"type": "object",
"additionalProperties": False,
"properties": {},
"required": []}}}]}
call = function_calls(legacy)[0]
assert call["name"] == "ping" and call["call_id"] == "call_9"
# A tool that takes nothing is legally called with an empty string, and a
# bare json.loads raises on it.
assert parse_arguments("") == ({}, None)
assert classify(call, declared_tools(legacy))[0] == "dispatchable"
def test_the_walker_and_the_readers_survive_junk():
assert parse_arguments(None)[1] == "the arguments field is absent"
assert parse_arguments("[1, 2]")[1] == "arguments parsed to list, not an object"
assert schema_violations({"a": 1}, None) == []
assert schema_violations({"a": 1}, {}) == []
assert schema_violations(True, {"type": "integer"}) == [
"arguments: expected integer, got bool"]
assert schema_violations({"rows": [{"sku": 1}]}, {
"type": "object", "properties": {"rows": {
"type": "array", "items": {"type": "object",
"properties": {"sku": {"type": "string"}}}}}}) == [
"arguments.rows[0].sku: expected string, got int"]
assert function_calls(None) == []
assert declared_tools(None) == {}
assert was_truncated(None) is False
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
classify, declaredTools, functionCalls, parseArguments, repairLines,
schemaViolations, wasTruncated,
} from './openai-tool-call-arguments.mjs';
const CHARGE = {
type: 'object',
additionalProperties: false,
required: ['account_id', 'amount_cents', 'currency'],
properties: {
account_id: { type: 'string' },
amount_cents: { type: 'integer' },
currency: { type: 'string', enum: ['usd', 'eur'] },
},
};
const response = (args, { name = 'charge', strict = true,
status = 'completed' } = {}) => ({
id: 'resp_t', status,
tools: [{ type: 'function', name: 'charge', parameters: CHARGE, strict }],
output: [{ type: 'function_call', name, call_id: 'call_1', arguments: args }],
});
test('arguments that parse and still break the contract', () => {
const stored = response('{"account_id": "acct_9", "amount_cents": "1200", ' +
'"currency": "gbp", "idempotency_key": "k1"}');
const call = functionCalls(stored)[0];
const [value, error] = parseArguments(call.arguments);
assert.equal(error, null);
assert.equal(typeof value, 'object');
const [state, detail] = classify(call, declaredTools(stored));
assert.equal(state, 'arguments-violate-schema');
assert.match(detail, /amount_cents: expected integer, got string/);
assert.match(detail, /currency: "gbp" is not one of the 2 declared value\(s\)/);
assert.match(detail, /idempotency_key: not declared/);
assert.match(repairLines(state)[0], /feed the validation error back to the model/);
});
test('a missing required argument is found before the handler is called', () => {
const stored = response('{"account_id": "acct_9", "currency": "usd"}');
const [state, detail] = classify(functionCalls(stored)[0], declaredTools(stored));
assert.equal(state, 'arguments-violate-schema');
assert.match(detail, /arguments\.amount_cents: required and missing/);
});
test('a cut argument string belongs to the truncation note', () => {
const stored = response('{"account_id": "acct_9", "amount_cent',
{ status: 'incomplete' });
stored.incomplete_details = { reason: 'max_output_tokens' };
assert.equal(wasTruncated(stored), true);
const [state, detail] = classify(functionCalls(stored)[0],
declaredTools(stored), wasTruncated(stored));
assert.equal(state, 'arguments-truncated');
assert.match(detail, /cut mid-write rather than written wrongly/);
assert.match(repairLines(state)[0], /Not a schema problem/);
});
test('a broken string on a completed response is the model own work', () => {
const stored = response('{{"account_id": "acct_9"}}');
assert.equal(wasTruncated(stored), false);
const [state, detail] = classify(functionCalls(stored)[0], declaredTools(stored));
assert.equal(state, 'arguments-unparseable');
assert.match(detail, /nothing was constraining the grammar/);
});
test('an unknown tool name is a lookup error not a parse error', () => {
const stored = response('{"account_id": "acct_9"}', { name: 'charge_v2' });
const [state, detail] = classify(functionCalls(stored)[0], declaredTools(stored));
assert.equal(state, 'unknown-tool');
assert.match(detail, /indexes a handler map by name raises here/);
assert.match(repairLines(state)[1], /renamed on one side only/);
});
test('a valid call is dispatchable and an unstrict one is flagged anyway', () => {
const good = '{"account_id": "acct_9", "amount_cents": 1200, "currency": "usd"}';
const stored = response(good);
assert.equal(classify(functionCalls(stored)[0], declaredTools(stored))[0],
'dispatchable');
const loose = response(good, { strict: false });
const [state, detail] = classify(functionCalls(loose)[0], declaredTools(loose));
assert.equal(state, 'dispatchable-unconstrained');
assert.match(detail, /nothing guaranteed that it would/);
});
test('the chat completions shape and the empty argument string', () => {
const legacy = {
choices: [{ finish_reason: 'tool_calls', message: { tool_calls: [
{ id: 'call_9', type: 'function',
function: { name: 'ping', arguments: '' } }] } }],
tools: [{ type: 'function', function: { name: 'ping', strict: true,
parameters: { type: 'object', additionalProperties: false,
properties: {}, required: [] } } }],
};
const call = functionCalls(legacy)[0];
assert.equal(call.name, 'ping');
assert.equal(call.callId, 'call_9');
assert.deepEqual(parseArguments(''), [{}, null]);
assert.equal(classify(call, declaredTools(legacy))[0], 'dispatchable');
});
test('the walker and the readers survive junk', () => {
assert.equal(parseArguments(null)[1], 'the arguments field is absent');
assert.equal(parseArguments('[1, 2]')[1], 'arguments parsed to array, not an object');
assert.deepEqual(schemaViolations({ a: 1 }, null), []);
assert.deepEqual(schemaViolations({ a: 1 }, {}), []);
assert.deepEqual(schemaViolations(true, { type: 'integer' }),
['arguments: expected integer, got boolean']);
assert.deepEqual(schemaViolations({ rows: [{ sku: 1 }] }, {
type: 'object',
properties: { rows: { type: 'array', items: { type: 'object',
properties: { sku: { type: 'string' } } } } },
}), ['arguments.rows[0].sku: expected string, got integer']);
assert.deepEqual(functionCalls(null), []);
assert.equal(declaredTools(null).size, 0);
assert.equal(wasTruncated(null), false);
});
FAQ
I already wrap the parse in try/except. Is that not enough?
It covers the half where the string is broken and misses the half where it is not. A missing required parameter, a wrong type, an undeclared key and an out-of-range enum value are all valid JSON, so the parse succeeds and the failure moves into your handler. That is where the KeyErrors and TypeErrors in agent loops come from, and no amount of care around json.loads reaches them.
Does strict: true make this go away?
It removes most of it and not all of it. Constrained decoding holds the grammar and the declared shape, which kills the unparseable strings and the wrong types. It does not help when the response is truncated mid-argument, and it does not help when the tool list and the handler map have drifted apart. Set it anyway; keep the validation step.
Why validate against the schema on the response instead of my own copy?
Because the schema on the response is the one that was actually sent. A wrapper library, a feature flag, or an older service can send a definition that differs from the constant you are reading in the repository. Checking the emitted call against the definition it was generated under is the only comparison that cannot be wrong about which version ran.
What should the handler do when validation fails?
Return the validation error to the model as the tool result. Models correct themselves well when told what was wrong with a call, and the turn survives. Raising through the turn loses the conversation state, and retrying the turn replays the same broken call, so the expensive failure mode is the one that looks like the careful one.
The arguments were cut off halfway. Is that this note?
Only as an attribution. When the response also reports stopping on the output ceiling, the argument string was truncated mid-write and the repair is a bigger ceiling, not a tighter schema. The script checks that before it reports a parse error, because a report that blames the schema for a token budget sends people to change the wrong file.
Related field notes
- A 200 whose JSON stops mid-object because the ceiling arrived
- A schema that was only ever a suggestion
- The ceiling checked against the model's own maximum before you send
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Function calling — OpenAI developer docs
- Responses — OpenAI API reference
- Structured Outputs — OpenAI developer docs
- Handling stop reasons — Claude API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.