Diagnostic LLM APIs
Parallel tool calls void the strict schema guarantee
The schemas are strict, every one of them, because somebody read the Structured Outputs page properly and did the work. The parser has no try/except around it, deliberately: the arguments are guaranteed to conform, so a failure there should be loud. It has been loud four times in three months, always on a Tuesday afternoon, always with a stack trace that says a required field was missing, and always unreproducible from the same prompt. The four turns have one thing in common that nobody looked at: each of them called two tools instead of one.
Read stored responses one turn at a time and look for fan-out. With a project key: GET /v1/responses/{response_id}. Flag any response whose output[] contains more than one item of type: "function_call" while the request it echoes has parallel_tool_calls true or absent and any tool declaring strict: true.
Structured Outputs is not supported together with parallel function calls. The documented guidance is to set parallel_tool_calls: false when you are relying on strict schemas. The field defaults to true, so the guarantee degrades silently, and it degrades exactly when the model decides to fan out rather than when anything changed in your code.
That is why it is deterministic in tests and flaky in production. A test sends a prompt that provokes one call. Real traffic sends whatever the user typed, and some fraction of it provokes two. The script reports that fraction as a rate, because a documented interaction becomes a priority only once it has a number attached.
Flag repeated calls to the same tool in one turn separately. That is a second fault with the same trigger: a handler written for one call per turn double-applies. Key every handler on call_id.
The problem in plain words
Every part of the configuration is correct in isolation, which is why this survives review. strict: true is right. The schemas are right. parallel_tool_calls was never set, because nobody sets it, and its default is the one that lets the model fan out. The interaction between the two is documented and it is one sentence in the middle of a guide about something else.
What comes back on a fan-out turn is an HTTP 200 with several function_call items whose arguments strings are no longer constrained by the schema you declared. Usually they are fine anyway — the model is good at this — which is worse than if they were always broken, because the parser that trusts the guarantee runs unprotected for months and then meets one argument object with a missing required field. The trace points at the parser. The cause is a boolean nobody wrote.
Why it happens
The default is the unsafe half of the pair. parallel_tool_calls defaults to true. Strict schemas are opt-in and take deliberate work; the setting that voids them is on by default and takes none. A codebase that did the hard part correctly and skipped the easy part is the normal case here, not a careless one.
A single call under the same configuration is not safe, it is lucky. The script reports those turns as at risk rather than as clean, because the configuration is loaded and simply did not fire. Counting them as passes is how a sample of a thousand responses with twelve fan-outs gets read as "99% fine". The right reading is that the guarantee is void on 1.2% of turns and the parser has no handling for any of them.
Fan-out without strict schemas is a different note and a real problem anyway. If no tool declares strict at all then there was never a guarantee to void, and the fault is that the arguments were never validated in the first place — a separate failure with its own repair. The script names that state rather than folding it in, because telling someone to set parallel_tool_calls: false when they never had strict schemas fixes nothing.
Duplicate calls to one tool are the second bug and they cost money rather than correctness. A turn that calls create_ticket twice will create two tickets, because the handler was written when a turn meant a call. It is not a schema problem and turning off parallel calls does fix it, but so does keying the handler on call_id, which is the change that survives someone turning parallel calls back on for latency next year.
You cannot check this from the aggregate, and you cannot enumerate the sample either. No usage report counts tool calls, so there is no shape in the buckets to find. And /v1/responses has no list endpoint, so the ids have to come from your own log. Every rate this script prints is a rate over the sample you handed it, which makes the sampling strategy part of the finding: sample the turns your users actually send, not the ones your fixtures do.
The fix, as a flow
This one is a documented interaction rather than a bug, which is exactly why it survives review: every part of the configuration is correct on its own. The guarantee holds for one call in a turn and stops holding for two, and the default is the setting that lets the model choose. A test suite sends one call and never sees it.
How to fix it
Sample stored responses from real traffic, not from tests
The whole finding is a rate over turns that fan out, and fixtures do not fan out. Take ids from production logs across a full week. Responses must have been stored to be readable at all.
Read the request configuration back off each response
The response object echoes tools, tool_choice and parallel_tool_calls. Collect the tools declaring strict: true, handling both shapes — top level on the Responses API, nested under function on Chat Completions — and treat an absent parallel_tool_calls as true, because that is what it is.
Count the function_call items in one turn
More than one function_call item in a single output[] array is a fan-out. That, plus strict declared, plus parallel allowed, is the finding. One call under the same configuration is at risk and gets its own state.
Compute the rate, not just the list
Fan-outs divided by turns that were at risk. That number is what makes this actionable: 0.4% and 22% get the same repair and deserve very different urgency, and neither is visible from a list of four incident ids.
Print the boolean first and the idempotency second
parallel_tool_calls: false restores the guarantee. If you need the fan-out for latency, drop strict and validate the arguments yourself rather than believing a promise that is not being kept. Either way, key every handler on call_id so a duplicate call cannot double-apply.
How to check it worked
Re-run on a fresh week after the flag ships. The at-risk count should go to zero and the serialised count should replace it; any remaining fan-out means one client or one code path was missed.
python3 openai_parallel_strict_calls.py --responses ids.txt
# strict-void resp_0f21a 3 function_call item(s) in one turn with strict declared and parallel_tool_calls left on
# calls: lookup_order, create_ticket, create_ticket
# duplicate: create_ticket called 2 time(s) in one turn; handlers keyed on the tool name will double apply
# repair: set parallel_tool_calls false whenever strict schemas matter.
# exposure: 12 of 1000 at-risk turn(s) fanned out (1.2%), covering 27 argument object(s) with no guarantee
# 1000 response(s) read, 12 finding(s)The full code
One GET per response id, and the unit of analysis is a single turn rather than a corpus — which is what keeps this apart from the coverage note that reads the same endpoint. Nine pure functions: the id parser and path guard; the name reader for both tool shapes; the strict-tool set; the parallel-calls reader, which has to treat an absent field as true; the call extractor, which keeps call_id because the repair depends on it; the duplicate detector; the classifier; the exposure rate over at-risk turns; and the count of argument objects that came back with no guarantee behind them.
"""Find OpenAI turns where parallel tool calls voided a strict schema.
Read only. One GET per stored response id, using a project key. No completion
is created and nothing is written; /v1/responses is read, never posted to.
Structured Outputs is not supported alongside parallel function calls, and
parallel_tool_calls defaults to true. So a turn that returns more than one
function_call item while any tool declares strict: true came back without the
guarantee the parser is relying on, and it did so with an HTTP 200.
The repair is printed, never performed. One boolean is still a deploy.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_parallel_strict_calls")
API = "https://api.openai.com/v1"
CALL_TYPES = ("function_call", "custom_tool_call")
FINDINGS = ("strict-void",)
def _int(value):
"""Read a count as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def parse_ids(text):
"""Response ids out of a plain text file. Pure. Order kept, duplicates dropped.
Also the guard that stops an arbitrary line of a file becoming a URL path
segment: anything that is not a plausible response id is discarded rather
than interpolated into a provider URL.
"""
out = []
seen = set()
for line in str(text or "").splitlines():
candidate = line.split("#", 1)[0].strip()
if not candidate or not candidate.startswith("resp_"):
continue
if not all(ch.isalnum() or ch in "_-" for ch in candidate):
continue
if candidate in seen:
continue
seen.add(candidate)
out.append(candidate)
return out
def tool_name(tool):
"""The function name out of either tool shape. Pure. None when absent."""
if not isinstance(tool, dict):
return None
name = tool.get("name")
if not name and isinstance(tool.get("function"), dict):
name = tool["function"].get("name")
name = str(name or "").strip()
return name or None
def declared_names(response):
"""Every named tool the request declared. Pure. Sorted."""
out = set()
for tool in (response or {}).get("tools") or []:
name = tool_name(tool)
if name:
out.add(name)
return sorted(out)
def strict_tools(response):
"""Tools declaring strict: true, in either shape. Pure. Sorted.
strict false and strict absent are the same thing here and neither counts.
A note about a voided guarantee has to be certain the guarantee was claimed.
"""
out = set()
for tool in (response or {}).get("tools") or []:
if not isinstance(tool, dict):
continue
strict = tool.get("strict")
if strict is not True and isinstance(tool.get("function"), dict):
strict = tool["function"].get("strict")
if strict is not True:
continue
name = tool_name(tool)
if name:
out.add(name)
return sorted(out)
def parallel_allowed(response):
"""Could the model return more than one tool call in this turn? Pure.
An absent parallel_tool_calls is true, and reading it as false is the exact
mistake that makes this whole class of failure invisible.
"""
value = (response or {}).get("parallel_tool_calls")
return value is not False
def function_calls(response):
"""The tool calls in one turn, in order. Pure.
call_id is kept because half the repair depends on it: a handler keyed on
call_id cannot double-apply when the same tool is called twice.
"""
out = []
for item in (response or {}).get("output") or []:
if not isinstance(item, dict) or item.get("type") not in CALL_TYPES:
continue
name = str(item.get("name") or "").strip()
if not name:
continue
out.append({"name": name, "call_id": str(item.get("call_id") or "")})
return out
def duplicate_names(calls):
"""Tool names called more than once in one turn. Pure.
A separate fault with the same trigger. It costs side effects rather than
correctness, and turning parallel calls off is not the only fix for it.
"""
counts = {}
for call in calls or []:
name = str((call or {}).get("name") or "")
if name:
counts[name] = counts.get(name, 0) + 1
return {name: n for name, n in counts.items() if n > 1}
def classify(response):
"""Classify one turn. Pure. Returns (state, detail).
The unit is the turn and not the corpus, because the guarantee is voided or
kept per response and a rate computed over anything else means nothing.
"""
declared = declared_names(response)
if not declared:
return ("no-tools", "no named tools declared in this turn")
strict = strict_tools(response)
calls = function_calls(response)
parallel = parallel_allowed(response)
names = ", ".join(c["name"] for c in calls) or "none"
if not strict:
if len(calls) > 1:
return ("fanout-no-strict",
"%d function_call item(s) in one turn (%s) and no tool "
"declares strict. There was no guarantee to void here: the "
"arguments were never validated by the API at all, which "
"is a different fault with a different repair."
% (len(calls), names))
return ("no-strict-declared",
"%d tool(s) declared, none of them strict. Nothing in this turn "
"was schema-guaranteed." % len(declared))
if not parallel:
return ("strict-serialised",
"strict declared on %d tool(s) and parallel_tool_calls is "
"false. The guarantee holds." % len(strict))
if len(calls) > 1:
return ("strict-void",
"%d function_call item(s) in one turn with strict declared and "
"parallel_tool_calls left on (%s). Structured Outputs is not "
"supported alongside parallel calls, so these argument objects "
"carry no schema guarantee." % (len(calls), names))
return ("strict-at-risk",
"strict declared on %d tool(s) with parallel_tool_calls left on, "
"and this turn happened to return %d call(s). The configuration is "
"loaded; it did not fire here." % (len(strict), len(calls)))
def exposure(states):
"""How often the fan-out that voids the guarantee actually happens. Pure.
The denominator is turns that were at risk, never all turns: a rate over
responses that declared no strict tools flatters the number by however much
unrelated traffic happened to be in the sample. None when nothing was at
risk, because a rate over an empty denominator invents a number.
"""
at_risk = sum(1 for s in states or [] if s in ("strict-void", "strict-at-risk"))
void = sum(1 for s in states or [] if s == "strict-void")
if at_risk <= 0:
return {"at_risk": 0, "void": void, "rate": None}
return {"at_risk": at_risk, "void": void, "rate": void / float(at_risk)}
def unvalidated_calls(rows):
"""Argument objects that came back with no guarantee behind them. Pure.
Counted only in the turns where the guarantee was actually void. The number
the parser cares about is objects, not turns.
"""
return sum(_int(row.get("calls")) for row in rows or []
if row.get("state") == "strict-void")
def repair_lines(state):
"""The repair for one classified turn. Pure."""
if state == "strict-void":
return [
"set parallel_tool_calls false whenever strict schemas matter. It "
"defaults to true, which is why this was never a decision anyone "
"made.",
"if you need the fan-out for latency, drop strict and validate the "
"arguments yourself. Do not keep a guarantee you know is not held.",
"key every tool handler on call_id and make it idempotent, so a "
"duplicate parallel call cannot double-apply.",
]
if state == "strict-at-risk":
return [
"this turn was fine and the configuration is not. The same request "
"shape returns several calls whenever the model decides to, so set "
"parallel_tool_calls false before it does.",
]
if state == "fanout-no-strict":
return [
"no schema guarantee was in place to lose. Validate tool arguments "
"in your own handler, or declare strict and serialise the calls.",
]
return []
def get(session, path):
r = session.get(API + path, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: OPENAI_API_KEY needs read access to "
"stored responses in this project" % r.status_code)
if r.status_code == 404:
return None
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--responses", metavar="FILE",
help="a text file of stored response ids, one per line")
ap.add_argument("--response-id", action="append", default=[],
help="a single response id; repeatable")
ap.add_argument("--show-all", action="store_true",
help="also print turns that are correctly configured")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key that can read stored "
"responses")
return 2
ids = list(args.response_id)
if args.responses:
try:
with open(args.responses, "r", encoding="utf-8") as fh:
ids.extend(parse_ids(fh.read()))
except OSError as exc:
log.error("could not read %s: %s", args.responses, exc)
return 2
ids = parse_ids("\n".join(ids))
if not ids:
log.error("no usable response ids. /v1/responses cannot be listed, so "
"the sample has to come from your own request log")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
rows = []
bad = 0
read = 0
for response_id in ids:
body = get(session, "/responses/" + response_id)
if body is None:
continue
read += 1
state, detail = classify(body)
calls = function_calls(body)
rows.append({"id": response_id, "state": state, "calls": len(calls)})
line = "%-19s %-14s %s" % (state, response_id, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
log.warning(" calls: %s", ", ".join(c["name"] for c in calls))
elif state == "fanout-no-strict":
log.warning(line)
elif args.show_all or state == "strict-at-risk":
log.info(line)
dupes = duplicate_names(calls)
if dupes:
log.warning(" duplicate: %s. Handlers keyed on the tool name "
"rather than call_id will double apply.",
"; ".join("%s called %d time(s) in one turn" % (n, c)
for n, c in sorted(dupes.items())))
if state in ("strict-void", "fanout-no-strict"):
for repair in repair_lines(state):
log.warning(" repair: %s", repair)
shape = exposure([r["state"] for r in rows])
if shape["rate"] is None:
log.info("no turn in this sample declared a strict tool with parallel "
"calls left on, so there is no exposure to report")
else:
log.info("exposure: %d of %d at-risk turn(s) fanned out (%.1f%%), "
"covering %d argument object(s) with no guarantee",
shape["void"], shape["at_risk"], shape["rate"] * 100,
unvalidated_calls(rows))
if shape["void"] == 0:
log.warning(" every at-risk turn happened to return one call. That "
"is luck, not configuration: set parallel_tool_calls "
"false before it stops being lucky.")
log.info("%d response(s) read, %d finding(s)", read, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI turns where parallel tool calls voided a strict schema.
*
* Read only. One GET per stored response id, using a project key. No
* completion is created: /v1/responses is read, never posted to.
*
* Structured Outputs is not supported alongside parallel function calls, and
* parallel_tool_calls defaults to true. A turn returning more than one
* function_call item while any tool declares strict came back without the
* guarantee the parser relies on, and it did so with an HTTP 200.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.openai.com/v1';
const CALL_TYPES = new Set(['function_call', 'custom_tool_call']);
const FINDINGS = new Set(['strict-void']);
/** Read a count as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Response ids out of a plain text file. Pure. Order kept, duplicates dropped.
* Also the guard that stops an arbitrary line becoming a URL path segment.
*/
export function parseIds(text) {
const out = [];
const seen = new Set();
for (const line of String(text ?? '').split('\n')) {
const candidate = line.split('#')[0].trim();
if (!candidate || !candidate.startsWith('resp_')) continue;
if (!/^[A-Za-z0-9_-]+$/.test(candidate)) continue;
if (seen.has(candidate)) continue;
seen.add(candidate);
out.push(candidate);
}
return out;
}
/** The function name out of either tool shape. Pure. Null when absent. */
export function toolName(tool) {
if (!tool || typeof tool !== 'object') return null;
let name = tool.name;
if (!name && tool.function && typeof tool.function === 'object') {
name = tool.function.name;
}
const text = String(name ?? '').trim();
return text || null;
}
/** Every named tool the request declared. Pure. Sorted. */
export function declaredNames(response) {
const out = new Set();
for (const tool of response?.tools ?? []) {
const name = toolName(tool);
if (name) out.add(name);
}
return [...out].sort();
}
/**
* Tools declaring strict true, in either shape. Pure. Sorted.
* strict false and strict absent are the same thing here and neither counts.
*/
export function strictTools(response) {
const out = new Set();
for (const tool of response?.tools ?? []) {
if (!tool || typeof tool !== 'object') continue;
let strict = tool.strict;
if (strict !== true && tool.function && typeof tool.function === 'object') {
strict = tool.function.strict;
}
if (strict !== true) continue;
const name = toolName(tool);
if (name) out.add(name);
}
return [...out].sort();
}
/**
* Could the model return more than one tool call in this turn? Pure.
* An absent parallel_tool_calls is true, and reading it as false is the exact
* mistake that makes this whole class of failure invisible.
*/
export function parallelAllowed(response) {
return response?.parallel_tool_calls !== false;
}
/** The tool calls in one turn, in order. Pure. call_id is kept deliberately. */
export function functionCalls(response) {
const out = [];
for (const item of response?.output ?? []) {
if (!item || typeof item !== 'object' || !CALL_TYPES.has(item.type)) continue;
const name = String(item.name ?? '').trim();
if (!name) continue;
out.push({ name, callId: String(item.call_id ?? '') });
}
return out;
}
/** Tool names called more than once in one turn. Pure. */
export function duplicateNames(calls) {
const counts = {};
for (const call of calls ?? []) {
const name = String(call?.name ?? '');
if (name) counts[name] = (counts[name] ?? 0) + 1;
}
return Object.fromEntries(Object.entries(counts).filter(([, n]) => n > 1));
}
/** Classify one turn. Pure. Returns [state, detail]. The unit is the turn. */
export function classify(response) {
const declared = declaredNames(response);
if (declared.length === 0) return ['no-tools', 'no named tools declared in this turn'];
const strict = strictTools(response);
const calls = functionCalls(response);
const parallel = parallelAllowed(response);
const names = calls.map((c) => c.name).join(', ') || 'none';
if (strict.length === 0) {
if (calls.length > 1) {
return ['fanout-no-strict',
`${calls.length} function_call item(s) in one turn (${names}) and no ` +
'tool declares strict. There was no guarantee to void here: the ' +
'arguments were never validated by the API at all, which is a ' +
'different fault with a different repair.'];
}
return ['no-strict-declared',
`${declared.length} tool(s) declared, none of them strict. Nothing in ` +
'this turn was schema-guaranteed.'];
}
if (!parallel) {
return ['strict-serialised',
`strict declared on ${strict.length} tool(s) and parallel_tool_calls is ` +
'false. The guarantee holds.'];
}
if (calls.length > 1) {
return ['strict-void',
`${calls.length} function_call item(s) in one turn with strict declared ` +
`and parallel_tool_calls left on (${names}). Structured Outputs is not ` +
'supported alongside parallel calls, so these argument objects carry no ' +
'schema guarantee.'];
}
return ['strict-at-risk',
`strict declared on ${strict.length} tool(s) with parallel_tool_calls left ` +
`on, and this turn happened to return ${calls.length} call(s). The ` +
'configuration is loaded; it did not fire here.'];
}
/**
* How often the fan-out that voids the guarantee actually happens. Pure.
* The denominator is turns that were at risk, never all turns, and it is null
* when nothing was at risk rather than a number invented over zero.
*/
export function exposure(states) {
const list = states ?? [];
const atRisk = list.filter((s) => s === 'strict-void' || s === 'strict-at-risk').length;
const voided = list.filter((s) => s === 'strict-void').length;
if (atRisk <= 0) return { atRisk: 0, void: voided, rate: null };
return { atRisk, void: voided, rate: voided / atRisk };
}
/** Argument objects that came back with no guarantee behind them. Pure. */
export function unvalidatedCalls(rows) {
let total = 0;
for (const row of rows ?? []) {
if (row?.state === 'strict-void') total += readInt(row?.calls);
}
return total;
}
/** The repair for one classified turn. Pure. */
export function repairLines(state) {
if (state === 'strict-void') {
return [
'set parallel_tool_calls false whenever strict schemas matter. It ' +
'defaults to true, which is why this was never a decision anyone made.',
'if you need the fan-out for latency, drop strict and validate the ' +
'arguments yourself. Do not keep a guarantee you know is not held.',
'key every tool handler on call_id and make it idempotent, so a ' +
'duplicate parallel call cannot double-apply.',
];
}
if (state === 'strict-at-risk') {
return ['this turn was fine and the configuration is not. The same request ' +
'shape returns several calls whenever the model decides to, so set ' +
'parallel_tool_calls false before it does.'];
}
if (state === 'fanout-no-strict') {
return ['no schema guarantee was in place to lose. Validate tool arguments ' +
'in your own handler, or declare strict and serialise the calls.'];
}
return [];
}
async function get(key, path) {
const res = await fetch(API + path, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: OPENAI_API_KEY needs read ` +
'access to stored responses in this project');
}
if (res.status === 404) return null;
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function main() {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key that can read stored responses');
process.exitCode = 2;
return;
}
const file = process.argv.slice(2).find((a) => !a.startsWith('--'));
if (!file) {
console.error('pass a text file of stored response ids, one per line');
process.exitCode = 2;
return;
}
const showAll = process.env.SHOW_ALL === '1';
const ids = parseIds(await readFile(file, 'utf8'));
if (ids.length === 0) {
console.error('no usable response ids. /v1/responses cannot be listed, so ' +
'the sample has to come from your own request log');
process.exitCode = 2;
return;
}
const rows = [];
let bad = 0;
let read = 0;
for (const id of ids) {
const body = await get(key, `/responses/${id}`);
if (body === null) continue;
read += 1;
const [state, detail] = classify(body);
const calls = functionCalls(body);
rows.push({ id, state, calls: calls.length });
const line = `${state.padEnd(19)} ${id.padEnd(14)} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
console.warn(` calls: ${calls.map((c) => c.name).join(', ')}`);
} else if (state === 'fanout-no-strict') {
console.warn(line);
} else if (showAll || state === 'strict-at-risk') {
console.log(line);
}
const dupes = duplicateNames(calls);
if (Object.keys(dupes).length > 0) {
console.warn(` duplicate: ${Object.entries(dupes).sort()
.map(([n, c]) => `${n} called ${c} time(s) in one turn`).join('; ')}. ` +
'Handlers keyed on the tool name rather than call_id will double apply.');
}
if (state === 'strict-void' || state === 'fanout-no-strict') {
for (const repair of repairLines(state)) console.warn(` repair: ${repair}`);
}
}
const shape = exposure(rows.map((r) => r.state));
if (shape.rate === null) {
console.log('no turn in this sample declared a strict tool with parallel ' +
'calls left on, so there is no exposure to report');
} else {
console.log(`exposure: ${shape.void} of ${shape.atRisk} at-risk turn(s) ` +
`fanned out (${(shape.rate * 100).toFixed(1)}%), covering ` +
`${unvalidatedCalls(rows)} argument object(s) with no guarantee`);
if (shape.void === 0) {
console.warn(' every at-risk turn happened to return one call. That is ' +
'luck, not configuration: set parallel_tool_calls false ' +
'before it stops being lucky.');
}
}
console.log(`${read} response(s) read, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is a pair of turns that came out of the same client with the same configuration: one returned a single tool call and one returned three, and the classifier has to call the second void and the first at risk rather than clean. The exposure test then puts a number on that pair — twelve fan-outs in a thousand at-risk turns is 1.2%, and the denominator deliberately excludes every turn that never declared a strict tool, because including them flatters the rate by however much unrelated traffic was in the sample. The rest pin the absent parallel_tool_calls that must read as true, the nested and flat strict shapes, the duplicate call that keeps both call_id values because the repair depends on them, and the fan-out with no strict tools anywhere, which is a different fault and must not be handed the same advice.
from openai_parallel_strict_calls import (classify, duplicate_names,
exposure, function_calls,
parallel_allowed, parse_ids,
repair_lines, strict_tools,
unvalidated_calls)
STRICT_TOOLS = [
{"type": "function", "name": "lookup_order", "strict": True},
{"type": "function", "name": "create_ticket", "strict": True},
]
def turn(calls, tools=None, parallel=None):
body = {"tools": tools if tools is not None else STRICT_TOOLS,
"output": [{"type": "function_call", "name": n,
"call_id": "call_%d" % i}
for i, n in enumerate(calls)]}
if parallel is not None:
body["parallel_tool_calls"] = parallel
return body
def test_a_turn_that_fans_out_under_strict_schemas_has_no_guarantee():
# The note in one assertion. Three calls, strict declared, and
# parallel_tool_calls never set, which means true.
body = turn(["lookup_order", "create_ticket", "create_ticket"])
assert parallel_allowed(body) is True
assert strict_tools(body) == ["create_ticket", "lookup_order"]
assert len(function_calls(body)) == 3
state, detail = classify(body)
assert state == "strict-void"
assert "3 function_call item(s) in one turn" in detail
assert "carry no schema guarantee" in detail
assert "parallel_tool_calls false" in repair_lines(state)[0]
def test_the_same_configuration_returning_one_call_is_at_risk_not_clean():
# The pair. Identical request, one call instead of three, and calling this
# a pass is how a thousand responses with twelve fan-outs read as fine.
state, detail = classify(turn(["lookup_order"]))
assert state == "strict-at-risk"
assert "The configuration is loaded; it did not fire here." in detail
states = ["strict-void"] * 12 + ["strict-at-risk"] * 988
# And 400 unrelated turns that never claimed a guarantee, which must not
# dilute the denominator.
states += ["no-strict-declared"] * 400
shape = exposure(states)
assert shape["at_risk"] == 1000 and shape["void"] == 12
assert round(shape["rate"], 4) == 0.012
rows = [{"state": "strict-void", "calls": 3} for _ in range(9)]
rows += [{"state": "strict-at-risk", "calls": 1} for _ in range(988)]
assert unvalidated_calls(rows) == 27
def test_turning_parallel_calls_off_restores_the_guarantee():
state, detail = classify(turn(["lookup_order"], parallel=False))
assert state == "strict-serialised"
assert "The guarantee holds." in detail
assert parallel_allowed({"parallel_tool_calls": False}) is False
assert parallel_allowed({"parallel_tool_calls": True}) is True
assert parallel_allowed({}) is True
assert exposure(["strict-serialised"] * 40)["rate"] is None
def test_the_same_tool_called_twice_keeps_both_call_ids():
calls = function_calls(turn(["create_ticket", "create_ticket"]))
assert duplicate_names(calls) == {"create_ticket": 2}
assert [c["call_id"] for c in calls] == ["call_0", "call_1"]
assert duplicate_names([{"name": "a"}, {"name": "b"}]) == {}
assert duplicate_names(None) == {}
def test_a_fan_out_with_no_strict_tools_is_a_different_fault():
loose = [{"type": "function", "name": "lookup_order"},
{"type": "function", "name": "create_ticket", "strict": False}]
state, detail = classify(turn(["lookup_order", "create_ticket"], tools=loose))
assert state == "fanout-no-strict"
assert "no tool declares strict" in detail
assert "different fault" in detail
assert "Validate tool arguments" in repair_lines(state)[0]
assert classify(turn([], tools=loose))[0] == "no-strict-declared"
assert strict_tools({"tools": loose}) == []
def test_strict_is_read_in_both_tool_shapes():
nested = [{"type": "function",
"function": {"name": "run_refund", "strict": True}}]
assert strict_tools({"tools": nested}) == ["run_refund"]
state, _ = classify({"tools": nested,
"output": [{"type": "function_call", "name": "run_refund",
"call_id": "c1"},
{"type": "function_call", "name": "run_refund",
"call_id": "c2"}]})
assert state == "strict-void"
def test_turns_without_tools_and_junk_do_not_become_findings():
assert classify({})[0] == "no-tools"
assert classify(None)[0] == "no-tools"
assert classify({"tools": [], "output": []})[0] == "no-tools"
# A message item is not a tool call.
body = turn([])
body["output"] = [{"type": "message", "content": []}, None, "nonsense"]
assert function_calls(body) == []
assert classify(body)[0] == "strict-at-risk"
assert unvalidated_calls(None) == 0
def test_response_ids_are_validated_before_they_reach_a_url():
text = "resp_abc123\n# note\n\nresp_abc123\nresp_def456\n../../etc\n"
assert parse_ids(text) == ["resp_abc123", "resp_def456"]
assert parse_ids("resp_bad/../x") == []
assert parse_ids(None) == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, duplicateNames, exposure, functionCalls, parallelAllowed,
parseIds, repairLines, strictTools, unvalidatedCalls }
from './openai-parallel-strict-calls.mjs';
const STRICT_TOOLS = [
{ type: 'function', name: 'lookup_order', strict: true },
{ type: 'function', name: 'create_ticket', strict: true },
];
const turn = (calls, tools, parallel) => {
const body = {
tools: tools ?? STRICT_TOOLS,
output: calls.map((name, i) => ({ type: 'function_call', name,
call_id: `call_${i}` })),
};
if (parallel !== undefined) body.parallel_tool_calls = parallel;
return body;
};
test('a turn that fans out under strict schemas has no guarantee', () => {
const body = turn(['lookup_order', 'create_ticket', 'create_ticket']);
assert.equal(parallelAllowed(body), true);
assert.deepEqual(strictTools(body), ['create_ticket', 'lookup_order']);
assert.equal(functionCalls(body).length, 3);
const [state, detail] = classify(body);
assert.equal(state, 'strict-void');
assert.match(detail, /3 function_call item/);
assert.match(detail, /carry no schema guarantee/);
assert.match(repairLines(state)[0], /parallel_tool_calls false/);
});
test('the same configuration returning one call is at risk not clean', () => {
const [state, detail] = classify(turn(['lookup_order']));
assert.equal(state, 'strict-at-risk');
assert.match(detail, /did not fire here/);
const states = [
...Array.from({ length: 12 }, () => 'strict-void'),
...Array.from({ length: 988 }, () => 'strict-at-risk'),
...Array.from({ length: 400 }, () => 'no-strict-declared'),
];
const shape = exposure(states);
assert.equal(shape.atRisk, 1000);
assert.equal(shape.void, 12);
assert.equal(Number(shape.rate.toFixed(4)), 0.012);
const rows = [
...Array.from({ length: 9 }, () => ({ state: 'strict-void', calls: 3 })),
...Array.from({ length: 988 }, () => ({ state: 'strict-at-risk', calls: 1 })),
];
assert.equal(unvalidatedCalls(rows), 27);
});
test('turning parallel calls off restores the guarantee', () => {
const [state, detail] = classify(turn(['lookup_order'], undefined, false));
assert.equal(state, 'strict-serialised');
assert.match(detail, /The guarantee holds/);
assert.equal(parallelAllowed({ parallel_tool_calls: false }), false);
assert.equal(parallelAllowed({ parallel_tool_calls: true }), true);
assert.equal(parallelAllowed({}), true);
assert.equal(exposure(Array.from({ length: 40 }, () => 'strict-serialised')).rate,
null);
});
test('the same tool called twice keeps both call ids', () => {
const calls = functionCalls(turn(['create_ticket', 'create_ticket']));
assert.deepEqual(duplicateNames(calls), { create_ticket: 2 });
assert.deepEqual(calls.map((c) => c.callId), ['call_0', 'call_1']);
assert.deepEqual(duplicateNames([{ name: 'a' }, { name: 'b' }]), {});
assert.deepEqual(duplicateNames(null), {});
});
test('a fan out with no strict tools is a different fault', () => {
const loose = [{ type: 'function', name: 'lookup_order' },
{ type: 'function', name: 'create_ticket', strict: false }];
const [state, detail] = classify(turn(['lookup_order', 'create_ticket'], loose));
assert.equal(state, 'fanout-no-strict');
assert.match(detail, /no tool declares strict/);
assert.match(detail, /different fault/);
assert.match(repairLines(state)[0], /Validate tool arguments/);
assert.equal(classify(turn([], loose))[0], 'no-strict-declared');
assert.deepEqual(strictTools({ tools: loose }), []);
});
test('strict is read in both tool shapes', () => {
const nested = [{ type: 'function',
function: { name: 'run_refund', strict: true } }];
assert.deepEqual(strictTools({ tools: nested }), ['run_refund']);
const [state] = classify({
tools: nested,
output: [{ type: 'function_call', name: 'run_refund', call_id: 'c1' },
{ type: 'function_call', name: 'run_refund', call_id: 'c2' }],
});
assert.equal(state, 'strict-void');
});
test('turns without tools and junk do not become findings', () => {
assert.equal(classify({})[0], 'no-tools');
assert.equal(classify(null)[0], 'no-tools');
assert.equal(classify({ tools: [], output: [] })[0], 'no-tools');
const body = turn([]);
body.output = [{ type: 'message', content: [] }, null, 'nonsense'];
assert.deepEqual(functionCalls(body), []);
assert.equal(classify(body)[0], 'strict-at-risk');
assert.equal(unvalidatedCalls(null), 0);
});
test('response ids are validated before they reach a url', () => {
const text = 'resp_abc123\n# note\n\nresp_abc123\nresp_def456\n../../etc\n';
assert.deepEqual(parseIds(text), ['resp_abc123', 'resp_def456']);
assert.deepEqual(parseIds('resp_bad/../x'), []);
assert.deepEqual(parseIds(null), []);
});
FAQ
Is this a bug in the API?
No, it is a documented interaction, which is precisely why it is dangerous. Structured Outputs is not supported alongside parallel function calls and the guidance is to set parallel_tool_calls to false when you depend on strict schemas. Nothing errors and nothing warns; the guarantee simply does not apply on the turns where the model fans out. The bug is in the code that assumed a default it never read.
Why is a turn with one tool call reported as at risk instead of fine?
Because the configuration is what is broken, not the turn. The same request shape returns several calls whenever the model decides to, so a single-call turn is a turn that happened not to fan out. Reporting those as clean is how a sample of a thousand responses with twelve fan-outs gets summarised as ninety-nine percent healthy, which is the reading that keeps the boolean unset for another quarter.
Can I keep parallel calls and keep strict schemas?
Not with the guarantee intact. If the fan-out matters for latency, the honest move is to drop strict and validate the arguments in your own handler, so the validation exists somewhere rather than being believed to exist in the API. Keeping strict declared while knowing it does not hold is the worst of the three options, because it is exactly what convinces the next person not to write a check.
What about the same tool being called twice in one turn?
Separate fault, same trigger, and it costs side effects rather than correctness. A handler written when a turn meant a call will create two tickets or issue two refunds. Turning off parallel calls fixes it today; keying the handler on call_id and making it idempotent fixes it permanently, including for whoever turns parallel calls back on next year for latency.
Why can the script not just scan all my responses?
There is no list endpoint. /v1/responses supports retrieval by id and nothing else, so the ids have to come from your own request log, and every rate the script prints is a rate over the sample you supplied. That makes sampling part of the method: take ids from real production traffic across a full week, because fixtures send the prompts that produce one call and that is the whole reason this survives testing.
Related field notes
- Which declared tools the model has never once chosen
- What the tools block actually weighs on every call
- A request-body field whose meaning is not what the code assumes
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Function calling — OpenAI developer docs
- Structured outputs — OpenAI developer docs
- Responses — OpenAI API reference
- Structured outputs — Microsoft Learn, Azure AI Foundry
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.