Diagnostic LLM APIs
strict omitted, so the JSON schema is only a suggestion
The validator threw about once every fifty calls, always in production, never in a test. Sometimes an extra key nobody had asked for; sometimes an optional field missing that the schema said was required; once a number arriving as a string. It read like flakiness, so it got a retry, and the retry mostly worked, which settled the matter for four months. The schema had been attached to every one of those calls. It had also never once been enforced, because somebody had taken the strict flag out eleven months earlier to make a 400 go away.
Read the flag off the response rather than out of your source tree. With a project key set to Read Only: GET /v1/responses/{response_id} and inspect the echoed text.format. If type is "json_schema" and strict is absent or false, the schema was a hint. If type is "json_object", it is legacy JSON mode: the output is guaranteed to be valid JSON and nothing whatever about its shape.
Structured Outputs guarantees adherence only with strict: true — in text.format on the Responses API, in response_format.json_schema on Chat Completions, and per tool in tools[].function.strict. Without it the model usually follows the schema, which is precisely why this survives testing: usually is enough for a test suite and not enough for production.
Then answer the question that actually unblocks anyone: why is the flag off? Almost always because the schema cannot satisfy the strict subset. Every object needs additionalProperties: false, every property must appear in required, the root must be a plain object, and a dozen familiar keywords are ignored entirely. The script walks the echoed schema and prints each rule it breaks, with the path.
The problem in plain words
Both forms of the request are legal, so nothing anywhere tells you which one you sent. The API accepts a schema with strict: true and a schema without it, and returns a 200 either way. The response looks the same. The output usually looks the same. There is no warning, no header, no deprecation notice, and no difference at all in the happy case, which is most cases.
The failure surfaces at your own validator, at a rate low enough to be mistaken for infrastructure. Pydantic or Zod throws on a field, somebody adds a retry, the retry usually succeeds because the model usually complies, and the incident closes. Meanwhile the records that did get written are the ones where the model complied — and where it complied differently, with an extra key or a coerced type, nobody checked at all.
Why it happens
Constrained decoding is either on or it is not. With strict: true the sampler cannot emit a token that would break the schema, so adherence is structural. Without it, the schema is text in the prompt and adherence is a behaviour. Those are not two settings on a dial; they are two different mechanisms, and only one of them has a guarantee attached.
The flag comes out because the schema cannot support it. The strict subset is narrower than JSON Schema: additionalProperties: false on every object, every property listed in required, a root that is a plain object and never an anyOf. A schema that breaks any of those is rejected outright with strict on, which is a 400 during development, which is when somebody removes the flag and the request starts working. That is why telling people to set strict: true is useless on its own, and why this script prints the blockers instead.
Optional fields are the usual blocker and the fix is counter-intuitive. Every property must be in required, so an optional field is expressed as a nullable type rather than by omission. Teams reach for leaving it out, get a 400, and conclude the strict subset cannot model their data. It can; the idiom is just unfamiliar.
Several keywords you rely on are silently unenforced. minLength, maxLength, pattern, format, minimum, maximum, minItems, maxItems and uniqueItems do not constrain decoding. Keeping them for your own validator is fine. Believing the model honours them is not, and the script lists the ones your schema carries so the belief can be corrected explicitly.
Legacy JSON mode is a third state and deserves its own name. json_object promises valid syntax and says nothing about shape. It is not a weaker schema; it is no schema. Reporting it as "strict is off" would understate it, because there is nothing to turn strict on.
Tools carry the flag separately. A response can declare a strict text format and a function definition with no strict flag at all, and the guarantee then covers exactly the half that asked for it. Tool arguments are constrained per tool. The script reports the gap per tool name rather than as one verdict for the response, which is also where this note stops: what an unconstrained tool actually emits is the next note along.
The fix, as a flow
The only note in this batch whose problem chain has no failing arrow at all. Every step works. The schema was attached, the request was accepted, the model followed the schema most of the time, and the one call in fifty that it did not follow is the one your validator sees. What is drawn here is the request that was accepted, because the guarantee was never bought and nothing along the way said so.
How to fix it
Read the format the response carries, not the constant in the repository
The echoed text.format is the only place outside your source tree where the flag can be read back, and it has the advantage of describing the deploy that actually ran. A wrapper library, a feature flag or an older service can all send something different from what the code you are reading appears to send.
Sort into three states, not two
json_schema with strict true is enforced. json_schema without it is advisory. json_object is no schema at all. The third is a different conversation from the second and collapsing them makes the report less useful, not shorter.
Walk the schema for every rule the strict subset requires
Per object: additionalProperties: false, and every property present in required. At the root: a plain object, never a composition keyword. Overall: at most five levels of nesting. The script reports each violation with its path, because "the schema is not eligible" is not a work item and "$.lines[].note is missing from required" is.
List the keywords that will not be enforced whatever you do
They are legal in the schema and ignored by the decoder. Print them explicitly, because a team that believes pattern is holding the line on an identifier format has a validation gap it does not know about, and that gap survives turning strict on.
Check every tool beside the format
strict is declared per tool. Report the loose ones by name. A response whose text format is strict and whose refund tool is not is not a passing response; it is a response with a specific, nameable hole in it.
How to check it worked
Fix the blockers, set the flag, and re-read the same ids. The state should move to enforced, and the intermittent validator failures should stop rather than get quieter.
python3 openai_advisory_schema.py --ids response_ids.txt
# advisory-schema resp_68e0b2 schema 'invoice' was attached with strict absent, so it is a hint the model usually follows rather than a guarantee.
# repair: strict: true would be refused for this schema until these are fixed:
# repair: $: needs additionalProperties: false
# repair: $: every property must be listed in required; missing lines, note.
# repair: $.invoice_id: minLength are silently unenforced under constrained decoding.
# 240 response(s) checked, 31 with a schema nobody was holding toThe full code
One GET per id, and the rest is a walk over data that came back with it. Seven pure functions. declared_format normalises two request shapes into one tuple, because a half-migrated codebase stores both. strict_state is three words long and is the whole finding. schema_blockers is where the work is: a recursive walk that reports, with paths, every reason strict mode would refuse this schema, which is the difference between advice somebody has already tried and a list they can work through. schema_size measures against the documented ceilings, loose_tools finds the per-tool gaps, and the classifier and repair lines assemble the verdict without ever looking at the output text — because whether this particular call happened to comply is not the question.
"""Find stored OpenAI responses whose JSON schema was never actually enforced.
Read only. GET /v1/responses/{response_id} for each id you supply, with a
project key set to Read Only. There is no list endpoint for /v1/responses, so
the ids come from your own records: one id per line in a file.
Structured Outputs guarantees schema adherence only when strict is true. With
strict absent or false the schema degrades to a hint the model usually follows,
and the request is accepted either way with no warning of any kind. The stored
response echoes the format it was given, which is the only place outside your
source tree where the flag can be read back.
When strict is off, the interesting question is why, and the answer is almost
always that the schema cannot satisfy the strict subset. So this script does
not stop at the flag: it walks the schema and prints every rule that would have
to be fixed before strict: true could be turned on.
The repair is printed, never performed.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_advisory_schema")
API = "https://api.openai.com/v1"
FINDINGS = ("advisory-schema", "no-schema", "advisory-tools")
# Constrained decoding ignores these entirely. A schema that carries them is
# not wrong, but the constraints they express are not enforced by anyone.
UNENFORCED_KEYWORDS = ("minLength", "maxLength", "pattern", "format", "minimum",
"maximum", "multipleOf", "minItems", "maxItems",
"uniqueItems", "default")
# The documented ceilings for a strict schema.
MAX_DEPTH = 5
MAX_PROPERTIES = 5000
MAX_ENUM_VALUES = 1000
def declared_format(response):
"""The output format the response was generated under. Pure.
Returns (kind, name, strict, schema). kind is json_schema, json_object,
text or none. Read from the response rather than from your source tree,
because the constant in the repository is not necessarily what the running
deploy sent.
"""
response = response or {}
fmt = ((response.get("text") or {}).get("format")
or response.get("response_format") or {})
if not isinstance(fmt, dict) or not fmt:
return ("none", None, None, None)
kind = str(fmt.get("type") or "none")
if kind == "json_schema":
# The Responses API flattens the schema onto the format object; Chat
# Completions nests it under json_schema. Both shapes are stored.
inner = fmt.get("json_schema") if isinstance(fmt.get("json_schema"), dict) else fmt
return ("json_schema", inner.get("name"), inner.get("strict"),
inner.get("schema"))
return (kind, None, None, None)
def strict_state(kind, strict):
"""What the declared format actually promises. Pure."""
if kind == "json_schema":
return "enforced" if strict is True else "advisory"
if kind == "json_object":
return "no-schema"
if kind in ("text", "none"):
return "free-text"
return "unknown-format"
def schema_size(schema, depth=1):
"""Count properties, depth and the largest enum in a schema. Pure.
Returned as a dict rather than printed, because the interesting comparison
is against the documented ceilings and those change more often than this
walk does.
"""
totals = {"properties": 0, "depth": depth, "enum": 0}
if not isinstance(schema, dict):
return totals
if isinstance(schema.get("enum"), list):
totals["enum"] = max(totals["enum"], len(schema["enum"]))
children = []
properties = schema.get("properties")
if isinstance(properties, dict):
totals["properties"] += len(properties)
children.extend(properties.values())
items = schema.get("items")
if isinstance(items, dict):
children.append(items)
for group in ("anyOf", "oneOf", "allOf"):
if isinstance(schema.get(group), list):
children.extend(x for x in schema[group] if isinstance(x, dict))
defs = schema.get("$defs")
if isinstance(defs, dict):
children.extend(x for x in defs.values() if isinstance(x, dict))
for child in children:
below = schema_size(child, depth + 1)
totals["properties"] += below["properties"]
totals["depth"] = max(totals["depth"], below["depth"])
totals["enum"] = max(totals["enum"], below["enum"])
return totals
def schema_blockers(schema, path="$", depth=1):
"""Every reason strict: true would be refused for this schema. Pure.
This is the part of the note that pays for itself. Telling somebody to set
strict: true is useless on its own, because they tried that, the request
400ed, and the flag came back out. The list of rules the schema breaks is
the actual work.
"""
problems = []
if not isinstance(schema, dict):
return ["%s: not a schema object" % path] if depth == 1 else problems
kinds = schema.get("type")
kinds = kinds if isinstance(kinds, list) else ([kinds] if kinds else [])
kinds = [str(k) for k in kinds]
if depth == 1:
if any(schema.get(group) for group in ("anyOf", "oneOf", "allOf")):
problems.append("$: the root may not be anyOf, oneOf or allOf; it "
"must be a plain object")
elif "object" not in kinds:
problems.append("$: the root type must be object, not %s"
% (", ".join(kinds) or "unset"))
if "object" in kinds:
if schema.get("additionalProperties") is not False:
problems.append("%s: needs additionalProperties: false" % path)
properties = schema.get("properties")
properties = properties if isinstance(properties, dict) else {}
required = schema.get("required")
required = set(required) if isinstance(required, list) else set()
missing = sorted(set(properties) - required)
if missing:
problems.append("%s: every property must be listed in required; "
"missing %s. Use a nullable type for the optional "
"ones rather than leaving them out."
% (path, ", ".join(missing)))
present = [k for k in UNENFORCED_KEYWORDS if k in schema]
if present:
problems.append("%s: %s are silently unenforced under constrained "
"decoding. Keep them for your own validator if you "
"like, but do not rely on the model honouring them."
% (path, ", ".join(present)))
if depth > MAX_DEPTH:
problems.append("%s: nested %d levels deep, past the limit of %d"
% (path, depth, MAX_DEPTH))
return problems
properties = schema.get("properties")
if isinstance(properties, dict):
for name in sorted(properties):
problems.extend(schema_blockers(properties[name],
"%s.%s" % (path, name), depth + 1))
items = schema.get("items")
if isinstance(items, dict):
problems.extend(schema_blockers(items, path + "[]", depth + 1))
return problems
def loose_tools(response):
"""Tools echoed on the response whose strict flag is not true. Pure.
Per tool, because strict is declared per tool. A response can carry a
strict text format and a function definition with no strict flag at all,
and the guarantee covers only the half that asked for it.
"""
loose = []
for tool in (response or {}).get("tools") or []:
if not isinstance(tool, dict):
continue
if str(tool.get("type") or "function") != "function":
continue
inner = tool.get("function") if isinstance(tool.get("function"), dict) else tool
if inner.get("strict") is not True:
loose.append(str(inner.get("name") or "unnamed"))
return loose
def classify(response):
"""Classify one stored response. Pure. Returns (state, detail).
Nothing here reads the output text. Whether this particular call happened
to produce a well-shaped object is not the point: the point is that no call
made under this format was ever obliged to.
"""
kind, name, strict, schema = declared_format(response)
state = strict_state(kind, strict)
loose = loose_tools(response)
label = ("schema %r" % name) if name else "the declared schema"
if state == "advisory":
return ("advisory-schema",
"%s was attached with strict %s, so it is a hint the model "
"usually follows rather than a guarantee. Valid JSON of the "
"wrong shape is a legal outcome here."
% (label, "false" if strict is False else "absent"))
if state == "no-schema":
return ("no-schema",
"Legacy json_object mode: the output is guaranteed to be valid "
"JSON and nothing else. No schema was ever attached, so no "
"shape was ever promised.")
if state == "free-text":
return ("free-text",
"No output format was declared, so there is no contract to "
"enforce and nothing to report.")
if state == "unknown-format":
return ("unknown-format",
"Format type %r is not one this script knows. Read the raw "
"record before drawing a conclusion." % kind)
if loose:
return ("advisory-tools",
"The text format is strict, but %d tool definition(s) are not: "
"%s. Tool arguments are constrained per tool, and an unstrict "
"tool is unconstrained." % (len(loose), ", ".join(loose)))
return ("enforced",
"%s was attached with strict: true, and no tool beside it is "
"loose." % label)
def repair_lines(response, state):
"""The repair, built from the schema this response actually carried. Pure."""
_kind, _name, _strict, schema = declared_format(response)
if state == "free-text" or state == "unknown-format":
return []
if state == "enforced":
return []
lines = []
if state == "no-schema":
lines.append("Move from json_object to a json_schema format with "
"strict: true. JSON mode promises syntax and nothing "
"about shape, which is why your validator is the first "
"thing that ever sees the mismatch.")
if state == "advisory-tools":
lines.append("Set strict: true on every tool as well as on the text "
"format, with additionalProperties: false and every "
"parameter listed in required.")
blockers = schema_blockers(schema) if schema else []
if blockers:
lines.append("strict: true would be refused for this schema until "
"these are fixed:")
lines.extend(" " + b for b in blockers)
elif state == "advisory-schema":
lines.append("This schema already satisfies the strict subset, so "
"setting strict: true is a one-line change. Somebody "
"dropped the flag and the request kept succeeding.")
if schema:
size = schema_size(schema)
if size["properties"] > MAX_PROPERTIES:
lines.append("The schema declares %d properties, past the limit of "
"%d." % (size["properties"], MAX_PROPERTIES))
if size["enum"] > MAX_ENUM_VALUES:
lines.append("The largest enum holds %d values, past the limit of "
"%d." % (size["enum"], MAX_ENUM_VALUES))
return lines
def read_ids(path):
"""One response id per line, blanks and # comments ignored."""
ids = []
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if line and not line.startswith("#"):
ids.append(line)
return ids
def fetch_response(session, response_id):
r = session.get(API + "/responses/" + response_id, timeout=60)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: this needs a project key that can "
"read stored responses" % r.status_code)
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ids", required=True,
help="file of stored response ids, one per line")
ap.add_argument("--show-all", action="store_true",
help="also print responses whose schema is enforced")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY, a project key set to Read Only")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
checked = 0
bad = 0
for response_id in read_ids(args.ids):
stored = fetch_response(session, response_id)
checked += 1
if stored is None:
log.warning("%-18s %s not found. Stored responses expire, and a "
"response created without storage was never readable.",
"unreadable", response_id)
continue
state, detail = classify(stored)
line = "%-18s %s %s" % (state, response_id, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
for repair in repair_lines(stored, state):
log.warning(" repair: %s", repair)
elif args.show_all or state not in ("enforced", "free-text"):
log.info(line)
log.info("%d response(s) checked, %d with a schema nobody was holding to",
checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find stored OpenAI responses whose JSON schema was never actually enforced.
*
* Read only. GET /v1/responses/{response_id} for each id you supply, with a
* project key set to Read Only. There is no list endpoint for /v1/responses,
* so the ids come from your own records: one id per line in a file.
*
* Structured Outputs guarantees schema adherence only when strict is true.
* With strict absent or false the schema degrades to a hint the model usually
* follows, and the request is accepted either way with no warning. The stored
* response echoes the format it was given, which is the only place outside
* your source tree where the flag can be read back.
*
* When strict is off, the interesting question is why, so this script walks
* the schema and prints every rule that would have to be fixed first.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set(['advisory-schema', 'no-schema', 'advisory-tools']);
// Constrained decoding ignores these entirely.
const UNENFORCED_KEYWORDS = ['minLength', 'maxLength', 'pattern', 'format',
'minimum', 'maximum', 'multipleOf', 'minItems', 'maxItems', 'uniqueItems',
'default'];
const MAX_DEPTH = 5;
const MAX_PROPERTIES = 5000;
const MAX_ENUM_VALUES = 1000;
/**
* The output format the response was generated under. Pure.
* Returns [kind, name, strict, schema]. Read from the response, not from your
* source tree: the constant in the repository is not necessarily what the
* running deploy sent.
*/
export function declaredFormat(response) {
const fmt = response?.text?.format ?? response?.response_format ?? {};
if (!fmt || typeof fmt !== 'object' || Object.keys(fmt).length === 0) {
return ['none', null, null, null];
}
const kind = String(fmt.type ?? 'none');
if (kind === 'json_schema') {
const inner = (fmt.json_schema && typeof fmt.json_schema === 'object')
? fmt.json_schema : fmt;
return ['json_schema', inner.name ?? null, inner.strict ?? null,
inner.schema ?? null];
}
return [kind, null, null, null];
}
/** What the declared format actually promises. Pure. */
export function strictState(kind, strict) {
if (kind === 'json_schema') return strict === true ? 'enforced' : 'advisory';
if (kind === 'json_object') return 'no-schema';
if (kind === 'text' || kind === 'none') return 'free-text';
return 'unknown-format';
}
/** Count properties, depth and the largest enum in a schema. Pure. */
export function schemaSize(schema, depth = 1) {
const totals = { properties: 0, depth, enum: 0 };
if (!schema || typeof schema !== 'object') return totals;
if (Array.isArray(schema.enum)) totals.enum = Math.max(totals.enum, schema.enum.length);
const children = [];
if (schema.properties && typeof schema.properties === 'object') {
const values = Object.values(schema.properties);
totals.properties += values.length;
children.push(...values);
}
if (schema.items && typeof schema.items === 'object') children.push(schema.items);
for (const group of ['anyOf', 'oneOf', 'allOf']) {
if (Array.isArray(schema[group])) {
children.push(...schema[group].filter((x) => x && typeof x === 'object'));
}
}
if (schema.$defs && typeof schema.$defs === 'object') {
children.push(...Object.values(schema.$defs).filter((x) => x && typeof x === 'object'));
}
for (const child of children) {
const below = schemaSize(child, depth + 1);
totals.properties += below.properties;
totals.depth = Math.max(totals.depth, below.depth);
totals.enum = Math.max(totals.enum, below.enum);
}
return totals;
}
/**
* Every reason strict: true would be refused for this schema. Pure.
* Telling somebody to set strict: true is useless on its own: they tried, the
* request 400ed, and the flag came back out. This list is the actual work.
*/
export function schemaBlockers(schema, path = '$', depth = 1) {
const problems = [];
if (!schema || typeof schema !== 'object' || Array.isArray(schema)) {
return depth === 1 ? [`${path}: not a schema object`] : problems;
}
const raw = schema.type;
const kinds = (Array.isArray(raw) ? raw : (raw ? [raw] : [])).map(String);
if (depth === 1) {
if (['anyOf', 'oneOf', 'allOf'].some((g) => schema[g])) {
problems.push('$: the root may not be anyOf, oneOf or allOf; it must be ' +
'a plain object');
} else if (!kinds.includes('object')) {
problems.push(`$: the root type must be object, not ${kinds.join(', ') || 'unset'}`);
}
}
if (kinds.includes('object')) {
if (schema.additionalProperties !== false) {
problems.push(`${path}: needs additionalProperties: false`);
}
const properties = (schema.properties && typeof schema.properties === 'object')
? schema.properties : {};
const required = new Set(Array.isArray(schema.required) ? schema.required : []);
const missing = Object.keys(properties).filter((k) => !required.has(k)).sort();
if (missing.length) {
problems.push(`${path}: every property must be listed in required; ` +
`missing ${missing.join(', ')}. Use a nullable type for the optional ` +
'ones rather than leaving them out.');
}
}
const present = UNENFORCED_KEYWORDS.filter((k) => k in schema);
if (present.length) {
problems.push(`${path}: ${present.join(', ')} are silently unenforced under ` +
'constrained decoding. Keep them for your own validator if you like, but ' +
'do not rely on the model honouring them.');
}
if (depth > MAX_DEPTH) {
problems.push(`${path}: nested ${depth} levels deep, past the limit of ${MAX_DEPTH}`);
return problems;
}
if (schema.properties && typeof schema.properties === 'object') {
for (const name of Object.keys(schema.properties).sort()) {
problems.push(...schemaBlockers(schema.properties[name], `${path}.${name}`, depth + 1));
}
}
if (schema.items && typeof schema.items === 'object') {
problems.push(...schemaBlockers(schema.items, `${path}[]`, depth + 1));
}
return problems;
}
/** Tools echoed on the response whose strict flag is not true. Pure. Per tool. */
export function looseTools(response) {
const loose = [];
for (const tool of response?.tools ?? []) {
if (!tool || typeof tool !== 'object') continue;
if (String(tool.type ?? 'function') !== 'function') continue;
const inner = (tool.function && typeof tool.function === 'object') ? tool.function : tool;
if (inner.strict !== true) loose.push(String(inner.name ?? 'unnamed'));
}
return loose;
}
/**
* Classify one stored response. Pure.
* Nothing here reads the output text: whether this call happened to produce a
* well-shaped object is not the point. No call under this format was obliged to.
*/
export function classify(response) {
const [kind, name, strict] = declaredFormat(response);
const state = strictState(kind, strict);
const loose = looseTools(response);
const label = name ? `schema '${name}'` : 'the declared schema';
if (state === 'advisory') {
return ['advisory-schema',
`${label} was attached with strict ${strict === false ? 'false' : 'absent'}, ` +
'so it is a hint the model usually follows rather than a guarantee. Valid ' +
'JSON of the wrong shape is a legal outcome here.'];
}
if (state === 'no-schema') {
return ['no-schema',
'Legacy json_object mode: the output is guaranteed to be valid JSON and ' +
'nothing else. No schema was ever attached, so no shape was ever promised.'];
}
if (state === 'free-text') {
return ['free-text',
'No output format was declared, so there is no contract to enforce and ' +
'nothing to report.'];
}
if (state === 'unknown-format') {
return ['unknown-format',
`Format type '${kind}' is not one this script knows. Read the raw record ` +
'before drawing a conclusion.'];
}
if (loose.length) {
return ['advisory-tools',
`The text format is strict, but ${loose.length} tool definition(s) are ` +
`not: ${loose.join(', ')}. Tool arguments are constrained per tool, and ` +
'an unstrict tool is unconstrained.'];
}
return ['enforced',
`${label} was attached with strict: true, and no tool beside it is loose.`];
}
/** The repair, built from the schema this response actually carried. Pure. */
export function repairLines(response, state) {
const schema = declaredFormat(response)[3];
if (state === 'free-text' || state === 'unknown-format' || state === 'enforced') {
return [];
}
const lines = [];
if (state === 'no-schema') {
lines.push('Move from json_object to a json_schema format with strict: true. ' +
'JSON mode promises syntax and nothing about shape, which is why your ' +
'validator is the first thing that ever sees the mismatch.');
}
if (state === 'advisory-tools') {
lines.push('Set strict: true on every tool as well as on the text format, ' +
'with additionalProperties: false and every parameter listed in required.');
}
const blockers = schema ? schemaBlockers(schema) : [];
if (blockers.length) {
lines.push('strict: true would be refused for this schema until these are fixed:');
lines.push(...blockers.map((b) => ` ${b}`));
} else if (state === 'advisory-schema') {
lines.push('This schema already satisfies the strict subset, so setting ' +
'strict: true is a one-line change. Somebody dropped the flag and the ' +
'request kept succeeding.');
}
if (schema) {
const size = schemaSize(schema);
if (size.properties > MAX_PROPERTIES) {
lines.push(`The schema declares ${size.properties} properties, past the ` +
`limit of ${MAX_PROPERTIES}.`);
}
if (size.enum > MAX_ENUM_VALUES) {
lines.push(`The largest enum holds ${size.enum} values, past the limit of ` +
`${MAX_ENUM_VALUES}.`);
}
}
return lines;
}
async function fetchResponse(key, responseId) {
const res = await fetch(`${API}/responses/${responseId}`,
{ headers: { Authorization: `Bearer ${key}` } });
if (res.status === 404) return null;
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: this needs a project key that ` +
'can read stored responses');
}
if (!res.ok) throw new Error(`${res.status} from /responses/${responseId}`);
return res.json();
}
async function main() {
const key = process.env.OPENAI_API_KEY;
const idsFile = process.env.RESPONSE_IDS;
if (!key || !idsFile) {
console.error('set OPENAI_API_KEY (a project key set to Read Only) and ' +
'RESPONSE_IDS (a file of stored response ids, one per line)');
process.exitCode = 2;
return;
}
const showAll = process.env.SHOW_ALL === '1';
const ids = (await readFile(idsFile, 'utf8')).split('\n')
.map((l) => l.trim()).filter((l) => l && !l.startsWith('#'));
let checked = 0;
let bad = 0;
for (const responseId of ids) {
const stored = await fetchResponse(key, responseId);
checked += 1;
if (stored === null) {
console.warn(`${'unreadable'.padEnd(18)} ${responseId} not found. Stored ` +
'responses expire, and a response created without storage was never readable.');
continue;
}
const [state, detail] = classify(stored);
const line = `${state.padEnd(18)} ${responseId} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
for (const repair of repairLines(stored, state)) console.warn(` repair: ${repair}`);
} else if (showAll || (state !== 'enforced' && state !== 'free-text')) {
console.log(line);
}
}
console.log(`${checked} response(s) checked, ${bad} with a schema nobody was ` +
'holding to');
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first test is the trap in one assertion: a response carrying perfectly valid JSON of exactly the right shape, which is nonetheless a finding, because nothing obliged it to be. Then strict false reading identically to strict missing, and legacy JSON mode getting its own name rather than being folded in. The schema walker is tested on a schema that breaks three rules at once and must report all three with paths — and, just as importantly, must stay quiet about the nested object that already complies, since a checker that flags everything gets muted. The last two cover the root that cannot be strict at all and a strict format sitting beside a loose tool.
from openai_advisory_schema import (classify, declared_format, loose_tools,
repair_lines, schema_blockers, schema_size,
strict_state)
TIGHT = {
"type": "object",
"additionalProperties": False,
"required": ["invoice_id", "total"],
"properties": {"invoice_id": {"type": "string"},
"total": {"type": "number"}},
}
def response(fmt, tools=None):
body = {"id": "resp_s", "status": "completed", "model": "gpt-5.1",
"text": {"format": fmt},
"output": [{"type": "message",
"content": [{"type": "output_text",
"text": '{"invoice_id": "INV-1", "total": 1}'}]}]}
if tools is not None:
body["tools"] = tools
return body
def test_a_schema_without_strict_is_advice_and_the_json_still_parsed():
# The trap: this response is a 200 carrying perfectly valid JSON of the
# right shape. Nothing about the output says the contract was optional.
stored = response({"type": "json_schema", "name": "invoice", "schema": TIGHT})
assert declared_format(stored)[:3] == ("json_schema", "invoice", None)
assert strict_state("json_schema", None) == "advisory"
state, detail = classify(stored)
assert state == "advisory-schema"
assert "strict absent" in detail
assert "wrong shape is a legal outcome" in detail
# The schema is already eligible, so the repair is one line rather than a
# rewrite. Saying so is the difference between a useful report and a chore.
assert "one-line change" in " ".join(repair_lines(stored, state))
def test_strict_false_reads_the_same_as_strict_missing():
stored = response({"type": "json_schema", "name": "invoice",
"strict": False, "schema": TIGHT})
state, detail = classify(stored)
assert state == "advisory-schema"
assert "strict false" in detail
def test_legacy_json_object_mode_is_named_as_its_own_thing():
stored = response({"type": "json_object"})
state, detail = classify(stored)
assert state == "no-schema"
assert "valid JSON and nothing else" in detail
assert "json_object to a json_schema" in repair_lines(stored, state)[0]
def test_schema_blockers_names_every_rule_the_subset_requires():
loose = {
"type": "object",
"required": ["invoice_id"],
"properties": {
"invoice_id": {"type": "string", "minLength": 3},
"note": {"type": "string"},
"lines": {"type": "array",
"items": {"type": "object",
"additionalProperties": False,
"properties": {"sku": {"type": "string"}},
"required": ["sku"]}},
},
}
found = " | ".join(schema_blockers(loose))
assert "$: needs additionalProperties: false" in found
assert "missing lines, note" in found
assert "minLength are silently unenforced" in found
# The nested object below the array already obeys the rules, so it must
# not be reported. A checker that flags everything gets muted.
assert "$.lines[]" not in found
assert schema_blockers(TIGHT) == []
def test_a_root_that_is_not_a_plain_object_cannot_be_strict_at_all():
assert "root may not be anyOf" in schema_blockers(
{"anyOf": [TIGHT, {"type": "object"}]})[0]
assert "root type must be object, not array" in schema_blockers(
{"type": "array", "items": TIGHT})[0]
assert "not a schema object" in schema_blockers(None)[0]
def test_depth_beyond_five_levels_is_reported_and_the_walk_stops():
schema = {"type": "object", "additionalProperties": False,
"required": ["a"], "properties": {"a": {"type": "string"}}}
for _ in range(6):
schema = {"type": "object", "additionalProperties": False,
"required": ["child"], "properties": {"child": schema}}
found = schema_blockers(schema)
assert any("past the limit of 5" in f for f in found)
assert schema_size(schema)["depth"] > 5
def test_a_strict_format_beside_a_loose_tool_is_still_a_gap():
tools = [{"type": "function", "name": "charge", "parameters": TIGHT,
"strict": True},
{"type": "function", "name": "refund", "parameters": TIGHT}]
stored = response({"type": "json_schema", "name": "invoice",
"strict": True, "schema": TIGHT}, tools=tools)
assert loose_tools(stored) == ["refund"]
state, detail = classify(stored)
assert state == "advisory-tools"
assert "refund" in detail
assert "every tool as well as on the text format" in repair_lines(stored, state)[0]
def test_the_chat_completions_shape_and_the_clean_cases():
legacy = {"response_format": {"type": "json_schema",
"json_schema": {"name": "invoice",
"strict": True,
"schema": TIGHT}}}
assert declared_format(legacy)[:3] == ("json_schema", "invoice", True)
assert classify(legacy)[0] == "enforced"
assert classify({})[0] == "free-text"
assert classify(None)[0] == "free-text"
assert repair_lines({}, "free-text") == []
assert loose_tools({}) == []
assert schema_size({}) == {"properties": 0, "depth": 1, "enum": 0}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
classify, declaredFormat, looseTools, repairLines, schemaBlockers, schemaSize,
strictState,
} from './openai-advisory-schema.mjs';
const TIGHT = {
type: 'object',
additionalProperties: false,
required: ['invoice_id', 'total'],
properties: { invoice_id: { type: 'string' }, total: { type: 'number' } },
};
const response = (format, tools) => {
const body = { id: 'resp_s', status: 'completed', model: 'gpt-5.1',
text: { format },
output: [{ type: 'message', content: [{ type: 'output_text',
text: '{"invoice_id": "INV-1", "total": 1}' }] }] };
if (tools !== undefined) body.tools = tools;
return body;
};
test('a schema without strict is advice and the json still parsed', () => {
const stored = response({ type: 'json_schema', name: 'invoice', schema: TIGHT });
assert.deepEqual(declaredFormat(stored).slice(0, 3), ['json_schema', 'invoice', null]);
assert.equal(strictState('json_schema', null), 'advisory');
const [state, detail] = classify(stored);
assert.equal(state, 'advisory-schema');
assert.match(detail, /strict absent/);
assert.match(detail, /wrong shape is a legal outcome/);
assert.match(repairLines(stored, state).join(' '), /one-line change/);
});
test('strict false reads the same as strict missing', () => {
const stored = response({ type: 'json_schema', name: 'invoice',
strict: false, schema: TIGHT });
const [state, detail] = classify(stored);
assert.equal(state, 'advisory-schema');
assert.match(detail, /strict false/);
});
test('legacy json_object mode is named as its own thing', () => {
const stored = response({ type: 'json_object' });
const [state, detail] = classify(stored);
assert.equal(state, 'no-schema');
assert.match(detail, /valid JSON and nothing else/);
assert.match(repairLines(stored, state)[0], /json_object to a json_schema/);
});
test('schemaBlockers names every rule the subset requires', () => {
const loose = {
type: 'object',
required: ['invoice_id'],
properties: {
invoice_id: { type: 'string', minLength: 3 },
note: { type: 'string' },
lines: { type: 'array',
items: { type: 'object', additionalProperties: false,
properties: { sku: { type: 'string' } }, required: ['sku'] } },
},
};
const found = schemaBlockers(loose).join(' | ');
assert.match(found, /\$: needs additionalProperties: false/);
assert.match(found, /missing lines, note/);
assert.match(found, /minLength are silently unenforced/);
assert.ok(!found.includes('$.lines[]'));
assert.deepEqual(schemaBlockers(TIGHT), []);
});
test('a root that is not a plain object cannot be strict at all', () => {
assert.match(schemaBlockers({ anyOf: [TIGHT, { type: 'object' }] })[0],
/root may not be anyOf/);
assert.match(schemaBlockers({ type: 'array', items: TIGHT })[0],
/root type must be object, not array/);
assert.match(schemaBlockers(null)[0], /not a schema object/);
});
test('depth beyond five levels is reported and the walk stops', () => {
let schema = { type: 'object', additionalProperties: false,
required: ['a'], properties: { a: { type: 'string' } } };
for (let i = 0; i < 6; i += 1) {
schema = { type: 'object', additionalProperties: false,
required: ['child'], properties: { child: schema } };
}
assert.ok(schemaBlockers(schema).some((f) => f.includes('past the limit of 5')));
assert.ok(schemaSize(schema).depth > 5);
});
test('a strict format beside a loose tool is still a gap', () => {
const tools = [
{ type: 'function', name: 'charge', parameters: TIGHT, strict: true },
{ type: 'function', name: 'refund', parameters: TIGHT },
];
const stored = response({ type: 'json_schema', name: 'invoice',
strict: true, schema: TIGHT }, tools);
assert.deepEqual(looseTools(stored), ['refund']);
const [state, detail] = classify(stored);
assert.equal(state, 'advisory-tools');
assert.match(detail, /refund/);
assert.match(repairLines(stored, state)[0], /every tool as well as on the text format/);
});
test('the chat completions shape and the clean cases', () => {
const legacy = { response_format: { type: 'json_schema',
json_schema: { name: 'invoice', strict: true, schema: TIGHT } } };
assert.deepEqual(declaredFormat(legacy).slice(0, 3), ['json_schema', 'invoice', true]);
assert.equal(classify(legacy)[0], 'enforced');
assert.equal(classify({})[0], 'free-text');
assert.equal(classify(null)[0], 'free-text');
assert.deepEqual(repairLines({}, 'free-text'), []);
assert.deepEqual(looseTools({}), []);
assert.deepEqual(schemaSize({}), { properties: 0, depth: 1, enum: 0 });
});
FAQ
If the model follows the schema anyway, does the flag matter?
It matters exactly as much as the tail matters to you. Without strict you are relying on a behaviour that holds most of the time and has no floor: it can shift with a model version, with prompt length, with how unusual the input is. If a wrong-shaped record costs nothing, live with it. If it reaches a database or a payment, the guarantee is the whole point of using structured outputs at all.
Why was strict removed in the first place?
Almost always because turning it on produced a 400. The strict subset is narrower than JSON Schema and a real-world schema usually breaks two or three of its rules on the first attempt. Removing the flag makes the error go away instantly and changes nothing visible, which is a nearly perfect trap. That is why this script prints the specific rules rather than the advice.
How do I express an optional field if everything must be required?
As a nullable type: list the property in required and give it a type of string or null. The field is then always present and may be null, which is the same information in a shape constrained decoding can hold. Leaving properties out of required is the single most common reason a schema cannot be made strict.
Is json_object mode just a weaker version of this?
No, it is a different thing and the script names it separately. JSON mode guarantees the output parses as JSON. It says nothing about keys, types, or structure, so there is no schema to enforce and no flag to set. Moving from json_object to a json_schema format is a change to the request, not a flag flip.
What about tools? Does the format flag cover them?
It does not. strict is declared per tool in tools[].function.strict, so a response can have a strict text format and an entirely unconstrained function definition beside it. The script reports the loose tools by name. What those tools then emit, and how it breaks a dispatcher, is the subject of the tool-arguments note.
Related field notes
- Tool arguments your dispatcher cannot use
- A 200 whose JSON stops mid-object because the ceiling arrived
- The model id that changes under you between deploys
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Structured Outputs — OpenAI developer docs
- Structured outputs — Azure OpenAI, Microsoft Learn
- Structured Outputs limits raised for larger schemas — OpenAI community
- Responses — OpenAI API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.