Diagnostic LLM APIs
The model refused and the refusal field was never read
The support summariser stopped producing summaries for a particular kind of ticket. Not all of them, and not with an error: the row was written, the summary column held an empty string, and the dashboard that counts summaries counted them. It took a week and a customer complaint before anyone read a stored response end to end, and there it was, in a content item nobody's code had ever touched: the model had declined, politely, and said why.
Look at the content types before you look at the content. With a project key set to Read Only: GET /v1/responses/{response_id}, then scan output[].content[] for an item whose type is "refusal" and read its refusal string. On Chat Completions the same event is a non-null choices[0].message.refusal with message.content set to null, which is why .parsed comes back as None.
Structured Outputs gives a refusal its own channel precisely so it does not have to be squeezed into your schema. That is the right design, and it is also why a parser that reaches straight for the text or the parsed object misses it completely. The request succeeded. The model answered. The answer was no.
Check the adjacent case in the same pass: status: "incomplete" with incomplete_details.reason == "content_filter". That is the platform halting generation rather than the model declining, and it needs the same caller-facing handling with different metrics behind it.
Then stop looking at single responses. One refusal is not a finding. A refusal rate per prompt template is, which is why the script groups by metadata before it judges and refuses to publish a rate for a group too small to have one.
The problem in plain words
A refusal is the only failure in this batch where everything worked. The input was read and billed, generation happened, the schema was respected in the sense that the model declined to produce a fabricated object rather than producing a bad one, and the response completed. There is no status code to catch, no reason field to branch on, and nothing anywhere in the response that looks like an error.
So the damage is quiet and it is data-shaped. Code that reaches for the parsed object gets None and either crashes on the attribute access or, far worse, writes an empty record and moves on. Empty records do not page anyone. They accumulate, they look like a source-data problem, and they are usually discovered by a human noticing that a particular category of thing never has a summary.
Why it happens
A refusal is not an error and treating it as one costs money. Retry logic that fires on empty output re-sends the same prompt and is declined again, at full price, as many times as the policy allows. The model was not flaky; it made a decision, and it will make the same decision. The only sane response is to surface the refusal text to whoever asked.
It is not a truncation either, and that is the confusion that costs a day. A cut answer and a refusal both return 200 with nothing your parser can use. One means the model was interrupted and the ceiling is the thing to change; the other means the model declined and the input is the thing to look at. The script reports them as different states so the reader does not go tuning a token budget to fix a policy decision.
The dangerous shape is a refusal with a preamble. When the turn produces some text and then refuses, a reader that concatenates output items ends up with a plausible-looking fragment and stores it as the answer. That is worse than an empty record, because it is wrong rather than missing, and nothing downstream can tell. The script reports it under its own state for exactly that reason.
A filter stop is a different event with the same user-facing shape. incomplete_details.reason == "content_filter" is the platform stopping the turn, not the model declining it. Users need the same message; you need different metrics, because the two move for different reasons and folding them together hides both.
The number worth having is a rate per prompt, not a count. Refusal rates are meaningful per template, because a template is the thing you can change. A template that refuses one call in three usually has a bad instruction or a bad input source feeding it, not bad users. The script therefore groups by whatever you put in metadata and falls back to the model id, which is the least useful grouping that is still true.
A rate from a handful of calls is a rumour. One refusal in one response is a hundred percent, and a report that says so is a report people learn to ignore within a fortnight. Below the floor the script counts and withholds the rate rather than printing a number it cannot stand behind.
The fix, as a flow
A refusal is not an error and it is not a truncation. The model was asked, it declined, and it said so in a content type built for the purpose so the refusal would not have to be squeezed into your schema. Everything about that works. The failure is one step later, in a parser that reaches straight for the text it expected and finds a channel it has never been taught to look at.
How to fix it
Gather the ids and expect some of them to be gone
Stored responses are reachable only by id and only if the call stored them at all; they also expire. A 404 here is not a finding, it is the retention window, and the script says so rather than counting it as a clean response.
Scan content types, not content
Walk output[].content[] and look at type on each item. refusal is a peer of output_text, not a variant of it. Read the Chat Completions shape in the same pass: message.refusal alongside a null message.content is the same event on the older surface, and a half-migrated codebase has both in its records.
Separate a refusal from a filter stop and from a truncation
Three different states, three different owners. A refusal points at the prompt and the input; a filter stop points at the platform; a truncation points at the ceiling. Reporting them under one heading is how a team ends up changing all three things at once and learning nothing.
Group by prompt template before counting anything
Tag your calls with a template name in metadata and this becomes sharp. Without tags the script groups by model id, which is honest and nearly useless: it will tell you that refusals happen, not which prompt causes them.
Print the rate only where there is enough of it
Below the floor, count and say the count. Above it, print a percentage. The point of the floor is that this report gets read every week for a year, and a report that cries wolf on a single call does not survive that.
How to check it worked
After the refusal branch is added, re-run. The refusals do not go away — they are not a bug — but the empty records do, and the rate per template becomes a number you can watch.
python3 openai_refusal_channel.py --ids response_ids.txt
# refused resp_68d1f0 Completed with a refusal and no answer: "I'm sorry, I can't help with that.". There is nothing to parse and nothing went wrong.
# repair: Handle refusal as a first-class branch before parsing: surface the refusal text to the caller and do not attempt schema parsing at all.
# group kyc-extract 30.0% of 30 response(s) refused or filtered
# 240 response(s) checked, 9 refused or filteredThe full code
One GET per id and no second call at all: everything this note needs is inside the stored object. Seven pure functions. refusals reads the channel itself on both API surfaces and returns the text rather than a boolean, because showing the reader what the model actually said is most of the value. visible_text is deliberately not called "the answer", since on a refused turn it is either empty or a preamble and treating it as the answer is the bug. Then the stop-reason reader that keeps a truncation out of this note, the grouping key with its honest fallback, the classifier, the rate function that withholds a percentage below the floor, and the repair lines.
"""Find stored OpenAI responses that carry a refusal nobody read.
Read only. GET /v1/responses/{response_id} for each id you supply, with a
project key set to Read Only. There is no list endpoint for /v1/responses, so
the ids come from your own records: one id per line in a file.
Structured Outputs gives a safety refusal its own content type so it does not
have to be squeezed into your schema. That is the right design and it is also
why a parser reaching straight for the text finds nothing: the refusal is not
an error, not a truncation, and not schema-shaped. The response completed.
One refusal is not a finding. A refusal rate per prompt template is, which is
why this script groups before it judges: a template that refuses one call in
three has a bad input source or a bad instruction, not bad users.
The repair is printed, never performed.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_refusal_channel")
API = "https://api.openai.com/v1"
# States this note owns. "truncated" is a handoff: an answer that was cut short
# is a different note with a different repair.
FINDINGS = ("refused", "refused-after-partial", "stopped-by-filter")
# Below this many responses in a group, a rate is a rumour.
GROUP_FLOOR = 20
def refusals(response):
"""Every refusal carried by a stored response. Pure.
Returns dicts with the output index and the refusal text, so a caller can
show the reader what the model actually said rather than the fact that it
said something. Both surfaces: the Responses API puts a refusal content
item in output[], Chat Completions puts a string on message.refusal.
"""
found = []
response = response or {}
for index, item in enumerate(response.get("output") or []):
for content in item.get("content") or []:
if content.get("type") == "refusal":
found.append({"index": index,
"text": str(content.get("refusal") or "").strip()})
for index, choice in enumerate(response.get("choices") or []):
text = (choice.get("message") or {}).get("refusal")
if text:
found.append({"index": index, "text": str(text).strip()})
return found
def visible_text(response):
"""The text a parser would have reached for. Pure.
Deliberately not "the answer": on a refused turn this is empty or a partial
preamble, and the whole bug is that the calling code treats emptiness as a
transport problem rather than as a decision the model made.
"""
parts = []
response = response or {}
for item in response.get("output") or []:
for content in item.get("content") or []:
if content.get("type") in ("output_text", "text"):
parts.append(str(content.get("text") or ""))
for choice in response.get("choices") or []:
text = (choice.get("message") or {}).get("content")
if isinstance(text, str):
parts.append(text)
return "".join(parts).strip()
def stop_reason(response):
"""Why the response stopped, in one vocabulary. Pure. None when it did not."""
response = response or {}
if str(response.get("status") or "") == "incomplete":
return str((response.get("incomplete_details") or {}).get("reason") or "unknown")
for choice in response.get("choices") or []:
finish = str(choice.get("finish_reason") or "")
if finish == "length":
return "max_output_tokens"
if finish == "content_filter":
return "content_filter"
return None
def group_key(response):
"""What to count refusals against. Pure.
A refusal rate is only interesting per prompt, so metadata wins over the
model id. Tag your calls and this script gets sharper; do not, and it still
works at model granularity, which is the least useful grouping that is
still true.
"""
response = response or {}
metadata = response.get("metadata") or {}
for field in ("template", "prompt_template", "prompt_id", "use_case"):
value = metadata.get(field)
if value:
return str(value)
prompt = response.get("prompt") or {}
if prompt.get("id"):
return "prompt:" + str(prompt["id"])
return "model:" + str(response.get("model") or "unknown")
def classify(response):
"""Classify one stored response. Pure. Returns (state, detail).
The distinction that costs people a day is refusal against truncation. Both
return 200 with no usable payload. One means the model declined and the
input is the thing to look at; the other means the model was interrupted
and the ceiling is the thing to look at.
"""
response = response or {}
declined = refusals(response)
text = visible_text(response)
reason = stop_reason(response)
if declined:
said = declined[0]["text"] or "(the refusal string was empty)"
if text:
return ("refused-after-partial",
"The turn produced %d character(s) of text and then "
"refused: %r. A reader that concatenates output items ends "
"up storing the preamble as if it were the answer."
% (len(text), said))
return ("refused",
"Completed with a refusal and no answer: %r. There is nothing "
"to parse and nothing went wrong." % said)
if reason == "content_filter":
return ("stopped-by-filter",
"Incomplete because the content filter halted generation. That "
"is the platform stopping the turn, not the model declining "
"it, and the two are worth separating in your metrics.")
if reason == "max_output_tokens":
return ("truncated",
"Incomplete because the output ceiling was reached. Nothing "
"was refused. Read the truncation note.")
if reason is not None:
return ("incomplete-other",
"Incomplete for reason %r, which is neither a refusal nor a "
"ceiling." % reason)
if not text:
return ("empty-answer",
"Completed, no refusal, and no text either. Check whether the "
"output items are a tool call rather than a message.")
return ("answered", "Completed with %d character(s) of text." % len(text))
def refusal_rate(rows, floor=GROUP_FLOOR):
"""Refusal rate per group. Pure. Rows are (group, state) pairs.
Returns rate None below the floor rather than a number, because one refusal
in one call is 100% and reporting it that way trains people to ignore the
report. Counting is still done, so a small group grows into a real one.
"""
totals = {}
for group, state in rows or []:
row = totals.setdefault(str(group), {"total": 0, "refused": 0,
"filtered": 0, "rate": None})
row["total"] += 1
if state in ("refused", "refused-after-partial"):
row["refused"] += 1
elif state == "stopped-by-filter":
row["filtered"] += 1
for row in totals.values():
if row["total"] >= floor:
row["rate"] = (row["refused"] + row["filtered"]) / float(row["total"])
return totals
def repair_lines(state):
"""The repair for one state. Pure."""
if state in ("refused", "refused-after-partial"):
return ["Handle refusal as a first-class branch before parsing: if any "
"output content item has type refusal, surface the refusal text "
"to the caller and do not attempt schema parsing at all.",
"Never treat an empty parsed value as a transport failure. A "
"refusal is a completed answer and retrying it unchanged spends "
"money to be told no again.",
"Log the refusal rate per prompt template. A spike is almost "
"always a prompt change or a bad input source, not a change in "
"who your users are."]
if state == "stopped-by-filter":
return ["Branch on incomplete_details.reason as well as on the refusal "
"content type. A filter stop is the platform halting the turn "
"and it needs the same caller-facing message as a refusal.",
"Count filter stops separately from model refusals. They move "
"for different reasons and folding them together hides both."]
if state == "truncated":
return ["Not a refusal. Check the output ceiling before you look at the "
"prompt: the model was interrupted, not unwilling."]
if state == "empty-answer":
return ["Not a refusal either. Inspect the output item types before "
"concluding anything: a function call is not a message."]
return []
def read_ids(path):
"""One response id per line, blanks and # comments ignored."""
ids = []
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if line and not line.startswith("#"):
ids.append(line)
return ids
def fetch_response(session, response_id):
r = session.get(API + "/responses/" + response_id, timeout=60)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: this needs a project key that can "
"read stored responses" % r.status_code)
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ids", required=True,
help="file of stored response ids, one per line")
ap.add_argument("--floor", type=int, default=GROUP_FLOOR,
help="responses a group needs before a rate is printed")
ap.add_argument("--show-all", action="store_true",
help="also print responses that were answered normally")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY, a project key set to Read Only")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
rows = []
checked = 0
bad = 0
for response_id in read_ids(args.ids):
stored = fetch_response(session, response_id)
checked += 1
if stored is None:
log.warning("%-22s %s not found. Stored responses expire, and a "
"response created without storage was never readable.",
"unreadable", response_id)
continue
state, detail = classify(stored)
rows.append((group_key(stored), state))
line = "%-22s %s %s" % (state, response_id, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
for repair in repair_lines(state):
log.warning(" repair: %s", repair)
elif state == "answered":
if args.show_all:
log.info(line)
else:
log.info(line)
for repair in repair_lines(state):
log.info(" note: %s", repair)
rates = refusal_rate(rows, args.floor)
for group in sorted(rates):
row = rates[group]
if row["rate"] is None:
log.info("%-22s %s %d response(s), under the floor of %d so no "
"rate is claimed", "group", group, row["total"], args.floor)
else:
log.warning("%-22s %s %.1f%% of %d response(s) refused or filtered",
"group", group, row["rate"] * 100, row["total"])
log.info("%d response(s) checked, %d refused or filtered", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find stored OpenAI responses that carry a refusal nobody read.
*
* Read only. GET /v1/responses/{response_id} for each id you supply, with a
* project key set to Read Only. There is no list endpoint for /v1/responses,
* so the ids come from your own records: one id per line in a file.
*
* Structured Outputs gives a safety refusal its own content type so it does
* not have to be squeezed into your schema. The response completed, nothing
* errored, and the field a parser reaches for is simply not where the answer
* went. One refusal is not a finding; a rate per prompt template is.
*
* The repair is printed, never performed.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.openai.com/v1';
const FINDINGS = new Set(['refused', 'refused-after-partial', 'stopped-by-filter']);
const GROUP_FLOOR = 20;
/** Every refusal carried by a stored response. Pure. Both surfaces. */
export function refusals(response) {
const found = [];
(response?.output ?? []).forEach((item, index) => {
for (const content of item?.content ?? []) {
if (content?.type === 'refusal') {
found.push({ index, text: String(content.refusal ?? '').trim() });
}
}
});
(response?.choices ?? []).forEach((choice, index) => {
const text = choice?.message?.refusal;
if (text) found.push({ index, text: String(text).trim() });
});
return found;
}
/** The text a parser would have reached for. Pure. Not "the answer". */
export function visibleText(response) {
const parts = [];
for (const item of response?.output ?? []) {
for (const content of item?.content ?? []) {
if (content?.type === 'output_text' || content?.type === 'text') {
parts.push(String(content.text ?? ''));
}
}
}
for (const choice of response?.choices ?? []) {
const text = choice?.message?.content;
if (typeof text === 'string') parts.push(text);
}
return parts.join('').trim();
}
/** Why the response stopped, in one vocabulary. Pure. Null when it did not. */
export function stopReason(response) {
if (String(response?.status ?? '') === 'incomplete') {
return String(response?.incomplete_details?.reason ?? 'unknown');
}
for (const choice of response?.choices ?? []) {
const finish = String(choice?.finish_reason ?? '');
if (finish === 'length') return 'max_output_tokens';
if (finish === 'content_filter') return 'content_filter';
}
return null;
}
/** What to count refusals against. Pure. Metadata wins over the model id. */
export function groupKey(response) {
const metadata = response?.metadata ?? {};
for (const field of ['template', 'prompt_template', 'prompt_id', 'use_case']) {
if (metadata[field]) return String(metadata[field]);
}
if (response?.prompt?.id) return `prompt:${response.prompt.id}`;
return `model:${String(response?.model ?? 'unknown')}`;
}
/** Classify one stored response. Pure. Refusal against truncation is the split. */
export function classify(response) {
const declined = refusals(response);
const text = visibleText(response);
const reason = stopReason(response);
if (declined.length) {
const said = declined[0].text || '(the refusal string was empty)';
if (text) {
return ['refused-after-partial',
`The turn produced ${text.length} character(s) of text and then ` +
`refused: '${said}'. A reader that concatenates output items ends up ` +
'storing the preamble as if it were the answer.'];
}
return ['refused',
`Completed with a refusal and no answer: '${said}'. There is nothing to ` +
'parse and nothing went wrong.'];
}
if (reason === 'content_filter') {
return ['stopped-by-filter',
'Incomplete because the content filter halted generation. That is the ' +
'platform stopping the turn, not the model declining it, and the two are ' +
'worth separating in your metrics.'];
}
if (reason === 'max_output_tokens') {
return ['truncated',
'Incomplete because the output ceiling was reached. Nothing was refused. ' +
'Read the truncation note.'];
}
if (reason !== null) {
return ['incomplete-other',
`Incomplete for reason '${reason}', which is neither a refusal nor a ceiling.`];
}
if (!text) {
return ['empty-answer',
'Completed, no refusal, and no text either. Check whether the output ' +
'items are a tool call rather than a message.'];
}
return ['answered', `Completed with ${text.length} character(s) of text.`];
}
/**
* Refusal rate per group. Pure. Rows are [group, state] pairs.
* Rate stays null below the floor: one refusal in one call is 100%, and
* printing that teaches people to ignore the report.
*/
export function refusalRate(rows, floor = GROUP_FLOOR) {
const totals = new Map();
for (const [group, state] of rows ?? []) {
const key = String(group);
if (!totals.has(key)) {
totals.set(key, { total: 0, refused: 0, filtered: 0, rate: null });
}
const row = totals.get(key);
row.total += 1;
if (state === 'refused' || state === 'refused-after-partial') row.refused += 1;
else if (state === 'stopped-by-filter') row.filtered += 1;
}
for (const row of totals.values()) {
if (row.total >= floor) row.rate = (row.refused + row.filtered) / row.total;
}
return totals;
}
/** The repair for one state. Pure. */
export function repairLines(state) {
if (state === 'refused' || state === 'refused-after-partial') {
return ['Handle refusal as a first-class branch before parsing: if any output ' +
'content item has type refusal, surface the refusal text to the caller and ' +
'do not attempt schema parsing at all.',
'Never treat an empty parsed value as a transport failure. A refusal is a ' +
'completed answer and retrying it unchanged spends money to be told no again.',
'Log the refusal rate per prompt template. A spike is almost always a prompt ' +
'change or a bad input source, not a change in who your users are.'];
}
if (state === 'stopped-by-filter') {
return ['Branch on incomplete_details.reason as well as on the refusal content ' +
'type. A filter stop is the platform halting the turn and it needs the same ' +
'caller-facing message as a refusal.',
'Count filter stops separately from model refusals. They move for different ' +
'reasons and folding them together hides both.'];
}
if (state === 'truncated') {
return ['Not a refusal. Check the output ceiling before you look at the prompt: ' +
'the model was interrupted, not unwilling.'];
}
if (state === 'empty-answer') {
return ['Not a refusal either. Inspect the output item types before concluding ' +
'anything: a function call is not a message.'];
}
return [];
}
async function fetchResponse(key, responseId) {
const res = await fetch(`${API}/responses/${responseId}`,
{ headers: { Authorization: `Bearer ${key}` } });
if (res.status === 404) return null;
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: this needs a project key that ` +
'can read stored responses');
}
if (!res.ok) throw new Error(`${res.status} from /responses/${responseId}`);
return res.json();
}
async function main() {
const key = process.env.OPENAI_API_KEY;
const idsFile = process.env.RESPONSE_IDS;
if (!key || !idsFile) {
console.error('set OPENAI_API_KEY (a project key set to Read Only) and ' +
'RESPONSE_IDS (a file of stored response ids, one per line)');
process.exitCode = 2;
return;
}
const floor = Number(process.env.FLOOR ?? GROUP_FLOOR);
const showAll = process.env.SHOW_ALL === '1';
const ids = (await readFile(idsFile, 'utf8')).split('\n')
.map((l) => l.trim()).filter((l) => l && !l.startsWith('#'));
const rows = [];
let checked = 0;
let bad = 0;
for (const responseId of ids) {
const stored = await fetchResponse(key, responseId);
checked += 1;
if (stored === null) {
console.warn(`${'unreadable'.padEnd(22)} ${responseId} not found. Stored ` +
'responses expire, and a response created without storage was never readable.');
continue;
}
const [state, detail] = classify(stored);
rows.push([groupKey(stored), state]);
const line = `${state.padEnd(22)} ${responseId} ${detail}`;
if (FINDINGS.has(state)) {
bad += 1;
console.warn(line);
for (const repair of repairLines(state)) console.warn(` repair: ${repair}`);
} else if (state === 'answered') {
if (showAll) console.log(line);
} else {
console.log(line);
for (const repair of repairLines(state)) console.log(` note: ${repair}`);
}
}
const rates = refusalRate(rows, floor);
for (const group of [...rates.keys()].sort()) {
const row = rates.get(group);
if (row.rate === null) {
console.log(`${'group'.padEnd(22)} ${group} ${row.total} response(s), ` +
`under the floor of ${floor} so no rate is claimed`);
} else {
console.warn(`${'group'.padEnd(22)} ${group} ${(row.rate * 100).toFixed(1)}% ` +
`of ${row.total} response(s) refused or filtered`);
}
}
console.log(`${checked} response(s) checked, ${bad} refused or filtered`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first test is the note itself: a completed response, no stop reason at all, no text, and a refusal item carrying a sentence. The second is the shape that does real damage — a preamble followed by a refusal, where naive concatenation produces something that looks like an answer. Then the Chat Completions form, a filter stop that has to stay a separate state, and a truncated response that has to be handed away to the other note rather than counted here. The last two pin the arithmetic: a rate per template over thirty calls, and a single refusal that must not be published as a hundred percent.
from openai_refusal_channel import (classify, group_key, refusal_rate,
refusals, repair_lines, stop_reason,
visible_text)
def refused(text="I'm sorry, I can't help with that.", preamble=None,
metadata=None):
content = []
if preamble:
content.append({"type": "output_text", "text": preamble})
content.append({"type": "refusal", "refusal": text})
return {"id": "resp_r", "status": "completed", "model": "gpt-5.1",
"metadata": metadata or {},
"output": [{"type": "message", "content": content}]}
def answered(text='{"ok": true}', metadata=None):
return {"id": "resp_a", "status": "completed", "model": "gpt-5.1",
"metadata": metadata or {},
"output": [{"type": "message",
"content": [{"type": "output_text", "text": text}]}]}
def test_a_refusal_is_a_completed_answer_with_nothing_to_parse():
# The note in one assertion: 200, status completed, and the payload the
# parser wanted is simply not the thing the model returned.
response = refused()
assert stop_reason(response) is None
assert visible_text(response) == ""
assert refusals(response) == [{"index": 0,
"text": "I'm sorry, I can't help with that."}]
state, detail = classify(response)
assert state == "refused"
assert "nothing went wrong" in detail
assert "first-class branch before parsing" in repair_lines(state)[0]
def test_a_refusal_that_follows_a_preamble_is_not_an_answer_either():
# The dangerous shape: concatenating the output items produces text, so a
# naive reader stores the preamble as though it were the record.
response = refused(preamble="Here is what I found so far. ")
state, detail = classify(response)
assert state == "refused-after-partial"
assert "storing the preamble" in detail
assert visible_text(response) == "Here is what I found so far."
def test_the_chat_completions_shape_is_read_as_well():
legacy = {"choices": [{"finish_reason": "stop",
"message": {"content": None,
"refusal": "I can't assist with that."}}]}
assert refusals(legacy)[0]["text"] == "I can't assist with that."
assert visible_text(legacy) == ""
assert classify(legacy)[0] == "refused"
def test_a_filter_stop_is_counted_apart_from_a_model_refusal():
filtered = {"status": "incomplete",
"incomplete_details": {"reason": "content_filter"},
"output": []}
state, detail = classify(filtered)
assert state == "stopped-by-filter"
assert "not the model declining it" in detail
assert "separately from model refusals" in repair_lines(state)[1]
def test_a_truncated_response_is_handed_to_the_other_note():
# Same 200, same missing payload, and the repair is a ceiling rather than
# a prompt. Getting these two confused costs an afternoon.
cut = {"status": "incomplete",
"incomplete_details": {"reason": "max_output_tokens"},
"output": [{"type": "message",
"content": [{"type": "output_text", "text": '{"a": 1'}]}]}
state, detail = classify(cut)
assert state == "truncated"
assert "Nothing was refused" in detail
assert "interrupted, not unwilling" in repair_lines(state)[0]
def test_the_rate_is_grouped_by_template_and_withheld_below_the_floor():
rows = ([(group_key(refused(metadata={"template": "kyc-extract"})), "refused")] * 9
+ [(group_key(answered(metadata={"template": "kyc-extract"})), "answered")] * 21
+ [(group_key(refused(metadata={"template": "rare-path"})), "refused")])
rates = refusal_rate(rows)
assert set(rates) == {"kyc-extract", "rare-path"}
assert rates["kyc-extract"]["total"] == 30
assert rates["kyc-extract"]["refused"] == 9
assert abs(rates["kyc-extract"]["rate"] - 0.3) < 1e-9
# One refusal in one call is 100%, and printing that teaches people to
# ignore the report.
assert rates["rare-path"]["total"] == 1
assert rates["rare-path"]["rate"] is None
def test_grouping_falls_back_without_pretending_it_is_sharp():
assert group_key(refused(metadata={"template": "kyc-extract"})) == "kyc-extract"
assert group_key({"prompt": {"id": "pmpt_9"}}) == "prompt:pmpt_9"
assert group_key({"model": "gpt-5.1"}) == "model:gpt-5.1"
assert group_key({}) == "model:unknown"
assert group_key(None) == "model:unknown"
def test_normal_and_empty_responses_are_left_alone():
assert classify(answered())[0] == "answered"
assert refusals(answered()) == []
assert refusals(None) == []
assert classify({"status": "completed", "output": []})[0] == "empty-answer"
assert refusal_rate([]) == {}
assert refusal_rate(None) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
classify, groupKey, refusalRate, refusals, repairLines, stopReason, visibleText,
} from './openai-refusal-channel.mjs';
const refused = ({ text = "I'm sorry, I can't help with that.",
preamble = null, metadata = {} } = {}) => {
const content = [];
if (preamble) content.push({ type: 'output_text', text: preamble });
content.push({ type: 'refusal', refusal: text });
return { id: 'resp_r', status: 'completed', model: 'gpt-5.1', metadata,
output: [{ type: 'message', content }] };
};
const answered = ({ text = '{"ok": true}', metadata = {} } = {}) => ({
id: 'resp_a', status: 'completed', model: 'gpt-5.1', metadata,
output: [{ type: 'message', content: [{ type: 'output_text', text }] }],
});
test('a refusal is a completed answer with nothing to parse', () => {
const response = refused();
assert.equal(stopReason(response), null);
assert.equal(visibleText(response), '');
assert.deepEqual(refusals(response),
[{ index: 0, text: "I'm sorry, I can't help with that." }]);
const [state, detail] = classify(response);
assert.equal(state, 'refused');
assert.match(detail, /nothing went wrong/);
assert.match(repairLines(state)[0], /first-class branch before parsing/);
});
test('a refusal that follows a preamble is not an answer either', () => {
const response = refused({ preamble: 'Here is what I found so far. ' });
const [state, detail] = classify(response);
assert.equal(state, 'refused-after-partial');
assert.match(detail, /storing the preamble/);
assert.equal(visibleText(response), 'Here is what I found so far.');
});
test('the chat completions shape is read as well', () => {
const legacy = { choices: [{ finish_reason: 'stop',
message: { content: null, refusal: "I can't assist with that." } }] };
assert.equal(refusals(legacy)[0].text, "I can't assist with that.");
assert.equal(visibleText(legacy), '');
assert.equal(classify(legacy)[0], 'refused');
});
test('a filter stop is counted apart from a model refusal', () => {
const filtered = { status: 'incomplete',
incomplete_details: { reason: 'content_filter' }, output: [] };
const [state, detail] = classify(filtered);
assert.equal(state, 'stopped-by-filter');
assert.match(detail, /not the model declining it/);
assert.match(repairLines(state)[1], /separately from model refusals/);
});
test('a truncated response is handed to the other note', () => {
const cut = { status: 'incomplete',
incomplete_details: { reason: 'max_output_tokens' },
output: [{ type: 'message', content: [{ type: 'output_text', text: '{"a": 1' }] }] };
const [state, detail] = classify(cut);
assert.equal(state, 'truncated');
assert.match(detail, /Nothing was refused/);
assert.match(repairLines(state)[0], /interrupted, not unwilling/);
});
test('the rate is grouped by template and withheld below the floor', () => {
const rows = [];
for (let i = 0; i < 9; i += 1) rows.push(['kyc-extract', 'refused']);
for (let i = 0; i < 21; i += 1) rows.push(['kyc-extract', 'answered']);
rows.push(['rare-path', 'refused']);
const rates = refusalRate(rows);
assert.deepEqual([...rates.keys()].sort(), ['kyc-extract', 'rare-path']);
assert.equal(rates.get('kyc-extract').total, 30);
assert.equal(rates.get('kyc-extract').refused, 9);
assert.ok(Math.abs(rates.get('kyc-extract').rate - 0.3) < 1e-9);
assert.equal(rates.get('rare-path').total, 1);
assert.equal(rates.get('rare-path').rate, null);
});
test('grouping falls back without pretending it is sharp', () => {
assert.equal(groupKey(refused({ metadata: { template: 'kyc-extract' } })), 'kyc-extract');
assert.equal(groupKey({ prompt: { id: 'pmpt_9' } }), 'prompt:pmpt_9');
assert.equal(groupKey({ model: 'gpt-5.1' }), 'model:gpt-5.1');
assert.equal(groupKey({}), 'model:unknown');
assert.equal(groupKey(null), 'model:unknown');
});
test('normal and empty responses are left alone', () => {
assert.equal(classify(answered())[0], 'answered');
assert.deepEqual(refusals(answered()), []);
assert.deepEqual(refusals(null), []);
assert.equal(classify({ status: 'completed', output: [] })[0], 'empty-answer');
assert.equal(refusalRate([]).size, 0);
assert.equal(refusalRate(null).size, 0);
});
FAQ
Is a refusal a failure I should alert on?
Not per call. A refusal is a legitimate outcome and alerting on each one produces a channel nobody reads. The rate per prompt template is the thing to watch: a step change there means a prompt shipped, an input source changed, or a new category of request started arriving. That is worth a look. One refusal is worth a log line.
Why is message.content null instead of holding the refusal text?
Because the refusal is deliberately kept out of the content channel. If it were text, a schema-shaped parse would either fail confusingly or, worse, succeed on something that was never a real answer. Putting it in its own field is what makes it detectable at all. The cost is that code written before the field existed cannot see it.
How is this different from the response being cut off?
A truncation was interrupted mid-answer and reports a reason for stopping; a refusal completed and reports nothing wrong at all. The repairs have nothing in common: one is a token ceiling, the other is the prompt or the input. Both look identical from a parser, which is why the script names them separately before anything else happens.
Does Anthropic have the same field?
Not the same shape. Claude reports a declined answer through stop_reason rather than through a distinct content type, so the equivalent check there is on the stop reason of the message. The failure mode is the same one: code that reads content without first reading why the model stopped.
Should I retry a refusal with a different prompt?
Automatically, no. Re-sending the same prompt is money spent to be told no again, and rewriting the prompt in a retry loop means your production behaviour is a prompt nobody reviewed. Surface the refusal, count it against the template, and change the template deliberately if the rate says it needs changing.
Related field notes
- A 200 whose JSON stops mid-object because the ceiling arrived
- A schema that was only ever a suggestion
- When the expensive half of the bill is the half you generate
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Structured Outputs — OpenAI developer docs
- Responses — OpenAI API reference
- Structured outputs — Azure OpenAI, Microsoft Learn
- Handling stop reasons — Claude API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.