Diagnostic LLM APIs
Tool shipped on every request and never once called
The tool registry has twenty-six entries. Nineteen of them were written in the same fortnight by four people, and the descriptions read like function signatures because that is what they were copied from. Every one of them goes out on every request, because the registry is built once at start-up and handed to the client whole. Nothing is broken. The handlers for six of those tools have not been entered in production since the day they were merged, and the tools are still being paid for on every turn.
Build two sets from a sample of stored responses and subtract. With a project key that can read them: GET /v1/responses/{response_id}. The response object echoes the tools the request declared, and its output[] array carries the items of type: "function_call" the model produced. Every name in the first that never appears in the second, across a large enough sample, is a definition you pay for and never use.
There is no list endpoint. /v1/responses cannot be enumerated, so the ids have to come from your own request log, and the sample has to be big enough that "never" means something. A few hundred turns is a finding; nine is a coincidence.
Split never called from never offered before you conclude anything. A response sent with tool_choice: "none", or with a named tool, never gave the model the chance, and counting those turns as evidence condemns tools that were simply not on the table. This script counts a tool as offered only in the turns where the model was free to pick it.
The repair is a rewrite before it is a deletion. A tool that is never selected usually has a description that says what it is rather than when to call it. If a call is mandatory, say so with tool_choice: "required" or a named tool rather than hoping.
The problem in plain words
Tool definitions are prompt. Names, descriptions and the whole JSON schema are serialised into the request and billed as input tokens on every single call, and unlike a system prompt nobody thinks of them as text at all — they are code, they live in a registry module, and they get added the way a route gets added. Nothing in the API charges you differently for a tool that is used and one that is not.
What makes it invisible is that the failure mode is an absence on both sides. The tool never fires, so its handler emits no logs, no metrics and no errors; a dashboard built on the handler shows a flat zero that looks exactly like a quiet week. And the cost never spikes, because it was there from the first deploy: a fixed addition to every request is a baseline, and baselines do not alert. The capability you thought you shipped is missing, and the only trace is a name in a request payload that no one reads.
Why it happens
Never called and never offered are different findings with different repairs. A tool that appears in tools on two hundred turns and was free to be chosen on none of them — because tool_choice was "none", or named another tool every time — has not been ignored by the model. It has been ruled out by your own request. Rewriting its description will change nothing. The script tracks a separate offered count per tool for exactly this, and refuses to call a tool dead on turns where it was never in the running.
Crowding is a real cause and it is the one nobody suspects. The guidance is to keep fewer than twenty tools available at the start of a turn. Past that, selection quality falls, and it falls unevenly: the tools with the vaguest descriptions lose first. A registry that grew to forty is not a configuration problem, it is a prompt problem, and the fix is allowed_tools or a per-turn subset rather than a better description on any one entry.
A description that says what a tool is reads as documentation and selects badly. "Looks up an order" is a signature. "Call this when the user asks about the status, contents or delivery date of an order they have already placed; do not call it for refunds" is a selection rule. The model is choosing between twenty-six of these under a token budget, and the ones that read like a rule win.
This note counts names; the cost of those names is a separate measurement. A character count of a JSON schema is not a token count, and printing one as if it were the other is worse than printing nothing. The script reports the share of the declared schema, in characters, that belongs to tools nobody calls, and says plainly that the token price belongs to the token-overhead note, which measures it exactly and for free.
"Never in this sample" is the only claim available, and the script says so. There is no list endpoint for stored responses and no aggregate that counts tool selections, so every statement here is bounded by the ids you supplied. A tool called once a month by one support workflow will look dead in a day of traffic. The output always carries the sample size beside the verdict, and a tool that was offered fewer than fifty times comes back as insufficient evidence rather than as a finding.
The fix, as a flow
Nothing here errors and nothing here is slow. A tool definition is part of the prompt, so it is re-sent and re-billed on every turn, and the only evidence that the model never picks it is a name that is present in the request and absent from the output. The set difference is the whole finding, and it needs a sample rather than a single call.
How to fix it
Collect a sample of stored response ids
/v1/responses has no list endpoint, so the ids come from your own request log. Sample across a full week rather than an afternoon: a tool used by one weekly workflow is dead on Tuesday and alive on Sunday. Responses have to have been stored in the first place — store defaults to true on the Responses API, but a client that turned it off leaves nothing to read.
Read each response and take the two sets
GET /v1/responses/{response_id}. The declared names come from tools[], handling both shapes: the Responses API puts name at the top level of the tool object, Chat Completions nests it under function. The called names come from output[] items whose type is function_call.
Count offers separately from declarations
Read tool_choice on each response. "none" means no tool was on the table in that turn and it counts for nothing. A named tool means only that one was on the table. "auto", "required" and an absent field mean every declared tool was in the running. Only the last group is evidence.
Sort by call count and read the zeros against the sample size
A tool offered four hundred times and never chosen is a finding. A tool offered eleven times and never chosen is nothing at all, and the script says so rather than padding the list. The rare bucket matters too: one call in five hundred turns is a tool to keep and to stop sending on every turn.
Print the repair per tool, not per registry
Rewrite the description as a selection rule; narrow the turn with allowed_tools so the model chooses among five rather than forty; force the call with tool_choice: "required" or a named tool where a call is mandatory; delete what nothing needs. Then measure what the surviving block weighs, because pruning six of twenty-six tools is a smaller saving than it feels like.
How to check it worked
Re-run on a fresh week of ids after the descriptions change. A tool that moves from zero to a handful of calls was a description problem; one that stays at zero after being offered freely a thousand times is a deletion.
python3 openai_dead_tool_definitions.py --responses ids.txt
# never-called escalate_to_human offered in 412 of 412 turn(s), called 0 time(s), 1180 schema char(s)
# repair: the description reads like a signature. Say when to call it, not what it is.
# never-offered run_refund declared in 412 turn(s), free to be chosen in 0 of them
# repair: tool_choice never let the model near this one. Fix the request before the description.
# rarely-called lookup_invoice offered in 412 turn(s), called 2 time(s) (0.5%)
# 26 declared tool(s) over 412 response(s), 6 finding(s)
# 31% of the declared schema, in characters, belongs to tools nothing ever calledThe full code
One GET per response id and no aggregate anywhere, because there is no aggregate that counts tool selections. Nine pure functions: the id parser, which is also the guard that stops an arbitrary string being pasted into a URL path; the name reader, which has to cope with both tool shapes; the declared and called readers; the tool_choice reader, which decides whether a turn is evidence at all; the fold; the coverage table; the classifier; the dead-weight share, which is measured in characters and says so; and the crowding check against the twenty-tool guidance.
"""Find OpenAI tool definitions that are sent on every call and never chosen.
Read only. One GET per stored response id, using a project key. No completion
is created and nothing is written; /v1/responses is read, never posted to.
There is no list endpoint for stored responses, so the sample comes from a file
of ids you supply. Every claim this script makes is bounded by that sample and
the output says so: "never called in 412 turns" is the finding, not "never
called".
The repair is printed, never performed. Pruning a tool registry is a deploy.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_dead_tool_definitions")
API = "https://api.openai.com/v1"
# Output item types that represent the model choosing a tool. Anything else in
# output[] is a message, a reasoning item or a hosted tool call, and none of
# those is evidence that one of your function definitions was selected.
CALL_TYPES = ("function_call", "custom_tool_call")
# The documented guidance is fewer than twenty tools available at the start of
# a turn. Past that, selection quality falls and it falls on the vaguest
# descriptions first.
CROWD_CEILING = 20
FINDINGS = ("never-called", "never-offered")
def _int(value):
"""Read a count as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def parse_ids(text):
"""Response ids out of a plain text file. Pure. Order kept, duplicates dropped.
Also the guard that stops an arbitrary line of a file becoming a URL path
segment. Anything that is not a plausible response id is discarded rather
than sent, because a script that interpolates unvalidated text into a
provider URL is one typo away from requesting something else entirely.
"""
out = []
seen = set()
for line in str(text or "").splitlines():
candidate = line.split("#", 1)[0].strip()
if not candidate or not candidate.startswith("resp_"):
continue
if not all(ch.isalnum() or ch in "_-" for ch in candidate):
continue
if candidate in seen:
continue
seen.add(candidate)
out.append(candidate)
return out
def tool_name(tool):
"""The function name out of either tool shape. Pure. None when absent.
The Responses API puts name at the top level of the tool object; Chat
Completions nests it under function. A reader that knows only one shape
reports every tool as undeclared on half the corpus.
"""
if not isinstance(tool, dict):
return None
name = tool.get("name")
if not name and isinstance(tool.get("function"), dict):
name = tool["function"].get("name")
name = str(name or "").strip()
return name or None
def declared_tools(response):
"""Every named tool the request declared, with its serialized size. Pure.
Size in characters, never tokens. Hosted tools carry no name and are
skipped: web search is not a definition you wrote and not one you can prune.
"""
out = {}
for tool in (response or {}).get("tools") or []:
name = tool_name(tool)
if name is None:
continue
try:
size = len(json.dumps(tool, separators=(",", ":"), sort_keys=True))
except (TypeError, ValueError):
size = 0
out[name] = max(out.get(name, 0), size)
return out
def called_tools(response):
"""Tool names the model actually chose in one response, counted. Pure."""
counts = {}
for item in (response or {}).get("output") or []:
if not isinstance(item, dict) or item.get("type") not in CALL_TYPES:
continue
name = str(item.get("name") or "").strip()
if not name:
continue
counts[name] = counts.get(name, 0) + 1
return counts
def choice_mode(response):
"""How free the model was to pick a tool in this turn. Pure.
Returns "free", "blocked", or "named:<tool>". An absent tool_choice is
auto, which is free. This is the difference between a tool the model
ignored and a tool your own request never put on the table.
"""
choice = (response or {}).get("tool_choice")
if choice is None:
return "free"
if isinstance(choice, str):
lowered = choice.strip().lower()
if lowered == "none":
return "blocked"
return "free"
if isinstance(choice, dict):
name = tool_name(choice)
if name:
return "named:" + name
return "free"
return "free"
def fold(responses):
"""Fold a sample of stored responses into one corpus. Pure.
Declarations and offers are counted separately on purpose. A tool declared
on four hundred turns and offered on none of them is not dead weight, it is
a tool_choice that never let the model near it, and the two have nothing in
common as repairs.
"""
corpus = {"sampled": 0, "with_tools": 0, "widest_turn": 0, "calls": 0,
"declared": {}, "offered": {}, "called": {}}
for response in responses or []:
if not isinstance(response, dict):
continue
corpus["sampled"] += 1
declared = declared_tools(response)
calls = called_tools(response)
for name, count in calls.items():
corpus["called"][name] = corpus["called"].get(name, 0) + count
corpus["calls"] += count
if not declared:
continue
corpus["with_tools"] += 1
corpus["widest_turn"] = max(corpus["widest_turn"], len(declared))
mode = choice_mode(response)
for name, size in declared.items():
row = corpus["declared"].setdefault(name, {"turns": 0, "chars": 0})
row["turns"] += 1
row["chars"] = max(row["chars"], size)
if mode == "blocked":
continue
if mode.startswith("named:") and mode[len("named:"):] != name:
continue
corpus["offered"][name] = corpus["offered"].get(name, 0) + 1
return corpus
def coverage(corpus):
"""One row per declared tool. Pure. Least used and most expensive first."""
rows = []
for name, row in ((corpus or {}).get("declared") or {}).items():
rows.append({
"name": name,
"turns": _int(row.get("turns")),
"chars": _int(row.get("chars")),
"offered": _int(((corpus or {}).get("offered") or {}).get(name)),
"calls": _int(((corpus or {}).get("called") or {}).get(name)),
})
rows.sort(key=lambda r: (r["calls"], -r["chars"], r["name"]))
return rows
def orphan_calls(corpus):
"""Names the model called that no sampled request declared. Pure.
Not a fault in the registry: it means the sample mixes two configurations,
and a set difference computed across two configurations is meaningless.
"""
declared = set(((corpus or {}).get("declared") or {}))
return sorted(n for n in ((corpus or {}).get("called") or {}) if n not in declared)
def classify(row, min_offered=50, rare=0.01):
"""Classify one tool's coverage across the sample. Pure. Returns (state, detail)."""
row = row or {}
name = str(row.get("name") or "unknown")
turns = _int(row.get("turns"))
offered = _int(row.get("offered"))
calls = _int(row.get("calls"))
if turns and offered == 0:
return ("never-offered",
"declared in %d turn(s), free to be chosen in 0 of them. "
"tool_choice ruled it out every time, so the model never "
"declined it and rewriting the description changes nothing."
% turns)
if offered < min_offered:
return ("too-small-a-sample",
"offered in %d turn(s), under the floor of %d. Not enough to "
"call anything dead." % (offered, min_offered))
if calls == 0:
return ("never-called",
"offered in %d of %d turn(s), called 0 time(s), %d schema "
"char(s). Sent and billed on every one of those turns."
% (offered, turns, _int(row.get("chars"))))
share = calls / float(offered)
if share < rare:
return ("rarely-called",
"offered in %d turn(s), called %d time(s) (%.1f%%). Worth "
"keeping and worth not sending on every turn."
% (offered, calls, share * 100))
return ("called",
"offered in %d turn(s), called %d time(s) (%.1f%%)."
% (offered, calls, share * 100))
def dead_weight(rows, min_offered=50, rare=0.01):
"""Share of the declared schema, in characters, that nothing ever calls. Pure.
Characters, and the docstring is the place to be blunt about it: this is
not a token count and must never be read as one. Tokens are measured
exactly and for free by the token-overhead note, and a character count
dressed up as a token count is worse than no number at all.
"""
total = 0
dead = 0
for row in rows or []:
chars = _int(row.get("chars"))
total += chars
if classify(row, min_offered, rare)[0] == "never-called":
dead += chars
if total <= 0:
return None
return dead / float(total)
def crowding(widest_turn, ceiling=CROWD_CEILING):
"""What the widest turn in the sample looked like. Pure.
Above the guidance the finding changes shape: the problem is no longer any
one description, it is that the model is choosing among too many at once,
and the repair is a narrower turn rather than better prose.
"""
widest = _int(widest_turn)
if widest <= 0:
return ("no-tools", "no sampled response declared any named tool")
if widest > ceiling:
return ("crowded",
"the widest turn offered %d tools, above the guidance of fewer "
"than %d. Selection quality falls with crowding and it falls on "
"the vaguest descriptions first." % (widest, ceiling))
return ("within-guidance",
"the widest turn offered %d tool(s), inside the guidance of fewer "
"than %d" % (widest, ceiling))
def repair_lines(state, name):
"""The repair for one classified tool. Pure."""
if state == "never-called":
return [
"the description probably reads like a signature. Rewrite it as a "
"selection rule: when to call %s, and when not to." % name,
"if a call is mandatory, say so with tool_choice required or a "
"named tool rather than hoping the model picks it up.",
"if nothing needs it, delete it. It is billed on every turn.",
]
if state == "never-offered":
return [
"tool_choice never let the model near %s. Fix the request before "
"you touch the description." % name,
]
if state == "rarely-called":
return [
"keep %s, but stop sending it on every turn. allowed_tools narrows "
"the set for the turns where it is plausible." % name,
]
return []
def get(session, path):
r = session.get(API + path, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: OPENAI_API_KEY needs read access to "
"stored responses in this project" % r.status_code)
if r.status_code == 404:
return None
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--responses", metavar="FILE",
help="a text file of stored response ids, one per line")
ap.add_argument("--response-id", action="append", default=[],
help="a single response id; repeatable")
ap.add_argument("--min-offered", type=int, default=50,
help="turns a tool must have been offered in before "
"silence counts as evidence (default 50)")
ap.add_argument("--show-all", action="store_true",
help="also print tools that are being called normally")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key that can read stored "
"responses")
return 2
ids = list(args.response_id)
if args.responses:
try:
with open(args.responses, "r", encoding="utf-8") as fh:
ids.extend(parse_ids(fh.read()))
except OSError as exc:
log.error("could not read %s: %s", args.responses, exc)
return 2
ids = parse_ids("\n".join(ids))
if not ids:
log.error("no usable response ids. /v1/responses cannot be listed, so "
"the sample has to come from your own request log")
return 2
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + key})
responses = []
missing = 0
for response_id in ids:
body = get(session, "/responses/" + response_id)
if body is None:
missing += 1
continue
responses.append(body)
if missing:
log.info("%d of %d id(s) no longer resolve; stored responses are not "
"kept forever", missing, len(ids))
corpus = fold(responses)
rows = coverage(corpus)
if not rows:
log.info("no named tools declared in %d sampled response(s)",
corpus["sampled"])
return 0
orphans = orphan_calls(corpus)
if orphans:
log.warning("called but never declared in this sample: %s. The sample "
"mixes two configurations, so the set difference below is "
"not reliable.", ", ".join(orphans))
bad = 0
for row in rows:
state, detail = classify(row, args.min_offered)
line = "%-19s %-22s %s" % (state, row["name"], detail)
if state in FINDINGS:
bad += 1
log.warning(line)
for repair in repair_lines(state, row["name"]):
log.warning(" repair: %s", repair)
elif state == "rarely-called":
log.warning(line)
for repair in repair_lines(state, row["name"]):
log.warning(" repair: %s", repair)
elif args.show_all or state == "too-small-a-sample":
log.info(line)
log.info("%d declared tool(s) over %d response(s), %d finding(s)",
len(rows), corpus["sampled"], bad)
share = dead_weight(rows, args.min_offered)
if share is not None:
log.info("%.0f%% of the declared schema, in characters, belongs to "
"tools nothing ever called. Characters are not tokens: count "
"the block for free against count_tokens before pricing it.",
share * 100)
state, detail = crowding(corpus["widest_turn"])
if state == "crowded":
log.warning("%-19s %s", state, detail)
log.warning(" repair: narrow the turn with allowed_tools rather than "
"rewriting one description at a time.")
else:
log.info("%-19s %s", state, detail)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI tool definitions that are sent on every call and never chosen.
*
* Read only. One GET per stored response id, using a project key. No
* completion is created: /v1/responses is read, never posted to.
*
* There is no list endpoint for stored responses, so the sample comes from a
* file of ids you supply, and every claim is bounded by that sample.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.openai.com/v1';
// Output items that represent the model choosing one of your function tools.
const CALL_TYPES = new Set(['function_call', 'custom_tool_call']);
const CROWD_CEILING = 20;
const FINDINGS = new Set(['never-called', 'never-offered']);
/** Read a count as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/**
* Response ids out of a plain text file. Pure. Order kept, duplicates dropped.
* Also the guard that stops an arbitrary line becoming a URL path segment.
*/
export function parseIds(text) {
const out = [];
const seen = new Set();
for (const line of String(text ?? '').split('\n')) {
const candidate = line.split('#')[0].trim();
if (!candidate || !candidate.startsWith('resp_')) continue;
if (!/^[A-Za-z0-9_-]+$/.test(candidate)) continue;
if (seen.has(candidate)) continue;
seen.add(candidate);
out.push(candidate);
}
return out;
}
/**
* The function name out of either tool shape. Pure. Null when absent.
* Responses puts name at the top level; Chat Completions nests it under
* function, and a reader that knows one shape is blind on half a corpus.
*/
export function toolName(tool) {
if (!tool || typeof tool !== 'object') return null;
let name = tool.name;
if (!name && tool.function && typeof tool.function === 'object') {
name = tool.function.name;
}
const text = String(name ?? '').trim();
return text || null;
}
/** Every named tool the request declared, with its size in characters. Pure. */
export function declaredTools(response) {
const out = {};
for (const tool of response?.tools ?? []) {
const name = toolName(tool);
if (name === null) continue;
let size = 0;
try {
size = JSON.stringify(tool, Object.keys(tool).sort()).length;
} catch {
size = 0;
}
out[name] = Math.max(out[name] ?? 0, size);
}
return out;
}
/** Tool names the model actually chose in one response, counted. Pure. */
export function calledTools(response) {
const counts = {};
for (const item of response?.output ?? []) {
if (!item || typeof item !== 'object' || !CALL_TYPES.has(item.type)) continue;
const name = String(item.name ?? '').trim();
if (!name) continue;
counts[name] = (counts[name] ?? 0) + 1;
}
return counts;
}
/**
* How free the model was to pick a tool in this turn. Pure.
* "free", "blocked", or "named:<tool>". Absent tool_choice is auto, which is
* free, and that is the line between a tool ignored and a tool ruled out.
*/
export function choiceMode(response) {
const choice = response?.tool_choice;
if (choice === null || choice === undefined) return 'free';
if (typeof choice === 'string') {
return choice.trim().toLowerCase() === 'none' ? 'blocked' : 'free';
}
if (typeof choice === 'object') {
const name = toolName(choice);
return name ? `named:${name}` : 'free';
}
return 'free';
}
/**
* Fold a sample of stored responses into one corpus. Pure.
* Declarations and offers are counted separately: a tool ruled out by
* tool_choice on every turn is not dead weight and needs a different repair.
*/
export function fold(responses) {
const corpus = { sampled: 0, withTools: 0, widestTurn: 0, calls: 0,
declared: {}, offered: {}, called: {} };
for (const response of responses ?? []) {
if (!response || typeof response !== 'object') continue;
corpus.sampled += 1;
const declared = declaredTools(response);
for (const [name, count] of Object.entries(calledTools(response))) {
corpus.called[name] = (corpus.called[name] ?? 0) + count;
corpus.calls += count;
}
const names = Object.keys(declared);
if (names.length === 0) continue;
corpus.withTools += 1;
corpus.widestTurn = Math.max(corpus.widestTurn, names.length);
const mode = choiceMode(response);
for (const name of names) {
const row = corpus.declared[name] ?? { turns: 0, chars: 0 };
row.turns += 1;
row.chars = Math.max(row.chars, declared[name]);
corpus.declared[name] = row;
if (mode === 'blocked') continue;
if (mode.startsWith('named:') && mode.slice('named:'.length) !== name) continue;
corpus.offered[name] = (corpus.offered[name] ?? 0) + 1;
}
}
return corpus;
}
/** One row per declared tool. Pure. Least used and most expensive first. */
export function coverage(corpus) {
const rows = [];
for (const [name, row] of Object.entries(corpus?.declared ?? {})) {
rows.push({ name,
turns: readInt(row?.turns),
chars: readInt(row?.chars),
offered: readInt(corpus?.offered?.[name]),
calls: readInt(corpus?.called?.[name]) });
}
rows.sort((a, b) => (a.calls - b.calls) || (b.chars - a.chars)
|| a.name.localeCompare(b.name));
return rows;
}
/** Names the model called that no sampled request declared. Pure. */
export function orphanCalls(corpus) {
const declared = new Set(Object.keys(corpus?.declared ?? {}));
return Object.keys(corpus?.called ?? {}).filter((n) => !declared.has(n)).sort();
}
/** Classify one tool's coverage across the sample. Pure. Returns [state, detail]. */
export function classify(row, minOffered = 50, rare = 0.01) {
const name = String(row?.name ?? 'unknown');
const turns = readInt(row?.turns);
const offered = readInt(row?.offered);
const calls = readInt(row?.calls);
if (turns > 0 && offered === 0) {
return ['never-offered',
`declared in ${turns} turn(s), free to be chosen in 0 of them. ` +
'tool_choice ruled it out every time, so the model never declined it ' +
'and rewriting the description changes nothing.'];
}
if (offered < minOffered) {
return ['too-small-a-sample',
`offered in ${offered} turn(s), under the floor of ${minOffered}. ` +
'Not enough to call anything dead.'];
}
if (calls === 0) {
return ['never-called',
`offered in ${offered} of ${turns} turn(s), called 0 time(s), ` +
`${readInt(row?.chars)} schema char(s). Sent and billed on every one ` +
'of those turns.'];
}
const share = calls / offered;
if (share < rare) {
return ['rarely-called',
`offered in ${offered} turn(s), called ${calls} time(s) ` +
`(${(share * 100).toFixed(1)}%). Worth keeping and worth not sending ` +
`on every turn. ${name} is the exception, not the default.`];
}
return ['called',
`offered in ${offered} turn(s), called ${calls} time(s) ` +
`(${(share * 100).toFixed(1)}%).`];
}
/**
* Share of the declared schema, in characters, that nothing ever calls. Pure.
* Characters, never tokens. The token price is measured exactly and for free
* elsewhere, and a character count dressed as a token count is worse than none.
*/
export function deadWeight(rows, minOffered = 50, rare = 0.01) {
let total = 0;
let dead = 0;
for (const row of rows ?? []) {
const chars = readInt(row?.chars);
total += chars;
if (classify(row, minOffered, rare)[0] === 'never-called') dead += chars;
}
if (total <= 0) return null;
return dead / total;
}
/** What the widest turn in the sample looked like. Pure. */
export function crowding(widestTurn, ceiling = CROWD_CEILING) {
const widest = readInt(widestTurn);
if (widest <= 0) return ['no-tools', 'no sampled response declared any named tool'];
if (widest > ceiling) {
return ['crowded',
`the widest turn offered ${widest} tools, above the guidance of fewer ` +
`than ${ceiling}. Selection quality falls with crowding and it falls on ` +
'the vaguest descriptions first.'];
}
return ['within-guidance',
`the widest turn offered ${widest} tool(s), inside the guidance of fewer ` +
`than ${ceiling}`];
}
/** The repair for one classified tool. Pure. */
export function repairLines(state, name) {
if (state === 'never-called') {
return [
`the description probably reads like a signature. Rewrite it as a ` +
`selection rule: when to call ${name}, and when not to.`,
'if a call is mandatory, say so with tool_choice required or a named ' +
'tool rather than hoping the model picks it up.',
'if nothing needs it, delete it. It is billed on every turn.',
];
}
if (state === 'never-offered') {
return [`tool_choice never let the model near ${name}. Fix the request ` +
'before you touch the description.'];
}
if (state === 'rarely-called') {
return [`keep ${name}, but stop sending it on every turn. allowed_tools ` +
'narrows the set for the turns where it is plausible.'];
}
return [];
}
async function get(key, path) {
const res = await fetch(API + path, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: OPENAI_API_KEY needs read ` +
'access to stored responses in this project');
}
if (res.status === 404) return null;
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function main() {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key that can read stored responses');
process.exitCode = 2;
return;
}
const file = process.argv.slice(2).find((a) => !a.startsWith('--'));
if (!file) {
console.error('pass a text file of stored response ids, one per line');
process.exitCode = 2;
return;
}
const minOffered = Number(process.env.MIN_OFFERED ?? 50);
const showAll = process.env.SHOW_ALL === '1';
const ids = parseIds(await readFile(file, 'utf8'));
if (ids.length === 0) {
console.error('no usable response ids. /v1/responses cannot be listed, so ' +
'the sample has to come from your own request log');
process.exitCode = 2;
return;
}
const responses = [];
let missing = 0;
for (const id of ids) {
const body = await get(key, `/responses/${id}`);
if (body === null) missing += 1;
else responses.push(body);
}
if (missing > 0) {
console.log(`${missing} of ${ids.length} id(s) no longer resolve; stored ` +
'responses are not kept forever');
}
const corpus = fold(responses);
const rows = coverage(corpus);
if (rows.length === 0) {
console.log(`no named tools declared in ${corpus.sampled} sampled response(s)`);
return;
}
const orphans = orphanCalls(corpus);
if (orphans.length > 0) {
console.warn(`called but never declared in this sample: ${orphans.join(', ')}. ` +
'The sample mixes two configurations, so the set difference ' +
'below is not reliable.');
}
let bad = 0;
for (const row of rows) {
const [state, detail] = classify(row, minOffered);
const line = `${state.padEnd(19)} ${row.name.padEnd(22)} ${detail}`;
if (FINDINGS.has(state) || state === 'rarely-called') {
if (FINDINGS.has(state)) bad += 1;
console.warn(line);
for (const repair of repairLines(state, row.name)) {
console.warn(` repair: ${repair}`);
}
} else if (showAll || state === 'too-small-a-sample') {
console.log(line);
}
}
console.log(`${rows.length} declared tool(s) over ${corpus.sampled} ` +
`response(s), ${bad} finding(s)`);
const share = deadWeight(rows, minOffered);
if (share !== null) {
console.log(`${(share * 100).toFixed(0)}% of the declared schema, in ` +
'characters, belongs to tools nothing ever called. Characters ' +
'are not tokens: count the block for free against count_tokens ' +
'before pricing it.');
}
const [state, detail] = crowding(corpus.widestTurn);
if (state === 'crowded') {
console.warn(`${state.padEnd(19)} ${detail}`);
console.warn(' repair: narrow the turn with allowed_tools rather than ' +
'rewriting one description at a time.');
} else {
console.log(`${state.padEnd(19)} ${detail}`);
}
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is the set difference itself: four hundred turns, six declared tools, one of them absent from every output[] array, and the classifier has to call that one dead while leaving the other five alone. Immediately beside it sits the case this note is most often wrong about — the same tool, the same four hundred turns, the same zero calls, but tool_choice named another tool every time — and it has to come back as a different state with a different repair, because the model never declined anything. The rest pin the two tool shapes, the sample floor that stops eleven turns becoming a verdict, the character share that is careful never to claim to be tokens, and the crowding check at exactly twenty.
from openai_dead_tool_definitions import (choice_mode, classify, coverage,
crowding, dead_weight,
declared_tools, fold, orphan_calls,
parse_ids, tool_name)
TOOLS = [
{"type": "function", "name": "lookup_order", "description": "x" * 200},
{"type": "function", "name": "cancel_order", "description": "x" * 200},
{"type": "function", "name": "lookup_invoice", "description": "x" * 200},
{"type": "function", "name": "escalate_to_human", "description": "x" * 1000},
]
def turn(calls, choice=None, tools=None):
body = {"tools": tools if tools is not None else TOOLS,
"output": [{"type": "function_call", "name": n, "call_id": "call_1"}
for n in calls]}
if choice is not None:
body["tool_choice"] = choice
return body
def test_a_tool_declared_on_every_turn_and_never_chosen_is_dead_weight():
# The note in one assertion. Four hundred turns, four tools, one of them
# absent from every output array.
sample = [turn(["lookup_order"]) for _ in range(300)]
sample += [turn(["cancel_order"]) for _ in range(98)]
sample += [turn(["lookup_invoice"]) for _ in range(2)]
corpus = fold(sample)
assert corpus["sampled"] == 400 and corpus["with_tools"] == 400
rows = {r["name"]: r for r in coverage(corpus)}
assert rows["escalate_to_human"]["offered"] == 400
assert rows["escalate_to_human"]["calls"] == 0
state, detail = classify(rows["escalate_to_human"])
assert state == "never-called"
assert "offered in 400 of 400 turn(s), called 0 time(s)" in detail
assert classify(rows["lookup_order"])[0] == "called"
assert classify(rows["lookup_invoice"])[0] == "rarely-called"
def test_a_tool_tool_choice_never_offered_is_a_different_finding():
# Same tool, same zero calls, and not this note: the model never had the
# chance to decline it, so its description is not the problem.
sample = [turn(["lookup_order"], choice={"type": "function",
"name": "lookup_order"})
for _ in range(400)]
rows = {r["name"]: r for r in coverage(fold(sample))}
state, detail = classify(rows["escalate_to_human"])
assert state == "never-offered"
assert "free to be chosen in 0 of them" in detail
# And the named tool itself was on the table every time.
assert rows["lookup_order"]["offered"] == 400
assert classify(rows["lookup_order"])[0] == "called"
def test_tool_choice_none_is_not_evidence_about_anything():
sample = [turn([], choice="none") for _ in range(400)]
rows = {r["name"]: r for r in coverage(fold(sample))}
assert rows["lookup_order"]["turns"] == 400
assert rows["lookup_order"]["offered"] == 0
assert classify(rows["lookup_order"])[0] == "never-offered"
assert choice_mode({"tool_choice": "none"}) == "blocked"
assert choice_mode({}) == "free"
assert choice_mode({"tool_choice": "auto"}) == "free"
assert choice_mode({"tool_choice": "required"}) == "free"
def test_both_tool_shapes_are_read():
nested = [{"type": "function", "function": {"name": "run_refund"}}]
assert tool_name(nested[0]) == "run_refund"
assert tool_name({"type": "function", "name": "flat"}) == "flat"
assert tool_name({"type": "web_search"}) is None
assert tool_name(None) is None
# A hosted tool carries no name and is not a definition you can prune.
assert declared_tools({"tools": [{"type": "web_search"}]}) == {}
assert set(declared_tools({"tools": nested})) == {"run_refund"}
def test_a_small_sample_is_not_a_verdict():
rows = {r["name"]: r for r in coverage(fold([turn([]) for _ in range(11)]))}
state, detail = classify(rows["lookup_order"])
assert state == "too-small-a-sample"
assert "under the floor of 50" in detail
assert classify(rows["lookup_order"], min_offered=5)[0] == "never-called"
def test_the_dead_weight_share_is_characters_and_stays_characters():
sample = [turn(["lookup_order", "cancel_order", "lookup_invoice"])
for _ in range(400)]
rows = coverage(fold(sample))
share = dead_weight(rows)
# escalate_to_human carries the 1000 character description; the other three
# carry 200 each, so the dead share is well over half.
assert 0.5 < share < 0.75
assert dead_weight([]) is None
assert dead_weight([{"name": "a", "chars": 0, "turns": 1, "offered": 1,
"calls": 0}]) is None
def test_a_crowded_turn_is_its_own_finding():
wide = [{"type": "function", "name": "tool_%d" % i} for i in range(26)]
corpus = fold([turn([], tools=wide) for _ in range(60)])
state, detail = crowding(corpus["widest_turn"])
assert state == "crowded"
assert "offered 26 tools" in detail
assert crowding(20)[0] == "within-guidance"
assert crowding(0)[0] == "no-tools"
def test_a_mixed_sample_is_reported_rather_than_silently_subtracted():
corpus = fold([turn(["from_another_config"])])
assert orphan_calls(corpus) == ["from_another_config"]
assert orphan_calls(fold([turn(["lookup_order"])])) == []
assert fold([]) == fold(None)
assert coverage(fold(None)) == []
def test_response_ids_are_validated_before_they_reach_a_url():
text = "resp_abc123\n# a comment\n\nresp_abc123\nresp_def456\n../../etc\n"
assert parse_ids(text) == ["resp_abc123", "resp_def456"]
assert parse_ids("resp_bad/../x") == []
assert parse_ids(None) == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { choiceMode, classify, coverage, crowding, deadWeight, declaredTools,
fold, orphanCalls, parseIds, toolName }
from './openai-dead-tool-definitions.mjs';
const TOOLS = [
{ type: 'function', name: 'lookup_order', description: 'x'.repeat(200) },
{ type: 'function', name: 'cancel_order', description: 'x'.repeat(200) },
{ type: 'function', name: 'lookup_invoice', description: 'x'.repeat(200) },
{ type: 'function', name: 'escalate_to_human', description: 'x'.repeat(1000) },
];
const turn = (calls, choice, tools) => {
const body = {
tools: tools ?? TOOLS,
output: calls.map((name) => ({ type: 'function_call', name, call_id: 'call_1' })),
};
if (choice !== undefined) body.tool_choice = choice;
return body;
};
const byName = (corpus) =>
Object.fromEntries(coverage(corpus).map((r) => [r.name, r]));
test('a tool declared on every turn and never chosen is dead weight', () => {
const sample = [
...Array.from({ length: 300 }, () => turn(['lookup_order'])),
...Array.from({ length: 98 }, () => turn(['cancel_order'])),
...Array.from({ length: 2 }, () => turn(['lookup_invoice'])),
];
const corpus = fold(sample);
assert.equal(corpus.sampled, 400);
assert.equal(corpus.withTools, 400);
const rows = byName(corpus);
assert.equal(rows.escalate_to_human.offered, 400);
assert.equal(rows.escalate_to_human.calls, 0);
const [state, detail] = classify(rows.escalate_to_human);
assert.equal(state, 'never-called');
assert.match(detail, /offered in 400 of 400 turn/);
assert.equal(classify(rows.lookup_order)[0], 'called');
assert.equal(classify(rows.lookup_invoice)[0], 'rarely-called');
});
test('a tool tool_choice never offered is a different finding', () => {
const sample = Array.from({ length: 400 },
() => turn(['lookup_order'], { type: 'function', name: 'lookup_order' }));
const rows = byName(fold(sample));
const [state, detail] = classify(rows.escalate_to_human);
assert.equal(state, 'never-offered');
assert.match(detail, /free to be chosen in 0 of them/);
assert.equal(rows.lookup_order.offered, 400);
assert.equal(classify(rows.lookup_order)[0], 'called');
});
test('tool_choice none is not evidence about anything', () => {
const rows = byName(fold(Array.from({ length: 400 }, () => turn([], 'none'))));
assert.equal(rows.lookup_order.turns, 400);
assert.equal(rows.lookup_order.offered, 0);
assert.equal(classify(rows.lookup_order)[0], 'never-offered');
assert.equal(choiceMode({ tool_choice: 'none' }), 'blocked');
assert.equal(choiceMode({}), 'free');
assert.equal(choiceMode({ tool_choice: 'auto' }), 'free');
assert.equal(choiceMode({ tool_choice: 'required' }), 'free');
});
test('both tool shapes are read', () => {
const nested = [{ type: 'function', function: { name: 'run_refund' } }];
assert.equal(toolName(nested[0]), 'run_refund');
assert.equal(toolName({ type: 'function', name: 'flat' }), 'flat');
assert.equal(toolName({ type: 'web_search' }), null);
assert.equal(toolName(null), null);
assert.deepEqual(declaredTools({ tools: [{ type: 'web_search' }] }), {});
assert.deepEqual(Object.keys(declaredTools({ tools: nested })), ['run_refund']);
});
test('a small sample is not a verdict', () => {
const rows = byName(fold(Array.from({ length: 11 }, () => turn([]))));
const [state, detail] = classify(rows.lookup_order);
assert.equal(state, 'too-small-a-sample');
assert.match(detail, /under the floor of 50/);
assert.equal(classify(rows.lookup_order, 5)[0], 'never-called');
});
test('the dead weight share is characters and stays characters', () => {
const sample = Array.from({ length: 400 },
() => turn(['lookup_order', 'cancel_order', 'lookup_invoice']));
const share = deadWeight(coverage(fold(sample)));
assert.ok(share > 0.5 && share < 0.75);
assert.equal(deadWeight([]), null);
assert.equal(deadWeight([{ name: 'a', chars: 0, turns: 1, offered: 1, calls: 0 }]),
null);
});
test('a crowded turn is its own finding', () => {
const wide = Array.from({ length: 26 },
(_, i) => ({ type: 'function', name: `tool_${i}` }));
const corpus = fold(Array.from({ length: 60 }, () => turn([], undefined, wide)));
const [state, detail] = crowding(corpus.widestTurn);
assert.equal(state, 'crowded');
assert.match(detail, /offered 26 tools/);
assert.equal(crowding(20)[0], 'within-guidance');
assert.equal(crowding(0)[0], 'no-tools');
});
test('a mixed sample is reported rather than silently subtracted', () => {
assert.deepEqual(orphanCalls(fold([turn(['from_another_config'])])),
['from_another_config']);
assert.deepEqual(orphanCalls(fold([turn(['lookup_order'])])), []);
assert.deepEqual(fold([]), fold(null));
assert.deepEqual(coverage(fold(null)), []);
});
test('response ids are validated before they reach a url', () => {
const text = 'resp_abc123\n# a comment\n\nresp_abc123\nresp_def456\n../../etc\n';
assert.deepEqual(parseIds(text), ['resp_abc123', 'resp_def456']);
assert.deepEqual(parseIds('resp_bad/../x'), []);
assert.deepEqual(parseIds(null), []);
});
FAQ
Why do I have to supply the response ids myself?
Because /v1/responses cannot be listed. OpenAI exposes retrieval by id and nothing else, so there is no way to ask the API for the last thousand responses in a project. The ids have to come from your own request log. That is a real limitation and it shapes the note: every verdict here is bounded by the sample you supplied, which is why the output prints the sample size next to every finding.
Does a tool cost anything if the model never calls it?
Yes, on every single request. Tool definitions are part of the prompt: the name, the description and the full JSON schema are serialised into the request and billed as input tokens whether or not anything is selected. That is the whole point of the note. A tool that never fires is not free capacity, it is a fixed line on every call, and because it was there from the first deploy it never shows up as a spike.
The script says a tool was never offered. What does that mean?
It means your own request took it off the table. A turn sent with tool_choice set to none gives the model no tools at all, and a turn that names one tool gives it exactly one. On those turns the other definitions were still sent and still billed, but the model was never allowed to choose them, so their silence says nothing about their descriptions. The repair is in the request, not in the prose.
How many tools is too many in one turn?
The documented guidance is fewer than twenty available at the start of a turn. It is not a hard limit and nothing 400s at twenty-one; selection quality just degrades, and it degrades first on the tools whose descriptions are vaguest. Once you are past it, no amount of description rewriting fixes the worst offenders, because the problem is the size of the choice rather than any one option in it.
Should I delete a tool that is called twice in five hundred turns?
Usually not. Rare is not dead, and a tool that handles an uncommon but important path is doing its job. What you should stop doing is sending it on all five hundred turns: narrow the set per turn with allowed_tools so the model chooses among a handful, and keep the rare tool in the turns where it is plausible. That saves the tokens without losing the capability.
Related field notes
- What the tools block actually weighs on every call
- The fan-out that voids a strict schema guarantee
- A capability that was provisioned and then never called
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Function calling — OpenAI developer docs
- Responses — OpenAI API reference
- Conversation state — OpenAI developer docs
- Completions usage — OpenAI API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.