Diagnostic LLM APIs
A model permission policy that has never excluded anything
The postmortem is short and everybody already knows how it ends. A nightly classification job that labels support tickets was pointed at the most capable model in the catalogue, because that is what the developer had in their editor from the last thing they built, and it ran that way for five weeks. The interesting question is not why they chose it. It is the one somebody asks at the end of the meeting, almost as an aside: what would have stopped it? And the answer is nothing, in that project or any other, because model access is open by default and the control that closes it is per project and opt-in.
One GET per project with an organization admin key. GET /v1/organization/projects/{project_id}/model_permissions returns a project.model_permissions object with mode — "allow_list" or "deny_list" — and model_ids[]. There are three ways for that to mean "everything is reachable":
No policy at all, where the endpoint has nothing to return, which is the default state of every project ever created. mode: "deny_list" with an empty model_ids, which is the interesting one: an object exists, the console shows a configured policy, and it denies nothing. And a deny list that names models, which is not unrestricted but fails open by construction — anything released next month is permitted the day it ships, without anybody deciding that.
Then grade the policy against reality. GET /v1/organization/usage/completions?group_by=project_id&group_by=model over thirty days gives the models each project actually called. An allow_list naming eleven models where one was used is a policy that has never excluded anything the project wanted, which is a different finding from having no policy and has a different repair.
GET /v1/organization/projects/{project_id}/hosted_tool_permissions is the same question for the billable hosted tools: code_interpreter, file_search, image_generation, mcp and web_search, each an {"enabled": bool}, each on by default.
The problem in plain words
Model access is open across an organization unless somebody closes it, and closing it is a per-project act. That means the control has the worst possible shape for surviving time: a policy written once, applied to the four projects that existed that afternoon, and silently absent from the eleventh project created two years later by somebody who has never heard of it.
The empty deny list is the version that does real damage, because it is indistinguishable from work having been done. Somebody opened the policy, chose deny list, intended to fill it in, and got pulled onto something else. From that day forward the project has a model permissions object, it appears configured in the console, and every model in the catalogue is reachable from it. An audit that checks whether a policy exists reports it as covered.
Hosted tools have the same default and a sharper cost profile. Web search, code interpreter, file search, image generation and MCP are all enabled on every project unless explicitly disabled, and they are billed on their own line items rather than as tokens. A project that has never made a web search call still has web search available to anything holding one of its keys.
Why it happens
This note is about the policy object, not about the model choice, and the code keeps that line. A published note already covers spending a frontier model's rate on trivial work; that one reads what was called and prices it. This one never says a model was the wrong pick, has no notion of a model being expensive, and produces no recommendation to switch anything. It asks a single structural question — is there a policy here, and has it ever excluded anything — and the observed model list appears only as the raw material for a least-privilege allow list. If you want to know whether the model was wrong, that is the other note.
An absent policy and an empty deny list are the same reachability and two different repairs. Both permit every model. But one of them was never touched, so the repair is "add a policy to project creation automation", and the other was touched by somebody who meant to finish, so the repair is "finish it, and find out whether anybody has been relying on the assumption that it was done". Collapsing them into one row loses the only fact that tells you which conversation to have.
A deny list that names models is graded, and graded as failing open. This is not a bug and it is not always wrong — a deny list is the right shape when you have one specific model to keep out. But its semantics are worth saying plainly in the report: every model that does not exist yet is permitted. A policy whose coverage of tomorrow's catalogue is unconditional is a different risk posture from an allow list, and a report that grades only "is a policy present" cannot tell you which one you have.
Three of the five hosted tools can be counted, one can be counted awkwardly, and one cannot be counted at all. web_search_calls, code_interpreter_sessions and file_search_calls each have a usage endpoint, so "enabled and never used" is a fact. image_generation shows up under the images usage report. mcp has no usage endpoint at all, so the script reports it as enabled-and-uncountable rather than pretending zero usage is evidence of anything. Saying which of the five it cannot measure is the difference between a report and a guess.
The fix, as a flow
The question at the end of the postmortem is not why somebody picked the expensive model. It is what would have stopped them, and the answer is a policy object that is opt in, per project, and does not inherit. The middle of this chain is the state worth naming: an empty deny list, which permits everything and looks configured to anyone who opens the console.
How to fix it
Use an organization admin key, provisioned read-only
The permission endpoints and the usage reports all reject a project key. Read scopes are enough. Nothing in the script constructs a request body, so there is no path by which it edits a policy.
List the projects and read both permission objects for each
GET /v1/organization/projects?limit=100, then model_permissions and hosted_tool_permissions per project. Treat a 404 on the model policy as "no policy configured", which is a real and common state, not an error.
Separate the three unrestricted shapes
Absent, deny_list with an empty model_ids, and deny_list with entries. The first two are unrestricted; the third is restricted today and open to whatever ships tomorrow. Grade an allow_list with an empty model_ids separately again — that one permits nothing and is a different kind of surprise.
Read the models actually used, per project
GET /v1/organization/usage/completions over thirty days with group_by=project_id and group_by=model, bucket_width=1d, limit=31. Fold num_model_requests per project and model. This is evidence for the allow list, not a judgement about the models.
Check the hosted tools against their own usage endpoints
web_search_calls, code_interpreter_sessions, file_search_calls and images. Enabled with nothing counted in the window is unused surface. mcp has no usage endpoint; report it as uncountable rather than as unused.
How to check it worked
Apply a least-privilege allow list to one project and re-run. That project should move from an unrestricted state to restricted, and the models list in the report should match the one the usage report shows. Then create a brand new project and run the script again without touching it: it should appear immediately as no-policy, which is the whole point — the control does not inherit, and that is the finding you want automated rather than remembered.
python3 openai_project_model_policy_audit.py --days 30
# 6 project(s), 4 finding(s)
# deny-list-empty proj_batch Nightly jobs
# a policy object exists, mode is deny_list, and model_ids is empty. This
# permits every model and looks configured in the console.
# repair: POST /v1/organization/projects/proj_batch/model_permissions with
# {"mode": "allow_list", "model_ids": ["gpt-4.1-mini"]}
# no-policy proj_demo Demo sandbox
# no model permissions policy is configured; every model the organization is
# entitled to is reachable from this project.
# repair: add the policy call to whatever creates projects. It does not inherit.
# allow-list-wider-than-use proj_web Customer web app
# allow_list names 11 model(s); 1 served any request in the last 30 day(s)
# hosted tools proj_demo web_search: enabled, and web_search_calls
# reports nothing in the window
# mcp: enabled, and no usage endpoint counts itThe full code
One paged GET for projects, two GETs per project, and five usage reads, against seven pure functions. The policy classifier, which distinguishes absent from an empty deny list from a deny list with entries from an allow list; the unrestricted predicate, which is deliberately narrow; the usage fold; unused_allowed, which subtracts the observed models from an allow list; the hosted-tool grader, which knows which tools have a usage endpoint and says so for the one that does not; the verdict; and the repair lines, which print a least-privilege body built from observed models and never name a model the project did not already call.
"""Find OpenAI projects whose model permission policy excludes nothing.
Read only. One paged GET for the project list, two GETs per project for the
permission objects, and five usage reads. Every request is a GET and no request
body is constructed; the least-privilege policy is printed as text.
This script has no opinion about which model is appropriate for which workload.
It answers one structural question: is there a policy here, and has it ever
excluded anything the project wanted.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_project_model_policy_audit")
API = "https://api.openai.com/v1"
DAY = 86400
# The hosted tools, and the usage endpoint that can count each one. mcp has no
# usage endpoint, so it is reported as uncountable rather than as unused.
TOOL_USAGE = {
"web_search": ("/organization/usage/web_search_calls", "num_requests"),
"code_interpreter": ("/organization/usage/code_interpreter_sessions", "num_sessions"),
"file_search": ("/organization/usage/file_search_calls", "num_requests"),
"image_generation": ("/organization/usage/images", "num_model_requests"),
}
FINDINGS = ("no-policy", "deny-list-empty", "allow-list-empty",
"deny-list-fails-open", "allow-list-wider-than-use",
"policy-unreadable")
SEVERITY = {"deny-list-empty": 0, "no-policy": 1, "allow-list-wider-than-use": 2,
"deny-list-fails-open": 3, "allow-list-empty": 4,
"policy-unreadable": 5}
def policy_ids(policy):
"""The non-empty model ids on a policy. Pure."""
out = []
for value in (policy or {}).get("model_ids") or []:
text = str(value or "").strip()
if text:
out.append(text)
return out
def policy_state(policy):
"""Shape of one model permissions object. Pure.
absent | deny-empty | deny-list | allow-empty | allow-list | unreadable.
An absent policy and an empty deny list permit exactly the same set and are
still two different states, because only one of them looks configured.
"""
if policy is None:
return "absent"
mode = str((policy or {}).get("mode") or "").strip().lower()
ids = policy_ids(policy)
if mode == "deny_list":
return "deny-empty" if not ids else "deny-list"
if mode == "allow_list":
return "allow-empty" if not ids else "allow-list"
return "unreadable"
def unrestricted(policy):
"""Does this policy permit every model? Pure. Narrow on purpose."""
return policy_state(policy) in ("absent", "deny-empty")
def fold_models(buckets, count_field="num_model_requests"):
"""{project_id: {model: requests}} across usage buckets. Pure."""
out = {}
for bucket in buckets or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
try:
n = int(row.get(count_field) or 0)
except (TypeError, ValueError):
n = 0
if n <= 0:
continue
pid = str(row.get("project_id") or "unattributed")
model = str(row.get("model") or "unknown")
entry = out.setdefault(pid, {})
entry[model] = entry.get(model, 0) + n
return out
def unused_allowed(policy, used):
"""Allow-listed models that served nothing in the window. Pure.
Only meaningful for an allow list. A deny list says nothing about what a
project is permitted to reach, so subtracting usage from it is nonsense.
"""
if policy_state(policy) != "allow-list":
return []
return sorted(set(policy_ids(policy)) - set((used or {}).keys()))
def unused_tools(perms, counts):
"""[(tool, why)] for enabled hosted tools. Pure.
A tool with no usage endpoint is reported as uncountable. Treating its
absence from a report as zero usage would be inventing evidence.
"""
out = []
for tool, block in sorted((perms or {}).items()):
if not isinstance(block, dict) or not block.get("enabled"):
continue
if tool not in TOOL_USAGE:
out.append((tool, "enabled, and no usage endpoint counts it"))
continue
if int((counts or {}).get(tool) or 0) <= 0:
out.append((tool, "enabled, and %s reports nothing in the window"
% TOOL_USAGE[tool][0].rsplit("/", 1)[-1]))
return out
def classify(policy, used, days=30):
"""Classify one project's model policy. Pure. Returns (state, detail)."""
shape = policy_state(policy)
seen = sorted((used or {}).keys())
if shape == "absent":
return ("no-policy",
"no model permissions policy is configured; every model the "
"organization is entitled to is reachable from this project")
if shape == "unreadable":
return ("policy-unreadable",
"the policy object has no recognisable mode and will not be "
"graded as restrictive")
if shape == "deny-empty":
return ("deny-list-empty",
"a policy object exists, mode is deny_list, and model_ids is "
"empty. This permits every model and looks configured")
if shape == "allow-empty":
return ("allow-list-empty",
"mode is allow_list with no model_ids, which permits nothing. If "
"this project is serving traffic, something else is going on")
if shape == "deny-list":
return ("deny-list-fails-open",
"deny_list naming %d model(s). Restrictive today and open by "
"construction to anything released tomorrow"
% len(policy_ids(policy)))
spare = unused_allowed(policy, used)
if spare:
return ("allow-list-wider-than-use",
"allow_list names %d model(s); %d served any request in the last "
"%d day(s). Unused: %s"
% (len(policy_ids(policy)), len(seen), days, ", ".join(spare)))
return ("restricted",
"allow_list of %d model(s), all of them in use"
% len(policy_ids(policy)))
def repair_lines(state, project_id, used):
"""The repair for one project. Pure. Printed, never performed.
The suggested allow list is exactly the set of models the project already
called. The script never proposes a model the project has not used.
"""
lines = []
if state not in FINDINGS:
return lines
if state == "no-policy":
lines.append("add the policy call to whatever creates projects. It does "
"not inherit from the organization or from any other "
"project.")
elif state == "deny-list-empty":
lines.append("somebody opened this policy and did not finish it. Find out "
"who, and whether anything downstream assumed it was done.")
elif state == "deny-list-fails-open":
lines.append("a deny list permits every model that does not exist yet. "
"Switch to an allow list unless keeping one named model out "
"is genuinely the whole requirement.")
elif state == "allow-list-empty":
lines.append("this permits nothing. Read it before changing it; an empty "
"allow list is more often a mistake than a lockdown.")
elif state == "policy-unreadable":
lines.append("read the policy object by hand. This audit will not call an "
"unrecognised mode restrictive.")
observed = sorted((used or {}).keys())
if observed:
lines.append('POST /v1/organization/projects/%s/model_permissions with '
'{"mode": "allow_list", "model_ids": %s}'
% (project_id, list(observed)))
lines.append("that list is what this project already called in the "
"window. It is a starting point, not a recommendation about "
"which model suits the work.")
else:
lines.append("this project called no model in the window, so there is no "
"observed set to build an allow list from. Decide it "
"deliberately rather than copying another project.")
return lines
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an organization "
"admin key, not a project key" % r.status_code)
if r.status_code == 404:
return None
r.raise_for_status()
return r.json()
def paged(session, path, **params):
params = dict(params)
while True:
page = get(session, path, params) or {}
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params["after"] = page.get("last_id") or (data[-1] or {}).get("id")
def usage(session, path, start, end):
params = {"start_time": start, "end_time": end, "bucket_width": "1d",
"limit": 31, "group_by": ["project_id", "model"]}
out = []
while True:
page = get(session, path, params) or {}
out.extend(page.get("data") or [])
cursor = page.get("next_page")
if not page.get("has_more") or not cursor:
return out
params = dict(params, page=cursor)
def tool_counts(session, start, end):
"""{project_id: {tool: count}} for the tools that have a usage endpoint."""
out = {}
for tool, (path, field) in TOOL_USAGE.items():
params = {"start_time": start, "end_time": end, "bucket_width": "1d",
"limit": 31, "group_by": ["project_id"]}
page = get(session, path, params) or {}
for bucket in page.get("data") or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
try:
n = int(row.get(field) or 0)
except (TypeError, ValueError):
n = 0
pid = str(row.get("project_id") or "unattributed")
out.setdefault(pid, {})
out[pid][tool] = out[pid].get(tool, 0) + n
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30, help="usage window to read")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key; a project "
"key cannot read the per-project permission endpoints")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + admin})
end = int(time.time())
start = end - max(1, args.days) * DAY
used = fold_models(usage(s, "/organization/usage/completions", start, end))
counts = tool_counts(s, start, end)
projects = list(paged(s, "/organization/projects", limit=100))
findings, tool_findings = [], []
for project in projects:
pid = str(project.get("id") or "")
policy = get(s, "/organization/projects/%s/model_permissions" % pid)
state, detail = classify(policy, used.get(pid), args.days)
if state in FINDINGS:
findings.append((project, state, detail))
perms = get(s, "/organization/projects/%s/hosted_tool_permissions" % pid) or {}
spare = unused_tools(perms, counts.get(pid))
if spare:
tool_findings.append((project, spare))
log.info("%d project(s), %d policy finding(s), %d project(s) with unused "
"hosted tools", len(projects), len(findings), len(tool_findings))
findings.sort(key=lambda r: (SEVERITY.get(r[1], 9), str(r[0].get("name") or "")))
for project, state, detail in findings:
pid = str(project.get("id") or "")
log.warning("%-26s %-14s %s", state, pid, project.get("name") or "(unnamed)")
log.warning(" %s", detail)
for line in repair_lines(state, pid, used.get(pid)):
log.warning(" repair: %s", line)
for project, spare in tool_findings:
log.warning("%-26s %-14s %s", "hosted tools", project.get("id"),
project.get("name") or "(unnamed)")
for tool, why in spare:
log.warning(" %s: %s", tool, why)
return 1 if (findings or tool_findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI projects whose model permission policy excludes nothing.
*
* Read only. One paged GET for the project list, two GETs per project for the
* permission objects, and five usage reads. No request body is constructed;
* the least-privilege policy is printed as text.
*
* The script has no opinion about which model suits which workload. It asks
* whether a policy exists and whether it has ever excluded anything.
*/
const API = 'https://api.openai.com/v1';
const DAY = 86400;
const TOOL_USAGE = {
web_search: ['/organization/usage/web_search_calls', 'num_requests'],
code_interpreter: ['/organization/usage/code_interpreter_sessions', 'num_sessions'],
file_search: ['/organization/usage/file_search_calls', 'num_requests'],
image_generation: ['/organization/usage/images', 'num_model_requests'],
};
const FINDINGS = new Set(['no-policy', 'deny-list-empty', 'allow-list-empty',
'deny-list-fails-open', 'allow-list-wider-than-use',
'policy-unreadable']);
const SEVERITY = { 'deny-list-empty': 0, 'no-policy': 1,
'allow-list-wider-than-use': 2, 'deny-list-fails-open': 3,
'allow-list-empty': 4, 'policy-unreadable': 5 };
/** The non-empty model ids on a policy. Pure. */
export function policyIds(policy) {
return (policy?.model_ids ?? [])
.map((v) => String(v ?? '').trim())
.filter(Boolean);
}
/** Shape of one model permissions object. Pure. */
export function policyState(policy) {
if (policy === null || policy === undefined) return 'absent';
const mode = String(policy?.mode ?? '').trim().toLowerCase();
const ids = policyIds(policy);
if (mode === 'deny_list') return ids.length ? 'deny-list' : 'deny-empty';
if (mode === 'allow_list') return ids.length ? 'allow-list' : 'allow-empty';
return 'unreadable';
}
/** Does this policy permit every model? Pure. Narrow on purpose. */
export function unrestricted(policy) {
const shape = policyState(policy);
return shape === 'absent' || shape === 'deny-empty';
}
/** {project_id: {model: requests}} across usage buckets. Pure. */
export function foldModels(buckets, countField = 'num_model_requests') {
const out = {};
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const n = Math.trunc(Number(result?.[countField] ?? 0));
if (!Number.isFinite(n) || n <= 0) continue;
const pid = String(result?.project_id ?? 'unattributed');
const model = String(result?.model ?? 'unknown');
const entry = (out[pid] ??= {});
entry[model] = (entry[model] ?? 0) + n;
}
}
return out;
}
/** Allow-listed models that served nothing in the window. Pure. */
export function unusedAllowed(policy, used) {
if (policyState(policy) !== 'allow-list') return [];
const seen = new Set(Object.keys(used ?? {}));
return policyIds(policy).filter((m) => !seen.has(m)).sort();
}
/** [[tool, why]] for enabled hosted tools. Pure. */
export function unusedTools(perms, counts) {
const out = [];
for (const tool of Object.keys(perms ?? {}).sort()) {
const block = perms[tool];
if (!block || typeof block !== 'object' || !block.enabled) continue;
if (!(tool in TOOL_USAGE)) {
out.push([tool, 'enabled, and no usage endpoint counts it']);
continue;
}
if (Math.trunc(Number((counts ?? {})[tool] ?? 0)) <= 0) {
const name = TOOL_USAGE[tool][0].split('/').pop();
out.push([tool, `enabled, and ${name} reports nothing in the window`]);
}
}
return out;
}
/** Classify one project's model policy. Pure. Returns [state, detail]. */
export function classify(policy, used, days = 30) {
const shape = policyState(policy);
const seen = Object.keys(used ?? {}).sort();
if (shape === 'absent') {
return ['no-policy',
'no model permissions policy is configured; every model the '
+ 'organization is entitled to is reachable from this project'];
}
if (shape === 'unreadable') {
return ['policy-unreadable',
'the policy object has no recognisable mode and will not be graded '
+ 'as restrictive'];
}
if (shape === 'deny-empty') {
return ['deny-list-empty',
'a policy object exists, mode is deny_list, and model_ids is empty. '
+ 'This permits every model and looks configured'];
}
if (shape === 'allow-empty') {
return ['allow-list-empty',
'mode is allow_list with no model_ids, which permits nothing. If this '
+ 'project is serving traffic, something else is going on'];
}
if (shape === 'deny-list') {
return ['deny-list-fails-open',
`deny_list naming ${policyIds(policy).length} model(s). Restrictive `
+ 'today and open by construction to anything released tomorrow'];
}
const spare = unusedAllowed(policy, used);
if (spare.length) {
return ['allow-list-wider-than-use',
`allow_list names ${policyIds(policy).length} model(s); ${seen.length} `
+ `served any request in the last ${days} day(s). Unused: ${spare.join(', ')}`];
}
return ['restricted',
`allow_list of ${policyIds(policy).length} model(s), all of them in use`];
}
/** The repair for one project. Pure. Printed, never performed. */
export function repairLines(state, projectId, used) {
const lines = [];
if (!FINDINGS.has(state)) return lines;
if (state === 'no-policy') {
lines.push('add the policy call to whatever creates projects. It does not '
+ 'inherit from the organization or from any other project.');
} else if (state === 'deny-list-empty') {
lines.push('somebody opened this policy and did not finish it. Find out who, '
+ 'and whether anything downstream assumed it was done.');
} else if (state === 'deny-list-fails-open') {
lines.push('a deny list permits every model that does not exist yet. Switch to '
+ 'an allow list unless keeping one named model out is genuinely the whole '
+ 'requirement.');
} else if (state === 'allow-list-empty') {
lines.push('this permits nothing. Read it before changing it; an empty allow '
+ 'list is more often a mistake than a lockdown.');
} else if (state === 'policy-unreadable') {
lines.push('read the policy object by hand. This audit will not call an '
+ 'unrecognised mode restrictive.');
}
const observed = Object.keys(used ?? {}).sort();
if (observed.length) {
lines.push(`POST /v1/organization/projects/${projectId}/model_permissions with `
+ `{"mode": "allow_list", "model_ids": ${JSON.stringify(observed)}}`);
lines.push('that list is what this project already called in the window. It is '
+ 'a starting point, not a recommendation about which model suits the work.');
} else {
lines.push('this project called no model in the window, so there is no observed '
+ 'set to build an allow list from. Decide it deliberately rather than '
+ 'copying another project.');
}
return lines;
}
async function read(key, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) {
if (Array.isArray(v)) for (const one of v) url.searchParams.append(k, String(one));
else url.searchParams.set(k, String(v));
}
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/organization/* needs an `
+ 'organization admin key, not a project key');
}
if (r.status === 404) return null;
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function paged(key, path, params) {
const out = [];
const q = { ...params };
for (;;) {
const page = (await read(key, path, q)) ?? {};
const data = page.data ?? [];
out.push(...data);
if (!page.has_more || data.length === 0) return out;
q.after = page.last_id ?? data[data.length - 1]?.id;
}
}
async function usage(key, path, start, end) {
const params = { start_time: start, end_time: end, bucket_width: '1d',
limit: 31, group_by: ['project_id', 'model'] };
const out = [];
for (;;) {
const page = (await read(key, path, params)) ?? {};
out.push(...(page.data ?? []));
if (!page.has_more || !page.next_page) return out;
params.page = page.next_page;
}
}
async function toolCounts(key, start, end) {
const out = {};
for (const [tool, [path, field]] of Object.entries(TOOL_USAGE)) {
const page = (await read(key, path, { start_time: start, end_time: end,
bucket_width: '1d', limit: 31, group_by: ['project_id'] })) ?? {};
for (const bucket of page.data ?? []) {
for (const result of bucket?.results ?? []) {
const n = Math.trunc(Number(result?.[field] ?? 0)) || 0;
const pid = String(result?.project_id ?? 'unattributed');
const entry = (out[pid] ??= {});
entry[tool] = (entry[tool] ?? 0) + n;
}
}
}
return out;
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key; a project '
+ 'key cannot read the per-project permission endpoints');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 30);
const end = Math.floor(Date.now() / 1000);
const start = end - Math.max(1, days) * DAY;
const used = foldModels(await usage(admin, '/organization/usage/completions', start, end));
const counts = await toolCounts(admin, start, end);
const projects = await paged(admin, '/organization/projects', { limit: 100 });
const findings = [];
const toolFindings = [];
for (const project of projects) {
const pid = String(project.id ?? '');
const policy = await read(admin, `/organization/projects/${pid}/model_permissions`);
const [state, detail] = classify(policy, used[pid], days);
if (FINDINGS.has(state)) findings.push([project, state, detail]);
const perms = (await read(admin,
`/organization/projects/${pid}/hosted_tool_permissions`)) ?? {};
const spare = unusedTools(perms, counts[pid]);
if (spare.length) toolFindings.push([project, spare]);
}
console.log(`${projects.length} project(s), ${findings.length} policy finding(s), `
+ `${toolFindings.length} project(s) with unused hosted tools`);
findings.sort(([pa, sa], [pb, sb]) =>
(SEVERITY[sa] ?? 9) - (SEVERITY[sb] ?? 9)
|| String(pa.name ?? '').localeCompare(String(pb.name ?? '')));
for (const [project, state, detail] of findings) {
const pid = String(project.id ?? '');
console.warn(`${state.padEnd(26)} ${pid.padEnd(14)} ${project.name ?? '(unnamed)'}`);
console.warn(` ${detail}`);
for (const line of repairLines(state, pid, used[pid])) {
console.warn(` repair: ${line}`);
}
}
for (const [project, spare] of toolFindings) {
console.warn(`${'hosted tools'.padEnd(26)} ${String(project.id).padEnd(14)} `
+ `${project.name ?? '(unnamed)'}`);
for (const [tool, why] of spare) console.warn(` ${tool}: ${why}`);
}
process.exitCode = (findings.length || toolFindings.length) ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The headline test is two projects with identical traffic and identical reachability: one with no policy at all, one with mode: "deny_list" and an empty model_ids. Both permit every model, both have to come back as findings, and they have to be different findings with different repairs, because only one of them looks configured to whoever set it. After that: a deny list that names models, which is restricted today and graded as failing open tomorrow; an allow list eleven wide over one used model, where the repair proposes exactly the observed set and nothing else; the hosted-tool grader, which reports mcp as uncountable rather than as unused; and one test whose only job is to prove this note stays off the neighbouring one — no repair line anywhere may recommend a model the project did not already call.
from openai_project_model_policy_audit import (classify, fold_models,
policy_ids, policy_state,
repair_lines, unrestricted,
unused_allowed, unused_tools)
USED = {"gpt-4.1-mini": 41208}
def bucket(*results):
return {"object": "bucket", "start_time": 0, "end_time": 86400,
"results": list(results)}
def test_an_absent_policy_and_an_empty_deny_list_are_two_findings():
# The note. Identical reachability, identical usage, different repairs,
# because one of these two looks configured in the console and one does not.
absent_state, absent_detail = classify(None, USED)
empty_state, empty_detail = classify({"mode": "deny_list", "model_ids": []}, USED)
assert unrestricted(None) is True
assert unrestricted({"mode": "deny_list", "model_ids": []}) is True
assert absent_state == "no-policy"
assert empty_state == "deny-list-empty"
assert "looks configured" in empty_detail
assert "reachable from this project" in absent_detail
absent_lines = repair_lines(absent_state, "proj_demo", USED)
empty_lines = repair_lines(empty_state, "proj_batch", USED)
assert any("does not inherit" in line for line in absent_lines)
assert any("did not finish it" in line for line in empty_lines)
assert absent_lines != empty_lines
def test_a_deny_list_with_entries_is_restrictive_today_and_open_tomorrow():
policy = {"mode": "deny_list", "model_ids": ["gpt-4.1"]}
assert unrestricted(policy) is False
state, detail = classify(policy, USED)
assert state == "deny-list-fails-open"
assert "released tomorrow" in detail
assert any("does not exist yet" in line
for line in repair_lines(state, "proj_x", USED))
# Subtracting usage from a deny list would be nonsense, so it is not done.
assert unused_allowed(policy, USED) == []
def test_an_allow_list_wider_than_use_names_only_what_it_measured():
policy = {"mode": "allow_list",
"model_ids": ["gpt-4.1-mini", "gpt-4.1", "o3", "gpt-4.1-nano"]}
state, detail = classify(policy, USED, days=30)
assert state == "allow-list-wider-than-use"
assert "names 4 model(s); 1 served any request" in detail
assert unused_allowed(policy, USED) == ["gpt-4.1", "gpt-4.1-nano", "o3"]
# Exactly the observed set, and nothing the project never called.
lines = repair_lines(state, "proj_web", USED)
assert any("'model_ids': ['gpt-4.1-mini']" in line
or '"model_ids": [\'gpt-4.1-mini\']' in line
or "['gpt-4.1-mini']" in line for line in lines)
assert not any("o3" in line for line in lines)
# An allow list matching use exactly is not a finding.
tight = {"mode": "allow_list", "model_ids": ["gpt-4.1-mini"]}
assert classify(tight, USED)[0] == "restricted"
assert repair_lines("restricted", "proj_web", USED) == []
def test_the_policy_shape_reader_handles_every_degenerate_case():
assert policy_state(None) == "absent"
assert policy_state({"mode": "allow_list", "model_ids": []}) == "allow-empty"
assert policy_state({"mode": "allow_list", "model_ids": [" "]}) == "allow-empty"
assert policy_state({"mode": "ALLOW_LIST", "model_ids": ["a"]}) == "allow-list"
assert policy_state({"mode": "something_new"}) == "unreadable"
assert policy_ids({"model_ids": ["a", "", None, " b "]}) == ["a", "b"]
state, detail = classify({"mode": "allow_list", "model_ids": []}, USED)
assert state == "allow-list-empty"
assert "permits nothing" in detail
assert classify({"mode": "?"}, USED)[0] == "policy-unreadable"
def test_a_tool_with_no_usage_endpoint_is_uncountable_not_unused():
perms = {"code_interpreter": {"enabled": False},
"file_search": {"enabled": True},
"image_generation": {"enabled": True},
"mcp": {"enabled": True},
"web_search": {"enabled": True}}
counts = {"web_search": 4120, "file_search": 0, "image_generation": 0}
found = dict(unused_tools(perms, counts))
assert "web_search" not in found # used, so not reported
assert "code_interpreter" not in found # disabled, so not reported
assert "file_search_calls reports nothing" in found["file_search"]
assert found["mcp"] == "enabled, and no usage endpoint counts it"
assert unused_tools(None, None) == []
assert unused_tools({"web_search": "not a block"}, {}) == []
def test_the_report_never_recommends_a_model_the_project_did_not_call():
# This note owns the policy object. Which model suits the workload belongs
# to a different note, and no repair line here may stray into it.
for state in ("no-policy", "deny-list-empty", "deny-list-fails-open",
"allow-list-wider-than-use"):
for line in repair_lines(state, "proj_a", USED):
assert "cheaper" not in line and "mini" not in line.replace(
"gpt-4.1-mini", "")
empty = repair_lines("no-policy", "proj_idle", {})
assert any("no observed set" in line for line in empty)
used = fold_models([bucket({"project_id": "p", "model": "m",
"num_model_requests": 4}),
bucket({"project_id": "p", "model": "m",
"num_model_requests": 0})])
assert used == {"p": {"m": 4}}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, foldModels, policyIds, policyState, repairLines, unrestricted,
unusedAllowed, unusedTools } from './openai-project-model-policy-audit.mjs';
const USED = { 'gpt-4.1-mini': 41208 };
const bucket = (...results) =>
({ object: 'bucket', start_time: 0, end_time: 86400, results });
test('an absent policy and an empty deny list are two findings', () => {
const [absentState, absentDetail] = classify(null, USED);
const [emptyState, emptyDetail] = classify({ mode: 'deny_list', model_ids: [] }, USED);
assert.equal(unrestricted(null), true);
assert.equal(unrestricted({ mode: 'deny_list', model_ids: [] }), true);
assert.equal(absentState, 'no-policy');
assert.equal(emptyState, 'deny-list-empty');
assert.ok(emptyDetail.includes('looks configured'));
assert.ok(absentDetail.includes('reachable from this project'));
const absentLines = repairLines(absentState, 'proj_demo', USED);
const emptyLines = repairLines(emptyState, 'proj_batch', USED);
assert.ok(absentLines.some((l) => l.includes('does not inherit')));
assert.ok(emptyLines.some((l) => l.includes('did not finish it')));
assert.notDeepEqual(absentLines, emptyLines);
});
test('a deny list with entries is restrictive today and open tomorrow', () => {
const policy = { mode: 'deny_list', model_ids: ['gpt-4.1'] };
assert.equal(unrestricted(policy), false);
const [state, detail] = classify(policy, USED);
assert.equal(state, 'deny-list-fails-open');
assert.ok(detail.includes('released tomorrow'));
assert.ok(repairLines(state, 'proj_x', USED)
.some((l) => l.includes('does not exist yet')));
assert.deepEqual(unusedAllowed(policy, USED), []);
});
test('an allow list wider than use names only what it measured', () => {
const policy = { mode: 'allow_list',
model_ids: ['gpt-4.1-mini', 'gpt-4.1', 'o3', 'gpt-4.1-nano'] };
const [state, detail] = classify(policy, USED, 30);
assert.equal(state, 'allow-list-wider-than-use');
assert.ok(detail.includes('names 4 model(s); 1 served any request'));
assert.deepEqual(unusedAllowed(policy, USED), ['gpt-4.1', 'gpt-4.1-nano', 'o3']);
const lines = repairLines(state, 'proj_web', USED);
assert.ok(lines.some((l) => l.includes('["gpt-4.1-mini"]')));
assert.ok(!lines.some((l) => l.includes('o3')));
const tight = { mode: 'allow_list', model_ids: ['gpt-4.1-mini'] };
assert.equal(classify(tight, USED)[0], 'restricted');
assert.deepEqual(repairLines('restricted', 'proj_web', USED), []);
});
test('the policy shape reader handles every degenerate case', () => {
assert.equal(policyState(null), 'absent');
assert.equal(policyState(undefined), 'absent');
assert.equal(policyState({ mode: 'allow_list', model_ids: [] }), 'allow-empty');
assert.equal(policyState({ mode: 'allow_list', model_ids: [' '] }), 'allow-empty');
assert.equal(policyState({ mode: 'ALLOW_LIST', model_ids: ['a'] }), 'allow-list');
assert.equal(policyState({ mode: 'something_new' }), 'unreadable');
assert.deepEqual(policyIds({ model_ids: ['a', '', null, ' b '] }), ['a', 'b']);
const [state, detail] = classify({ mode: 'allow_list', model_ids: [] }, USED);
assert.equal(state, 'allow-list-empty');
assert.ok(detail.includes('permits nothing'));
assert.equal(classify({ mode: '?' }, USED)[0], 'policy-unreadable');
});
test('a tool with no usage endpoint is uncountable, not unused', () => {
const perms = { code_interpreter: { enabled: false },
file_search: { enabled: true },
image_generation: { enabled: true },
mcp: { enabled: true },
web_search: { enabled: true } };
const counts = { web_search: 4120, file_search: 0, image_generation: 0 };
const found = Object.fromEntries(unusedTools(perms, counts));
assert.equal(found.web_search, undefined);
assert.equal(found.code_interpreter, undefined);
assert.ok(found.file_search.includes('file_search_calls reports nothing'));
assert.equal(found.mcp, 'enabled, and no usage endpoint counts it');
assert.deepEqual(unusedTools(null, null), []);
assert.deepEqual(unusedTools({ web_search: 'not a block' }, {}), []);
});
test('the report never recommends a model the project did not call', () => {
for (const state of ['no-policy', 'deny-list-empty', 'deny-list-fails-open',
'allow-list-wider-than-use']) {
for (const line of repairLines(state, 'proj_a', USED)) {
assert.ok(!line.includes('cheaper'));
}
}
assert.ok(repairLines('no-policy', 'proj_idle', {})
.some((l) => l.includes('no observed set')));
const used = foldModels([
bucket({ project_id: 'p', model: 'm', num_model_requests: 4 }),
bucket({ project_id: 'p', model: 'm', num_model_requests: 0 })]);
assert.deepEqual(used, { p: { m: 4 } });
});
FAQ
How is this different from the note about frontier models on trivial workloads?
That note reads what was called and prices it: it is a spending finding about a usage pattern. This one reads the policy object and asks whether anything would have stopped the call. They meet at exactly one point — the list of models a project actually used — and they use it for opposite purposes. The other note uses it to say a model was overkill for the work. This one uses it only as raw material for a least-privilege allow list, and deliberately never says a model was the wrong choice. If the postmortem question is why did we spend that, read the other note. If it is what would have prevented it, this is the one.
Why is an empty deny list worse than no policy at all?
Both permit every model, so the reachability is identical. The difference is what a human sees. A project with no policy shows as unconfigured, which is honest. A project with mode deny_list and an empty model_ids shows as having a model permissions policy, and anybody glancing at it — including whoever writes the next compliance answer — will conclude the control is in place. It is the most common way for this to look done. The script grades them as separate states with separate repairs because the follow-up conversation is different: one is a gap in project-creation automation, the other is a half-finished change with a person attached to it.
Is a deny list wrong?
No, and the script does not say it is. A deny list is the right instrument when the requirement is genuinely to keep one named model out. What the report states plainly is its semantics: everything not on the list is permitted, including every model released after the policy was written, so its coverage of next quarter's catalogue is unconditional. That is a fine trade in some projects and a bad one in a project that exists to run one nightly job. Knowing which one you have is the value; the script names the shape and leaves the decision where it belongs.
Why not just check the hosted tools by looking at cost?
Because cost cannot distinguish enabled-and-unused from disabled, and unused surface is the finding. A tool that is enabled and never called costs nothing, which is exactly why nobody turns it off, and which is why it is still available to anything holding a key for that project. So the script reads the permission object for the enabled flag and the matching usage endpoint for the count, and reports the pair. It also refuses to guess about mcp, which has no usage endpoint at all: that one is reported as enabled and uncountable, because calling zero evidence of usage zero usage would be inventing a measurement.
Can I run the repair the script prints?
You can, and you should read it first. The printed body is an allow list built from exactly the models that project called in the window, which is a starting point rather than a policy: a project that ran a quarterly job outside the window will be missing a model it genuinely needs, and applying the list unread will break it on the next run. Widen the window, or add the model deliberately. The script does not execute the call under any flag, and the same applies to the hosted tool body it prints alongside.
Related field notes
- The spending pattern this policy is the structural answer to
- The ceiling that is missing rather than empty
- What a hosted tool costs once something starts calling it
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Admin APIs — OpenAI platform docs
- Projects — OpenAI API reference
- Models — OpenAI platform docs
- openai-python admin API surface
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.