Diagnostic LLM APIs
Nothing has ever called the free moderation endpoint
The product takes free text from anyone with the link and has done since launch. Nobody thinks of it as a moderation problem, because nothing has gone wrong yet and the model refuses most of what it should refuse on its own. Then a support ticket arrives with a screenshot, and somebody asks the only question that matters in the first ten minutes: what do we already screen? The honest answer takes an afternoon to establish, and it turns out to be nothing — not because anyone decided against it, but because moderation is a separate endpoint you have to go and call, and no line of code ever did.
Two GETs with an organization admin key. GET /v1/organization/usage/moderations?start_time={now-30d}&bucket_width=1d&limit=31&group_by=project_id&group_by=model, and the same query against /v1/organization/usage/completions. Both return buckets of results carrying num_model_requests. Fold them per project and compare.
Two findings come out of that comparison, and the second one is the better one. A project with real completion volume and no moderations bucket at all has never called the endpoint — which is the default state, because moderation is opt-in, nothing routes input through it automatically, and it is free, so cost was never the reason it was skipped. And a project whose moderation requests are attributed to a text-moderation-* model id is screening through a retired family that predates omni-moderation-latest, which is the current id and the only one that reads images as well as text. That one is worse than it looks on a product that started accepting uploads after the code was written.
The script never calls POST /v1/moderations. Sending content to a model to prove the model answers is generating, and nothing in this section generates. The endpoint is read entirely through the usage report the organization already has.
The problem in plain words
Moderation is a separate endpoint, not a property of the completion. There is no flag on a chat request that turns it on, no default that screens input for you, and no error if you never call it. The safety behaviour people actually observe in testing is the model's own refusal, which is a different mechanism with a different purpose: it decides what the model will say, not what your users are allowed to send you, and it tells you nothing you can log, threshold or report on.
So the endpoint gets skipped, and it gets skipped for a reason that sounds better than it is: the model seems to handle it. That holds right up until the thing you needed to catch was in the input rather than the output — a user pasting something into a support widget, an image upload, a public-facing form — and at that point there is no record of the input having been assessed at all, because assessing it was never a step.
The retired-id half is a quieter failure with the same shape. Code written before omni-moderation-latest shipped calls text-moderation-latest or a pinned text-moderation-007, and it keeps working, because a moderation call with an old id still returns a verdict. What it does not do is read images. A product that added uploads two years after it added the moderation call has a screening layer that silently covers half its input surface, and nothing anywhere reports that as a gap.
Why it happens
The zero is the finding, and the ratio is not. A completions:moderations ratio looks like it should be the measurement here and it is not a good one in either direction. It understates coverage, because one moderation request can carry an array of inputs and still counts as one num_model_requests. And it overstates the gap, because the completions figure includes everything — batch jobs, evaluation suites, internal tooling — none of which has user input in it at all. So the script grades the ratio as a soft signal with a name that says so, and reserves the word "finding" for the two facts that survive contact with reality: no moderation requests whatsoever against real traffic, and moderation requests on an id that no longer belongs there.
It is free, which removes the only excuse and changes what the report should say. Almost every other note in this section ends in a trade-off, because the fix costs tokens or latency or engineering time. This one does not. The moderations endpoint bills nothing, so a report that hedges — "consider whether moderation is appropriate for your workload" — is worse than useless. The script says the round trip is the entire cost and moves on.
Grouping by project is what makes the report actionable rather than a scold. An organization-wide zero is not interesting; a zero on the one project serving the public form, next to a non-zero on the internal one, is a work item with an owner. The usage endpoints support group_by=project_id and group_by=model together, so the same two reads produce the per-project split and the per-model split at once, and a floor on completion volume keeps a nine-request scratch project out of the list.
What the usage report cannot tell you, it does not claim. It knows how many moderation requests were made and under which model id. It does not know whether they were made before the completion or after it, whether the code branched on flagged or logged the result and continued, or whether the input assessed was the user's or something the application generated itself. Those live in your source tree. The script reports the request counts and says which questions it is not answering.
The fix, as a flow
Nothing in this chain raises an error, and the middle of it is the reason: the model refuses most of what it should refuse on its own, so the absence of a screening step looks like a working screening step for as long as nothing arrives that needed one. The endpoint that would have caught it is free, opt in, and reachable only by code somebody has to write on purpose.
How to fix it
Use an organization admin key, provisioned read-only
Both /v1/organization/usage/moderations and /v1/organization/usage/completions need api.usage.read on an organization admin key. A project key is rejected. The script never sends anything to a model.
Read both usage reports over the same window
start_time in unix seconds, bucket_width=1d, limit=31 — the daily bucket cap is 31, so a 30-day window at limit=30 quietly drops a day. Group by project_id and model together and paginate on next_page using the page parameter, which is not the after cursor the rest of the admin API uses.
Fold per project and set a volume floor
Sum num_model_requests across every bucket for each project. Then drop projects under a completion floor, because a project with forty requests in a month is somebody's laptop and grading it teaches people to ignore the report.
Check the model ids before you check the counts
A project can be moderating every single request and still be a finding, if those requests carry a text-moderation-* id. Test the ids first: a healthy ratio on a retired model is the case a count-based audit reports as fine.
Print what is knowable and name what is not
Per project: completion requests, moderation requests, the model ids seen. The repair is POST /v1/moderations on user input before the completion, branching on flagged and logging category_scores rather than the boolean — printed, never performed.
How to check it worked
Add the call, deploy, and re-read the same two reports a day later. The moderations bucket for that project should appear with a request count in the same order of magnitude as the user-facing share of its completions, and the model column should read omni-moderation-latest. If the count arrives but the id is still the old one, the deploy picked up an existing helper rather than the new call site.
python3 openai_moderation_coverage_audit.py --days 30 --min-completions 500
# 4 project(s) over the 500 request floor, 2 finding(s)
# never-called proj_public 41,208 completion request(s) and no moderation
# request at all
# repair: route user input through POST /v1/moderations before the completion.
# The endpoint is free, so a round trip is the entire cost.
# repair: branch on flagged, and log category_scores rather than the single
# boolean, so a threshold can be tuned per category later.
# retired-model-id proj_intake 3,904 moderation request(s), 100% of them on
# text-moderation-latest
# repair: move text-moderation-latest to omni-moderation-latest, which is
# current and is the only moderation model that reads images.
# not graded: this report counts requests. Whether the code branched on flagged,
# and whether the input assessed was the user's, are not in the API.The full code
Two paged GETs and five pure functions. The bucket fold, which sums num_model_requests per project and per model and ignores a result whose count is zero rather than creating a phantom entry for it; the retired-id test, which matches the whole text-moderation family by prefix so a pinned -007 is caught alongside -latest; the coverage join, which keeps projects with no moderation entry at all rather than dropping them, since those are the note; the classifier, which tests the model ids before it tests any count; and the repair lines. Nothing here constructs a request body, and POST /v1/moderations appears only as printed text.
"""Find an OpenAI organization whose moderation endpoint is never called.
Read only. Two paged GETs against /v1/organization/usage/moderations and
/v1/organization/usage/completions with an organization admin key. Every
request is a GET and no request body is constructed.
The script deliberately does not call the moderations endpoint to prove it
works. Sending content to a model to see what comes back is generating, and
nothing in this section generates. The finding comes entirely from two request
counts the organization already has.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_moderation_coverage_audit")
API = "https://api.openai.com/v1"
DAY = 86400
# The whole text-moderation-* family is retired: -latest, -stable and the
# pinned -006 / -007 snapshots. Matched by prefix so a pin is caught too.
RETIRED_PREFIX = "text-moderation"
CURRENT = "omni-moderation-latest"
FINDINGS = ("never-called", "retired-model-id", "thin-coverage")
# An unmoderated public surface outranks a stale model id, which outranks a
# ratio, because the ratio is the weakest of the three signals by some way.
SEVERITY = {"never-called": 0, "retired-model-id": 1, "thin-coverage": 2}
def fold(buckets, count_field="num_model_requests"):
"""{project_id: {"requests": n, "models": {id: n}}} across buckets. Pure.
A result carrying zero requests creates no entry. That matters: the whole
detection rests on a project being absent from the moderations fold, and a
zero-valued row would make it present with a count of nothing.
"""
out = {}
for bucket in buckets or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
try:
n = int(row.get(count_field) or 0)
except (TypeError, ValueError):
n = 0
if n <= 0:
continue
pid = str(row.get("project_id") or "unattributed")
model = str(row.get("model") or "unknown")
entry = out.setdefault(pid, {"requests": 0, "models": {}})
entry["requests"] += n
entry["models"][model] = entry["models"].get(model, 0) + n
return out
def is_retired(model):
"""Is this a retired moderation model id? Pure. Prefix match, so pins count."""
return str(model or "").strip().lower().startswith(RETIRED_PREFIX)
def retired_ids(models):
"""Sorted retired ids inside one {model: requests} mapping. Pure."""
return sorted(m for m in (models or {}) if is_retired(m))
def coverage(completions, moderations):
"""[(project_id, completions, moderations, models)] busiest first. Pure.
Driven by the completions side, so a project with no moderations entry is
still a row. Dropping it is exactly the bug this note is about.
"""
rows = []
for pid, entry in (completions or {}).items():
mod = (moderations or {}).get(pid) or {}
rows.append((pid,
int((entry or {}).get("requests") or 0),
int(mod.get("requests") or 0),
dict(mod.get("models") or {})))
rows.sort(key=lambda r: (-r[1], r[0]))
return rows
def classify(row, min_completions=500, min_ratio=0.2):
"""Classify one coverage row. Pure. Returns (state, detail).
The model ids are tested BEFORE any count. A project can be moderating
every request it serves and still be a finding, because a healthy ratio on
a retired id is exactly what a count-based audit calls fine.
"""
pid, completions, moderations, models = row
if completions < min_completions:
return ("below-floor",
"%d completion request(s), under the %d floor"
% (completions, min_completions))
retired = retired_ids(models)
if retired:
share = sum(models[m] for m in retired) / float(max(1, moderations))
return ("retired-model-id",
"%d moderation request(s), %d%% of them on %s"
% (moderations, round(share * 100), ", ".join(retired)))
if moderations <= 0:
return ("never-called",
"%d completion request(s) and no moderation request at all"
% completions)
ratio = moderations / float(completions)
if ratio < min_ratio:
return ("thin-coverage",
"%d moderation request(s) against %d completion request(s), a "
"ratio of %.2f" % (moderations, completions, ratio))
return ("covered",
"%d moderation request(s), ratio %.2f" % (moderations, ratio))
def repair_lines(state, row):
"""The repair for one classified project. Pure. Printed, never performed."""
pid, completions, moderations, models = row
lines = []
if state not in FINDINGS:
return lines
if state == "never-called":
lines.append("route user input through the moderations endpoint before "
"the completion. It bills nothing, so a round trip is the "
"entire cost.")
lines.append("branch on flagged, and log category_scores rather than the "
"single boolean, so a threshold can be tuned per category "
"later without another deploy.")
elif state == "retired-model-id":
lines.append("move %s to %s, which is current and is the only moderation "
"model that reads images as well as text."
% (", ".join(retired_ids(models)), CURRENT))
lines.append("if this product accepts uploads, the retired id has been "
"screening the text half only.")
else:
lines.append("moderation is being called on a small share of the traffic. "
"Find the call sites that skip it before tuning anything; "
"the ratio alone cannot tell you which they are.")
lines.append("re-read project %s with the same two usage reports after the "
"deploy, and check the model column, not only the count" % pid)
return lines
def get(session, path, params):
r = session.get(API + path, params=params, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: the usage reports need an organization "
"admin key with api.usage.read, not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def usage(session, path, start, end):
"""Every bucket in the window. Paginates on next_page via the page param."""
params = {"start_time": start, "end_time": end, "bucket_width": "1d",
"limit": 31, "group_by": ["project_id", "model"]}
out = []
while True:
page = get(session, path, params)
out.extend(page.get("data") or [])
cursor = page.get("next_page")
if not page.get("has_more") or not cursor:
return out
params = dict(params, page=cursor)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30, help="window to read")
ap.add_argument("--min-completions", type=int, default=500,
help="completion requests a project needs before it is graded")
ap.add_argument("--min-ratio", type=float, default=0.2,
help="soft floor on moderations per completion")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key; a project "
"key cannot read /v1/organization/usage/*")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + admin})
end = int(time.time())
start = end - max(1, args.days) * DAY
completions = fold(usage(s, "/organization/usage/completions", start, end))
moderations = fold(usage(s, "/organization/usage/moderations", start, end))
rows = coverage(completions, moderations)
graded = [(row, classify(row, args.min_completions, args.min_ratio))
for row in rows]
bad = [(row, state, detail) for row, (state, detail) in graded
if state in FINDINGS]
over_floor = sum(1 for _, (state, _) in graded if state != "below-floor")
log.info("%d project(s) over the %d request floor, %d finding(s)",
over_floor, args.min_completions, len(bad))
bad.sort(key=lambda r: (SEVERITY.get(r[1], 9), -r[0][1]))
for row, state, detail in bad:
log.warning("%-18s %-14s %s", state, row[0], detail)
for line in repair_lines(state, row):
log.warning(" repair: %s", line)
log.info("not graded: this report counts requests. Whether the code branched "
"on flagged, and whether the input assessed was the user's, are not "
"in the API.")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find an OpenAI organization whose moderation endpoint is never called.
*
* Read only. Two paged GETs against /v1/organization/usage/moderations and
* /v1/organization/usage/completions. No request body is constructed.
*
* The script deliberately does not call the moderations endpoint to prove it
* works: sending content to a model is generating. The finding comes from two
* request counts the organization already has.
*/
const API = 'https://api.openai.com/v1';
const DAY = 86400;
const RETIRED_PREFIX = 'text-moderation';
const CURRENT = 'omni-moderation-latest';
const FINDINGS = new Set(['never-called', 'retired-model-id', 'thin-coverage']);
const SEVERITY = { 'never-called': 0, 'retired-model-id': 1, 'thin-coverage': 2 };
/** {project_id: {requests, models}} across buckets. Pure. Zero rows create nothing. */
export function fold(buckets, countField = 'num_model_requests') {
const out = {};
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const n = Math.trunc(Number(result?.[countField] ?? 0));
if (!Number.isFinite(n) || n <= 0) continue;
const pid = String(result?.project_id ?? 'unattributed');
const model = String(result?.model ?? 'unknown');
const entry = (out[pid] ??= { requests: 0, models: {} });
entry.requests += n;
entry.models[model] = (entry.models[model] ?? 0) + n;
}
}
return out;
}
/** Is this a retired moderation model id? Pure. Prefix match, so pins count. */
export function isRetired(model) {
return String(model ?? '').trim().toLowerCase().startsWith(RETIRED_PREFIX);
}
/** Sorted retired ids inside one {model: requests} mapping. Pure. */
export function retiredIds(models) {
return Object.keys(models ?? {}).filter(isRetired).sort();
}
/** [[project, completions, moderations, models]] busiest first. Pure. */
export function coverage(completions, moderations) {
const rows = [];
for (const [pid, entry] of Object.entries(completions ?? {})) {
const mod = (moderations ?? {})[pid] ?? {};
rows.push([pid, Math.trunc(Number(entry?.requests ?? 0)),
Math.trunc(Number(mod.requests ?? 0)), { ...(mod.models ?? {}) }]);
}
rows.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
return rows;
}
/** Classify one coverage row. Pure. Model ids are tested before any count. */
export function classify(row, minCompletions = 500, minRatio = 0.2) {
const [pid, completions, moderations, models] = row;
if (completions < minCompletions) {
return ['below-floor',
`${completions} completion request(s), under the ${minCompletions} floor`];
}
const retired = retiredIds(models);
if (retired.length) {
const share = retired.reduce((a, m) => a + models[m], 0) / Math.max(1, moderations);
return ['retired-model-id',
`${moderations} moderation request(s), ${Math.round(share * 100)}% of `
+ `them on ${retired.join(', ')}`];
}
if (moderations <= 0) {
return ['never-called',
`${completions} completion request(s) and no moderation request at all`];
}
const ratio = moderations / completions;
if (ratio < minRatio) {
return ['thin-coverage',
`${moderations} moderation request(s) against ${completions} completion `
+ `request(s), a ratio of ${ratio.toFixed(2)}`];
}
return ['covered', `${moderations} moderation request(s), ratio ${ratio.toFixed(2)}`];
}
/** The repair for one classified project. Pure. Printed, never performed. */
export function repairLines(state, row) {
const [pid, , , models] = row;
const lines = [];
if (!FINDINGS.has(state)) return lines;
if (state === 'never-called') {
lines.push('route user input through the moderations endpoint before the '
+ 'completion. It bills nothing, so a round trip is the entire cost.');
lines.push('branch on flagged, and log category_scores rather than the single '
+ 'boolean, so a threshold can be tuned per category later without another '
+ 'deploy.');
} else if (state === 'retired-model-id') {
lines.push(`move ${retiredIds(models).join(', ')} to ${CURRENT}, which is `
+ 'current and is the only moderation model that reads images as well as text.');
lines.push('if this product accepts uploads, the retired id has been screening '
+ 'the text half only.');
} else {
lines.push('moderation is being called on a small share of the traffic. Find '
+ 'the call sites that skip it before tuning anything; the ratio alone cannot '
+ 'tell you which they are.');
}
lines.push(`re-read project ${pid} with the same two usage reports after the `
+ 'deploy, and check the model column, not only the count');
return lines;
}
async function read(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) {
if (Array.isArray(v)) for (const one of v) url.searchParams.append(k, String(one));
else url.searchParams.set(k, String(v));
}
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: the usage reports need an `
+ 'organization admin key with api.usage.read, not a project key');
}
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function usage(key, path, start, end) {
const params = { start_time: start, end_time: end, bucket_width: '1d',
limit: 31, group_by: ['project_id', 'model'] };
const out = [];
for (;;) {
const page = await read(key, path, params);
out.push(...(page.data ?? []));
if (!page.has_more || !page.next_page) return out;
params.page = page.next_page;
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key; a project '
+ 'key cannot read /v1/organization/usage/*');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 30);
const minCompletions = Number(process.env.MIN_COMPLETIONS ?? 500);
const minRatio = Number(process.env.MIN_RATIO ?? 0.2);
const end = Math.floor(Date.now() / 1000);
const start = end - Math.max(1, days) * DAY;
const completions = fold(await usage(admin, '/organization/usage/completions', start, end));
const moderations = fold(await usage(admin, '/organization/usage/moderations', start, end));
const graded = coverage(completions, moderations)
.map((row) => [row, classify(row, minCompletions, minRatio)]);
const bad = graded.filter(([, [state]]) => FINDINGS.has(state));
const overFloor = graded.filter(([, [state]]) => state !== 'below-floor').length;
console.log(`${overFloor} project(s) over the ${minCompletions} request floor, `
+ `${bad.length} finding(s)`);
bad.sort(([ra, [sa]], [rb, [sb]]) =>
(SEVERITY[sa] ?? 9) - (SEVERITY[sb] ?? 9) || rb[1] - ra[1]);
for (const [row, [state, detail]] of bad) {
console.warn(`${state.padEnd(18)} ${row[0].padEnd(14)} ${detail}`);
for (const line of repairLines(state, row)) console.warn(` repair: ${line}`);
}
console.log('not graded: this report counts requests. Whether the code branched '
+ "on flagged, and whether the input assessed was the user's, are not "
+ 'in the API.');
process.exitCode = bad.length ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first two tests are the note. A project with forty-one thousand completions and no moderations entry at all has to survive the join and come back as never-called, which is the case a fold that drops empty projects silently deletes. And a project moderating every request through text-moderation-latest has to come back as a finding rather than as coverage, because a count-based audit calls that one fine. Around them: the prefix match, so a pinned text-moderation-007 is caught alongside -latest and omni-moderation-latest is not; the volume floor, so a scratch project is never graded; a zero-valued result row, which must not create a phantom entry; and the repair for a retired id, which has to mention images, because that is the half of the gap nobody expects.
from openai_moderation_coverage_audit import (classify, coverage, fold,
is_retired, repair_lines,
retired_ids)
COMPLETION = "organization.usage.completions.result"
MODERATION = "organization.usage.moderations.result"
def bucket(*results):
return {"object": "bucket", "start_time": 0, "end_time": 86400,
"results": list(results)}
def row(project, model, n, obj=MODERATION):
return {"object": obj, "project_id": project, "model": model,
"num_model_requests": n, "input_tokens": n * 12}
def test_a_busy_project_with_no_moderations_entry_survives_the_join():
# The note. proj_public never appears in the moderations report at all, so
# a join driven by the moderation side would report nothing wrong.
completions = fold([bucket(row("proj_public", "gpt-4.1-mini", 20604, COMPLETION),
row("proj_intake", "gpt-4.1", 2000, COMPLETION)),
bucket(row("proj_public", "gpt-4.1-mini", 20604, COMPLETION))])
moderations = fold([bucket(row("proj_intake", "omni-moderation-latest", 1900))])
rows = coverage(completions, moderations)
assert [r[0] for r in rows] == ["proj_public", "proj_intake"]
state, detail = classify(rows[0])
assert state == "never-called"
assert "41208 completion request(s)" in detail
lines = repair_lines(state, rows[0])
assert any("bills nothing" in line for line in lines)
assert any("category_scores" in line for line in lines)
assert classify(rows[1])[0] == "covered"
def test_full_coverage_on_a_retired_id_is_a_finding_not_coverage():
# Every request moderated, ratio near 1.0, and still wrong. A count-based
# audit reports this project as healthy.
completions = fold([bucket(row("proj_old", "gpt-4.1", 4000, COMPLETION))])
moderations = fold([bucket(row("proj_old", "text-moderation-latest", 3904))])
rows = coverage(completions, moderations)
state, detail = classify(rows[0])
assert state == "retired-model-id"
assert "100% of them on text-moderation-latest" in detail
lines = repair_lines(state, rows[0])
assert any("omni-moderation-latest" in line for line in lines)
assert any("images" in line for line in lines)
def test_a_pinned_snapshot_is_caught_and_the_current_id_is_not():
assert is_retired("text-moderation-007") is True
assert is_retired("text-moderation-stable") is True
assert is_retired("TEXT-MODERATION-LATEST") is True
assert is_retired("omni-moderation-latest") is False
assert is_retired("omni-moderation-2024-09-26") is False
assert is_retired(None) is False
assert retired_ids({"omni-moderation-latest": 5, "text-moderation-007": 2}) == \
["text-moderation-007"]
# A part-migrated project is still a finding, and both ids are named.
mixed = ("proj_half", 4000, 3900,
{"omni-moderation-latest": 3000, "text-moderation-007": 900})
state, detail = classify(mixed)
assert state == "retired-model-id"
assert "23% of them" in detail
def test_a_low_volume_project_is_never_graded():
quiet = ("proj_scratch", 41, 0, {})
state, detail = classify(quiet)
assert state == "below-floor"
assert "under the 500 floor" in detail
assert repair_lines(state, quiet) == []
# And it becomes gradeable once the floor is lowered deliberately.
assert classify(quiet, min_completions=10)[0] == "never-called"
def test_a_zero_valued_result_row_creates_no_entry():
# A project present in the report with a zero count must not look moderated.
moderations = fold([bucket(row("proj_a", "omni-moderation-latest", 0),
row("proj_b", "omni-moderation-latest", 7))])
assert "proj_a" not in moderations
assert moderations["proj_b"]["requests"] == 7
assert fold(None) == {}
assert fold([{"results": None}]) == {}
assert fold([bucket({"num_model_requests": "not a number"})]) == {}
# A result with no project_id is kept under an explicit name, not dropped.
assert "unattributed" in fold([bucket({"num_model_requests": 3})])
def test_the_ratio_is_graded_as_the_soft_signal_it_is():
thin = ("proj_thin", 10000, 400, {"omni-moderation-latest": 400})
state, detail = classify(thin)
assert state == "thin-coverage"
assert "ratio of 0.04" in detail
assert any("cannot tell you which" in line for line in repair_lines(state, thin))
assert classify(thin, min_ratio=0.01)[0] == "covered"
assert repair_lines("covered", thin) == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, coverage, fold, isRetired, repairLines, retiredIds }
from './openai-moderation-coverage-audit.mjs';
const COMPLETION = 'organization.usage.completions.result';
const MODERATION = 'organization.usage.moderations.result';
const bucket = (...results) =>
({ object: 'bucket', start_time: 0, end_time: 86400, results });
const row = (project, model, n, obj = MODERATION) =>
({ object: obj, project_id: project, model, num_model_requests: n,
input_tokens: n * 12 });
test('a busy project with no moderations entry survives the join', () => {
const completions = fold([
bucket(row('proj_public', 'gpt-4.1-mini', 20604, COMPLETION),
row('proj_intake', 'gpt-4.1', 2000, COMPLETION)),
bucket(row('proj_public', 'gpt-4.1-mini', 20604, COMPLETION))]);
const moderations = fold([bucket(row('proj_intake', 'omni-moderation-latest', 1900))]);
const rows = coverage(completions, moderations);
assert.deepEqual(rows.map((r) => r[0]), ['proj_public', 'proj_intake']);
const [state, detail] = classify(rows[0]);
assert.equal(state, 'never-called');
assert.ok(detail.includes('41208 completion request(s)'));
const lines = repairLines(state, rows[0]);
assert.ok(lines.some((l) => l.includes('bills nothing')));
assert.ok(lines.some((l) => l.includes('category_scores')));
assert.equal(classify(rows[1])[0], 'covered');
});
test('full coverage on a retired id is a finding, not coverage', () => {
const completions = fold([bucket(row('proj_old', 'gpt-4.1', 4000, COMPLETION))]);
const moderations = fold([bucket(row('proj_old', 'text-moderation-latest', 3904))]);
const rows = coverage(completions, moderations);
const [state, detail] = classify(rows[0]);
assert.equal(state, 'retired-model-id');
assert.ok(detail.includes('100% of them on text-moderation-latest'));
const lines = repairLines(state, rows[0]);
assert.ok(lines.some((l) => l.includes('omni-moderation-latest')));
assert.ok(lines.some((l) => l.includes('images')));
});
test('a pinned snapshot is caught and the current id is not', () => {
assert.equal(isRetired('text-moderation-007'), true);
assert.equal(isRetired('text-moderation-stable'), true);
assert.equal(isRetired('TEXT-MODERATION-LATEST'), true);
assert.equal(isRetired('omni-moderation-latest'), false);
assert.equal(isRetired('omni-moderation-2024-09-26'), false);
assert.equal(isRetired(null), false);
assert.deepEqual(retiredIds({ 'omni-moderation-latest': 5, 'text-moderation-007': 2 }),
['text-moderation-007']);
const mixed = ['proj_half', 4000, 3900,
{ 'omni-moderation-latest': 3000, 'text-moderation-007': 900 }];
const [state, detail] = classify(mixed);
assert.equal(state, 'retired-model-id');
assert.ok(detail.includes('23% of them'));
});
test('a low volume project is never graded', () => {
const quiet = ['proj_scratch', 41, 0, {}];
const [state, detail] = classify(quiet);
assert.equal(state, 'below-floor');
assert.ok(detail.includes('under the 500 floor'));
assert.deepEqual(repairLines(state, quiet), []);
assert.equal(classify(quiet, 10)[0], 'never-called');
});
test('a zero valued result row creates no entry', () => {
const moderations = fold([bucket(row('proj_a', 'omni-moderation-latest', 0),
row('proj_b', 'omni-moderation-latest', 7))]);
assert.equal(moderations.proj_a, undefined);
assert.equal(moderations.proj_b.requests, 7);
assert.deepEqual(fold(null), {});
assert.deepEqual(fold([{ results: null }]), {});
assert.deepEqual(fold([bucket({ num_model_requests: 'not a number' })]), {});
assert.ok('unattributed' in fold([bucket({ num_model_requests: 3 })]));
});
test('the ratio is graded as the soft signal it is', () => {
const thin = ['proj_thin', 10000, 400, { 'omni-moderation-latest': 400 }];
const [state, detail] = classify(thin);
assert.equal(state, 'thin-coverage');
assert.ok(detail.includes('ratio of 0.04'));
assert.ok(repairLines(state, thin).some((l) => l.includes('cannot tell you which')));
assert.equal(classify(thin, 500, 0.01)[0], 'covered');
assert.deepEqual(repairLines('covered', thin), []);
});
FAQ
Why does the script not just call the moderations endpoint to test it?
Because calling it is generating, and nothing in this section generates. The moderations endpoint takes content and runs a model over it; a script that sends it a test string to prove the endpoint responds has made an inference request on a key that can make inference requests, which is exactly the line every script here refuses to cross. It costs nothing in money, which is not the point. The finding is available without it: the usage report already counts every moderation request the organization has made, per project and per model id, and that is what the script reads.
Is the completions-to-moderations ratio really meaningful?
Only weakly, and the script grades it as the weak signal it is. It understates coverage because one moderation request can carry an array of inputs and still counts as a single num_model_requests. It overstates the gap because the completions figure includes batch jobs, evaluation runs and internal tooling, none of which has user input in it. So a ratio below the floor is reported under the name thin-coverage with a repair that says to go and find the call sites, rather than as a number to hit. The two states that are worth acting on are the ones that do not depend on the ratio at all: no moderation requests whatsoever, and requests on a retired model id.
What is actually wrong with text-moderation-latest? It still returns a verdict.
It does, and that is why nobody notices. The text-moderation family is retired in favour of omni-moderation-latest, and the difference that bites is not the accuracy of the categories, it is that the old family reads text only. A product that added image uploads after the moderation call was written has a screening layer covering one half of its input surface, with no error and no log line anywhere saying so. The newer id also returns category_applied_input_types, which tells you which modality triggered a flag, and that is the field you want when somebody asks why a particular item was blocked.
Does Anthropic have an equivalent endpoint to audit?
Not a free standalone classifier of this shape, which is why this note is written against OpenAI. Anthropic's safety behaviour is carried in the model's own refusals and in stop_reason, and its usage report has no request-count field at all, so there is nothing to compare a moderation count against even in principle. The nearest Anthropic reading is a separate question about refusals in your own traffic, and it is a different note with a different mechanism.
The report says never-called, but we do screen input with our own classifier. Is it wrong?
It is right about what it measured and it does not know about your classifier. The usage report can only see requests made to OpenAI, so an in-house model, a third-party service or a rules engine is invisible to it. Read the finding as what it literally says: this project has never used the provider's moderation endpoint. If a different layer covers it, the useful follow-up is whether that layer reads images, whether it produces per-category scores you can threshold, and whether it costs anything, because the endpoint it is replacing does not.
Related field notes
- The model's own refusal, which is not a moderation decision
- Whether any policy would have stopped the call
- A usage report reading zero for a much duller reason
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Moderation — OpenAI platform docs
- Usage and costs — OpenAI API reference
- Admin APIs — OpenAI platform docs
- openai-python admin API surface
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.