Diagnostic LLM APIs
A live project's usage buckets have been empty for days
A customer asks, politely, when the summaries are coming back. Nobody on the call knows what they mean, because the feature works: the page renders, the job runs, the queue is empty, the error rate is zero and the latency graph is the flattest it has been all year. It is flat because for eleven days that project has sent the API nothing at all. A feature flag went the wrong way in an unrelated release, and every dashboard the team owns measures things that only exist when requests do.
Ask for a fortnight of daily buckets per project and look for the ones that stop. With an organization admin key: GET /v1/organization/usage/completions?start_time={now-14d}&bucket_width=1d&limit=14&group_by=project_id. Buckets come back for the whole range whether or not there was traffic, so a project that has gone dark shows results as an empty array for the recent days while the earlier ones are full.
The rule is: non-zero in the first part of the window, zero for the last 48 hours, and the project still active. Drop today's bucket, which is always partial, and treat a project whose traffic starts in the tail as a launch rather than a death — those two look identical if you only compare halves.
Then sweep the other usage surfaces, because completions is one of eight: embeddings, images, audio speeches, audio transcriptions, moderations, file search and web search all take the same parameters. Quiet everywhere is a credential or a deploy. Quiet on one surface while another is still busy is one code path, which is a much smaller search.
The problem in plain words
Every alarm a team owns is a ceiling. Error rate above a threshold, latency above a threshold, spend above a threshold, queue depth above a threshold. All of them read perfectly when the number is zero, and an integration that stops calling the API produces zeroes everywhere at once. There is no exception to catch, because nothing was attempted; no 4xx, no 5xx, no timeout, no retry. Silence is the one state monitoring built on thresholds cannot see.
So these get found by customers, or by a quarterly reconciliation, or not at all. The causes are dull and common: a feature flag flipped in an unrelated release, a consumer that died and was never restarted, a config that now points at a key in a different project, a refactor that left a condition permanently false, an upstream producer that stopped emitting the events which triggered the calls. None of them announces itself, and all of them look the same from the outside.
Why it happens
The endpoint answers with a shape, not with a zero. A project with no traffic in a bucket does not come back as num_model_requests: 0; it comes back as a bucket whose results array is empty, or with that project absent from the results entirely. A parser that assumes every project appears in every bucket silently skips exactly the case this note is about, so the day axis has to come from the window you asked for rather than from what the response happened to contain.
A launch and a death are the same shape read backwards. Traffic in one half of the window and none in the other is the finding, and which half decides everything. The check is directional or it fires on every new project in the organization, once, and then gets muted.
The key roster corroborates the silence and says how wide it is. GET /v1/organization/projects/{project_id}/api_keys?owner_project_access=any returns last_used_at per key. Frozen at about the hour the buckets stopped means the whole integration went quiet. Still moving while usage is empty means something is authenticating and not inferring — a health check, a listing call, a surface this sweep did not read — and that is a much narrower fault. The owner_project_access=any parameter matters: without it the roster can be filtered to keys the caller can see and an audit quietly reads a subset.
Usage data lags, so never alert on the current bucket. Today is always partial and cost and usage can both be revised as late events land. A 48-hour quiet window is short enough to be useful and long enough not to fire on a slow Sunday plus reporting lag, and the current day is dropped before anything is compared.
This is not the orphaned-key check. When a key's owner loses access to a project, the platform sets a flag and there is a note about reading it. Here nothing is flagged, nothing is disabled, the key is valid, the project is active and the credential would work perfectly if anything called it. The provider has no opinion to read, so the only evidence is the absence, which is why this check has to be built rather than subscribed to.
It is also not a spend anomaly. A week-over-week cost check classifies the shape of a change in one org-wide series, and a fall in dollars is one of the things it reports. This one runs per project, at day granularity, and its only interesting output is a floor being crossed: it is the check whose entire value is that it fires on zero.
The fix, as a flow
The only note in the batch whose finding is an absence. Every alarm a team owns is a ceiling, and a ceiling reads perfectly at zero. The endpoint returns a bucket for every day you asked for, so the day axis comes from the window rather than from the response, and the check is directional because a launch is a death read backwards.
How to fix it
List the projects and keep the active ones
GET /v1/organization/projects?limit=100, paging on after. Archived projects going quiet is expected and is a different note's business, so filter to status == "active" before anything else. A project that has never had traffic in the window is also not a finding — it is either new or dormant, and both are reported as their own state.
Read fourteen daily buckets per usage surface
bucket_width=1d, limit=14, group_by=project_id, repeated across the surfaces the organization actually uses. Build the day axis from the window you requested rather than from the days that came back, because the missing days are the finding.
Drop today before comparing anything
The current bucket is partial by definition and usage data arrives with some lag. Compare complete days only. This is the same discipline the cost checks need and it matters more here, because the whole signal is a run of zeroes and a partial day is a small zero.
Split the window and require the traffic to be in the early half
Non-zero before the quiet window, zero inside it. Reverse that and you have a launch. Report the last day with traffic, how many complete days ago it was, and the mean daily volume before it stopped, because those three numbers are what turn "it is quiet" into "it stopped on the sixteenth, and it was doing four thousand calls a day".
Corroborate with the keys, then print an alarm with a floor
Check last_used_at across the project's keys to separate a dead integration from a live one that stopped inferring. The repair is not a code change the script can name, because the cause is in your deploy: it is a scheduled liveness check that treats absence as an alert condition, per project, with a floor rather than a ceiling. Print it, including the endpoint and the threshold to use.
How to check it worked
Once the floor alarm exists, this script stops being interesting, which is the goal. Until then run it daily; a project that has resumed moves to live on the next run.
python3 openai_project_went_quiet.py --days 14 --quiet-days 2
# went-quiet proj_summaries completions: last traffic on 2026-08-16, 2 complete day(s) ago, after a prior mean of 4102 request(s) a day
# the newest key use is 2.1 day(s) ago, which lines up with the buckets. The integration went quiet, not one call site.
# still live on: embeddings
# 6 active project(s) checked, 1 finding(s)The full code
Three GETs: the project roster, one usage sweep per surface, and the key list only for projects that turn up quiet. Seven pure functions: the day key and the complete-day axis, which is built from the window rather than from the response because the missing days are the point; the fold; the classifier, which is directional so a launch is not reported as a death; the key reader; the corroboration that separates a dead integration from one that is still authenticating; and the surface split, which turns “the project is quiet” into “one code path is quiet” when another surface is still busy.
"""Find OpenAI projects whose usage buckets went empty while the project is live.
Read only. GET requests against the organization endpoints, which reject
project keys: this needs an organization admin key (sk-admin-), and read-only
scopes are enough.
The finding is an absence. Nothing errored, because nothing was sent, so there
is no status code anywhere to look up. The usage endpoint returns buckets for
the whole window whether or not there was traffic, which makes the empty ones
readable, and the day axis is built from the window requested rather than from
the days that came back.
The repair is printed, never performed. What is missing is an alarm with a
floor instead of a ceiling, and that lives in your monitoring, not here.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_project_went_quiet")
API = "https://api.openai.com/v1"
# Completions is one surface of eight, and a project can go quiet on one while
# staying busy on another. Quiet everywhere is a credential or a deploy; quiet
# on one is a code path, which is a far smaller thing to search.
SURFACES = ("completions", "embeddings", "images", "audio_speeches",
"audio_transcriptions", "moderations", "file_search_calls",
"web_search_calls")
# Each surface counts a different thing, and exactly one of these appears on any
# given result.
COUNT_FIELDS = ("num_model_requests", "num_requests", "num_images",
"num_seconds", "num_characters")
FINDINGS = ("went-quiet",)
def _int(value):
"""Read a usage field as an int. Pure. Missing and unreadable both mean 0."""
try:
return int(value or 0)
except (TypeError, ValueError):
return 0
def day_key(epoch):
"""The UTC day a bucket start belongs to. Pure. None if unreadable."""
try:
return dt.datetime.fromtimestamp(int(epoch), dt.timezone.utc).strftime("%Y-%m-%d")
except (TypeError, ValueError, OSError, OverflowError):
return None
def complete_days(now_epoch, days):
"""The last N complete UTC days, oldest first. Pure.
Today is excluded. The current bucket is partial by definition and usage
data lags, so a run of zeroes that includes today is one day shorter than
it looks, and the whole finding is a run of zeroes.
Built here rather than read off the response, because a project with no
traffic may be absent from a bucket entirely and the missing days are the
thing being looked for.
"""
out = []
for offset in range(int(days), 0, -1):
key = day_key(int(now_epoch) - offset * 86400)
if key is not None:
out.append(key)
return out
def daily(buckets):
"""{project_id: {day: count}} from one usage surface. Pure.
Surfaces count different things, so the first recognised field wins rather
than being summed: a result carrying both would otherwise be counted twice.
"""
out = {}
for bucket in buckets or []:
day = day_key(bucket.get("start_time"))
if day is None:
continue
for result in bucket.get("results") or []:
project = str(result.get("project_id") or "unknown")
count = 0
for field in COUNT_FIELDS:
if field in result:
count = _int(result.get(field))
break
row = out.setdefault(project, {})
row[day] = row.get(day, 0) + count
return out
def classify(series, days, quiet_days=2, min_requests=100):
"""Classify one project's daily series. Pure. Returns (state, detail).
Directional on purpose. Traffic in the early days and none in the last two
is a project that stopped; the reverse is a project that started, and a
check that cannot tell them apart fires on every launch and gets muted.
"""
days = list(days or [])
if len(days) <= quiet_days:
return ("window-too-short",
"%d complete day(s) is not enough to hold a %d day quiet "
"window" % (len(days), quiet_days))
series = series or {}
head, tail = days[:-quiet_days], days[-quiet_days:]
prior = sum(_int(series.get(day)) for day in head)
recent = sum(_int(series.get(day)) for day in tail)
active = [day for day in days if _int(series.get(day)) > 0]
if not active:
return ("never-active",
"no traffic at all across %d complete day(s)" % len(days))
if prior == 0:
return ("new-traffic",
"first traffic in this window landed on %s, inside the last %d "
"day(s). A launch reads exactly like a death if you only "
"compare halves." % (active[0], quiet_days))
if recent > 0:
return ("live",
"%d request(s) in the last %d day(s), against a prior mean of "
"%d a day" % (recent, quiet_days, prior / float(len(head))))
if prior < min_requests:
return ("too-little-traffic",
"%d request(s) before the quiet window, under the floor of %d. "
"Too sporadic for a gap to mean anything." % (prior, min_requests))
since = len(days) - 1 - days.index(active[-1])
return ("went-quiet",
"last traffic on %s, %d complete day(s) ago, after a prior mean of "
"%d request(s) a day"
% (active[-1], since, prior / float(len(head))))
def key_activity(keys, now_epoch):
"""The newest last_used_at across a project's keys. Pure.
Returns (epoch, days_since), or (None, None) when no key reports a use.
"""
best = None
for key in keys or []:
try:
used = key.get("last_used_at")
except AttributeError:
continue
if used is None:
continue
try:
used = int(used)
except (TypeError, ValueError):
continue
if best is None or used > best:
best = used
if best is None:
return (None, None)
return (best, max(0.0, (int(now_epoch) - best) / 86400.0))
def corroborate(days_since, quiet_days=2):
"""Line the key roster up against the silence. Pure. Returns (state, detail).
A key still in use while the buckets are empty is a much narrower fault
than a key that went quiet at the same moment: something is authenticating
and not inferring.
"""
if days_since is None:
return ("no-key-use",
"no key on this project reports a last use, so there is "
"nothing here to corroborate the silence with")
if days_since <= quiet_days:
return ("key-still-used",
"a key on this project was used %.1f day(s) ago while the "
"usage buckets were empty. Something is still authenticating "
"and not inferring: a health check, or a surface this sweep "
"did not read." % days_since)
return ("key-quiet-too",
"the newest key use is %.1f day(s) ago, which lines up with the "
"buckets. The integration went quiet, not one call site."
% days_since)
def surface_split(states):
"""(quiet, live) surface names for one project. Pure.
Quiet on one surface while another is still busy is a code path rather than
a credential, and that difference is worth more than the finding itself.
"""
quiet = sorted(name for name, state in (states or {}).items()
if state == "went-quiet")
live = sorted(name for name, state in (states or {}).items()
if state == "live")
return (quiet, live)
def get(session, path, params=None):
r = session.get(API + path, params=params or {}, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key (sk-admin-), not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def pages(session, path, params, max_pages=40):
"""Walk a usage report, which paginates on an opaque page cursor."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params = dict(params)
params["page"] = page["next_page"]
def listing(session, path, params, max_pages=20):
"""Walk a list endpoint, which paginates on an object id."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, params)
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params = dict(params)
params["after"] = data[-1].get("id")
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=14,
help="complete days to read (default 14)")
ap.add_argument("--quiet-days", type=int, default=2,
help="days of silence that make a finding (default 2)")
ap.add_argument("--min-requests", type=int, default=100,
help="ignore projects quieter than this before the gap "
"(default 100)")
ap.add_argument("--show-all", action="store_true",
help="also print projects that are still live")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY (an organization admin key; read-only "
"scopes are enough)")
return 2
now = int(time.time())
days = complete_days(now, max(3, min(int(args.days), 30)))
session = requests.Session()
session.headers.update({"Authorization": "Bearer " + admin})
projects = [p for p in listing(session, "/organization/projects", {"limit": 100})
if str(p.get("status") or "") == "active"]
if not projects:
log.info("no active projects in this organization")
return 0
per_surface = {}
for surface in SURFACES:
try:
per_surface[surface] = daily(pages(
session, "/organization/usage/" + surface,
{"start_time": now - (len(days) + 1) * 86400,
"bucket_width": "1d", "limit": len(days) + 1,
"group_by": ["project_id"]}))
except requests.HTTPError:
# A surface the organization has never used can 400 rather than
# returning an empty window. Not a finding, and not fatal.
log.info("skipped the %s usage surface", surface)
checked = 0
bad = 0
for project in projects:
project_id = str(project.get("id") or "")
states = {}
details = {}
for surface, rows in per_surface.items():
state, detail = classify(rows.get(project_id), days,
args.quiet_days, args.min_requests)
states[surface] = state
details[surface] = detail
checked += 1
quiet, live = surface_split(states)
if not quiet:
if args.show_all:
log.info("%-18s %s no surface went quiet", "live", project_id)
continue
bad += 1
log.warning("%-18s %s %s: %s", "went-quiet", project_id, quiet[0],
details[quiet[0]])
keys = list(listing(session,
"/organization/projects/%s/api_keys" % project_id,
{"limit": 100, "owner_project_access": "any"}))
_, note = corroborate(key_activity(keys, now)[1], args.quiet_days)
log.warning(" %s", note)
if live:
log.warning(" still live on: %s", ", ".join(live))
log.warning(" repair: one code path stopped calling, not the "
"credential. Look at the deploy that touched it rather "
"than at the key.")
else:
log.warning(" repair: every surface is quiet, so look at the "
"credential, the feature flag or the consumer before "
"the call site.")
log.warning(" repair: add a scheduled liveness check that alerts on "
"absence. Read /v1/organization/usage/completions daily "
"with group_by=project_id and page on next_page, and alert "
"when a project falls below a floor rather than above a "
"ceiling. This is the one check whose value is that it "
"fires on zero.")
log.info("%d active project(s) checked, %d finding(s)", checked, bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find OpenAI projects whose usage buckets went empty while the project is live.
*
* Read only. GET requests against the organization endpoints, which reject
* project keys: this needs an organization admin key (sk-admin-).
*
* The finding is an absence, so the day axis is built from the window that was
* requested rather than from the days that came back. The repair is printed,
* never performed: what is missing is an alarm with a floor instead of a
* ceiling, and that lives in your monitoring.
*/
const API = 'https://api.openai.com/v1';
const SURFACES = ['completions', 'embeddings', 'images', 'audio_speeches',
'audio_transcriptions', 'moderations', 'file_search_calls',
'web_search_calls'];
const COUNT_FIELDS = ['num_model_requests', 'num_requests', 'num_images',
'num_seconds', 'num_characters'];
/** Read a usage field as an integer. Pure. Missing and unreadable both mean 0. */
export function readInt(value) {
const n = Number(value ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/** The UTC day a bucket start belongs to. Pure. Null if unreadable. */
export function dayKey(epoch) {
const n = Number(epoch);
if (!Number.isFinite(n)) return null;
const when = new Date(Math.trunc(n) * 1000);
if (Number.isNaN(when.getTime())) return null;
return when.toISOString().slice(0, 10);
}
/**
* The last N complete UTC days, oldest first. Pure.
* Today is excluded: the current bucket is partial and usage data lags, so a
* run of zeroes that includes it is one day shorter than it looks.
*/
export function completeDays(nowEpoch, days) {
const out = [];
for (let offset = Math.trunc(days); offset > 0; offset -= 1) {
const key = dayKey(Math.trunc(nowEpoch) - offset * 86400);
if (key !== null) out.push(key);
}
return out;
}
/**
* {project_id: {day: count}} from one usage surface. Pure.
* First recognised count field wins rather than being summed.
*/
export function daily(buckets) {
const out = new Map();
for (const bucket of buckets ?? []) {
const day = dayKey(bucket?.start_time);
if (day === null) continue;
for (const result of bucket?.results ?? []) {
const project = String(result?.project_id ?? 'unknown');
let count = 0;
for (const field of COUNT_FIELDS) {
if (result && field in result) { count = readInt(result[field]); break; }
}
if (!out.has(project)) out.set(project, new Map());
const row = out.get(project);
row.set(day, (row.get(day) ?? 0) + count);
}
}
return out;
}
/**
* Classify one project's daily series. Pure. Returns [state, detail].
* Directional on purpose: traffic early and none late is a project that
* stopped, the reverse is one that started, and a check that cannot tell them
* apart fires on every launch and gets muted.
*/
export function classify(series, days, quietDays = 2, minRequests = 100) {
const axis = [...(days ?? [])];
if (axis.length <= quietDays) {
return ['window-too-short',
`${axis.length} complete day(s) is not enough to hold a ${quietDays} ` +
'day quiet window'];
}
const at = (day) => readInt(series?.get ? series.get(day) : series?.[day]);
const head = axis.slice(0, axis.length - quietDays);
const tail = axis.slice(axis.length - quietDays);
const prior = head.reduce((sum, day) => sum + at(day), 0);
const recent = tail.reduce((sum, day) => sum + at(day), 0);
const active = axis.filter((day) => at(day) > 0);
if (active.length === 0) {
return ['never-active', `no traffic at all across ${axis.length} complete day(s)`];
}
if (prior === 0) {
return ['new-traffic',
`first traffic in this window landed on ${active[0]}, inside the last ` +
`${quietDays} day(s). A launch reads exactly like a death if you only ` +
'compare halves.'];
}
if (recent > 0) {
return ['live',
`${recent} request(s) in the last ${quietDays} day(s), against a prior ` +
`mean of ${Math.trunc(prior / head.length)} a day`];
}
if (prior < minRequests) {
return ['too-little-traffic',
`${prior} request(s) before the quiet window, under the floor of ` +
`${minRequests}. Too sporadic for a gap to mean anything.`];
}
const last = active[active.length - 1];
const since = axis.length - 1 - axis.indexOf(last);
return ['went-quiet',
`last traffic on ${last}, ${since} complete day(s) ago, after a prior mean ` +
`of ${Math.trunc(prior / head.length)} request(s) a day`];
}
/** The newest last_used_at across a project's keys. Pure. [epoch, daysSince]. */
export function keyActivity(keys, nowEpoch) {
let best = null;
for (const key of keys ?? []) {
const used = key?.last_used_at;
if (used === null || used === undefined) continue;
const n = Number(used);
if (!Number.isFinite(n)) continue;
if (best === null || n > best) best = Math.trunc(n);
}
if (best === null) return [null, null];
return [best, Math.max(0, (Math.trunc(nowEpoch) - best) / 86400)];
}
/**
* Line the key roster up against the silence. Pure. Returns [state, detail].
* A key still in use while the buckets are empty is a much narrower fault.
*/
export function corroborate(daysSince, quietDays = 2) {
if (daysSince === null || daysSince === undefined) {
return ['no-key-use',
'no key on this project reports a last use, so there is nothing here to ' +
'corroborate the silence with'];
}
if (daysSince <= quietDays) {
return ['key-still-used',
`a key on this project was used ${daysSince.toFixed(1)} day(s) ago while ` +
'the usage buckets were empty. Something is still authenticating and not ' +
'inferring: a health check, or a surface this sweep did not read.'];
}
return ['key-quiet-too',
`the newest key use is ${daysSince.toFixed(1)} day(s) ago, which lines up ` +
'with the buckets. The integration went quiet, not one call site.'];
}
/** [quiet, live] surface names for one project. Pure. */
export function surfaceSplit(states) {
const entries = states instanceof Map ? [...states] : Object.entries(states ?? {});
const quiet = entries.filter(([, s]) => s === 'went-quiet').map(([n]) => n).sort();
const live = entries.filter(([, s]) => s === 'live').map(([n]) => n).sort();
return [quiet, live];
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params ?? {})) {
if (Array.isArray(v)) v.forEach((item) => url.searchParams.append(k, item));
else url.searchParams.set(k, String(v));
}
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from OpenAI: /v1/organization/* needs an ` +
'organization admin key (sk-admin-), not a project key');
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
async function* pages(key, path, params, maxPages = 40) {
let query = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await get(key, path, query);
for (const bucket of page?.data ?? []) yield bucket;
if (!page?.has_more || !page?.next_page) return;
query = { ...params, page: page.next_page };
}
}
async function* listing(key, path, params, maxPages = 20) {
let query = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await get(key, path, query);
const data = page?.data ?? [];
for (const item of data) yield item;
if (!page?.has_more || data.length === 0) return;
query = { ...params, after: data[data.length - 1]?.id };
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY (an organization admin key; read-only ' +
'scopes are enough)');
process.exitCode = 2;
return;
}
const now = Math.floor(Date.now() / 1000);
const quietDays = Number(process.env.QUIET_DAYS ?? 2);
const minRequests = Number(process.env.MIN_REQUESTS ?? 100);
const showAll = process.env.SHOW_ALL === '1';
const days = completeDays(now, Math.max(3, Math.min(Number(process.env.DAYS ?? 14), 30)));
const projects = [];
for await (const project of listing(admin, '/organization/projects', { limit: 100 })) {
if (String(project?.status ?? '') === 'active') projects.push(project);
}
if (projects.length === 0) {
console.log('no active projects in this organization');
return;
}
const perSurface = new Map();
for (const surface of SURFACES) {
try {
const buckets = [];
for await (const bucket of pages(admin, `/organization/usage/${surface}`, {
start_time: now - (days.length + 1) * 86400,
bucket_width: '1d',
limit: days.length + 1,
group_by: ['project_id'],
})) buckets.push(bucket);
perSurface.set(surface, daily(buckets));
} catch {
console.log(`skipped the ${surface} usage surface`);
}
}
let checked = 0;
let bad = 0;
for (const project of projects) {
const projectId = String(project?.id ?? '');
const states = new Map();
const details = new Map();
for (const [surface, rows] of perSurface) {
const [state, detail] = classify(rows.get(projectId), days, quietDays, minRequests);
states.set(surface, state);
details.set(surface, detail);
}
checked += 1;
const [quiet, live] = surfaceSplit(states);
if (quiet.length === 0) {
if (showAll) console.log(`live ${projectId} no surface went quiet`);
continue;
}
bad += 1;
console.warn(`went-quiet ${projectId} ${quiet[0]}: ${details.get(quiet[0])}`);
const keys = [];
for await (const key of listing(admin, `/organization/projects/${projectId}/api_keys`,
{ limit: 100, owner_project_access: 'any' })) {
keys.push(key);
}
const [, note] = corroborate(keyActivity(keys, now)[1], quietDays);
console.warn(` ${note}`);
if (live.length > 0) {
console.warn(` still live on: ${live.join(', ')}`);
console.warn(' repair: one code path stopped calling, not the credential. ' +
'Look at the deploy that touched it rather than at the key.');
} else {
console.warn(' repair: every surface is quiet, so look at the credential, ' +
'the feature flag or the consumer before the call site.');
}
console.warn(' repair: add a scheduled liveness check that alerts on absence. ' +
'Read /v1/organization/usage/completions daily with ' +
'group_by=project_id and page on next_page, and alert when a ' +
'project falls below a floor rather than above a ceiling. This is ' +
'the one check whose value is that it fires on zero.');
}
console.log(`${checked} active project(s) checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing test is a fortnight where one project stops on the sixteenth and its neighbour does not, and it asserts the report says the date, how many complete days ago it was, and what the project used to do daily — because “quiet” on its own sends nobody anywhere. Immediately after it is the test that stops this check from being muted within a week: the same shape reversed, a project whose traffic starts in the last two days, which has to come back as a launch. The rest pin the day axis excluding today, the key roster telling a dead integration apart from one that is still authenticating, and the surface split that narrows a credential problem down to a single code path.
from openai_project_went_quiet import (classify, complete_days, corroborate,
daily, day_key, key_activity,
surface_split)
DAYS = ["2026-08-%02d" % n for n in range(5, 19)] # 14 complete days
NOW = 1787097600 # 2026-08-19T00:00:00Z
def test_a_project_that_stops_is_named_with_a_date_and_a_volume():
# The note in one assertion. Twelve busy days, then two empty ones, and the
# report has to say when it stopped and what it used to do.
series = {day: 4102 for day in DAYS[:12]}
state, detail = classify(series, DAYS)
assert state == "went-quiet"
assert "last traffic on 2026-08-16" in detail
assert "2 complete day(s) ago" in detail
assert "prior mean of 4102 request(s) a day" in detail
# The project next to it never stopped, and must not be reported.
assert classify({day: 4102 for day in DAYS}, DAYS)[0] == "live"
def test_a_launch_is_not_a_death_read_backwards():
# The same shape, reversed. Get this wrong and the check fires on every new
# project once and is muted by the end of the week.
state, detail = classify({DAYS[12]: 900, DAYS[13]: 1200}, DAYS)
assert state == "new-traffic"
assert "first traffic in this window landed on 2026-08-17" in detail
def test_the_quiet_states_that_are_not_findings():
assert classify({}, DAYS)[0] == "never-active"
assert classify({DAYS[0]: 4}, DAYS)[0] == "too-little-traffic"
assert classify({DAYS[0]: 4102}, DAYS[:2])[0] == "window-too-short"
assert classify(None, DAYS)[0] == "never-active"
def test_today_is_never_in_the_axis():
days = complete_days(NOW, 14)
assert days == DAYS
assert day_key(NOW) == "2026-08-19"
assert day_key(NOW) not in days
assert day_key("not an epoch") is None
def test_a_project_absent_from_a_bucket_is_a_zero_not_a_gap():
# Buckets come back for the whole range; a project with no traffic is
# simply not in the results. The day axis has to come from the window.
buckets = [{"start_time": 1786579200,
"results": [{"project_id": "proj_busy", "num_model_requests": 10}]},
{"start_time": 1786665600, "results": []}]
rows = daily(buckets)
assert rows == {"proj_busy": {"2026-08-13": 10}}
assert rows.get("proj_quiet") is None
# Other surfaces count other things, and only one field is ever present.
assert daily([{"start_time": 1786579200,
"results": [{"project_id": "p", "num_images": 7}]}]) \
== {"p": {"2026-08-13": 7}}
assert daily([]) == {}
def test_a_key_still_in_use_means_something_is_authenticating():
keys = [{"last_used_at": NOW - 3600}, {"last_used_at": None},
{"last_used_at": NOW - 900000}]
used, since = key_activity(keys, NOW)
assert used == NOW - 3600
assert round(since, 2) == 0.04
state, detail = corroborate(since)
assert state == "key-still-used"
assert "authenticating and not inferring" in detail
def test_a_key_frozen_with_the_buckets_means_the_integration_died():
_, since = key_activity([{"last_used_at": NOW - 11 * 86400}], NOW)
state, detail = corroborate(since)
assert state == "key-quiet-too"
assert "11.0 day(s) ago" in detail
assert key_activity([], NOW) == (None, None)
assert key_activity([{"last_used_at": "never"}], NOW) == (None, None)
assert corroborate(None)[0] == "no-key-use"
def test_one_quiet_surface_beside_a_live_one_is_a_code_path():
quiet, live = surface_split({"completions": "went-quiet",
"embeddings": "live",
"images": "never-active"})
assert quiet == ["completions"]
assert live == ["embeddings"]
assert surface_split({}) == ([], [])
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, completeDays, corroborate, daily, dayKey, keyActivity,
surfaceSplit } from './openai-project-went-quiet.mjs';
const DAYS = Array.from({ length: 14 }, (_, i) => `2026-08-${String(i + 5).padStart(2, '0')}`);
const NOW = 1787097600; // 2026-08-19T00:00:00Z
test('a project that stops is named with a date and a volume', () => {
const series = new Map(DAYS.slice(0, 12).map((day) => [day, 4102]));
const [state, detail] = classify(series, DAYS);
assert.equal(state, 'went-quiet');
assert.match(detail, /last traffic on 2026-08-16/);
assert.match(detail, /2 complete day\(s\) ago/);
assert.match(detail, /prior mean of 4102 request\(s\) a day/);
const busy = new Map(DAYS.map((day) => [day, 4102]));
assert.equal(classify(busy, DAYS)[0], 'live');
});
test('a launch is not a death read backwards', () => {
const [state, detail] = classify(new Map([[DAYS[12], 900], [DAYS[13], 1200]]), DAYS);
assert.equal(state, 'new-traffic');
assert.match(detail, /first traffic in this window landed on 2026-08-17/);
});
test('the quiet states that are not findings', () => {
assert.equal(classify(new Map(), DAYS)[0], 'never-active');
assert.equal(classify(new Map([[DAYS[0], 4]]), DAYS)[0], 'too-little-traffic');
assert.equal(classify(new Map([[DAYS[0], 4102]]), DAYS.slice(0, 2))[0],
'window-too-short');
assert.equal(classify(null, DAYS)[0], 'never-active');
});
test('today is never in the axis', () => {
const days = completeDays(NOW, 14);
assert.deepEqual(days, DAYS);
assert.equal(dayKey(NOW), '2026-08-19');
assert.ok(!days.includes(dayKey(NOW)));
assert.equal(dayKey('not an epoch'), null);
});
test('a project absent from a bucket is a zero not a gap', () => {
const buckets = [
{ start_time: 1786579200,
results: [{ project_id: 'proj_busy', num_model_requests: 10 }] },
{ start_time: 1786665600, results: [] },
];
const rows = daily(buckets);
assert.deepEqual([...rows.get('proj_busy')], [['2026-08-13', 10]]);
assert.equal(rows.get('proj_quiet'), undefined);
const images = daily([{ start_time: 1786579200,
results: [{ project_id: 'p', num_images: 7 }] }]);
assert.deepEqual([...images.get('p')], [['2026-08-13', 7]]);
assert.equal(daily([]).size, 0);
});
test('a key still in use means something is authenticating', () => {
const keys = [{ last_used_at: NOW - 3600 }, { last_used_at: null },
{ last_used_at: NOW - 900000 }];
const [used, since] = keyActivity(keys, NOW);
assert.equal(used, NOW - 3600);
assert.equal(Number(since.toFixed(2)), 0.04);
const [state, detail] = corroborate(since);
assert.equal(state, 'key-still-used');
assert.match(detail, /authenticating and not inferring/);
});
test('a key frozen with the buckets means the integration died', () => {
const [, since] = keyActivity([{ last_used_at: NOW - 11 * 86400 }], NOW);
const [state, detail] = corroborate(since);
assert.equal(state, 'key-quiet-too');
assert.match(detail, /11\.0 day\(s\) ago/);
assert.deepEqual(keyActivity([], NOW), [null, null]);
assert.deepEqual(keyActivity([{ last_used_at: 'never' }], NOW), [null, null]);
assert.equal(corroborate(null)[0], 'no-key-use');
});
test('one quiet surface beside a live one is a code path', () => {
const [quiet, live] = surfaceSplit({ completions: 'went-quiet',
embeddings: 'live',
images: 'never-active' });
assert.deepEqual(quiet, ['completions']);
assert.deepEqual(live, ['embeddings']);
assert.deepEqual(surfaceSplit({}), [[], []]);
});
FAQ
Does the usage endpoint really return buckets for days with no traffic?
It returns a bucket for every interval in the range you asked for, and for an interval with nothing in it the results array comes back empty. That is what makes the silence readable at all. It also means a project with no traffic is simply absent from the results rather than present with a zero, so the day axis has to be built from the window you requested and not from the days the response happened to contain.
Why not just alert on spend dropping?
Because spend is an organization-level series and this failure is project-level and often small. A project that does four thousand calls a day inside an organization doing four hundred thousand can go completely dark without moving the invoice enough to notice. Cost anomaly detection and a per-project floor answer different questions, and only one of them fires when a feature quietly stops working.
How long a quiet window should I use?
Long enough to survive the traffic pattern and the reporting lag, short enough to be worth having. Forty-eight hours suits a service that runs every day; a batch job that only runs on Mondays needs a window measured in weeks, or a schedule-aware check instead of a flat one. The wrong answer is a window so short that a quiet Sunday pages somebody, because that is how the alarm gets turned off.
Isn't this the same as finding a key whose owner left?
No. That check reads a flag the platform sets for you, on a key whose owner lost access to the project. Here nothing is flagged: the key is valid, the project is active, and the credential would work perfectly if anything called it. The provider has no opinion to read, which is why the evidence has to be assembled out of an absence.
The project is quiet on completions but busy on embeddings. What does that mean?
That the credential is fine and one code path stopped. It is a much narrower search than a dead project: look at what shipped near the last day with traffic, at the feature flag guarding that call, and at whatever produces the events that used to trigger it. The script reports the surfaces separately for exactly this reason.
Related field notes
- A live key whose owner is no longer on the project
- Keys that survive the project being archived
- The shape of a change in spend: a spike, a step or a ramp
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Usage — OpenAI API reference
- Projects — OpenAI API reference
- Project API keys — OpenAI API reference
- Administration and the Admin APIs — OpenAI developer docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.