Diagnostic LLM APIs
API keys that no request has ever used
Nobody left. Nobody was removed from anything. The engineer who minted the key in March is sitting eight feet away, still employed, still on the project, and could tell you within a second why she made it: a vendor evaluation that went nowhere. The key has authenticated exactly zero requests in the five months since, and it has full access to everything in that project, and there is no report anywhere that will ever mention it, because a credential with no traffic produces no cost line, no log entry and no alert. Creating it took one click. Deleting it requires somebody to be confident nothing breaks.
With an organization admin key, walk every project with GET /v1/organization/projects?limit=100&include_archived=true, then read each one's keys with GET /v1/organization/projects/{project_id}/api_keys?limit=100&owner_project_access=any. Flag every key where last_used_at is null and created_at is more than 30 days old, and separately every key whose last_used_at is real but older than 90 days. Sweep GET /v1/organization/admin_api_keys the same way, because that object carries last_used_at too.
The two query parameters matter more than either threshold. owner_project_access=any and include_archived=true both default to the narrow answer, and neither omission produces an error or an obviously short list. Leave them off and the sweep returns a confident, clean result over a subset of your organization.
On Anthropic the same question cannot be asked the same way, because the Anthropic API key object has no last_used_at field at all. Its schema is id, created_at, created_by, expires_at, name, partial_key_hint, principal, scope, status, type. So you list the active keys, then collect every non-null api_key_id from GET /v1/organizations/usage_report/messages?group_by[]=api_key_id over the retrievable window, and take the set difference. That answers "unused in the last N days". It cannot answer "never", and the script says so rather than rounding one to the other.
The problem in plain words
An unused credential is the only kind of problem in this section that generates no signal whatsoever. A retired model 404s. A truncated response has a field on it. A runaway key shows up in the cost report. A key that nothing uses emits nothing at all: no requests, no tokens, no dollars, no audit entries after the one that created it. The single fact that it exists lives in a list nobody opens.
So the population grows monotonically. Keys get minted for a spike, for onboarding, for a vendor trial, for a debugging session at 2am, for a notebook, for the CI job that was replaced six weeks later by a different CI job with a different key. Every one of them is still valid, still carries the project's full access, and still sits in whatever Slack thread, .env.example, laptop backup or password manager it was pasted into on the day it was created. Project API keys have no expiry, so nothing ever removes one on its own.
What makes the sweep itself unreliable is subtler and worse. Both of the parameters that control what you can see default to hiding things. Ask a project for its keys without owner_project_access=any and the endpoint applies membership-based visibility rules that can omit enabled keys. Ask the organization for its projects without include_archived=true and archived projects are not in the list at all, along with every key inside them. Neither of those is an error. You get back a smaller number that looks exactly like good news.
Why it happens
This note never looks at the owner, which is what separates it from the offboarding sweep. The published note on keys whose owner lost project access reads owner_project_access == "inactive", a flag the provider raises about a person. Its finding is that somebody left and their credential did not. This one reads traffic and nothing else, so its central case is the opposite: a key belonging to a current employee in good standing, on a project they still work on, that has never authenticated anything. The offboarding sweep is structurally incapable of seeing that key, and this one is structurally incapable of caring who owns it.
Never-used and dormant are the same field and different conversations. A last_used_at of null on a key three months old means nothing was ever built on it, which makes it the safest object in the organization to revoke: there is no traffic to break, by definition. A last_used_at from fourteen months ago means something was built on it and has since been decommissioned, or has been quietly failing, or runs annually. The first is a cleanup. The second deserves a question asked out loud before anything is deleted, and the script grades and orders them separately for that reason.
The absence of the key from the cost report is corroboration, not proof. GET /v1/organization/costs?group_by=api_key_id only covers its reporting window, and it does not emit zero rows, so a key you cannot find there is consistent with dormancy rather than demonstrating it. The key object's last_used_at is the only usage signal that is actually attached to the key, and neither provider exposes a per-request log that would let you do better.
Anthropic answers a weaker question, and the script refuses to overstate it. With no last_used_at on the key object, "unused" has to be reconstructed by joining the active key list against api_key_id values in the usage report, which reaches back only as far as the report does. A key that last ran a job thirteen months ago is indistinguishable there from a key that has never run anything. The Anthropic half of the output therefore says unused-in-window and states the window, and it will not print the word "never".
Admin keys are the ones with real blast radius, and they are on a separate endpoint. GET /v1/organization/admin_api_keys returns objects that also carry last_used_at as well as an optional expires_at that project keys do not have. An idle admin key can enumerate every credential in the organization. A sweep that walks projects and stops there has skipped the most powerful credentials you own.
The fix, as a flow
An unused credential is the only fault in this section that emits nothing at all: no requests, no tokens, no cost line, no audit entry after the one that created it. The only place it exists is a list, and the two parameters that decide what that list contains both default to showing you less of it.
How to fix it
Use an admin key, provisioned read-only
Every path under /v1/organization/* and /v1/organizations/* rejects a project or workspace key, so this cannot run on the credential your application holds. Mint an OpenAI admin key (sk-admin-) with read scopes, or an Anthropic Admin key (sk-ant-admin), which can also be provisioned read-only. Treat it as the most sensitive secret you have: it can list every key in the organization.
Ask for the whole universe before you ask for the finding
include_archived=true on the project listing and owner_project_access=any on every key listing. Run the project call once with the parameter and once without and compare the counts; the difference is the number of projects your existing audits have never seen. Doing that once teaches the lesson better than being told the parameter exists.
Grade OpenAI keys off last_used_at, in two buckets
last_used_at is a unix timestamp and null on a key that has never authenticated anything. Never-used and older than 30 days is one finding; used at least once but idle for more than 90 days is a different one. Report id, name, redacted_value, created_at and owner.type, and print nothing else about the key.
Reconstruct the same question on Anthropic, and name the window
GET /v1/organizations/api_keys?status=active&limit=1000 for the roster, paging on has_more and last_id. Then GET /v1/organizations/usage_report/messages?starting_at={T-30d}&bucket_width=1d&limit=31&group_by[]=api_key_id and collect every non-null api_key_id. The set difference is the finding, and it is a finding about the window, not about the key's whole life.
Order by revocation safety and print, never delete
Never-used keys first, because nothing can break; then dormant keys ordered by how long they have been idle. For each, print the revocation call as text: DELETE /v1/organization/projects/{project_id}/api_keys/{api_key_id} on OpenAI, or on Anthropic a status change to archived on the key object. Run them yourself, after somebody has confirmed what each key was for.
How to check it worked
Re-run after the cleanup. Every project should report zero never-used keys older than the threshold, and the admin key sweep should be empty too.
python3 llm_idle_key_audit.py --never-after 30 --dormant-after 90
# openai: 6 project(s) read, 2 archived, 38 key(s) including 3 admin key(s)
# never-used proj_evals vendor-trial sk-...4f7a created 154 day(s) ago, never used
# repair: nothing has ever authenticated with this key, so revoking it cannot break traffic.
# dormant proj_prod nightly-export sk-...c19b last used 412 day(s) ago
# repair: something was built on this one. Ask before revoking, then watch last_used_at stop moving.
# anthropic: 21 active key(s), 14 seen in the usage report over 30 day(s)
# unused-in-window ingest-worker sk-ant-...igAA no traffic in the last 30 day(s)
# note: the Anthropic key object has no last_used_at, so this is unused in the window, not never used.
# 59 key(s) read, 3 finding(s)The full code
Two providers, one report, and the difference between them is the reason the pure functions are split rather than shared. openai_verdict reads a field. anthropic_verdict reads a set membership and is given the window so it can state it in the finding. Around them: age_days, which has to accept a unix integer from one provider and an RFC 3339 string from the other, because a reader that handles only one silently treats every key from the other as ageless; audit_gaps, which turns the two defaulted parameters into an assertion instead of a comment; revocation_order, which sorts by how safe each row is to delete rather than by age; and safe_hint, which is the one function here that exists purely to make a mistake impossible. It passes through a provider-redacted hint and refuses anything that does not look like one.
"""Find API keys that no request has ever used.
Read only. Every request is a GET against the OpenAI Administration API or the
Anthropic Admin API. Nothing is created, changed or removed, and no key value
is printed: the providers return a redacted hint and that hint is all that
reaches the output.
The two providers answer the same question with different evidence, and the
difference is the point of the script. OpenAI carries last_used_at on the key
object, so "never used" is a field you read. Anthropic has no such field, so
"unused" has to be computed as a set difference between the active key list and
the api_key_id values appearing in the usage report, which reaches back only as
far as the report does. On Anthropic this script reports "unused in the last N
days" and will not say "never".
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("llm_idle_key_audit")
OPENAI = "https://api.openai.com/v1"
ANTHROPIC = "https://api.anthropic.com/v1"
ANTHROPIC_VERSION = "2023-06-01"
NEVER = "never-used"
DORMANT = "dormant"
UNUSED_IN_WINDOW = "unused-in-window"
IN_USE = "in-use"
SEEN = "seen-in-window"
TOO_NEW = "too-new"
UNREADABLE = "unreadable-dates"
NOT_ACTIVE = "not-active"
FINDINGS = (NEVER, DORMANT, UNUSED_IN_WINDOW)
# Sort weight for the revocation queue. Lower goes first, and the order is by
# how safe the row is to delete rather than by how old it is: a key nothing has
# ever used cannot break anything when it is revoked, and a key that ran a job
# last spring can.
SAFETY = {NEVER: 0, UNUSED_IN_WINDOW: 1, DORMANT: 2}
def safe_hint(value):
"""Return a key hint that is safe to print. Pure.
Both providers hand back a redacted form: OpenAI's redacted_value and
Anthropic's partial_key_hint. This passes those through and refuses
anything else, because the one unrecoverable mistake an audit script can
make is to print a live credential into a log that then gets shipped
somewhere. A hint with no ellipsis or star in it is not a hint.
"""
text = str(value or "").strip()
if not text:
return "(no hint)"
if "..." not in text and "*" not in text:
return "(hint withheld)"
if len(text) > 40:
return "(hint withheld)"
return text
def age_days(stamp, now):
"""Whole days between a timestamp and now. Pure. None when unreadable.
Accepts a unix integer (OpenAI) or an RFC 3339 string (Anthropic). A reader
that handles only one of the two treats every key from the other provider
as ageless, which reads as "too new to judge" and quietly drops it from the
report.
"""
if stamp is None or stamp == "":
return None
when = None
if isinstance(stamp, bool):
return None
if isinstance(stamp, (int, float)):
when = dt.datetime.fromtimestamp(float(stamp), dt.timezone.utc)
else:
text = str(stamp).strip()
if text.isdigit():
when = dt.datetime.fromtimestamp(float(text), dt.timezone.utc)
else:
try:
when = dt.datetime.fromisoformat(text.replace("Z", "+00:00"))
except ValueError:
return None
if when.tzinfo is None:
when = when.replace(tzinfo=dt.timezone.utc)
return int((now - when).total_seconds() // 86400)
def openai_verdict(key, now, never_after=30, dormant_after=90):
"""Classify one OpenAI key off last_used_at. Pure. Returns (state, detail).
last_used_at is null on a key that has never authenticated anything, and a
unix timestamp otherwise. Zero is treated as absent: it is not a plausible
last-use time and reading it as one would date the key to 1970 and file it
under dormant instead of never used.
"""
row = key or {}
created = age_days(row.get("created_at"), now)
last = row.get("last_used_at")
if last in (None, "", 0):
if created is None:
return (UNREADABLE,
"never used, and created_at cannot be read, so no age can "
"be given for it")
if created < never_after:
return (TOO_NEW,
"never used, but only %d day(s) old" % created)
return (NEVER,
"never used in the %d day(s) since it was created" % created)
idle = age_days(last, now)
if idle is None:
return (UNREADABLE, "last_used_at is present but cannot be read")
if idle >= dormant_after:
return (DORMANT, "last used %d day(s) ago" % idle)
return (IN_USE, "last used %d day(s) ago" % idle)
def anthropic_verdict(key, seen_ids, window_days, now, never_after=30):
"""Classify one Anthropic key off usage-report membership. Pure.
There is no last_used_at on the Anthropic key object, so the strongest
available claim is bounded by the usage report's window. The detail string
says so on every row, because "unused in 30 days" and "never used" are
different facts and only one of them is in evidence here.
"""
row = key or {}
status = str(row.get("status") or "active").strip().lower()
if status != "active":
return (NOT_ACTIVE, "status is %s, so it cannot authenticate" % status)
created = age_days(row.get("created_at"), now)
if created is not None and created < never_after:
return (TOO_NEW, "only %d day(s) old" % created)
if str(row.get("id") or "") in (seen_ids or set()):
return (SEEN, "carried traffic inside the last %d day(s)" % window_days)
return (UNUSED_IN_WINDOW,
"no traffic in the last %d day(s). The Anthropic key object has no "
"last_used_at field, so this is unused within the retrievable "
"window and not a claim that it was never used." % window_days)
def audit_gaps(project_params, key_params):
"""Warn about a sweep that will silently under-report. Pure.
Both parameters default to the narrower answer, and neither omission
produces an error or a visibly short list. You get a clean report over a
partial universe, which is the most convincing kind of wrong answer, so the
check is an assertion in the code rather than a sentence in a comment.
"""
gaps = []
if str((project_params or {}).get("include_archived", "")).lower() != "true":
gaps.append("include_archived is not true: archived projects are "
"omitted from the project listing, and every key inside "
"them with it")
if str((key_params or {}).get("owner_project_access", "")) != "any":
gaps.append("owner_project_access is not 'any': the key listing "
"applies membership visibility rules and can hide enabled "
"keys from this audit")
return gaps
def seen_key_ids(pages):
"""Every non-null api_key_id in an Anthropic usage report. Pure."""
out = set()
for page in pages or []:
for bucket in (page or {}).get("data") or []:
for result in (bucket or {}).get("results") or []:
key_id = (result or {}).get("api_key_id")
if key_id:
out.add(str(key_id))
return out
def revocation_order(rows):
"""Order findings by how safe each is to revoke. Pure.
Never-used first: nothing has authenticated with it, so revocation cannot
break traffic. Then the window-bounded Anthropic rows, then dormant keys
longest-idle first, because those are the ones where something was built
and somebody has to be asked before anything is deleted.
"""
findings = [r for r in (rows or []) if (r or {}).get("state") in SAFETY]
return sorted(findings,
key=lambda r: (SAFETY[r["state"]], -int(r.get("idle") or 0),
str(r.get("name") or "")))
def repair_lines(state, row):
"""The repair for one classified key. Pure. Printed, never performed."""
data = row or {}
if state == NEVER:
return [
"nothing has ever authenticated with this key, so revoking it "
"cannot break traffic. These are the safest credentials in the "
"organization to remove.",
"revoke with a DELETE on /v1/organization/projects/%s/api_keys/%s "
"once somebody confirms what it was minted for."
% (data.get("container") or "{project_id}", data.get("id") or "{key_id}"),
]
if state == DORMANT:
return [
"something was built on this key and has since stopped calling. "
"Ask what it was before revoking: annual jobs and disaster-recovery "
"paths look exactly like this.",
"if it is genuinely dead, revoke it and confirm last_used_at stops "
"advancing rather than assuming it will.",
]
if state == UNUSED_IN_WINDOW:
return [
"this is unused within the report window, not proven unused. "
"Widen the window as far as the report allows before concluding "
"anything, then archive the key rather than deleting it.",
"the Anthropic key object carries an optional expires_at. Set one "
"on the replacement so the next idle key expires itself.",
]
return []
def get(session, url, params, who):
r = session.get(url, params=params, timeout=60)
if r.status_code in (401, 403):
raise SystemExit("%d from %s: this endpoint needs an administration "
"key, not a project or workspace key"
% (r.status_code, who))
r.raise_for_status()
return r.json()
def openai_paged(session, path, params):
"""Walk an OpenAI administration listing on has_more / last_id."""
params = dict(params)
while True:
page = get(session, OPENAI + path, params, "OpenAI")
yield page
if not page.get("has_more") or not page.get("last_id"):
return
params["after"] = page["last_id"]
def anthropic_paged(session, path, params):
"""Walk an Anthropic Admin listing on has_more / last_id."""
params = dict(params)
while True:
page = get(session, ANTHROPIC + path, params, "Anthropic")
yield page
if not page.get("has_more") or not page.get("last_id"):
return
params["after_id"] = page["last_id"]
def anthropic_report(session, path, params):
"""Walk the Anthropic usage report on has_more / next_page."""
params = dict(params)
while True:
page = get(session, ANTHROPIC + path, params, "Anthropic")
yield page
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def window_start(days, now):
"""Floor to midnight UTC: starting_at must sit on a bucket boundary."""
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
return (midnight - dt.timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
def sweep_openai(session, now, args):
"""Read every project, every project key and every admin key."""
project_params = {"limit": 100, "include_archived": "true"}
key_params = {"limit": 100, "owner_project_access": "any"}
for gap in audit_gaps(project_params, key_params):
log.warning("audit gap: %s", gap)
projects = []
for page in openai_paged(session, "/organization/projects", project_params):
projects.extend(page.get("data") or [])
archived = sum(1 for p in projects if p.get("status") == "archived")
rows = []
for project in projects:
pid = project.get("id")
if not pid:
continue
for page in openai_paged(session,
"/organization/projects/%s/api_keys" % pid,
key_params):
for key in page.get("data") or []:
state, detail = openai_verdict(key, now, args.never_after,
args.dormant_after)
rows.append({"provider": "openai", "state": state,
"detail": detail, "id": key.get("id"),
"name": key.get("name") or "(unnamed)",
"hint": safe_hint(key.get("redacted_value")),
"container": pid,
"label": project.get("name") or pid,
"idle": age_days(key.get("last_used_at"), now)
or age_days(key.get("created_at"), now) or 0})
admin_keys = []
for page in openai_paged(session, "/organization/admin_api_keys", {"limit": 100}):
admin_keys.extend(page.get("data") or [])
for key in admin_keys:
state, detail = openai_verdict(key, now, args.never_after,
args.dormant_after)
rows.append({"provider": "openai", "state": state, "detail": detail,
"id": key.get("id"), "name": key.get("name") or "(unnamed)",
"hint": safe_hint(key.get("redacted_value")),
"container": "organization", "label": "admin key",
"idle": age_days(key.get("last_used_at"), now)
or age_days(key.get("created_at"), now) or 0})
log.info("openai: %d project(s) read, %d archived, %d key(s) including %d "
"admin key(s)", len(projects), archived, len(rows), len(admin_keys))
return rows
def sweep_anthropic(session, now, args):
"""Read the active key roster, then the usage report it must be joined to."""
keys = []
for page in anthropic_paged(session, "/organizations/api_keys",
{"status": "active", "limit": 1000}):
keys.extend(page.get("data") or [])
seen = seen_key_ids(anthropic_report(
session, "/organizations/usage_report/messages",
{"starting_at": window_start(args.days, now), "bucket_width": "1d",
"limit": min(args.days + 1, 31), "group_by[]": ["api_key_id"]}))
rows = []
for key in keys:
state, detail = anthropic_verdict(key, seen, args.days, now,
args.never_after)
rows.append({"provider": "anthropic", "state": state, "detail": detail,
"id": key.get("id"), "name": key.get("name") or "(unnamed)",
"hint": safe_hint(key.get("partial_key_hint")),
"container": key.get("id"), "label": "anthropic",
"idle": age_days(key.get("created_at"), now) or 0})
log.info("anthropic: %d active key(s), %d seen in the usage report over "
"%d day(s)", len(keys), len(seen), args.days)
return rows
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--never-after", type=int, default=30,
help="days a never-used key must exist before it is a finding")
ap.add_argument("--dormant-after", type=int, default=90,
help="days since last use that counts as dormant")
ap.add_argument("--days", type=int, default=30,
help="Anthropic usage-report window, which bounds that half")
args = ap.parse_args()
openai_key = os.environ.get("OPENAI_ADMIN_KEY")
anthropic_key = os.environ.get("ANTHROPIC_ADMIN_KEY")
if not openai_key and not anthropic_key:
log.error("set OPENAI_ADMIN_KEY (sk-admin-, read scopes) or "
"ANTHROPIC_ADMIN_KEY (sk-ant-admin), or both; a project or "
"workspace key cannot read the administration endpoints")
return 2
now = dt.datetime.now(dt.timezone.utc)
rows = []
if openai_key:
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + openai_key})
rows.extend(sweep_openai(s, now, args))
if anthropic_key:
s = requests.Session()
s.headers.update({"x-api-key": anthropic_key,
"anthropic-version": ANTHROPIC_VERSION})
rows.extend(sweep_anthropic(s, now, args))
queue = revocation_order(rows)
for row in queue:
log.warning("%-16s %-14s %-18s %s %s", row["state"], row["label"],
row["name"], row["hint"], row["detail"])
for repair in repair_lines(row["state"], row):
log.warning(" repair: %s", repair)
log.info("%d key(s) read, %d finding(s)", len(rows), len(queue))
log.info("no key value appears above: both providers return a redacted "
"hint and the hint is all this script will print")
return 1 if queue else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find API keys that no request has ever used.
*
* Read only. Every request is a GET against the OpenAI Administration API or
* the Anthropic Admin API. No key value is printed: the providers return a
* redacted hint and that hint is all that reaches the output.
*
* OpenAI carries last_used_at on the key object, so "never used" is a field.
* Anthropic has no such field, so "unused" is a set difference against the
* usage report and is bounded by that report's window. The Anthropic half of
* the output says "unused in the last N days" and never says "never".
*/
const OPENAI = 'https://api.openai.com/v1';
const ANTHROPIC = 'https://api.anthropic.com/v1';
const ANTHROPIC_VERSION = '2023-06-01';
export const NEVER = 'never-used';
export const DORMANT = 'dormant';
export const UNUSED_IN_WINDOW = 'unused-in-window';
export const IN_USE = 'in-use';
export const SEEN = 'seen-in-window';
export const TOO_NEW = 'too-new';
export const UNREADABLE = 'unreadable-dates';
export const NOT_ACTIVE = 'not-active';
// Lower sorts first, and the order is revocation safety rather than age.
const SAFETY = { [NEVER]: 0, [UNUSED_IN_WINDOW]: 1, [DORMANT]: 2 };
/** A key hint that is safe to print. Pure. Anything unredacted is withheld. */
export function safeHint(value) {
const text = String(value ?? '').trim();
if (!text) return '(no hint)';
if (!text.includes('...') && !text.includes('*')) return '(hint withheld)';
if (text.length > 40) return '(hint withheld)';
return text;
}
/** Whole days between a timestamp and now. Pure. null when unreadable.
* Accepts a unix integer (OpenAI) or an RFC 3339 string (Anthropic). */
export function ageDays(stamp, now) {
if (stamp === null || stamp === undefined || stamp === '' ||
typeof stamp === 'boolean') return null;
let when;
if (typeof stamp === 'number') {
when = new Date(stamp * 1000);
} else if (/^\d+$/.test(String(stamp).trim())) {
when = new Date(Number(String(stamp).trim()) * 1000);
} else {
when = new Date(String(stamp).trim());
}
if (Number.isNaN(when.getTime())) return null;
return Math.floor((now.getTime() - when.getTime()) / 86400000);
}
/** Classify one OpenAI key off last_used_at. Pure. Returns [state, detail]. */
export function openaiVerdict(key, now, neverAfter = 30, dormantAfter = 90) {
const row = key ?? {};
const created = ageDays(row.created_at, now);
const last = row.last_used_at;
if (last === null || last === undefined || last === '' || last === 0) {
if (created === null) {
return [UNREADABLE,
'never used, and created_at cannot be read, so no age can be given for it'];
}
if (created < neverAfter) {
return [TOO_NEW, `never used, but only ${created} day(s) old`];
}
return [NEVER, `never used in the ${created} day(s) since it was created`];
}
const idle = ageDays(last, now);
if (idle === null) return [UNREADABLE, 'last_used_at is present but cannot be read'];
if (idle >= dormantAfter) return [DORMANT, `last used ${idle} day(s) ago`];
return [IN_USE, `last used ${idle} day(s) ago`];
}
/** Classify one Anthropic key off usage-report membership. Pure. */
export function anthropicVerdict(key, seenIds, windowDays, now, neverAfter = 30) {
const row = key ?? {};
const status = String(row.status ?? 'active').trim().toLowerCase();
if (status !== 'active') {
return [NOT_ACTIVE, `status is ${status}, so it cannot authenticate`];
}
const created = ageDays(row.created_at, now);
if (created !== null && created < neverAfter) {
return [TOO_NEW, `only ${created} day(s) old`];
}
if ((seenIds ?? new Set()).has(String(row.id ?? ''))) {
return [SEEN, `carried traffic inside the last ${windowDays} day(s)`];
}
return [UNUSED_IN_WINDOW,
`no traffic in the last ${windowDays} day(s). The Anthropic key object ` +
'has no last_used_at field, so this is unused within the retrievable ' +
'window and not a claim that it was never used.'];
}
/** Warn about a sweep that will silently under-report. Pure. */
export function auditGaps(projectParams, keyParams) {
const gaps = [];
if (String((projectParams ?? {}).include_archived ?? '').toLowerCase() !== 'true') {
gaps.push('include_archived is not true: archived projects are omitted ' +
'from the project listing, and every key inside them with it');
}
if (String((keyParams ?? {}).owner_project_access ?? '') !== 'any') {
gaps.push("owner_project_access is not 'any': the key listing applies " +
'membership visibility rules and can hide enabled keys from ' +
'this audit');
}
return gaps;
}
/** Every non-null api_key_id in an Anthropic usage report. Pure. */
export function seenKeyIds(pages) {
const out = new Set();
for (const page of pages ?? []) {
for (const bucket of page?.data ?? []) {
for (const result of bucket?.results ?? []) {
if (result?.api_key_id) out.add(String(result.api_key_id));
}
}
}
return out;
}
/** Order findings by how safe each is to revoke. Pure. */
export function revocationOrder(rows) {
return (rows ?? [])
.filter((r) => r?.state in SAFETY)
.slice()
.sort((a, b) => (SAFETY[a.state] - SAFETY[b.state])
|| (Number(b.idle ?? 0) - Number(a.idle ?? 0))
|| String(a.name ?? '').localeCompare(String(b.name ?? '')));
}
/** The repair for one classified key. Pure. Printed, never performed. */
export function repairLines(state, row) {
const data = row ?? {};
if (state === NEVER) {
return [
'nothing has ever authenticated with this key, so revoking it cannot ' +
'break traffic. These are the safest credentials in the organization to remove.',
`revoke with a DELETE on /v1/organization/projects/${data.container ?? '{project_id}'}` +
`/api_keys/${data.id ?? '{key_id}'} once somebody confirms what it was minted for.`,
];
}
if (state === DORMANT) {
return [
'something was built on this key and has since stopped calling. Ask ' +
'what it was before revoking: annual jobs and disaster-recovery paths ' +
'look exactly like this.',
'if it is genuinely dead, revoke it and confirm last_used_at stops ' +
'advancing rather than assuming it will.',
];
}
if (state === UNUSED_IN_WINDOW) {
return [
'this is unused within the report window, not proven unused. Widen the ' +
'window as far as the report allows before concluding anything, then ' +
'archive the key rather than deleting it.',
'the Anthropic key object carries an optional expires_at. Set one on ' +
'the replacement so the next idle key expires itself.',
];
}
return [];
}
/** Floor to midnight UTC: starting_at must sit on a bucket boundary. */
export function windowStart(days, now = new Date()) {
const midnight = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(),
now.getUTCDate()));
midnight.setUTCDate(midnight.getUTCDate() - days);
return `${midnight.toISOString().slice(0, 19)}Z`;
}
async function getJson(url, headers, who) {
const res = await fetch(url, { headers });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from ${who}: this endpoint needs an ` +
'administration key, not a project or workspace key');
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
async function openaiPaged(key, path, params) {
const headers = { Authorization: `Bearer ${key}` };
const out = [];
let after = null;
for (;;) {
const url = new URL(OPENAI + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
if (after) url.searchParams.set('after', after);
const page = await getJson(url, headers, 'OpenAI');
out.push(...(page.data ?? []));
if (!page.has_more || !page.last_id) return out;
after = page.last_id;
}
}
async function anthropicPaged(key, path, params) {
const headers = { 'x-api-key': key, 'anthropic-version': ANTHROPIC_VERSION };
const out = [];
let afterId = null;
for (;;) {
const url = new URL(ANTHROPIC + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
if (afterId) url.searchParams.set('after_id', afterId);
const page = await getJson(url, headers, 'Anthropic');
out.push(...(page.data ?? []));
if (!page.has_more || !page.last_id) return out;
afterId = page.last_id;
}
}
async function anthropicReport(key, params) {
const headers = { 'x-api-key': key, 'anthropic-version': ANTHROPIC_VERSION };
const pages = [];
let page = null;
for (;;) {
const url = new URL(`${ANTHROPIC}/organizations/usage_report/messages`);
for (const [k, v] of params) url.searchParams.append(k, v);
if (page) url.searchParams.set('page', page);
const body = await getJson(url, headers, 'Anthropic');
pages.push(body);
if (!body.has_more || !body.next_page) return pages;
page = body.next_page;
}
}
async function main() {
const openaiKey = process.env.OPENAI_ADMIN_KEY;
const anthropicKey = process.env.ANTHROPIC_ADMIN_KEY;
if (!openaiKey && !anthropicKey) {
console.error('set OPENAI_ADMIN_KEY (sk-admin-, read scopes) or ' +
'ANTHROPIC_ADMIN_KEY (sk-ant-admin), or both; a project or ' +
'workspace key cannot read the administration endpoints');
process.exitCode = 2;
return;
}
const neverAfter = Number(process.env.NEVER_AFTER ?? 30);
const dormantAfter = Number(process.env.DORMANT_AFTER ?? 90);
const days = Number(process.env.DAYS ?? 30);
const now = new Date();
const rows = [];
if (openaiKey) {
const projectParams = { limit: 100, include_archived: 'true' };
const keyParams = { limit: 100, owner_project_access: 'any' };
for (const gap of auditGaps(projectParams, keyParams)) {
console.warn(`audit gap: ${gap}`);
}
const projects = await openaiPaged(openaiKey, '/organization/projects', projectParams);
for (const project of projects) {
if (!project.id) continue;
const keys = await openaiPaged(
openaiKey, `/organization/projects/${project.id}/api_keys`, keyParams);
for (const key of keys) {
const [state, detail] = openaiVerdict(key, now, neverAfter, dormantAfter);
rows.push({ state, detail, id: key.id, name: key.name ?? '(unnamed)',
hint: safeHint(key.redacted_value), container: project.id,
label: project.name ?? project.id,
idle: ageDays(key.last_used_at, now) ?? ageDays(key.created_at, now) ?? 0 });
}
}
const adminKeys = await openaiPaged(openaiKey, '/organization/admin_api_keys', { limit: 100 });
for (const key of adminKeys) {
const [state, detail] = openaiVerdict(key, now, neverAfter, dormantAfter);
rows.push({ state, detail, id: key.id, name: key.name ?? '(unnamed)',
hint: safeHint(key.redacted_value), container: 'organization',
label: 'admin key',
idle: ageDays(key.last_used_at, now) ?? ageDays(key.created_at, now) ?? 0 });
}
console.log(`openai: ${projects.length} project(s) read, ${rows.length} key(s) ` +
`including ${adminKeys.length} admin key(s)`);
}
if (anthropicKey) {
const keys = await anthropicPaged(anthropicKey, '/organizations/api_keys',
{ status: 'active', limit: 1000 });
const seen = seenKeyIds(await anthropicReport(anthropicKey, [
['starting_at', windowStart(days, now)], ['bucket_width', '1d'],
['limit', String(Math.min(days + 1, 31))], ['group_by[]', 'api_key_id'],
]));
for (const key of keys) {
const [state, detail] = anthropicVerdict(key, seen, days, now, neverAfter);
rows.push({ state, detail, id: key.id, name: key.name ?? '(unnamed)',
hint: safeHint(key.partial_key_hint), container: key.id,
label: 'anthropic', idle: ageDays(key.created_at, now) ?? 0 });
}
console.log(`anthropic: ${keys.length} active key(s), ${seen.size} seen in ` +
`the usage report over ${days} day(s)`);
}
const queue = revocationOrder(rows);
for (const row of queue) {
console.warn(`${row.state.padEnd(16)} ${String(row.label).padEnd(14)} ` +
`${String(row.name).padEnd(18)} ${row.hint} ${row.detail}`);
for (const repair of repairLines(row.state, row)) {
console.warn(` repair: ${repair}`);
}
}
console.log(`${rows.length} key(s) read, ${queue.length} finding(s)`);
console.log('no key value appears above: both providers return a redacted ' +
'hint and the hint is all this script will print');
process.exitCode = queue.length ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first test is the note itself, and it is written so that it would fail if this drifted into the offboarding sweep: a key whose owner is present, active and entirely fine, that has never authenticated anything, has to come back as never-used. The second is the provider split, asserted as a difference in wording rather than in state alone: the Anthropic verdict must contain the window and must not contain the word never, because the field that would justify it does not exist. After that: audit_gaps, which has to complain about exactly the two parameters that default to hiding things; age_days against a unix integer, an RFC 3339 string and a last_used_at of zero, which must not date a key to 1970; the revocation queue, which must put a never-used key ahead of a longer-idle dormant one; and safe_hint, which has to withhold anything that does not already look redacted.
import datetime as dt
from llm_idle_key_audit import (age_days, anthropic_verdict, audit_gaps,
openai_verdict, repair_lines, revocation_order,
safe_hint, seen_key_ids)
NOW = dt.datetime(2026, 8, 31, 12, 0, 0, tzinfo=dt.timezone.utc)
def unix(days_ago):
return int((NOW - dt.timedelta(days=days_ago)).timestamp())
def test_a_key_whose_owner_is_perfectly_fine_is_still_the_finding():
# The note in one assertion, and the line that keeps it away from the
# published offboarding note. This owner is present, active and employed.
# The verdict does not read the owner at all.
key = {"id": "key_a1", "name": "vendor-trial",
"redacted_value": "sk-...4f7a", "created_at": unix(154),
"last_used_at": None,
"owner": {"type": "user", "user": {"email": "dev@example.test"}},
"owner_project_access": "active"}
state, detail = openai_verdict(key, NOW)
assert state == "never-used"
assert "154 day(s)" in detail
assert any("cannot break traffic" in line
for line in repair_lines(state, {"container": "proj_1",
"id": "key_a1"}))
def test_the_two_providers_answer_different_strengths_of_the_question():
# OpenAI reads a field and may say "never". Anthropic reads a set
# difference over a window and must not.
openai_state, openai_detail = openai_verdict(
{"created_at": unix(200), "last_used_at": None}, NOW)
assert openai_state == "never-used"
assert "never used" in openai_detail
anthropic_state, anthropic_detail = anthropic_verdict(
{"id": "apikey_z9", "status": "active",
"created_at": "2025-01-04T09:12:00Z"}, set(), 30, NOW)
assert anthropic_state == "unused-in-window"
assert "last 30 day(s)" in anthropic_detail
assert "no last_used_at field" in anthropic_detail
assert "never used" not in anthropic_detail.split("not a claim")[0]
def test_the_two_defaulted_parameters_are_the_audit_and_are_asserted():
assert audit_gaps({"include_archived": "true"},
{"owner_project_access": "any"}) == []
gaps = audit_gaps({"limit": 100}, {"limit": 100})
assert len(gaps) == 2
assert any("include_archived" in g for g in gaps)
assert any("owner_project_access" in g for g in gaps)
# The dangerous middle case: one parameter remembered, one forgotten.
assert len(audit_gaps({"include_archived": "true"}, {"limit": 100})) == 1
def test_dates_arrive_in_two_shapes_and_a_zero_is_not_1970():
assert age_days(unix(45), NOW) == 45
assert age_days("2026-08-01T00:00:00Z", NOW) == 30
assert age_days("2026-08-01T00:00:00+00:00", NOW) == 30
assert age_days(str(unix(7)), NOW) == 7
assert age_days(None, NOW) is None
assert age_days("not a date", NOW) is None
assert age_days(True, NOW) is None
# last_used_at of 0 is absent, not a use in 1970.
state, _ = openai_verdict({"created_at": unix(100), "last_used_at": 0}, NOW)
assert state == "never-used"
def test_dormant_and_never_used_are_graded_and_ordered_apart():
fresh = openai_verdict({"created_at": unix(120), "last_used_at": unix(3)}, NOW)
assert fresh[0] == "in-use"
old = openai_verdict({"created_at": unix(900), "last_used_at": unix(412)}, NOW)
assert old[0] == "dormant"
assert "412 day(s) ago" in old[1]
young = openai_verdict({"created_at": unix(9), "last_used_at": None}, NOW)
assert young[0] == "too-new"
order = revocation_order([
{"state": "dormant", "idle": 412, "name": "nightly"},
{"state": "in-use", "idle": 1, "name": "prod"},
{"state": "never-used", "idle": 154, "name": "vendor-trial"},
{"state": "unused-in-window", "idle": 300, "name": "ingest"},
])
assert [r["state"] for r in order] == ["never-used", "unused-in-window",
"dormant"]
def test_an_anthropic_key_seen_in_the_report_is_not_a_finding():
pages = [{"data": [{"results": [{"api_key_id": "apikey_a"},
{"api_key_id": None},
{"api_key_id": "apikey_b"}]}]}]
seen = seen_key_ids(pages)
assert seen == {"apikey_a", "apikey_b"}
assert anthropic_verdict({"id": "apikey_a", "status": "active",
"created_at": "2024-02-02T00:00:00Z"},
seen, 30, NOW)[0] == "seen-in-window"
assert anthropic_verdict({"id": "apikey_c", "status": "archived"},
seen, 30, NOW)[0] == "not-active"
assert seen_key_ids([]) == set()
assert seen_key_ids(None) == set()
def test_no_key_value_can_reach_the_output():
assert safe_hint("sk-...4f7a") == "sk-...4f7a"
assert safe_hint("sk-ant-...igAA") == "sk-ant-...igAA"
assert safe_hint("sk-abcd****wxyz") == "sk-abcd****wxyz"
# Anything that is not already redacted is refused, whatever it is.
assert safe_hint("sk-fake-not-redacted-value") == "(hint withheld)"
assert safe_hint("...." + "x" * 60) == "(hint withheld)"
assert safe_hint(None) == "(no hint)"
assert safe_hint("") == "(no hint)"
def test_the_repairs_say_different_things_for_the_two_findings():
never = repair_lines("never-used", {"container": "proj_1", "id": "key_1"})
dormant = repair_lines("dormant", {})
window = repair_lines("unused-in-window", {})
assert any("safest credentials" in line for line in never)
assert any("Ask what it was before revoking" in line for line in dormant)
assert any("not proven unused" in line for line in window)
assert repair_lines("in-use", {}) == []
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { ageDays, anthropicVerdict, auditGaps, openaiVerdict, repairLines,
revocationOrder, safeHint, seenKeyIds, windowStart }
from './llm-idle-key-audit.mjs';
const NOW = new Date('2026-08-31T12:00:00Z');
const unix = (daysAgo) => Math.floor(NOW.getTime() / 1000) - daysAgo * 86400;
test('a key whose owner is perfectly fine is still the finding', () => {
const key = { id: 'key_a1', name: 'vendor-trial', redacted_value: 'sk-...4f7a',
created_at: unix(154), last_used_at: null,
owner: { type: 'user', user: { email: 'dev@example.test' } },
owner_project_access: 'active' };
const [state, detail] = openaiVerdict(key, NOW);
assert.equal(state, 'never-used');
assert.match(detail, /154 day\(s\)/);
assert.ok(repairLines(state, { container: 'proj_1', id: 'key_a1' })
.some((line) => line.includes('cannot break traffic')));
});
test('the two providers answer different strengths of the question', () => {
const [openaiState, openaiDetail] =
openaiVerdict({ created_at: unix(200), last_used_at: null }, NOW);
assert.equal(openaiState, 'never-used');
assert.match(openaiDetail, /never used/);
const [anthropicState, anthropicDetail] = anthropicVerdict(
{ id: 'apikey_z9', status: 'active', created_at: '2025-01-04T09:12:00Z' },
new Set(), 30, NOW);
assert.equal(anthropicState, 'unused-in-window');
assert.match(anthropicDetail, /last 30 day\(s\)/);
assert.match(anthropicDetail, /no last_used_at field/);
assert.ok(!anthropicDetail.split('not a claim')[0].includes('never used'));
});
test('the two defaulted parameters are the audit and are asserted', () => {
assert.deepEqual(auditGaps({ include_archived: 'true' },
{ owner_project_access: 'any' }), []);
const gaps = auditGaps({ limit: 100 }, { limit: 100 });
assert.equal(gaps.length, 2);
assert.ok(gaps.some((g) => g.includes('include_archived')));
assert.ok(gaps.some((g) => g.includes('owner_project_access')));
assert.equal(auditGaps({ include_archived: 'true' }, { limit: 100 }).length, 1);
});
test('dates arrive in two shapes and a zero is not 1970', () => {
assert.equal(ageDays(unix(45), NOW), 45);
assert.equal(ageDays('2026-08-01T00:00:00Z', NOW), 30);
assert.equal(ageDays(String(unix(7)), NOW), 7);
assert.equal(ageDays(null, NOW), null);
assert.equal(ageDays('not a date', NOW), null);
assert.equal(ageDays(true, NOW), null);
assert.equal(openaiVerdict({ created_at: unix(100), last_used_at: 0 }, NOW)[0],
'never-used');
});
test('dormant and never used are graded and ordered apart', () => {
assert.equal(openaiVerdict({ created_at: unix(120), last_used_at: unix(3) }, NOW)[0],
'in-use');
const [state, detail] =
openaiVerdict({ created_at: unix(900), last_used_at: unix(412) }, NOW);
assert.equal(state, 'dormant');
assert.match(detail, /412 day\(s\) ago/);
assert.equal(openaiVerdict({ created_at: unix(9), last_used_at: null }, NOW)[0],
'too-new');
const order = revocationOrder([
{ state: 'dormant', idle: 412, name: 'nightly' },
{ state: 'in-use', idle: 1, name: 'prod' },
{ state: 'never-used', idle: 154, name: 'vendor-trial' },
{ state: 'unused-in-window', idle: 300, name: 'ingest' },
]);
assert.deepEqual(order.map((r) => r.state),
['never-used', 'unused-in-window', 'dormant']);
});
test('an anthropic key seen in the report is not a finding', () => {
const seen = seenKeyIds([{ data: [{ results: [{ api_key_id: 'apikey_a' },
{ api_key_id: null },
{ api_key_id: 'apikey_b' }] }] }]);
assert.deepEqual([...seen].sort(), ['apikey_a', 'apikey_b']);
assert.equal(anthropicVerdict({ id: 'apikey_a', status: 'active',
created_at: '2024-02-02T00:00:00Z' },
seen, 30, NOW)[0], 'seen-in-window');
assert.equal(anthropicVerdict({ id: 'apikey_c', status: 'archived' },
seen, 30, NOW)[0], 'not-active');
assert.equal(seenKeyIds([]).size, 0);
assert.equal(seenKeyIds(null).size, 0);
});
test('no key value can reach the output', () => {
assert.equal(safeHint('sk-...4f7a'), 'sk-...4f7a');
assert.equal(safeHint('sk-ant-...igAA'), 'sk-ant-...igAA');
assert.equal(safeHint('sk-abcd****wxyz'), 'sk-abcd****wxyz');
assert.equal(safeHint('sk-fake-not-redacted-value'), '(hint withheld)');
assert.equal(safeHint(`....${'x'.repeat(60)}`), '(hint withheld)');
assert.equal(safeHint(null), '(no hint)');
assert.equal(safeHint(''), '(no hint)');
});
test('the window start is floored to midnight utc', () => {
assert.equal(windowStart(30, new Date('2026-08-31T17:45:12Z')),
'2026-08-01T00:00:00Z');
});
FAQ
How is this different from auditing keys whose owner has left?
It never looks at the owner. The offboarding sweep reads owner_project_access, a flag the provider raises when the person behind a key loses access to the project, so its subject is a human. This one reads last_used_at, so its subject is traffic. The case that only this note can see is a key belonging to somebody who is present, employed and still on the project, minted for a reason that evaporated, that has authenticated nothing since. Nothing about that key is irregular from the offboarding sweep's point of view, and it is a live credential with full project access all the same.
Why does the Anthropic half refuse to say a key was never used?
Because the field that would justify it does not exist. The Anthropic API key object carries id, created_at, created_by, expires_at, name, partial_key_hint, principal, scope, status and type, and no last-used timestamp of any kind. So the only way to ask the question is to take the active key list and subtract every api_key_id that appears in the usage report, which reaches back exactly as far as the report does. A key that last ran something thirteen months ago and a key that has never run anything are the same row in that set difference. Reporting the weaker fact accurately is better than reporting the stronger one hopefully.
What do owner_project_access=any and include_archived=true actually change?
What the audit can see, and neither omission tells you it happened. Without owner_project_access=any the key listing applies membership-based visibility rules that can leave enabled keys out of the response. Without include_archived=true the project listing omits archived projects entirely, and archived projects still hold live keys. You do not get an error or an obviously truncated list; you get a smaller number that reads as good news. The script asserts both parameters rather than mentioning them in a comment.
Is a never-used key really safe to delete?
It is the safest object in the organization to delete, and that is a statement about traffic rather than about process. If last_used_at is null, nothing has ever authenticated with this credential, so there is no request path that can break when it is revoked. A dormant key is the opposite: something was built on it, and annual reports, disaster-recovery paths and quarterly exports all look identical to abandonment for eleven months of the year. The script orders never-used ahead of dormant for exactly that reason, and it still only prints the revocation call.
Why sweep the admin keys separately?
Because they live on their own endpoint and they are the credentials with real blast radius. GET /v1/organization/admin_api_keys returns objects that carry last_used_at just as project keys do, plus an optional expires_at that project keys do not have at all. An idle admin key can enumerate every credential in the organization and read every usage and cost report in it. A sweep that walks projects and stops has skipped the most powerful keys you own, and it will report a clean result while doing so.
Related field notes
- The key whose owner lost access, which is a fact about a person rather than about traffic
- The projects your key sweep never enumerated in the first place
- The same roster read for how long ago each key was minted
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Project API keys — OpenAI API reference
- Projects — OpenAI API reference
- List API keys — Claude Admin API
- Get messages usage report — Claude Admin API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.