Diagnostic LLM APIs
One project holds every environment, so nothing can be capped
Somebody asks what production costs. Not the API bill, which everyone knows to the dollar, but the production share of it, and the answer takes four days to arrive because there is no answer. Every request the company has ever made to OpenAI — the customer-facing assistant, the nightly evaluation suite, the CI job that regenerates fixtures, eleven laptops, and the prototype somebody wrote in March and never turned off — landed in a project called Default project that was created before anybody had an opinion about structure.
Two GETs with an organization admin key. GET /v1/organization/projects?limit=100&include_archived=true for the containers, and GET /v1/organization/costs?start_time={now-30d}&bucket_width=1d&limit=30&group_by=project_id for the money. The finding is the first number: how many projects have status of "active".
One is the whole note. The project is the unit of nearly every control the platform offers — spend limits, spend alerts, rate limits, model permissions, hosted tool permissions, data retention — and every one of those is configured per project. An organization with one active project cannot apply any of them differentially. You cannot cap the experiments without capping the product, and you cannot grant a frontier model to research without granting it to the batch job.
Do the count before the ranking, because the ranking cannot tell you this. Grouped by project_id, a single-project org returns one row at a hundred per cent, which is arithmetic rather than a discovery, and it looks identical in a table to a genuine concentration finding in an org that has nine projects and puts everything in one of them. Those are different problems with different repairs, and the concentration note is the one for the second.
Archived projects do not count as a boundary. They cannot receive new traffic, they are excluded from the default listing, and include_archived=true is passed here only so they can be counted and then set aside honestly.
The problem in plain words
The default project is created for you and works from the first request, so there is never a moment at which anyone is forced to choose a structure. That is a good onboarding decision and a bad steady state, because it means the structural choice gets made by not making it, at the exact time when nobody yet knows what the environments will be.
What is lost is not visibility. Visibility is recoverable: the usage and cost reports will happily group by API key, and a disciplined team can name its keys well enough to reconstruct a rough split after the fact. What is lost is enforcement. A spend limit on the only project is a spend limit on production. A rate-limit override on the only project throttles the customer-facing path along with the evaluation run that caused the problem. A model permission that stops an experiment reaching an expensive model stops the product reaching it too. Every control the platform gives you is scoped to a container, and there is only one container.
And there is a one-way door in it. Projects can be archived; they cannot be deleted. The split you make is permanent in the sense that the names you choose are permanent, which is worth ten minutes of thought before the first one is created and is the usual reason the ten minutes never happen.
Why it happens
Count containers first, and rank rows second, because the ranking is uninformative when the count is one. A share of total is a comparison, and a comparison needs something to compare with. In a single-project org the top row is a hundred per cent of spend for the same reason the only runner wins the race. The script therefore establishes the boundary count before it looks at a single dollar, and when it finds a dominant project in an org that does have several, it says explicitly that this is not a topology finding and points at the other reading.
An ungrouped cost row is not a giant project, and mistaking it for one manufactures this exact finding. GET /v1/organization/costs without group_by returns project_id: null on every result. A reader that folds those nulls into a bucket and ranks it will report one enormous project in an organization that has twelve, which is a false positive in the most embarrassing direction. The fold keeps null rows out of the ranking and reports them separately as what they are: a call that was not grouped.
Projects that exist and never receive traffic are a different state from projects that do not exist. Plenty of organizations created staging and dev during a security review and then never issued a key for either. The containers are there, every control on them is configured, and none of it does anything, because no traffic routes to them. That reads as a healthy topology in a project count and as a broken one in the cost report, so the script grades it separately and prints a different repair: the boundary is not the problem, the routing is.
Key names inside the single project are corroboration, not evidence. When one project holds keys called prod-worker, local-adam and ci-fixtures, the environments already exist as a fact about how people work and are simply not represented in the platform. The script reads key names only, never values, matches environment words as whole tokens so that devops-runner is not read as a development key, and treats a match as a supporting line rather than a verdict.
The fix, as a flow
Nothing in this chain is a mistake at the moment it happens. The default project is created for you and serves the first request, which is a good onboarding decision, and every environment that arrives afterwards arrives one key at a time. The count of containers is the reading, not the size of the bill: a share of total needs something to compare against, and a single project is a hundred per cent of everything by construction.
How to fix it
Use an organization admin key, provisioned read-only
Every path under /v1/organization/* rejects a project key with a 401. Read scopes are enough for all three calls here, and this script has no other kind.
List the projects with include_archived=true, then set the archived ones aside
GET /v1/organization/projects?limit=100&include_archived=true, paginated on after. Archived projects are excluded by default; asking for them and then excluding them deliberately is different from never seeing them. An archived project cannot take new traffic, so it is not a boundary — but if one is still billing, that is its own note.
Count the active projects before you look at any money
A count of one is the finding, whatever the spend report says. A count above one moves the question to whether the extra containers are actually used, which is a different grading path.
Group thirty days of cost by project_id, and drop the null rows
GET /v1/organization/costs?start_time={now-30d}&bucket_width=1d&limit=30&group_by=project_id. Results with a null project_id are ungrouped rows, not a project; they are reported separately and never ranked.
Read the key names in the dominant project, and print the repair
GET /v1/organization/projects/{project_id}/api_keys?limit=100&owner_project_access=any, names only. Two or more environment words inside one project is the split that already exists in people's heads. The repair is a project per environment, each with its own service account, spend limit and rate limits — printed, never performed, and worth naming carefully because archiving is the only undo.
How to check it worked
Re-run after the second project has a key and has served real traffic. The state moves from no-boundary to boundary-unused the moment the container exists, and only to separated once cost actually lands in more than one of them — which is the point, because a project with no traffic in it enforces nothing.
python3 openai_project_boundary_audit.py --days 30
# 1 active project(s), 2 archived, $18,406.11 in the last 30 day(s)
# no-boundary 1 active project holds 100% of $18,406.11. There is no second
# container to cap, alert on, rate limit or attribute against.
# corroboration: key names in Default project already name 3 environment(s): ci, local, prod
# repair: create prod, staging and dev with POST /v1/organization/projects
# repair: give each a service account and its own key, then move traffic key by key
# repair: a spend limit, spend alerts and rate limits only exist per project
# repair: projects can be archived but never deleted, so get the names right once
# 1 finding(s)The full code
Two paged GETs, one optional third for key names, and six pure functions. active, which drops archived projects by both status and archived_at because the two disagree in older responses; the cost fold, which keeps ungrouped null rows out of the ranking so a forgotten group_by cannot fabricate a giant project; the share ranking; a whole-token environment matcher; the verdict, which counts containers before it compares any money and hands concentration back to the other note when the org has a boundary; and the repair lines.
"""Find an OpenAI organization with no project boundary to enforce anything on.
Read only. Two paged GETs against /v1/organization/projects and
/v1/organization/costs, plus one per dominant project for its key NAMES. Every
request is a GET and no request body is ever built.
The finding is the absence of a boundary, not the concentration of spend. A
single active project holds 100% of cost by construction, which is arithmetic;
a dominant project in an organization that has nine is a different reading with
a different repair, and this script names that reading rather than claiming it.
Key values are never read or printed. The key listing is used for the `name`
field only, and only as corroboration.
"""
import argparse
import datetime as dt
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_project_boundary_audit")
API = "https://api.openai.com/v1"
UNGROUPED = "ungrouped"
# Environment words, matched as WHOLE tokens after splitting on anything that is
# not a letter or a digit. Substring matching here is how "devops-runner" gets
# reported as a development key and "provider-proxy" as a production one, which
# is a false positive attached to a person's naming habits.
ENV_WORDS = {
"prod": "prod", "production": "prod", "live": "prod",
"stage": "staging", "staging": "staging", "preprod": "staging",
"dev": "dev", "development": "dev",
"local": "local", "laptop": "local",
"test": "test", "testing": "test", "qa": "test",
"ci": "ci", "build": "ci",
"sandbox": "sandbox", "scratch": "sandbox", "playground": "sandbox",
}
FINDINGS = ("no-boundary", "boundary-unused")
def active(projects):
"""Projects that can still receive traffic. Pure.
Archived projects are dropped on either signal. `status` is the documented
field and `archived_at` is the one that is reliably present, and a listing
that carries one without the other is common enough that trusting a single
field over-counts the boundary.
"""
out = []
for project in projects or []:
row = project or {}
if str(row.get("status") or "").strip().lower() == "archived":
continue
if row.get("archived_at"):
continue
out.append(row)
return out
def spend_by_project(buckets):
"""{project_id: dollars} from the cost report. Pure.
A result with a null project_id is an UNGROUPED row, not a project. Folding
those into the ranking is how a forgotten group_by turns into a confident
report of one enormous project in an organization that has twelve.
"""
rows = {}
for bucket in buckets or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
name = row.get("project_id") or UNGROUPED
try:
value = float((row.get("amount") or {}).get("value") or 0.0)
except (TypeError, ValueError):
continue
rows[str(name)] = rows.get(str(name), 0.0) + value
return rows
def shares(spend):
"""[(project_id, dollars, share)] over real projects only. Pure.
Sorted by dollars descending. UNGROUPED is excluded from both the ranking
and the denominator, so a share is always a share of attributable spend.
"""
rows = {k: v for k, v in (spend or {}).items() if k != UNGROUPED}
total = sum(rows.values())
out = [(k, round(v, 2), (v / total) if total > 0 else 0.0)
for k, v in rows.items()]
out.sort(key=lambda r: (-r[1], r[0]))
return out
def environments(name):
"""The environment classes named in one identifier. Pure.
Tokenised on non-alphanumerics and matched whole, so "devops" is a team and
"provider" is a noun. Returns a set, possibly empty.
"""
tokens = re.split(r"[^a-z0-9]+", str(name or "").strip().lower())
return {ENV_WORDS[t] for t in tokens if t in ENV_WORDS}
def mixed(names):
"""Every environment class named across a set of identifiers. Pure."""
found = set()
for name in names or []:
found |= environments(name)
return found
def verdict(active_count, ranked, min_spend=1.0, dominant=0.95):
"""Classify the organization's topology. Pure. Returns (state, detail).
The container count is read before any money, because a share of total is a
comparison and a single-project organization has nothing to compare with.
"""
rows = list(ranked or [])
total = round(sum(row[1] for row in rows), 2)
if active_count <= 0:
return ("no-active-projects",
"the listing returned no active project at all, which usually "
"means the key could not see them rather than that none exist")
if active_count == 1:
return ("no-boundary",
"1 active project holds 100%% of $%s. There is no second "
"container to cap, alert on, rate limit or attribute against."
% format(total, ",.2f"))
if total < min_spend:
return ("no-spend-yet",
"%d active project(s) and $%s of attributable spend in the "
"window. The boundary exists and nothing has tested it yet."
% (active_count, format(total, ",.2f")))
top_id, top_amount, top_share = rows[0]
quiet = [row for row in rows[1:] if row[1] <= 0.0]
if top_share >= dominant and len(quiet) == len(rows) - 1:
return ("boundary-unused",
"%d active project(s), and %s carries %.0f%% of $%s while every "
"other project has no spend at all. The containers exist and no "
"traffic routes to them, so the controls on them enforce nothing."
% (active_count, top_id, top_share * 100, format(total, ",.2f")))
if top_share >= dominant:
return ("concentration-not-topology",
"%d active project(s), and %s carries %.0f%% of $%s. This "
"organization has a boundary, so that is a concentration "
"reading rather than a topology one and has a different repair."
% (active_count, top_id, top_share * 100, format(total, ",.2f")))
return ("separated",
"%d active project(s) sharing $%s, top project at %.0f%%"
% (active_count, format(total, ",.2f"), top_share * 100))
def repair_lines(state, envs=()):
"""The repair for one topology verdict. Pure. Printed, never performed."""
found = sorted(envs or ())
if state == "no-boundary":
lines = [
"create prod, staging and dev with POST /v1/organization/projects, "
"which is the smallest split that lets any control differ.",
"give each project its own service account and key, then move "
"traffic one key at a time rather than in one cutover.",
"spend limits, spend alerts, rate limits, model permissions and "
"data retention are all configured per project and cannot differ "
"until the projects do.",
"projects can be archived but never deleted, so the names are "
"permanent. Spend ten minutes on them once.",
]
if found:
lines.insert(0, "the environments already exist in your key names "
"(%s); they are simply not represented in the "
"platform." % ", ".join(found))
return lines
if state == "boundary-unused":
return [
"the projects are not the problem. Nothing routes to them.",
"issue a key in the quiet projects and move the traffic that "
"belongs there, then set the limits per project afterwards.",
"until traffic actually lands in a project, every control "
"configured on it is inert.",
]
if state == "concentration-not-topology":
return [
"do not restructure on this reading. Rank the cost rows by share "
"of total and ask which line item is expensive instead.",
]
return []
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key, not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def paged(session, path, **params):
"""Walk an after/last_id cursor listing."""
params = dict(params)
while True:
page = get(session, path, **params)
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params["after"] = page.get("last_id") or (data[-1] or {}).get("id")
def cost_buckets(session, params, max_pages=40):
"""Walk the paged cost report."""
params = dict(params)
for _ in range(max_pages):
page = get(session, "/organization/costs", **params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def window_start(days, now=None):
"""Unix seconds at midnight UTC, `days` ago."""
now = now or dt.datetime.now(dt.timezone.utc)
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
return int((midnight - dt.timedelta(days=days)).timestamp())
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30,
help="days of cost to read (default 30)")
ap.add_argument("--dominant", type=float, default=0.95,
help="share above which one project is called dominant")
ap.add_argument("--no-key-names", action="store_true",
help="skip the key-name corroboration read")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key; a "
"project key cannot read /v1/organization/*")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + admin})
projects = list(paged(s, "/organization/projects", limit=100,
include_archived="true"))
live = active(projects)
spend = spend_by_project(cost_buckets(
s, {"start_time": window_start(args.days), "bucket_width": "1d",
"limit": min(args.days, 30), "group_by": "project_id"}))
ranked = shares(spend)
total = round(sum(row[1] for row in ranked), 2)
log.info("%d active project(s), %d archived, $%s in the last %d day(s)",
len(live), len(projects) - len(live), format(total, ",.2f"),
args.days)
if spend.get(UNGROUPED):
log.info("$%s of cost came back ungrouped and is not counted as a "
"project", format(spend[UNGROUPED], ",.2f"))
envs = set()
if not args.no_key_names and live:
target = live[0]
if ranked:
by_id = {p.get("id"): p for p in live}
target = by_id.get(ranked[0][0], target)
names = [(k or {}).get("name") or ""
for k in paged(s, "/organization/projects/%s/api_keys"
% target.get("id"), limit=100,
owner_project_access="any")]
envs = mixed(names)
if envs:
log.info("key names in %s already name %d environment(s): %s",
target.get("name") or target.get("id"), len(envs),
", ".join(sorted(envs)))
state, detail = verdict(len(live), ranked, dominant=args.dominant)
if state in FINDINGS:
log.warning("%-26s %s", state, detail)
for line in repair_lines(state, envs):
log.warning(" repair: %s", line)
log.info("1 finding(s)")
return 1
log.info("%-26s %s", state, detail)
for line in repair_lines(state, envs):
log.info(" note: %s", line)
log.info("0 finding(s)")
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find an OpenAI organization with no project boundary to enforce anything on.
*
* Read only. Two paged GETs plus one for key NAMES. No request body is built,
* no key value is read or printed.
*
* The finding is the absence of a boundary rather than the concentration of
* spend: a single active project holds 100% by construction, and a dominant
* project in an org that has nine is a different reading with a different
* repair.
*/
const API = 'https://api.openai.com/v1';
const UNGROUPED = 'ungrouped';
// Whole-token matches only. Substring matching reports "devops-runner" as a
// development key and "provider-proxy" as a production one.
const ENV_WORDS = {
prod: 'prod', production: 'prod', live: 'prod',
stage: 'staging', staging: 'staging', preprod: 'staging',
dev: 'dev', development: 'dev',
local: 'local', laptop: 'local',
test: 'test', testing: 'test', qa: 'test',
ci: 'ci', build: 'ci',
sandbox: 'sandbox', scratch: 'sandbox', playground: 'sandbox',
};
const FINDINGS = new Set(['no-boundary', 'boundary-unused']);
const money = (n) => Number(n).toLocaleString('en-US',
{ minimumFractionDigits: 2, maximumFractionDigits: 2 });
/** Projects that can still receive traffic. Pure. Archived drops on either signal. */
export function active(projects) {
return (projects ?? []).filter((project) => {
const row = project ?? {};
if (String(row.status ?? '').trim().toLowerCase() === 'archived') return false;
if (row.archived_at) return false;
return true;
});
}
/** {project_id: dollars} from the cost report. Pure. Null project_id is ungrouped. */
export function spendByProject(buckets) {
const rows = {};
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const name = String(result?.project_id ?? UNGROUPED);
const value = Number(result?.amount?.value ?? 0);
if (!Number.isFinite(value)) continue;
rows[name] = (rows[name] ?? 0) + value;
}
}
return rows;
}
/** [[project_id, dollars, share]] over real projects only. Pure. */
export function shares(spend) {
const rows = Object.entries(spend ?? {}).filter(([k]) => k !== UNGROUPED);
const total = rows.reduce((a, [, v]) => a + v, 0);
const out = rows.map(([k, v]) => [k, Math.round(v * 100) / 100,
total > 0 ? v / total : 0]);
out.sort((a, b) => (b[1] - a[1]) || a[0].localeCompare(b[0]));
return out;
}
/** The environment classes named in one identifier. Pure. Whole tokens only. */
export function environments(name) {
const tokens = String(name ?? '').trim().toLowerCase().split(/[^a-z0-9]+/);
return new Set(tokens.filter((t) => ENV_WORDS[t]).map((t) => ENV_WORDS[t]));
}
/** Every environment class named across a set of identifiers. Pure. */
export function mixed(names) {
const found = new Set();
for (const name of names ?? []) for (const e of environments(name)) found.add(e);
return found;
}
/** Classify the organization's topology. Pure. Returns [state, detail]. */
export function verdict(activeCount, ranked, minSpend = 1.0, dominant = 0.95) {
const rows = [...(ranked ?? [])];
const total = Math.round(rows.reduce((a, r) => a + r[1], 0) * 100) / 100;
if (activeCount <= 0) {
return ['no-active-projects',
'the listing returned no active project at all, which usually means '
+ 'the key could not see them rather than that none exist'];
}
if (activeCount === 1) {
return ['no-boundary',
`1 active project holds 100% of $${money(total)}. There is no second `
+ 'container to cap, alert on, rate limit or attribute against.'];
}
if (total < minSpend) {
return ['no-spend-yet',
`${activeCount} active project(s) and $${money(total)} of attributable `
+ 'spend in the window. The boundary exists and nothing has tested it yet.'];
}
const [topId, , topShare] = rows[0];
const quiet = rows.slice(1).filter((r) => r[1] <= 0);
if (topShare >= dominant && quiet.length === rows.length - 1) {
return ['boundary-unused',
`${activeCount} active project(s), and ${topId} carries `
+ `${(topShare * 100).toFixed(0)}% of $${money(total)} while every other `
+ 'project has no spend at all. The containers exist and no traffic '
+ 'routes to them, so the controls on them enforce nothing.'];
}
if (topShare >= dominant) {
return ['concentration-not-topology',
`${activeCount} active project(s), and ${topId} carries `
+ `${(topShare * 100).toFixed(0)}% of $${money(total)}. This organization `
+ 'has a boundary, so that is a concentration reading rather than a '
+ 'topology one and has a different repair.'];
}
return ['separated',
`${activeCount} active project(s) sharing $${money(total)}, top project at `
+ `${(topShare * 100).toFixed(0)}%`];
}
/** The repair for one topology verdict. Pure. Printed, never performed. */
export function repairLines(state, envs = []) {
const found = [...envs].sort();
if (state === 'no-boundary') {
const lines = [
'create prod, staging and dev with POST /v1/organization/projects, which is '
+ 'the smallest split that lets any control differ.',
'give each project its own service account and key, then move traffic one '
+ 'key at a time rather than in one cutover.',
'spend limits, spend alerts, rate limits, model permissions and data '
+ 'retention are all configured per project and cannot differ until the '
+ 'projects do.',
'projects can be archived but never deleted, so the names are permanent. '
+ 'Spend ten minutes on them once.',
];
if (found.length) {
lines.unshift(`the environments already exist in your key names (${found.join(', ')}); `
+ 'they are simply not represented in the platform.');
}
return lines;
}
if (state === 'boundary-unused') {
return [
'the projects are not the problem. Nothing routes to them.',
'issue a key in the quiet projects and move the traffic that belongs there, '
+ 'then set the limits per project afterwards.',
'until traffic actually lands in a project, every control configured on it '
+ 'is inert.',
];
}
if (state === 'concentration-not-topology') {
return ['do not restructure on this reading. Rank the cost rows by share of '
+ 'total and ask which line item is expensive instead.'];
}
return [];
}
/** Unix seconds at midnight UTC, `days` ago. Pure given `now`. */
export function windowStart(days, now = new Date()) {
const midnight = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate());
return Math.floor(midnight / 1000) - days * 86400;
}
async function read(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/organization/* needs an `
+ 'organization admin key, not a project key');
}
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function* paged(key, path, params) {
const q = { ...params };
for (;;) {
const page = await read(key, path, q);
const data = page.data ?? [];
for (const item of data) yield item;
if (!page.has_more || data.length === 0) return;
q.after = page.last_id ?? data[data.length - 1]?.id;
}
}
async function* costBuckets(key, params, maxPages = 40) {
const q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, '/organization/costs', q);
for (const bucket of page.data ?? []) yield bucket;
if (!page.has_more || !page.next_page) return;
q.page = page.next_page;
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key; a project '
+ 'key cannot read /v1/organization/*');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 30);
const projects = [];
for await (const p of paged(admin, '/organization/projects',
{ limit: 100, include_archived: 'true' })) {
projects.push(p);
}
const live = active(projects);
const buckets = [];
for await (const b of costBuckets(admin, {
start_time: windowStart(days), bucket_width: '1d',
limit: Math.min(days, 30), group_by: 'project_id',
})) buckets.push(b);
const spend = spendByProject(buckets);
const ranked = shares(spend);
const total = Math.round(ranked.reduce((a, r) => a + r[1], 0) * 100) / 100;
console.log(`${live.length} active project(s), ${projects.length - live.length} `
+ `archived, $${money(total)} in the last ${days} day(s)`);
if (spend[UNGROUPED]) {
console.log(`$${money(spend[UNGROUPED])} of cost came back ungrouped and is not `
+ 'counted as a project');
}
let envs = new Set();
if (live.length) {
const byId = new Map(live.map((p) => [p.id, p]));
const target = byId.get(ranked[0]?.[0]) ?? live[0];
const names = [];
for await (const k of paged(admin, `/organization/projects/${target.id}/api_keys`,
{ limit: 100, owner_project_access: 'any' })) {
names.push(k?.name ?? '');
}
envs = mixed(names);
if (envs.size) {
console.log(`key names in ${target.name ?? target.id} already name ${envs.size} `
+ `environment(s): ${[...envs].sort().join(', ')}`);
}
}
const [state, detail] = verdict(live.length, ranked);
console.log(`${state.padEnd(26)} ${detail}`);
for (const line of repairLines(state, envs)) console.log(` repair: ${line}`);
process.exitCode = FINDINGS.has(state) ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The two tests at the top are the same organization read twice, and they are the note. One active project holding everything has to come back as no-boundary; four active projects with one of them at ninety-six per cent has to come back as concentration-not-topology, and must not be a finding, because that org has a boundary and needs the other reading. Next to them: containers that exist and never receive traffic, which is a third state with a third repair; archived projects dropped on either signal; the ungrouped null row that a careless fold turns into one enormous fictional project; and the environment matcher, which has to catch prod-worker and leave devops-runner alone.
from openai_project_boundary_audit import (active, environments, mixed,
repair_lines, shares,
spend_by_project, verdict,
window_start)
def project(pid, name, status="active", archived_at=None):
return {"id": pid, "name": name, "status": status,
"archived_at": archived_at}
def cost(pid, value):
return {"project_id": pid, "amount": {"value": value, "currency": "usd"}}
def buckets(*results):
return [{"results": list(results)}]
def test_one_active_project_is_the_finding_whatever_the_bill_says():
# The note in one assertion. There is nothing wrong with the spend; there
# is nowhere to put a limit, an alert or a model permission.
live = active([project("proj_a", "Default project"),
project("proj_old", "Prototype", status="archived")])
assert len(live) == 1
ranked = shares(spend_by_project(buckets(cost("proj_a", 18406.11))))
state, detail = verdict(len(live), ranked)
assert state == "no-boundary"
assert "no second container" in detail
repairs = repair_lines(state, {"prod", "ci", "local"})
assert any("archived but never deleted" in line for line in repairs)
assert any("key names" in line for line in repairs)
def test_a_dominant_project_in_a_split_org_is_the_other_note():
# Identical arithmetic, opposite conclusion. This organization already has
# the boundary this note is about, so the finding belongs to the
# concentration reading and the script says so rather than claiming it.
ranked = shares(spend_by_project(buckets(
cost("proj_prod", 96_000.0), cost("proj_stage", 2_400.0),
cost("proj_dev", 1_100.0), cost("proj_ci", 500.0))))
state, detail = verdict(4, ranked)
assert state == "concentration-not-topology"
assert "different repair" in detail
assert any("Rank the cost rows" in line for line in repair_lines(state))
def test_projects_that_exist_and_never_receive_traffic():
ranked = shares(spend_by_project(buckets(
cost("proj_prod", 9_900.0), cost("proj_stage", 0.0),
cost("proj_dev", 0.0))))
state, detail = verdict(3, ranked)
assert state == "boundary-unused"
assert "no traffic routes to them" in detail
assert any("Nothing routes to them" in line for line in repair_lines(state))
def test_archived_projects_are_dropped_on_either_signal():
rows = [project("a", "live"),
project("b", "by status", status="archived"),
project("c", "by timestamp", archived_at=1_700_000_000),
project("d", "shouty", status="ARCHIVED")]
assert [p["id"] for p in active(rows)] == ["a"]
assert active([]) == [] and active(None) == []
def test_an_ungrouped_row_is_never_ranked_as_a_project():
# The failure this guards: forget group_by, every project_id comes back
# null, and the fold reports one enormous project in an org that has three.
spend = spend_by_project(buckets(cost(None, 41_000.0), cost("proj_a", 900.0),
cost("proj_b", 100.0)))
assert spend["ungrouped"] == 41_000.0
ranked = shares(spend)
assert [row[0] for row in ranked] == ["proj_a", "proj_b"]
assert round(ranked[0][2], 2) == 0.90
assert verdict(3, ranked)[0] == "separated"
def test_environment_words_match_whole_tokens_only():
assert environments("prod-worker") == {"prod"}
assert environments("Local Adam") == {"local"}
assert environments("ci-fixtures") == {"ci"}
# The ones a substring test destroys.
assert environments("devops-runner") == set()
assert environments("provider-proxy") == set()
assert environments("protest") == set()
assert environments(None) == set()
assert mixed(["prod-worker", "local-adam", "ci-fixtures"]) == \
{"prod", "local", "ci"}
assert mixed([]) == set()
def test_no_spend_and_no_projects_are_never_verdicts():
assert verdict(0, [])[0] == "no-active-projects"
state, detail = verdict(3, shares(spend_by_project(buckets(cost("a", 0.2)))))
assert state == "no-spend-yet"
assert "nothing has tested it" in detail
assert repair_lines("separated") == []
assert spend_by_project(None) == {} and shares(None) == []
def test_the_window_starts_at_midnight_utc():
import datetime as dt
now = dt.datetime(2026, 8, 31, 17, 45, 12, tzinfo=dt.timezone.utc)
assert window_start(30, now) == int(
dt.datetime(2026, 8, 1, tzinfo=dt.timezone.utc).timestamp())
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { active, environments, mixed, repairLines, shares, spendByProject,
verdict, windowStart } from './openai-project-boundary-audit.mjs';
const project = (id, name, status = 'active', archivedAt = null) =>
({ id, name, status, archived_at: archivedAt });
const cost = (projectId, value) =>
({ project_id: projectId, amount: { value, currency: 'usd' } });
const buckets = (...results) => [{ results }];
test('one active project is the finding whatever the bill says', () => {
const live = active([project('proj_a', 'Default project'),
project('proj_old', 'Prototype', 'archived')]);
assert.equal(live.length, 1);
const ranked = shares(spendByProject(buckets(cost('proj_a', 18406.11))));
const [state, detail] = verdict(live.length, ranked);
assert.equal(state, 'no-boundary');
assert.match(detail, /no second container/);
const repairs = repairLines(state, new Set(['prod', 'ci', 'local']));
assert.ok(repairs.some((l) => l.includes('archived but never deleted')));
assert.ok(repairs.some((l) => l.includes('key names')));
});
test('a dominant project in a split org is the other note', () => {
const ranked = shares(spendByProject(buckets(
cost('proj_prod', 96000), cost('proj_stage', 2400),
cost('proj_dev', 1100), cost('proj_ci', 500))));
const [state, detail] = verdict(4, ranked);
assert.equal(state, 'concentration-not-topology');
assert.match(detail, /different repair/);
assert.ok(repairLines(state).some((l) => l.includes('Rank the cost rows')));
});
test('projects that exist and never receive traffic', () => {
const ranked = shares(spendByProject(buckets(
cost('proj_prod', 9900), cost('proj_stage', 0), cost('proj_dev', 0))));
const [state, detail] = verdict(3, ranked);
assert.equal(state, 'boundary-unused');
assert.match(detail, /no traffic routes to them/);
});
test('archived projects are dropped on either signal', () => {
const rows = [project('a', 'live'), project('b', 'by status', 'archived'),
project('c', 'by timestamp', 'active', 1700000000),
project('d', 'shouty', 'ARCHIVED')];
assert.deepEqual(active(rows).map((p) => p.id), ['a']);
assert.deepEqual(active(null), []);
});
test('an ungrouped row is never ranked as a project', () => {
const spend = spendByProject(buckets(cost(null, 41000), cost('proj_a', 900),
cost('proj_b', 100)));
assert.equal(spend.ungrouped, 41000);
const ranked = shares(spend);
assert.deepEqual(ranked.map((r) => r[0]), ['proj_a', 'proj_b']);
assert.equal(Math.round(ranked[0][2] * 100) / 100, 0.9);
assert.equal(verdict(3, ranked)[0], 'separated');
});
test('environment words match whole tokens only', () => {
assert.deepEqual([...environments('prod-worker')], ['prod']);
assert.deepEqual([...environments('Local Adam')], ['local']);
assert.deepEqual([...environments('devops-runner')], []);
assert.deepEqual([...environments('provider-proxy')], []);
assert.deepEqual([...environments('protest')], []);
assert.deepEqual([...environments(null)], []);
assert.deepEqual([...mixed(['prod-worker', 'local-adam', 'ci-fixtures'])].sort(),
['ci', 'local', 'prod']);
});
test('no spend and no projects are never verdicts', () => {
assert.equal(verdict(0, [])[0], 'no-active-projects');
const [state, detail] = verdict(3, shares(spendByProject(buckets(cost('a', 0.2)))));
assert.equal(state, 'no-spend-yet');
assert.match(detail, /nothing has tested it/);
assert.deepEqual(repairLines('separated'), []);
assert.deepEqual(spendByProject(null), {});
assert.deepEqual(shares(null), []);
});
test('the window starts at midnight utc', () => {
assert.equal(windowStart(30, new Date('2026-08-31T17:45:12Z')),
Date.UTC(2026, 7, 1) / 1000);
});
FAQ
How is this different from finding out that one project dominates the bill?
By what it counts. The concentration reading ranks cost rows by share of total and tells you which line item or project is expensive, and it is a useful reading in an organization that has several projects, because you can act inside the structure you already have: cap that project, move that workload, change that model. In a single-project organization the top row is a hundred per cent by construction, so the ranking has told you nothing you did not know, and the repair is not rebalancing but creating containers that do not exist yet. The script counts active projects before it looks at any money, and when it finds a dominant project in an org that has a boundary it names the concentration reading and refuses to report a topology fault.
What actually breaks with one project? The bill is the same either way.
The bill is the same. The controls are not. Spend limits, spend alerts, rate limits, model permissions, hosted tool permissions and data retention are all configured per project, so with one project every one of them is an all-or-nothing switch over your entire organization. You cannot cap the experiments without capping production, cannot throttle the evaluation run without throttling the customer path, and cannot stop a prototype reaching an expensive model without stopping the product reaching it too. Attribution is the symptom people notice; enforcement is what is missing.
We created staging and dev months ago. Why is the script still complaining?
Because nothing routes to them. That is a separate state with a separate name in the output, boundary-unused, and it comes with a different repair: the containers are fine and the traffic is the problem. A project with no keys and no spend enforces nothing at all, and every limit configured on it is inert until a request actually arrives there. This is common after a security review, where the projects get created as an artefact of the review and the key migration never follows.
Does the script read my API keys to work out which environments exist?
It reads key names, never key values, and the platform would not return a value anyway. The name read is corroboration only and never changes the verdict: when one project holds keys called prod-worker, local-adam and ci-fixtures, the environments already exist as a fact about how people work and are simply not represented in the platform. Environment words are matched as whole tokens after splitting on punctuation, so devops-runner is a team name rather than a development key, and provider-proxy is not production.
Is there an Anthropic version of this?
The equivalent container is a workspace, and the same reasoning applies: one workspace means no boundary to attribute or rate limit against. The reason it is not this script is that Anthropic's topology has a second problem OpenAI does not have at all, the default workspace, whose cost reports with a null workspace id and which cannot carry a rate-limit override even if you want one. That is the subject of the sibling note rather than a footnote to this one.
Related field notes
- The concentration reading, which needs a second container to mean anything
- The only ceiling an org without projects has left
- The Anthropic half of the same topology, read from the cost report
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Projects — OpenAI API reference
- Managing projects in the API platform — OpenAI help
- Usage and costs — OpenAI API reference
- Admin APIs — OpenAI platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.