Diagnostic LLM APIs
Vector store bytes grow while nobody queries the index
Somebody finally asks about the small line. It has been on the invoice for fourteen months, it has never been more than a couple of hundred dollars, and it is the only line that has gone up every single month regardless of what shipped. It is vector store storage. It is billed on bytes retained per hour rather than on anything anybody did, and a good deal of it is a corpus indexed for a demo in the spring of last year that has not been searched since the demo.
Three GETs with an organization admin key, and the reading is a slope rather than a share. GET /v1/organization/usage/vector_stores?start_time={now-90d}&bucket_width=1d&limit=31&group_by=project_id, paged on next_page, gives a daily usage_bytes series per project. Fit a trend across it.
Then the denominator that makes the slope mean something: GET /v1/organization/usage/file_search_calls?start_time={now-90d}&bucket_width=1d&limit=31&group_by=project_id&group_by=vector_store_id, whose results carry num_requests and vector_store_id. Bytes climbing while queries stay flat is the finding. Bytes climbing alongside queries is a corpus doing its job and is graded as such.
Price it off GET /v1/organization/costs?start_time={now-90d}&bucket_width=1d&limit=31&group_by=line_item, selecting on quantity_unit == "gibibyte_hours" rather than on the line item's name. The unit is the thing that identifies storage, it is the thing that will not be renamed, and gibibyte-hours is the billing model stated out loud: you are paying for bytes multiplied by time.
One asymmetry decides the shape of the script. The bytes endpoint groups by project_id and nothing else, so there is no per-store byte series in the usage API at all. Queries can be grouped per store. To name the store rather than the project you need the current snapshot from GET /v1/vector_stores with a project key, joined against the per-store query counts — a store holding real bytes with zero num_requests across ninety days is retained waste, and that join is the only way to see it.
The problem in plain words
Every other line on an LLM invoice is a flow. Tokens, tool calls, audio seconds, image counts: they are all driven by requests, so they fall to zero when the traffic does, and they are all things somebody chose to do. Storage is a stock. It is charged on how many bytes you are holding and for how long, so the bill continues at exactly the same rate through a quiet quarter, a code freeze and a product being retired.
The behaviour that produces it is not careless, which is why it persists. Indexing a corpus is a normal part of building retrieval and it is meant to be easy. Deleting it afterwards was never a ticket, because at the moment the work stops nobody has decided the work is over. The prototype might come back. The evaluation corpus might get re-run. And the monthly cost of keeping it is small enough that it never crosses the threshold at which anybody would ask.
So it compounds, and it compounds invisibly, because the number that would reveal it is a slope rather than a level. Any single month's storage line looks negligible. The same line plotted across ninety days against a flat query count is the whole argument, and nothing in the console or the invoice draws that comparison for you.
The reverse case matters just as much and is easy to trample. A corpus that is growing because the product is growing is supposed to cost more, and reporting it as waste is how a cost report gets ignored. The finding is not growth; it is growth without use.
Why it happens
Bytes and queries do not come back at the same granularity, and the script is shaped around that. /v1/organization/usage/vector_stores supports exactly one grouping, project_id, so the usage API has no per-store byte series and never will produce one by asking harder. /v1/organization/usage/file_search_calls does support vector_store_id. So the trend is a per-project reading, and naming the individual store requires joining the per-store query counts against the current snapshot from GET /v1/vector_stores, which needs a project key rather than the admin key everything else here uses. Run it with only the admin key and you get a correct trend and no culprit, which the output says rather than implying the store list was empty.
Selecting the storage cost by quantity_unit rather than by line-item name is what makes this survive a rename. quantity_unit is an enumerated field, and gibibyte_hours appears on exactly the storage lines. Matching a name string means the reconciliation quietly returns zero the first time the platform relabels something, and returning zero from a cost check is the failure mode that never gets noticed. This also keeps the note away from the line-item reconciliation note, which is about a dashboard's coverage of the whole bill; this one reads one unit and one slope.
Ninety days of daily buckets does not fit in one response. The usage endpoints cap limit at 31 buckets when bucket_width is 1d, so the window has to be walked with the page parameter against next_page. A script that asks for ninety and reads what comes back gets a month, computes a slope over it, and reports a trend that is genuinely a third of the one you asked for.
A slope needs a floor under it or it reports rounding. A project holding forty megabytes can double its storage in a week and the finding is worth nothing at all. The script requires an absolute size before it grades a growth rate, and reports the money next to the percentage every time, because a percentage with no dollars attached is how cost reports get argued with rather than acted on.
This is not a per-token reading and does not belong next to one. The output-token note is about the price of generating; this is about the price of holding. They move independently, they are fixed by different changes, and the only thing storage has in common with the rest of the bill is that it appears on it.
The fix, as a flow
Every other line on the bill is a flow: it falls to zero when the traffic stops. Storage is a stock, so it keeps billing whether anyone queries it or not, and the shape that gives it is a line that only ever goes up. The reading needs two series rather than one, and they do not have the same granularity: bytes come back per project, while file search calls can be grouped per store.
How to fix it
Use an organization admin key, provisioned read-only
Every /v1/organization/* path rejects a project key. Add a project key as well if you want the per-store snapshot, because GET /v1/vector_stores is project-scoped and the admin key cannot reach it.
Walk ninety days of daily byte buckets, one page at a time
GET /v1/organization/usage/vector_stores?start_time={now-90d}&bucket_width=1d&limit=31&group_by=project_id. limit caps at 31 for daily buckets, so page on next_page until it is null. Each result carries usage_bytes and project_id.
Pull the query volume over the same window, grouped two ways
GET /v1/organization/usage/file_search_calls with group_by=project_id&group_by=vector_store_id. Results carry num_requests, and both grouping fields are null on rows the API could not attribute, which are summed separately rather than folded into a store.
Price the storage by its unit, not by its name
GET /v1/organization/costs?start_time={now-90d}&bucket_width=1d&limit=31&group_by=line_item, keeping only results whose quantity_unit is gibibyte_hours. That unit is the billing model written down: bytes multiplied by time.
Join the snapshot to name the stores, and print the repair
GET /v1/vector_stores?limit=100 with a project key for each store's current usage_bytes, last_active_at and name. A store above the size floor with zero num_requests across the window is retained waste. The repair is a deletion for the dead ones and an expiration policy at creation for the rest, printed rather than run.
How to check it worked
Delete one dead store and re-run a week later. The project's byte series should step down and stay down, and the gibibyte_hours quantity should fall in proportion. The reading that tells you the fix is durable is a second run a month after that: a project whose slope is flat while its query count is not has stopped accumulating, which is the actual goal rather than a one-off deletion.
python3 openai_vector_store_storage_trend.py --days 90
# 90 day(s) of daily buckets across 3 project(s), 4 store(s) in the snapshot
# storage cost in the window: $412.88 over 41,288.0 gibibyte_hours
# bytes-growing-queries-flat proj_research: 8.1 GiB -> 31.4 GiB (+288%), 0
# file search call(s) in 90 day(s)
# repair: no query has touched this project's stores in the window. The bytes
# are being retained, not used.
# repair: idle stores holding real bytes:
# vs_c3 march-demo 12.4 GiB, last active 148 day(s) ago
# vs_e5 eval-corpus-v1 9.8 GiB, last active 96 day(s) ago
# repair: delete the dead ones with DELETE /v1/vector_stores/{vector_store_id}
# after archiving anything you still need.
# repair: set an expiration policy at creation on stores that are meant to be
# temporary, so the next prototype ages out on its own.
# bytes-and-queries-growing proj_prod: 44.0 GiB -> 61.2 GiB (+39%), 1,204,551
# file search call(s). Growth, priced correctly.
# below-threshold proj_ci: 0.1 GiB, under the 1.0 GiB floor
# 1 finding(s)The full code
Three paged GETs, one optional fourth, and seven pure functions. byte_series and query_series, which fold the usage buckets into per-project daily series and keep the unattributed rows under an explicit sentinel; slope, a least-squares fit in bytes per day that returns zero rather than raising on a single point; growth, which returns first, last, delta and percentage together so no caller has to recompute one from another; searches_by_store, the per-store query totals that the byte series cannot provide; storage_lines, which selects cost results on quantity_unit being gibibyte_hours rather than on a name; idle_stores, the join; and verdict, which puts an absolute size floor under every growth rate.
"""Trend retained vector store bytes against the queries that justify them.
Read only. Three paged GETs against /v1/organization/* with an admin key, plus
one optional GET of /v1/vector_stores with a project key for the per-store
snapshot. No request body is constructed and no file_search query is ever run.
Storage is a stock rather than a flow: it bills on bytes retained per unit of
time, so it does not fall when traffic does. The finding is therefore a slope
rather than a share, and it is only a finding when the slope is not matched by
query volume. Bytes growing alongside searches is a corpus doing its job.
One asymmetry shapes everything below. The vector stores usage endpoint groups
by project_id and nothing else, so there is no per-store byte series to ask
for; file search calls can be grouped by vector_store_id. Naming an individual
store therefore requires the current snapshot, which needs a project key.
"""
import argparse
import datetime as dt
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_vector_store_storage_trend")
API = "https://api.openai.com/v1"
BETA = {"OpenAI-Beta": "assistants=v2"}
# Rows the report could not attribute to a project or a store. Kept under an
# explicit name and never folded into a real id, because a null that becomes a
# key is how one enormous fictional project gets reported.
UNGROUPED = "ungrouped"
# The unit that identifies storage on the cost report. Selecting on this rather
# than on a line item's display name is the difference between a check that
# survives a relabel and one that silently starts returning zero.
STORAGE_UNIT = "gibibyte_hours"
GIB = 1073741824.0
DAY = 86400
FINDINGS = ("bytes-growing-queries-flat", "bytes-growing-never-queried")
def byte_series(buckets):
"""{project_id: [(start_time, usage_bytes)]} sorted by time. Pure."""
rows = {}
for bucket in buckets or []:
start = (bucket or {}).get("start_time")
for result in (bucket or {}).get("results") or []:
row = result or {}
key = str(row.get("project_id") or UNGROUPED)
try:
value = int(row.get("usage_bytes") or 0)
except (TypeError, ValueError):
continue
rows.setdefault(key, []).append((int(start or 0), value))
for points in rows.values():
points.sort()
return rows
def query_series(buckets):
"""{project_id: [(start_time, num_requests)]} sorted by time. Pure."""
rows = {}
for bucket in buckets or []:
start = (bucket or {}).get("start_time")
for result in (bucket or {}).get("results") or []:
row = result or {}
key = str(row.get("project_id") or UNGROUPED)
try:
value = int(row.get("num_requests") or 0)
except (TypeError, ValueError):
continue
rows.setdefault(key, []).append((int(start or 0), value))
for points in rows.values():
points.sort()
return rows
def searches_by_store(buckets):
"""{vector_store_id: total num_requests}. Pure.
The one per-store number available anywhere in the usage API. There is no
matching per-store byte series: the vector stores endpoint groups by
project_id only.
"""
rows = {}
for bucket in buckets or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
key = str(row.get("vector_store_id") or UNGROUPED)
try:
value = int(row.get("num_requests") or 0)
except (TypeError, ValueError):
continue
rows[key] = rows.get(key, 0) + value
return rows
def slope(points):
"""Least-squares trend in units per day. Pure. Zero on fewer than 2 points."""
rows = sorted(points or [])
if len(rows) < 2:
return 0.0
base = rows[0][0]
xs = [(t - base) / float(DAY) for t, _ in rows]
ys = [float(v) for _, v in rows]
n = float(len(rows))
mx = sum(xs) / n
my = sum(ys) / n
denom = sum((x - mx) ** 2 for x in xs)
if denom <= 0:
return 0.0
return sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / denom
def growth(points):
"""(first, last, delta, fraction) over a series. Pure.
The fraction is delta over first, and is 0.0 rather than infinity when the
series starts at zero, because "grew infinitely from nothing" is a division
artefact rather than a reading anybody can act on.
"""
rows = sorted(points or [])
if not rows:
return (0, 0, 0, 0.0)
first = rows[0][1]
last = rows[-1][1]
delta = last - first
fraction = (float(delta) / float(first)) if first > 0 else 0.0
return (first, last, delta, fraction)
def storage_lines(buckets):
"""{line_item: {"dollars": x, "gibibyte_hours": q}} for storage only. Pure.
Selected on quantity_unit, never on the line item's name.
"""
rows = {}
for bucket in buckets or []:
for result in (bucket or {}).get("results") or []:
row = result or {}
if str(row.get("quantity_unit") or "") != STORAGE_UNIT:
continue
name = str(row.get("line_item") or "unlabelled")
try:
dollars = float((row.get("amount") or {}).get("value") or 0.0)
except (TypeError, ValueError):
dollars = 0.0
try:
quantity = float(row.get("quantity") or 0.0)
except (TypeError, ValueError):
quantity = 0.0
entry = rows.setdefault(name, {"dollars": 0.0, STORAGE_UNIT: 0.0})
entry["dollars"] += dollars
entry[STORAGE_UNIT] += quantity
return rows
def idle_stores(stores, searches, now, min_bytes=1073741824):
"""[(id, name, bytes, idle_days)] for stores nothing searched. Pure.
The join the usage API cannot do for you: per-store query counts against a
current snapshot. A store under the size floor is skipped, because a
finding about 40 MiB is a finding about nothing.
"""
out = []
for store in stores or []:
row = store or {}
sid = str(row.get("id") or "")
try:
size = int(row.get("usage_bytes") or 0)
except (TypeError, ValueError):
continue
if not sid or size < min_bytes:
continue
if int((searches or {}).get(sid, 0)) > 0:
continue
try:
last = int(row.get("last_active_at") or 0)
except (TypeError, ValueError):
last = 0
idle = int((now - last) / DAY) if last > 0 else -1
out.append((sid, str(row.get("name") or "(unnamed)"), size, idle))
out.sort(key=lambda r: (-r[2], r[0]))
return out
def verdict(bytes_points, query_points, days, min_gib=1.0, min_growth=0.25):
"""Classify one project. Pure. Returns (state, detail).
The absolute size floor comes before the growth rate, always. A project
holding forty megabytes can triple its storage in a week and the reading is
worth nothing.
"""
first, last, _delta, fraction = growth(bytes_points)
queries = sum(v for _, v in (query_points or []))
if last < min_gib * GIB:
return ("below-threshold",
"%.1f GiB, under the %.1f GiB floor" % (last / GIB, min_gib))
if fraction < min_growth:
return ("flat",
"%.1f GiB, %+.0f%% over %d day(s), %s file search call(s)"
% (last / GIB, fraction * 100, days, format(queries, ",")))
shape = ("%.1f GiB -> %.1f GiB (%+.0f%%)"
% (first / GIB, last / GIB, fraction * 100))
if queries <= 0:
return ("bytes-growing-never-queried",
"%s, 0 file search call(s) in %d day(s)" % (shape, days))
if slope(query_points) <= 0:
return ("bytes-growing-queries-flat",
"%s while file search calls are flat or falling across the same "
"window" % shape)
return ("bytes-and-queries-growing",
"%s, %s file search call(s). Growth, priced correctly."
% (shape, format(queries, ",")))
def repair_lines(state, idle=()):
"""The repair for one verdict. Pure. Printed, never performed."""
idle = list(idle or [])
if state in FINDINGS:
lines = []
if state == "bytes-growing-never-queried":
lines.append("no query has touched this project's stores in the "
"window. The bytes are being retained, not used.")
else:
lines.append("the corpus is growing and the query volume is not, "
"so you are paying more each month for the same "
"amount of retrieval.")
if idle:
lines.append("idle stores holding real bytes: " + "; ".join(
"%s %s %.1f GiB%s" % (sid, name, size / GIB,
"" if days < 0 else
", last active %d day(s) ago" % days)
for sid, name, size, days in idle[:8]))
else:
lines.append("no per-store snapshot was read, so the project is "
"named and the store is not. Add a project key to "
"join the query counts against GET /v1/vector_stores.")
lines.append("delete the dead ones with "
"DELETE /v1/vector_stores/{vector_store_id} after "
"archiving anything you still need.")
lines.append("set an expiration policy at creation on stores that are "
"meant to be temporary, so the next prototype ages out on "
"its own rather than being somebody's future ticket.")
return lines
if state == "bytes-and-queries-growing":
return ["nothing to do. This is a corpus that is being used more, and "
"the storage line is supposed to follow it."]
return []
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/organization/* needs an "
"organization admin key, not a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def usage_buckets(session, path, params, max_pages=40):
"""Walk a usage report. limit caps at 31 daily buckets, so this pages."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, **params)
for bucket in page.get("data") or []:
yield bucket
if not page.get("has_more") or not page.get("next_page"):
return
params["page"] = page["next_page"]
def paged(session, path, max_pages=200, **params):
"""Walk an after/last_id cursor listing."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, **params)
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params["after"] = page.get("last_id") or (data[-1] or {}).get("id")
def window_start(days, now=None):
"""Unix seconds at midnight UTC, `days` ago."""
now = now or dt.datetime.now(dt.timezone.utc)
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
return int((midnight - dt.timedelta(days=days)).timestamp())
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=90,
help="days of daily buckets to trend (default 90)")
ap.add_argument("--min-gib", type=float, default=1.0,
help="size floor below which growth is not graded")
ap.add_argument("--min-growth", type=float, default=0.25,
help="fractional growth above which a slope is a finding")
args = ap.parse_args()
admin = os.environ.get("OPENAI_ADMIN_KEY")
if not admin:
log.error("set OPENAI_ADMIN_KEY to an organization admin key; a "
"project key cannot read /v1/organization/*")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + admin})
start = window_start(args.days)
common = {"start_time": start, "bucket_width": "1d", "limit": 31}
bytes_buckets = list(usage_buckets(
s, "/organization/usage/vector_stores",
dict(common, group_by="project_id")))
search_buckets = list(usage_buckets(
s, "/organization/usage/file_search_calls",
dict(common, group_by=["project_id", "vector_store_id"])))
cost_buckets = list(usage_buckets(
s, "/organization/costs", dict(common, group_by="line_item")))
by_project = byte_series(bytes_buckets)
queries = query_series(search_buckets)
per_store = searches_by_store(search_buckets)
stores = []
project_key = os.environ.get("OPENAI_API_KEY")
if project_key:
p = requests.Session()
p.headers.update({"Authorization": "Bearer " + project_key, **BETA})
stores = list(paged(p, "/vector_stores", limit=100))
log.info("%d day(s) of daily buckets across %d project(s), %d store(s) in "
"the snapshot", args.days, len(by_project), len(stores))
lines = storage_lines(cost_buckets)
dollars = sum(v["dollars"] for v in lines.values())
hours = sum(v[STORAGE_UNIT] for v in lines.values())
if lines:
log.info("storage cost in the window: $%s over %s %s",
format(round(dollars, 2), ",.2f"), format(round(hours, 1), ","),
STORAGE_UNIT)
else:
log.info("no cost result carried quantity_unit %r in the window, so "
"nothing is being billed for storage yet", STORAGE_UNIT)
now = int(dt.datetime.now(dt.timezone.utc).timestamp())
idle = idle_stores(stores, per_store, now,
min_bytes=int(args.min_gib * GIB))
findings = 0
for project in sorted(by_project):
state, detail = verdict(by_project[project], queries.get(project, []),
args.days, args.min_gib, args.min_growth)
emit = log.warning if state in FINDINGS else log.info
emit("%-27s %s: %s", state, project, detail)
for line in repair_lines(state, idle if state in FINDINGS else ()):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
if per_store.get(UNGROUPED):
log.info("%s file search call(s) came back with no vector_store_id and "
"are not attributed to a store",
format(per_store[UNGROUPED], ","))
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Trend retained vector store bytes against the queries that justify them.
*
* Read only. Three paged GETs against /v1/organization/* with an admin key,
* plus one optional GET of /v1/vector_stores with a project key. No request
* body is constructed and no file_search query is ever run.
*
* Storage is a stock rather than a flow, so the finding is a slope rather than
* a share, and only when the slope is not matched by query volume.
*
* The vector stores usage endpoint groups by project_id and nothing else, so
* naming an individual store needs the snapshot joined to per-store query
* counts from the file search calls report.
*/
const API = 'https://api.openai.com/v1';
const BETA = { 'OpenAI-Beta': 'assistants=v2' };
/** Rows the report could not attribute. Never folded into a real id. */
export const UNGROUPED = 'ungrouped';
/** The unit that identifies storage on the cost report. Not a name match. */
export const STORAGE_UNIT = 'gibibyte_hours';
const GIB = 1073741824;
const DAY = 86400;
const FINDINGS = new Set(['bytes-growing-queries-flat',
'bytes-growing-never-queried']);
const num = (n) => Number(n).toLocaleString('en-US');
/** {projectId: [[startTime, usageBytes]]} sorted by time. Pure. */
export function byteSeries(buckets) {
const rows = {};
for (const bucket of buckets ?? []) {
const start = Math.trunc(Number(bucket?.start_time ?? 0));
for (const result of bucket?.results ?? []) {
const key = String(result?.project_id ?? UNGROUPED);
const value = Number(result?.usage_bytes ?? 0);
if (!Number.isFinite(value)) continue;
(rows[key] ??= []).push([start, Math.trunc(value)]);
}
}
for (const points of Object.values(rows)) points.sort((a, b) => a[0] - b[0]);
return rows;
}
/** {projectId: [[startTime, numRequests]]} sorted by time. Pure. */
export function querySeries(buckets) {
const rows = {};
for (const bucket of buckets ?? []) {
const start = Math.trunc(Number(bucket?.start_time ?? 0));
for (const result of bucket?.results ?? []) {
const key = String(result?.project_id ?? UNGROUPED);
const value = Number(result?.num_requests ?? 0);
if (!Number.isFinite(value)) continue;
(rows[key] ??= []).push([start, Math.trunc(value)]);
}
}
for (const points of Object.values(rows)) points.sort((a, b) => a[0] - b[0]);
return rows;
}
/** {vectorStoreId: total numRequests}. Pure. The only per-store number there is. */
export function searchesByStore(buckets) {
const rows = {};
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
const key = String(result?.vector_store_id ?? UNGROUPED);
const value = Number(result?.num_requests ?? 0);
if (!Number.isFinite(value)) continue;
rows[key] = (rows[key] ?? 0) + Math.trunc(value);
}
}
return rows;
}
/** Least-squares trend in units per day. Pure. Zero on fewer than 2 points. */
export function slope(points) {
const rows = [...(points ?? [])].sort((a, b) => a[0] - b[0]);
if (rows.length < 2) return 0;
const base = rows[0][0];
const xs = rows.map(([t]) => (t - base) / DAY);
const ys = rows.map(([, v]) => Number(v));
const n = rows.length;
const mx = xs.reduce((a, x) => a + x, 0) / n;
const my = ys.reduce((a, y) => a + y, 0) / n;
const denom = xs.reduce((a, x) => a + (x - mx) ** 2, 0);
if (denom <= 0) return 0;
let cov = 0;
for (let i = 0; i < n; i += 1) cov += (xs[i] - mx) * (ys[i] - my);
return cov / denom;
}
/** [first, last, delta, fraction] over a series. Pure. */
export function growth(points) {
const rows = [...(points ?? [])].sort((a, b) => a[0] - b[0]);
if (!rows.length) return [0, 0, 0, 0];
const first = rows[0][1];
const last = rows[rows.length - 1][1];
const delta = last - first;
return [first, last, delta, first > 0 ? delta / first : 0];
}
/** {lineItem: {dollars, gibibyte_hours}} for storage only. Pure. */
export function storageLines(buckets) {
const rows = {};
for (const bucket of buckets ?? []) {
for (const result of bucket?.results ?? []) {
if (String(result?.quantity_unit ?? '') !== STORAGE_UNIT) continue;
const name = String(result?.line_item ?? 'unlabelled');
const dollars = Number(result?.amount?.value ?? 0);
const quantity = Number(result?.quantity ?? 0);
const entry = (rows[name] ??= { dollars: 0, [STORAGE_UNIT]: 0 });
if (Number.isFinite(dollars)) entry.dollars += dollars;
if (Number.isFinite(quantity)) entry[STORAGE_UNIT] += quantity;
}
}
return rows;
}
/** [[id, name, bytes, idleDays]] for stores nothing searched. Pure. */
export function idleStores(stores, searches, now, minBytes = GIB) {
const out = [];
for (const store of stores ?? []) {
const sid = String(store?.id ?? '');
const size = Number(store?.usage_bytes ?? 0);
if (!sid || !Number.isFinite(size) || size < minBytes) continue;
if (Number((searches ?? {})[sid] ?? 0) > 0) continue;
const last = Number(store?.last_active_at ?? 0);
const idle = Number.isFinite(last) && last > 0
? Math.trunc((now - last) / DAY) : -1;
out.push([sid, String(store?.name ?? '(unnamed)'), Math.trunc(size), idle]);
}
out.sort((a, b) => (b[2] - a[2]) || a[0].localeCompare(b[0]));
return out;
}
/** Classify one project. Pure. Returns [state, detail]. */
export function verdict(bytesPoints, queryPoints, days, minGib = 1.0,
minGrowth = 0.25) {
const [first, last, , fraction] = growth(bytesPoints);
const queries = (queryPoints ?? []).reduce((a, [, v]) => a + v, 0);
if (last < minGib * GIB) {
return ['below-threshold',
`${(last / GIB).toFixed(1)} GiB, under the ${minGib.toFixed(1)} GiB floor`];
}
if (fraction < minGrowth) {
return ['flat',
`${(last / GIB).toFixed(1)} GiB, ${fraction >= 0 ? '+' : ''}`
+ `${(fraction * 100).toFixed(0)}% over ${days} day(s), `
+ `${num(queries)} file search call(s)`];
}
const shape = `${(first / GIB).toFixed(1)} GiB -> ${(last / GIB).toFixed(1)} GiB `
+ `(${fraction >= 0 ? '+' : ''}${(fraction * 100).toFixed(0)}%)`;
if (queries <= 0) {
return ['bytes-growing-never-queried',
`${shape}, 0 file search call(s) in ${days} day(s)`];
}
if (slope(queryPoints) <= 0) {
return ['bytes-growing-queries-flat',
`${shape} while file search calls are flat or falling across the same window`];
}
return ['bytes-and-queries-growing',
`${shape}, ${num(queries)} file search call(s). Growth, priced correctly.`];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, idle = []) {
const rows = [...(idle ?? [])];
if (FINDINGS.has(state)) {
const lines = [];
if (state === 'bytes-growing-never-queried') {
lines.push("no query has touched this project's stores in the window. The "
+ 'bytes are being retained, not used.');
} else {
lines.push('the corpus is growing and the query volume is not, so you are '
+ 'paying more each month for the same amount of retrieval.');
}
if (rows.length) {
lines.push('idle stores holding real bytes: ' + rows.slice(0, 8)
.map(([sid, name, size, days]) => `${sid} ${name} ${(size / GIB).toFixed(1)} GiB`
+ (days < 0 ? '' : `, last active ${days} day(s) ago`)).join('; '));
} else {
lines.push('no per-store snapshot was read, so the project is named and the '
+ 'store is not. Add a project key to join the query counts against '
+ 'GET /v1/vector_stores.');
}
lines.push('delete the dead ones with DELETE /v1/vector_stores/{vector_store_id} '
+ 'after archiving anything you still need.');
lines.push('set an expiration policy at creation on stores that are meant to be '
+ 'temporary, so the next prototype ages out on its own rather than being '
+ "somebody's future ticket.");
return lines;
}
if (state === 'bytes-and-queries-growing') {
return ['nothing to do. This is a corpus that is being used more, and the '
+ 'storage line is supposed to follow it.'];
}
return [];
}
/** Unix seconds at midnight UTC, `days` ago. Pure given `now`. */
export function windowStart(days, now = new Date()) {
const midnight = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate());
return Math.floor(midnight / 1000) - days * DAY;
}
async function read(key, path, params, extra = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) {
if (Array.isArray(v)) for (const one of v) url.searchParams.append(k, String(one));
else url.searchParams.set(k, String(v));
}
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}`, ...extra } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/organization/* needs an `
+ 'organization admin key, not a project key');
}
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function* usageBuckets(key, path, params, maxPages = 40) {
const q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, path, q);
for (const bucket of page.data ?? []) yield bucket;
if (!page.has_more || !page.next_page) return;
q.page = page.next_page;
}
}
async function* paged(key, path, params, maxPages = 200) {
const q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, path, q, BETA);
const data = page.data ?? [];
for (const item of data) yield item;
if (!page.has_more || data.length === 0) return;
q.after = page.last_id ?? data[data.length - 1]?.id;
}
}
async function main() {
const admin = process.env.OPENAI_ADMIN_KEY;
if (!admin) {
console.error('set OPENAI_ADMIN_KEY to an organization admin key; a project '
+ 'key cannot read /v1/organization/*');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 90);
const minGib = Number(process.env.MIN_GIB ?? 1);
const minGrowth = Number(process.env.MIN_GROWTH ?? 0.25);
const common = { start_time: windowStart(days), bucket_width: '1d', limit: 31 };
const collect = async (path, params) => {
const out = [];
for await (const b of usageBuckets(admin, path, params)) out.push(b);
return out;
};
const bytesBuckets = await collect('/organization/usage/vector_stores',
{ ...common, group_by: 'project_id' });
const searchBuckets = await collect('/organization/usage/file_search_calls',
{ ...common, group_by: ['project_id', 'vector_store_id'] });
const costBuckets = await collect('/organization/costs',
{ ...common, group_by: 'line_item' });
const byProject = byteSeries(bytesBuckets);
const queries = querySeries(searchBuckets);
const perStore = searchesByStore(searchBuckets);
const stores = [];
if (process.env.OPENAI_API_KEY) {
for await (const st of paged(process.env.OPENAI_API_KEY, '/vector_stores',
{ limit: 100 })) stores.push(st);
}
console.log(`${days} day(s) of daily buckets across `
+ `${Object.keys(byProject).length} project(s), ${stores.length} `
+ 'store(s) in the snapshot');
const lines = storageLines(costBuckets);
const dollars = Object.values(lines).reduce((a, v) => a + v.dollars, 0);
const hours = Object.values(lines).reduce((a, v) => a + v[STORAGE_UNIT], 0);
if (Object.keys(lines).length) {
console.log(`storage cost in the window: $${dollars.toFixed(2)} over `
+ `${hours.toFixed(1)} ${STORAGE_UNIT}`);
} else {
console.log(`no cost result carried quantity_unit '${STORAGE_UNIT}' in the `
+ 'window, so nothing is being billed for storage yet');
}
const now = Math.floor(Date.now() / 1000);
const idle = idleStores(stores, perStore, now, Math.trunc(minGib * GIB));
let findings = 0;
for (const project of Object.keys(byProject).sort()) {
const [state, detail] = verdict(byProject[project], queries[project] ?? [],
days, minGib, minGrowth);
console.log(`${state.padEnd(27)} ${project}: ${detail}`);
for (const line of repairLines(state, FINDINGS.has(state) ? idle : [])) {
console.log(` repair: ${line}`);
}
if (FINDINGS.has(state)) findings += 1;
}
if (perStore[UNGROUPED]) {
console.log(`${num(perStore[UNGROUPED])} file search call(s) came back with no `
+ 'vector_store_id and are not attributed to a store');
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the note: bytes tripling across ninety days with not one file search call has to be a finding, and the repair has to name the store rather than the project, which only works because the snapshot was joined in. The second is the case the finding must not swallow — identical byte growth alongside rising query volume, which is a corpus doing its job and is graded as such. Then the size floor, which keeps a project holding a hundred megabytes out of the report however fast it grew; the cost selection, which has to pick storage by quantity_unit and ignore a token line whatever it is called; the null vector_store_id, which must not become a store; and the slope, checked on a flat series and on a single point.
from openai_vector_store_storage_trend import (GIB, UNGROUPED, byte_series,
growth, idle_stores,
query_series, repair_lines,
searches_by_store, slope,
storage_lines, verdict,
window_start)
DAY = 86400
T0 = 1_800_000_000
def series(first, last, points=90, key="usage_bytes", project="proj_research"):
"""A straight line from first to last, as usage buckets."""
out = []
for i in range(points):
value = first + (last - first) * i // max(points - 1, 1)
out.append({"start_time": T0 + i * DAY,
"results": [{"object": "organization.usage.vector_stores.result",
key: value, "project_id": project}]})
return out
def test_bytes_tripling_with_no_queries_at_all_is_the_finding():
# The note. Nothing is wrong with the index; the money is being spent on
# holding it rather than on using it.
points = byte_series(series(int(8.1 * GIB), int(31.4 * GIB)))["proj_research"]
state, detail = verdict(points, [], 90)
assert state == "bytes-growing-never-queried"
assert "8.1 GiB -> 31.4 GiB" in detail and "+288%" in detail
idle = idle_stores(
[{"id": "vs_c3", "name": "march-demo", "usage_bytes": int(12.4 * GIB),
"last_active_at": T0 - 148 * DAY}],
{"vs_c3": 0}, T0)
lines = repair_lines(state, idle)
assert any("march-demo" in line and "12.4 GiB" in line for line in lines)
assert any("expiration policy at creation" in line for line in lines)
def test_the_same_growth_with_rising_queries_is_not_a_finding():
# The reading this note must not trample. A corpus that is growing because
# it is being used more is supposed to cost more.
points = byte_series(series(int(44 * GIB), int(61 * GIB)))["proj_research"]
queries = query_series(series(400, 14_000, key="num_requests"))["proj_research"]
state, detail = verdict(points, queries, 90)
assert state == "bytes-and-queries-growing"
assert "Growth, priced correctly" in detail
assert repair_lines(state)[0].startswith("nothing to do")
def test_the_size_floor_comes_before_the_growth_rate():
tiny = byte_series(series(int(0.02 * GIB), int(0.12 * GIB)))["proj_research"]
state, detail = verdict(tiny, [], 90)
assert state == "below-threshold"
assert "0.1 GiB" in detail
assert repair_lines(state) == []
def test_storage_is_selected_by_unit_and_never_by_name():
buckets = [{"results": [
{"line_item": "Vector store storage", "quantity_unit": "gibibyte_hours",
"quantity": 41_288.0, "amount": {"value": 412.88, "currency": "usd"}},
{"line_item": "gpt-5, input", "quantity_unit": "tokens",
"quantity": 9_000_000, "amount": {"value": 18_402.11, "currency": "usd"}},
{"line_item": "Storage, renamed next quarter",
"quantity_unit": "gibibyte_hours", "quantity": 10.0,
"amount": {"value": 0.1, "currency": "usd"}}]}]
lines = storage_lines(buckets)
assert set(lines) == {"Vector store storage", "Storage, renamed next quarter"}
assert round(sum(v["dollars"] for v in lines.values()), 2) == 412.98
assert storage_lines([]) == {}
def test_an_unattributed_row_never_becomes_a_store():
buckets = [{"results": [
{"num_requests": 12, "vector_store_id": "vs_a1", "project_id": "proj_a"},
{"num_requests": 3, "vector_store_id": None, "project_id": None}]}]
per_store = searches_by_store(buckets)
assert per_store == {"vs_a1": 12, UNGROUPED: 3}
assert byte_series([{"start_time": T0, "results": [
{"usage_bytes": 5, "project_id": None}]}]) == {UNGROUPED: [(T0, 5)]}
def test_idle_stores_need_real_bytes_and_zero_searches():
stores = [
{"id": "vs_big", "name": "corpus", "usage_bytes": int(9 * GIB),
"last_active_at": T0 - 96 * DAY},
{"id": "vs_busy", "name": "live", "usage_bytes": int(9 * GIB),
"last_active_at": T0},
{"id": "vs_small", "name": "scratch", "usage_bytes": 40 * 1024 * 1024,
"last_active_at": T0 - 400 * DAY},
{"id": "vs_never", "name": "no-timestamp", "usage_bytes": int(2 * GIB),
"last_active_at": None}]
rows = idle_stores(stores, {"vs_busy": 900}, T0)
assert [r[0] for r in rows] == ["vs_big", "vs_never"]
assert rows[0][3] == 96
assert rows[1][3] == -1
assert idle_stores(None, None, T0) == []
def test_the_slope_is_zero_on_a_flat_series_and_on_one_point():
flat = [(T0 + i * DAY, 1000) for i in range(30)]
assert slope(flat) == 0.0
assert slope([(T0, 5)]) == 0.0
assert slope([]) == 0.0
rising = [(T0 + i * DAY, 100 * i) for i in range(10)]
assert round(slope(rising), 3) == 100.0
assert growth([]) == (0, 0, 0, 0.0)
assert growth([(T0, 0), (T0 + DAY, 50)])[3] == 0.0
def test_the_window_starts_at_midnight_utc():
import datetime as dt
now = dt.datetime(2026, 8, 31, 17, 45, 12, tzinfo=dt.timezone.utc)
assert window_start(90, now) == int(
dt.datetime(2026, 6, 2, tzinfo=dt.timezone.utc).timestamp())
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { UNGROUPED, byteSeries, growth, idleStores, querySeries, repairLines,
searchesByStore, slope, storageLines, verdict, windowStart }
from './openai-vector-store-storage-trend.mjs';
const GIB = 1073741824;
const DAY = 86400;
const T0 = 1800000000;
const series = (first, last, points = 90, key = 'usage_bytes',
project = 'proj_research') => {
const out = [];
for (let i = 0; i < points; i += 1) {
const value = first + Math.trunc((last - first) * i / Math.max(points - 1, 1));
out.push({ start_time: T0 + i * DAY,
results: [{ [key]: value, project_id: project }] });
}
return out;
};
test('bytes tripling with no queries at all is the finding', () => {
const points = byteSeries(series(Math.trunc(8.1 * GIB),
Math.trunc(31.4 * GIB))).proj_research;
const [state, detail] = verdict(points, [], 90);
assert.equal(state, 'bytes-growing-never-queried');
assert.match(detail, /8\.1 GiB -> 31\.4 GiB/);
assert.match(detail, /\+288%/);
const idle = idleStores(
[{ id: 'vs_c3', name: 'march-demo', usage_bytes: Math.trunc(12.4 * GIB),
last_active_at: T0 - 148 * DAY }], { vs_c3: 0 }, T0);
const lines = repairLines(state, idle);
assert.ok(lines.some((l) => l.includes('march-demo') && l.includes('12.4 GiB')));
assert.ok(lines.some((l) => l.includes('expiration policy at creation')));
});
test('the same growth with rising queries is not a finding', () => {
const points = byteSeries(series(44 * GIB, 61 * GIB)).proj_research;
const queries = querySeries(series(400, 14000, 90, 'num_requests')).proj_research;
const [state, detail] = verdict(points, queries, 90);
assert.equal(state, 'bytes-and-queries-growing');
assert.match(detail, /Growth, priced correctly/);
assert.ok(repairLines(state)[0].startsWith('nothing to do'));
});
test('the size floor comes before the growth rate', () => {
const tiny = byteSeries(series(Math.trunc(0.02 * GIB),
Math.trunc(0.12 * GIB))).proj_research;
const [state, detail] = verdict(tiny, [], 90);
assert.equal(state, 'below-threshold');
assert.match(detail, /0\.1 GiB/);
assert.deepEqual(repairLines(state), []);
});
test('storage is selected by unit and never by name', () => {
const buckets = [{ results: [
{ line_item: 'Vector store storage', quantity_unit: 'gibibyte_hours',
quantity: 41288, amount: { value: 412.88, currency: 'usd' } },
{ line_item: 'gpt-5, input', quantity_unit: 'tokens', quantity: 9000000,
amount: { value: 18402.11, currency: 'usd' } },
{ line_item: 'Storage, renamed next quarter', quantity_unit: 'gibibyte_hours',
quantity: 10, amount: { value: 0.1, currency: 'usd' } }] }];
const lines = storageLines(buckets);
assert.deepEqual(Object.keys(lines).sort(),
['Storage, renamed next quarter', 'Vector store storage']);
const total = Object.values(lines).reduce((a, v) => a + v.dollars, 0);
assert.equal(Math.round(total * 100) / 100, 412.98);
assert.deepEqual(storageLines([]), {});
});
test('an unattributed row never becomes a store', () => {
const buckets = [{ results: [
{ num_requests: 12, vector_store_id: 'vs_a1', project_id: 'proj_a' },
{ num_requests: 3, vector_store_id: null, project_id: null }] }];
assert.deepEqual(searchesByStore(buckets), { vs_a1: 12, [UNGROUPED]: 3 });
assert.deepEqual(byteSeries([{ start_time: T0,
results: [{ usage_bytes: 5, project_id: null }] }]),
{ [UNGROUPED]: [[T0, 5]] });
});
test('idle stores need real bytes and zero searches', () => {
const stores = [
{ id: 'vs_big', name: 'corpus', usage_bytes: 9 * GIB,
last_active_at: T0 - 96 * DAY },
{ id: 'vs_busy', name: 'live', usage_bytes: 9 * GIB, last_active_at: T0 },
{ id: 'vs_small', name: 'scratch', usage_bytes: 40 * 1024 * 1024,
last_active_at: T0 - 400 * DAY },
{ id: 'vs_never', name: 'no-timestamp', usage_bytes: 2 * GIB,
last_active_at: null }];
const rows = idleStores(stores, { vs_busy: 900 }, T0);
assert.deepEqual(rows.map((r) => r[0]), ['vs_big', 'vs_never']);
assert.equal(rows[0][3], 96);
assert.equal(rows[1][3], -1);
assert.deepEqual(idleStores(null, null, T0), []);
});
test('the slope is zero on a flat series and on one point', () => {
const flat = [];
for (let i = 0; i < 30; i += 1) flat.push([T0 + i * DAY, 1000]);
assert.equal(slope(flat), 0);
assert.equal(slope([[T0, 5]]), 0);
assert.equal(slope([]), 0);
const rising = [];
for (let i = 0; i < 10; i += 1) rising.push([T0 + i * DAY, 100 * i]);
assert.equal(Math.round(slope(rising) * 1000) / 1000, 100);
assert.deepEqual(growth([]), [0, 0, 0, 0]);
assert.equal(growth([[T0, 0], [T0 + DAY, 50]])[3], 0);
});
test('the window starts at midnight utc', () => {
assert.equal(windowStart(90, new Date('2026-08-31T17:45:12Z')),
Date.UTC(2026, 5, 2) / 1000);
});
FAQ
Can I get a byte trend for one vector store rather than a whole project?
Not from the usage API. The vector stores usage endpoint accepts exactly one grouping, project_id, so there is no per-store byte series and asking harder will not produce one. What you can group per store is the query volume: the file search calls endpoint accepts vector_store_id and returns num_requests against it. So the trend is a project-level reading and the culprit is identified by joining those per-store query counts against the current snapshot from GET /v1/vector_stores, which needs a project key. The script says out loud when it ran without one, rather than reporting an empty store list as though there were no idle stores.
Why match on quantity_unit instead of the line item name?
Because names get relabelled and units do not. quantity_unit is an enumerated field on the cost result and gibibyte_hours appears on the storage lines specifically. A check written against a display string keeps working right up until somebody in the billing team renames a product, at which point it starts reporting zero dollars of storage cost, which looks exactly like good news. Matching the unit also states the billing model in the output: gibibyte-hours is bytes multiplied by time, which is the whole reason this line behaves differently from every other one.
How is this different from reconciling the line items on the bill?
Different question and different shape. The line-item note asks whether your cost dashboard renders the whole invoice, and it answers by subtracting what you cover from what you were charged in one window. This one takes a single unit and asks whether it is trending, over ninety days, against the thing that is supposed to justify it. A storage line can be fully covered by a dashboard, perfectly reconciled, and still be the wrong number, because nothing about the reconciliation asks whether anybody searched the bytes.
The bytes are growing. Is that not just what a growing product does?
Frequently, and the script grades that separately for exactly that reason. Bytes climbing alongside a climbing query count is a corpus being used more, the storage line is supposed to follow it, and reporting it as waste is how a cost report gets ignored. The finding is growth without use: bytes rising while file search calls are flat, falling, or zero. Two of the five states this script emits exist purely to keep normal growth out of the finding column.
Why ninety days, and why does the script page the usage endpoints?
Ninety days because a slope needs enough points to be a slope rather than a coincidence, and a monthly billing cycle means anything shorter than two of them cannot show you a trend that survives a month boundary. The paging is not optional: the usage endpoints cap limit at 31 buckets when bucket_width is 1d, so a request for ninety days returns thirty-one of them and a next_page cursor. A script that reads the first response and stops computes a real slope over a third of the window and reports it as though it covered all of it.
Related field notes
- The index that deletes itself on a schedule nobody diaried
- The line items a token dashboard never renders at all
- The per-token reading, which storage is not
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Usage and costs — OpenAI API reference
- Vector stores — OpenAI API reference
- openai-openapi — the published OpenAPI specification
- Pricing — OpenAI platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.