Diagnostic LLM APIs
An empty vector store is still named in vector_store_ids
The retrieval feature is live and nobody has complained, which is the part that should worry you. Every request carries a file_search tool with a vector_store_ids array copied out of the config months ago; every response comes back 200 with no citations attached; and the model, asked what the refund window is, says thirty days in a confident and well-structured paragraph. It is thirty days in the training data. Your policy changed to fourteen in March, it is in the document you indexed, and the store that was supposed to hold that document has nothing in it.
One GET per configured id with a project key, and the configured ids are the input: GET /v1/vector_stores/{vector_store_id} for each id your application actually passes in vector_store_ids, plus GET /v1/vector_stores?limit=100 for the wider picture. Flag file_counts.total == 0, file_counts.completed == 0, or usage_bytes == 0.
Emptiness on its own is not a fault. A store nobody references is litter: it grounds nothing because nothing points at it, and it costs nothing because it holds no bytes. The finding only exists when an empty store is named in the tool configuration, which is why this script takes the ids from you rather than grading everything it can see.
The fourth case is an id that does not resolve at all — a 404 from a store deleted, expired out of existence, or created under a different project than the key you deployed with. That is the same failure one step further along, and file_search handles it about as loudly.
Then say why the store is empty, because that decides who fixes it. file_counts.total == 0 means nothing was ever attached and the ingest is the repair. total > 0 with completed == 0 means files were attached and none of them indexed, which is the attach-failure note and is repaired per error code. And status == "expired" means the store emptied itself on a schedule, which is the expiry note and will happen again.
The durable fix is a startup assertion. For every id in vector_store_ids, read the store back and refuse to boot when file_counts.completed == 0. A retrieval feature that cannot retrieve should fail at deploy, not in an answer.
The problem in plain words
The file search tool does not error on an empty index. It searches, finds nothing, and returns nothing, and that is a successful tool call. The model then does what a model does with a tool that returned no context: it answers from what it already knows. The output is fluent, plausible and ungrounded, and it looks exactly like the output you were hoping for, minus the citations that nobody is checking.
Stores get into this state by several ordinary routes and they all look the same afterwards. A create-then-attach sequence broke between the two calls, so the store exists with an id worth copying and nothing in it. An ingest ran against the wrong project. Every attach failed. Or the store hit its expiration policy, and expiry deletes the contained file objects, which turns a working index into an empty one without touching the id.
What keeps it alive is that the id is real. Everything downstream validates: the config parses, the tool schema is well formed, the API accepts the array, the request succeeds. Nothing in the chain has an opinion about whether the id points at anything, and the one component that could tell you — the store object, with five integers on it — is not on the request path at all.
And an id can stop resolving entirely. Stores are deletable, expiry eventually removes them, and a project key deployed to the wrong project will not see stores that exist perfectly well elsewhere. All three arrive as an id that does not answer, which your application discovers at request time if it discovers it at all.
Why it happens
The configured ids are the input, and that is what makes this a different note from the rest of the batch. Every other script here reads the platform and grades what it finds. This one starts from what your application claims and checks it against the platform, which is the only order that can produce this finding: an empty store is a completely ordinary object right up until something names it. Pass the ids you deploy with, from the same source your application reads them from, or the script is grading a list you made up.
An empty store that nobody references is not a finding and must not be reported as one. It holds no bytes so it bills nothing, it grounds nothing because no request mentions it, and every prototype leaves a few behind. Reporting them at the same severity as a referenced one trains people to skim the output, and the one line that mattered goes past with the nine that did not.
Three fields say three different things about why the store is empty, and only one of them is repaired here. total == 0 is "the ingest never ran", which this note owns. total > 0, completed == 0 is "the ingest ran and everything failed", which belongs to the attach-failure note and is fixed per error code. status == "expired" is "the store deleted its own contents", which belongs to the expiry note and will recur on the same schedule unless the policy changes. The script prints the cause with the finding, because otherwise all three land on the same engineer with the same useless instruction to look at the store.
usage_bytes == 0 alongside completed files is an anomaly rather than a synonym. The three emptiness tests are usually redundant and occasionally are not, and the case where they disagree — completed files reported with no bytes retained — is worth its own state rather than being folded into the same word. It is graded, named, and left to a human, because guessing what it means would be worse than saying it is odd.
An unresolvable id is a project problem far more often than a deletion. Vector stores are project-scoped, so a key issued in the wrong project sees a clean 404 for a store that is alive and well next door. The repair line says so first, because "your store was deleted" sends somebody to re-ingest a corpus that already exists.
The fix, as a flow
This one is only visible from the application side, because an empty store is a perfectly ordinary object until you know that something still names it. So the script takes the ids your code configures as input and reads the platform for the answer, which is the reverse of every other note in the batch. An id that does not resolve at all is the same failure one step further along.
How to fix it
Get the ids from the same place your application gets them
VECTOR_STORE_IDS as a comma-separated list, or repeated --store-id. Read them out of the deployed configuration rather than from memory: an audit of the ids you believe are configured proves nothing about the ids that ship.
Read each configured store back with a project key
GET /v1/vector_stores/{vector_store_id}. A 404 is a finding rather than an error — usually the wrong project for the key, sometimes a store that expired or was deleted. Everything else returns the object with file_counts, usage_bytes and status.
Apply the three emptiness tests in order
file_counts.total == 0 first, then file_counts.completed == 0, then usage_bytes == 0. The order is the point: the first says nothing was attached, the second says the attach failed, and running them the other way around reports every failed ingest as an empty store.
Name the cause from status and the counts
status == "expired" means the store deleted its own files. A non-zero failed count means the attaches failed. A non-zero in_progress count means it is still working and you are early. Anything else means the ingest never ran at all.
Sweep the rest of the listing, and print the assertion
GET /v1/vector_stores?limit=100, paged on after, for the empty stores nothing references. Report them quietly as litter. The repair for the real finding is a startup assertion over vector_store_ids that refuses to boot when file_counts.completed == 0.
How to check it worked
Re-run after the ingest, with the same id list. Every configured store should read grounded, and the abandoned ones should still be listed and still not be findings. The most useful re-run is the one you do from a deploy environment rather than a laptop, because a key from the wrong project turns every configured id into referenced-missing and that is exactly the failure this catches.
VECTOR_STORE_IDS=vs_a1,vs_b2,vs_c3,vs_d4 \
python3 openai_empty_vector_store_audit.py
# 4 configured id(s), 9 store(s) visible to this key
# referenced-empty vs_a1 handbook: 0 file(s) attached, 0 bytes
# cause: the ingest never ran against this store
# repair: run the ingest, then re-read the store before shipping the id
# repair: assert file_counts.completed > 0 for every id in vector_store_ids at
# startup and refuse to boot. A retrieval feature that cannot retrieve
# should fail at deploy, not in an answer.
# referenced-nothing-indexed vs_b2 policies: 40 attached, 0 completed, 40 failed
# cause: files were attached and none of them indexed
# repair: this is the attach failure note. Bucket the children by
# last_error.code and repair per bucket, not per store.
# referenced-missing vs_c3: no such store for this key
# repair: check the project first. Vector stores are project scoped, so a key
# from the wrong project 404s on a store that is alive next door.
# grounded vs_d4 pricing: 812 file(s) completed, 41.2 MiB
# abandoned-empty 2 empty store(s) nothing references, which is litter
# 3 finding(s)The full code
One GET per configured id, one paged listing, and five pure functions. configured_ids, which splits on commas or whitespace and de-duplicates while keeping order, so a trailing comma in an environment variable cannot become an empty id that 404s; counts, the same coercion of the five integers as its sibling note; emptiness, which runs the three tests in the order that keeps them meaning different things; cause, which reads status and the counts to say which note owns the repair; and classify, which grades a store only against whether something references it.
"""Check that the vector store ids your application configures index anything.
Read only. One GET per configured id against /v1/vector_stores/{id}, plus a
paged GET of /v1/vector_stores for the wider picture. No request body is
constructed and no file_search query is ever run, because a retrieval query is
a generation and this script exists to say whether the index is empty, not to
find out what it would answer.
The configured ids are the input, and that is the whole design. An empty vector
store is an ordinary object; it only becomes a fault when something still names
it in vector_store_ids. So this reads your configuration first and the platform
second, which is the reverse of every other note in this batch.
"""
import argparse
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_empty_vector_store_audit")
API = "https://api.openai.com/v1"
# The official client still sends this on every vector store call, so this
# script does too. It is a GET either way.
BETA = {"OpenAI-Beta": "assistants=v2"}
FINDINGS = ("referenced-empty", "referenced-nothing-indexed",
"referenced-zero-bytes", "referenced-missing")
CAUSES = {
"expired": "the store passed its expiration policy and deleted its own "
"files. That is the expiry note, and it will happen again on "
"the same schedule.",
"attach-failed": "files were attached and none of them indexed. That is "
"the attach failure note: bucket the children by "
"last_error.code and repair per bucket, not per store.",
"still-ingesting": "files are still processing. You are early rather than "
"broken; re-read once file_counts.in_progress is zero.",
"never-ingested": "the ingest never ran against this store. Nothing was "
"ever attached to it.",
}
def configured_ids(*raw):
"""The store ids the application claims to use. Pure.
Split on commas or whitespace, blanks dropped, order preserved, duplicates
removed. A trailing comma in an environment variable is the common way an
empty string becomes an id that 404s and gets reported as a missing store.
"""
out, seen = [], set()
for chunk in raw:
if not chunk:
continue
items = chunk if isinstance(chunk, (list, tuple)) else [chunk]
for item in items:
for token in re.split(r"[,\s]+", str(item or "").strip()):
token = token.strip()
if token and token not in seen:
seen.add(token)
out.append(token)
return out
def counts(store):
"""The five file_counts integers, coerced. Pure."""
raw = (store or {}).get("file_counts") or {}
out = {}
for key in ("in_progress", "completed", "failed", "cancelled", "total"):
try:
out[key] = int(raw.get(key) or 0)
except (TypeError, ValueError):
out[key] = 0
return out
def usage_bytes(store):
"""usage_bytes as an integer. Pure. Missing or unparseable reads as 0."""
try:
return int((store or {}).get("usage_bytes") or 0)
except (TypeError, ValueError):
return 0
def emptiness(store):
"""How empty one store is. Pure. One of four words, tested in order.
The order carries the meaning. total == 0 says nothing was ever attached;
completed == 0 with files present says the attach failed. Running the tests
the other way round reports every failed ingest as an empty store and sends
the repair to the wrong place.
"""
c = counts(store)
if c["total"] <= 0:
return "no-files"
if c["completed"] <= 0:
return "nothing-completed"
if usage_bytes(store) <= 0:
return "zero-bytes"
return "indexed"
def cause(store):
"""Why the store is empty, as far as the object can say. Pure.
Returns a key into CAUSES. status is read first because expiry is the one
cause that recurs: an expired store deletes its contained files, so the
counts afterwards look exactly like an ingest that never ran.
"""
if str((store or {}).get("status") or "").strip().lower() == "expired":
return "expired"
c = counts(store)
if c["failed"] > 0:
return "attach-failed"
if c["in_progress"] > 0:
return "still-ingesting"
return "never-ingested"
def classify(store, referenced):
"""Grade one store. Pure. Returns (state, detail).
A store is only graded against whether something references it. Emptiness
on its own bills nothing and grounds nothing, and reporting it at finding
severity is how a report teaches people to skim it.
"""
if store is None:
if referenced:
return ("referenced-missing",
"no such store for this key. Vector stores are project "
"scoped, so the usual cause is a key from the wrong "
"project rather than a deleted store.")
return ("not-found", "no such store")
c = counts(store)
kind = emptiness(store)
size = usage_bytes(store)
if not referenced:
if kind == "indexed":
return ("unreferenced",
"%d file(s) completed, and nothing you passed names it"
% c["completed"])
return ("abandoned-empty",
"empty and unreferenced, which is litter rather than an outage")
if kind == "no-files":
return ("referenced-empty",
"0 file(s) attached, 0 bytes")
if kind == "nothing-completed":
return ("referenced-nothing-indexed",
"%d attached, 0 completed, %d failed, %d in progress"
% (c["total"], c["failed"], c["in_progress"]))
if kind == "zero-bytes":
return ("referenced-zero-bytes",
"%d file(s) report completed and usage_bytes is 0, which the "
"three emptiness tests disagree about. Read it before acting."
% c["completed"])
return ("grounded",
"%d file(s) completed, %.1f MiB" % (c["completed"], size / 1048576.0))
def repair_lines(state, why=None):
"""The repair for one verdict. Pure. Printed, never performed."""
assertion = ("assert file_counts.completed > 0 for every id in "
"vector_store_ids at startup and refuse to boot. A retrieval "
"feature that cannot retrieve should fail at deploy, not in "
"an answer.")
if state == "referenced-empty":
lines = []
if why == "expired":
lines.append(CAUSES["expired"])
lines.append("run the ingest, then re-read the store before shipping "
"the id.")
lines.append(assertion)
return lines
if state == "referenced-nothing-indexed":
return [CAUSES.get(why or "attach-failed", CAUSES["attach-failed"]),
assertion]
if state == "referenced-zero-bytes":
return ["do not delete this one on the strength of a byte count. Read "
"the store and one of its files before deciding what it is.",
assertion]
if state == "referenced-missing":
return ["check the project first. A project key cannot see a store "
"that lives in another project, and that 404 is identical to "
"the one a deleted store returns.",
"if the store really is gone, re-ingest and update the "
"configured id in the same change.",
assertion]
if state == "abandoned-empty":
return ["nothing references it and it holds no bytes, so it is not "
"costing you anything. Delete it when convenient with "
"DELETE /v1/vector_stores/{vector_store_id}."]
return []
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/vector_stores needs a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def get_optional(session, path):
"""One store, or None when it does not resolve for this key."""
r = session.get(API + path, timeout=90)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/vector_stores needs a project key"
% r.status_code)
r.raise_for_status()
return r.json()
def paged(session, path, max_pages=200, **params):
"""Walk an after/last_id cursor listing."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, **params)
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params["after"] = page.get("last_id") or (data[-1] or {}).get("id")
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--store-id", action="append", default=[],
help="a store id your application configures (repeatable)")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key for the project that "
"owns the vector stores")
return 2
wanted = configured_ids(os.environ.get("VECTOR_STORE_IDS"), args.store_id)
if not wanted:
log.error("pass the store ids your application configures, as "
"VECTOR_STORE_IDS or repeated --store-id. Without them this "
"script has nothing to grade: an empty store is only a "
"finding when something still names it.")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + key, **BETA})
stores = list(paged(s, "/vector_stores", limit=100))
by_id = {(st or {}).get("id"): st for st in stores}
log.info("%d configured id(s), %d store(s) visible to this key",
len(wanted), len(stores))
findings = 0
for sid in wanted:
store = by_id.get(sid)
if store is None:
store = get_optional(s, "/vector_stores/%s" % sid)
state, detail = classify(store, referenced=True)
why = cause(store) if store is not None else None
name = (store or {}).get("name") or ""
emit = log.warning if state in FINDINGS else log.info
emit("%-26s %s %s: %s", state, sid, name, detail)
if state in FINDINGS and store is not None:
emit(" cause: %s", CAUSES[why])
for line in repair_lines(state, why):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
litter = [st for st in stores
if (st or {}).get("id") not in set(wanted)
and emptiness(st) != "indexed"]
if litter:
log.info("%-26s %d empty store(s) nothing references, which is litter",
"abandoned-empty", len(litter))
for line in repair_lines("abandoned-empty"):
log.info(" note: %s", line)
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Check that the vector store ids your application configures index anything.
*
* Read only. One GET per configured id plus a paged listing. No request body,
* and no file_search query is ever run: a retrieval query is a generation, and
* the question here is whether the index is empty rather than what it answers.
*
* The configured ids are the input. An empty vector store is an ordinary
* object; it becomes a fault only when something still names it.
*/
const API = 'https://api.openai.com/v1';
const BETA = { 'OpenAI-Beta': 'assistants=v2' };
const FINDINGS = new Set(['referenced-empty', 'referenced-nothing-indexed',
'referenced-zero-bytes', 'referenced-missing']);
export const CAUSES = {
expired:
'the store passed its expiration policy and deleted its own files. That is '
+ 'the expiry note, and it will happen again on the same schedule.',
'attach-failed':
'files were attached and none of them indexed. That is the attach failure '
+ 'note: bucket the children by last_error.code and repair per bucket, not '
+ 'per store.',
'still-ingesting':
'files are still processing. You are early rather than broken; re-read once '
+ 'file_counts.in_progress is zero.',
'never-ingested':
'the ingest never ran against this store. Nothing was ever attached to it.',
};
/** The store ids the application claims to use. Pure. Order kept, dupes dropped. */
export function configuredIds(...raw) {
const out = [];
const seen = new Set();
for (const chunk of raw) {
if (!chunk) continue;
const items = Array.isArray(chunk) ? chunk : [chunk];
for (const item of items) {
for (const token of String(item ?? '').trim().split(/[,\s]+/)) {
if (token && !seen.has(token)) { seen.add(token); out.push(token); }
}
}
}
return out;
}
/** The five file_counts integers, coerced. Pure. */
export function counts(store) {
const raw = store?.file_counts ?? {};
const out = {};
for (const key of ['in_progress', 'completed', 'failed', 'cancelled', 'total']) {
const n = Number(raw[key] ?? 0);
out[key] = Number.isFinite(n) ? Math.trunc(n) : 0;
}
return out;
}
/** usage_bytes as an integer. Pure. Missing or unparseable reads as 0. */
export function usageBytes(store) {
const n = Number(store?.usage_bytes ?? 0);
return Number.isFinite(n) ? Math.trunc(n) : 0;
}
/** How empty one store is. Pure. Four words, tested in a load-bearing order. */
export function emptiness(store) {
const c = counts(store);
if (c.total <= 0) return 'no-files';
if (c.completed <= 0) return 'nothing-completed';
if (usageBytes(store) <= 0) return 'zero-bytes';
return 'indexed';
}
/** Why the store is empty, as far as the object can say. Pure. */
export function cause(store) {
if (String(store?.status ?? '').trim().toLowerCase() === 'expired') return 'expired';
const c = counts(store);
if (c.failed > 0) return 'attach-failed';
if (c.in_progress > 0) return 'still-ingesting';
return 'never-ingested';
}
/** Grade one store. Pure. Returns [state, detail]. */
export function classify(store, referenced) {
if (store === null || store === undefined) {
if (referenced) {
return ['referenced-missing',
'no such store for this key. Vector stores are project scoped, so '
+ 'the usual cause is a key from the wrong project rather than a '
+ 'deleted store.'];
}
return ['not-found', 'no such store'];
}
const c = counts(store);
const kind = emptiness(store);
const size = usageBytes(store);
if (!referenced) {
if (kind === 'indexed') {
return ['unreferenced',
`${c.completed} file(s) completed, and nothing you passed names it`];
}
return ['abandoned-empty',
'empty and unreferenced, which is litter rather than an outage'];
}
if (kind === 'no-files') return ['referenced-empty', '0 file(s) attached, 0 bytes'];
if (kind === 'nothing-completed') {
return ['referenced-nothing-indexed',
`${c.total} attached, 0 completed, ${c.failed} failed, `
+ `${c.in_progress} in progress`];
}
if (kind === 'zero-bytes') {
return ['referenced-zero-bytes',
`${c.completed} file(s) report completed and usage_bytes is 0, which `
+ 'the three emptiness tests disagree about. Read it before acting.'];
}
return ['grounded',
`${c.completed} file(s) completed, ${(size / 1048576).toFixed(1)} MiB`];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, why = null) {
const assertion = 'assert file_counts.completed > 0 for every id in '
+ 'vector_store_ids at startup and refuse to boot. A retrieval feature that '
+ 'cannot retrieve should fail at deploy, not in an answer.';
if (state === 'referenced-empty') {
const lines = [];
if (why === 'expired') lines.push(CAUSES.expired);
lines.push('run the ingest, then re-read the store before shipping the id.');
lines.push(assertion);
return lines;
}
if (state === 'referenced-nothing-indexed') {
return [CAUSES[why] ?? CAUSES['attach-failed'], assertion];
}
if (state === 'referenced-zero-bytes') {
return ['do not delete this one on the strength of a byte count. Read the '
+ 'store and one of its files before deciding what it is.', assertion];
}
if (state === 'referenced-missing') {
return ['check the project first. A project key cannot see a store that '
+ 'lives in another project, and that 404 is identical to the one a '
+ 'deleted store returns.',
'if the store really is gone, re-ingest and update the configured id '
+ 'in the same change.',
assertion];
}
if (state === 'abandoned-empty') {
return ['nothing references it and it holds no bytes, so it is not costing '
+ 'you anything. Delete it when convenient with '
+ 'DELETE /v1/vector_stores/{vector_store_id}.'];
}
return [];
}
async function read(key, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}`, ...BETA } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/vector_stores needs a project key`);
}
if (r.status === 404) return null;
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function* paged(key, path, params, maxPages = 200) {
const q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, path, q);
const data = page?.data ?? [];
for (const item of data) yield item;
if (!page?.has_more || data.length === 0) return;
q.after = page.last_id ?? data[data.length - 1]?.id;
}
}
async function main() {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key for the project that owns '
+ 'the vector stores');
process.exitCode = 2;
return;
}
const wanted = configuredIds(process.env.VECTOR_STORE_IDS);
if (!wanted.length) {
console.error('pass the store ids your application configures as '
+ 'VECTOR_STORE_IDS. Without them this script has nothing to '
+ 'grade: an empty store is only a finding when something '
+ 'still names it.');
process.exitCode = 2;
return;
}
const stores = [];
for await (const st of paged(key, '/vector_stores', { limit: 100 })) stores.push(st);
const byId = new Map(stores.map((st) => [st?.id, st]));
console.log(`${wanted.length} configured id(s), ${stores.length} store(s) `
+ 'visible to this key');
let findings = 0;
for (const sid of wanted) {
const store = byId.get(sid) ?? await read(key, `/vector_stores/${sid}`);
const [state, detail] = classify(store, true);
const why = store ? cause(store) : null;
console.log(`${state.padEnd(26)} ${sid} ${store?.name ?? ''}: ${detail}`);
if (FINDINGS.has(state) && store) console.log(` cause: ${CAUSES[why]}`);
for (const line of repairLines(state, why)) console.log(` repair: ${line}`);
if (FINDINGS.has(state)) findings += 1;
}
const configured = new Set(wanted);
const litter = stores.filter((st) => !configured.has(st?.id)
&& emptiness(st) !== 'indexed');
if (litter.length) {
console.log(`${'abandoned-empty'.padEnd(26)} ${litter.length} empty store(s) `
+ 'nothing references, which is litter');
for (const line of repairLines('abandoned-empty')) console.log(` note: ${line}`);
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first two tests are the boundary this note shares with its neighbour, written as assertions rather than as prose: a store with total == 0 is referenced-empty and its cause is that the ingest never ran, while a store with forty files attached and none completed is referenced-nothing-indexed and its repair line names the attach-failure note. The third is the same empty store whose status is expired, which has to attribute the emptiness to the schedule rather than to a missing ingest. Then the unreferenced empty store, which must not be a finding; the unresolvable id, whose repair has to mention the project before it mentions deletion; and configured_ids against the trailing comma that turns an environment variable into an empty id.
from openai_empty_vector_store_audit import (cause, classify, configured_ids,
counts, emptiness, repair_lines,
usage_bytes)
def store(total=0, completed=0, failed=0, in_progress=0, bytes_=0,
status="completed", sid="vs_a1", name="handbook"):
return {"id": sid, "name": name, "status": status, "usage_bytes": bytes_,
"file_counts": {"total": total, "completed": completed,
"failed": failed, "in_progress": in_progress,
"cancelled": 0}}
def test_a_configured_store_with_nothing_in_it_is_the_finding():
empty = store(total=0)
assert emptiness(empty) == "no-files"
state, detail = classify(empty, referenced=True)
assert state == "referenced-empty"
assert "0 file(s) attached" in detail
assert cause(empty) == "never-ingested"
assert any("refuse to boot" in line
for line in repair_lines(state, cause(empty)))
def test_attached_but_never_indexed_is_the_other_note():
# The boundary. Forty files went in and none came out, which is an attach
# failure wearing an empty store's symptoms, and it is repaired per
# last_error.code rather than by re-running the ingest.
broken = store(total=40, completed=0, failed=40)
assert emptiness(broken) == "nothing-completed"
state, detail = classify(broken, referenced=True)
assert state == "referenced-nothing-indexed"
assert "40 attached, 0 completed" in detail
assert cause(broken) == "attach-failed"
assert any("last_error.code" in line
for line in repair_lines(state, cause(broken)))
def test_an_expired_store_is_empty_for_a_reason_that_will_recur():
gone = store(total=0, status="expired")
assert cause(gone) == "expired"
lines = repair_lines("referenced-empty", cause(gone))
assert any("same schedule" in line for line in lines)
# And the counts alone cannot tell you: they are identical either way.
assert counts(gone) == counts(store(total=0))
def test_an_empty_store_nobody_references_is_not_a_finding():
state, detail = classify(store(total=0), referenced=False)
assert state == "abandoned-empty"
assert "litter" in detail
assert classify(store(total=9, completed=9, bytes_=1024),
referenced=False)[0] == "unreferenced"
def test_an_id_that_does_not_resolve_blames_the_project_first():
state, detail = classify(None, referenced=True)
assert state == "referenced-missing"
assert "project scoped" in detail
lines = repair_lines(state)
assert "project" in lines[0]
assert classify(None, referenced=False)[0] == "not-found"
def test_completed_files_with_no_bytes_is_named_rather_than_guessed():
odd = store(total=9, completed=9, bytes_=0)
assert emptiness(odd) == "zero-bytes"
state, detail = classify(odd, referenced=True)
assert state == "referenced-zero-bytes"
assert "disagree" in detail
assert any("before deciding" in line for line in repair_lines(state))
def test_configured_ids_survives_the_trailing_comma():
assert configured_ids("vs_a1,vs_b2,") == ["vs_a1", "vs_b2"]
assert configured_ids("vs_a1 vs_b2\nvs_a1") == ["vs_a1", "vs_b2"]
assert configured_ids(None, ["vs_c3"], "vs_c3") == ["vs_c3"]
assert configured_ids("") == [] and configured_ids() == []
def test_a_grounded_store_reports_its_size():
good = store(total=812, completed=812, bytes_=43_200_512)
state, detail = classify(good, referenced=True)
assert state == "grounded"
assert "41.2 MiB" in detail
assert repair_lines(state) == []
assert usage_bytes({"usage_bytes": "nope"}) == 0
assert emptiness(None) == "no-files"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cause, classify, configuredIds, counts, emptiness, repairLines,
usageBytes } from './openai-empty-vector-store-audit.mjs';
const store = ({ total = 0, completed = 0, failed = 0, in_progress = 0,
bytes = 0, status = 'completed', id = 'vs_a1',
name = 'handbook' } = {}) =>
({ id, name, status, usage_bytes: bytes,
file_counts: { total, completed, failed, in_progress, cancelled: 0 } });
test('a configured store with nothing in it is the finding', () => {
const empty = store({ total: 0 });
assert.equal(emptiness(empty), 'no-files');
const [state, detail] = classify(empty, true);
assert.equal(state, 'referenced-empty');
assert.match(detail, /0 file\(s\) attached/);
assert.equal(cause(empty), 'never-ingested');
assert.ok(repairLines(state, cause(empty)).some((l) => l.includes('refuse to boot')));
});
test('attached but never indexed is the other note', () => {
const broken = store({ total: 40, completed: 0, failed: 40 });
assert.equal(emptiness(broken), 'nothing-completed');
const [state, detail] = classify(broken, true);
assert.equal(state, 'referenced-nothing-indexed');
assert.match(detail, /40 attached, 0 completed/);
assert.equal(cause(broken), 'attach-failed');
assert.ok(repairLines(state, cause(broken)).some((l) => l.includes('last_error.code')));
});
test('an expired store is empty for a reason that will recur', () => {
const gone = store({ total: 0, status: 'expired' });
assert.equal(cause(gone), 'expired');
assert.ok(repairLines('referenced-empty', cause(gone))
.some((l) => l.includes('same schedule')));
assert.deepEqual(counts(gone), counts(store({ total: 0 })));
});
test('an empty store nobody references is not a finding', () => {
const [state, detail] = classify(store({ total: 0 }), false);
assert.equal(state, 'abandoned-empty');
assert.match(detail, /litter/);
assert.equal(classify(store({ total: 9, completed: 9, bytes: 1024 }), false)[0],
'unreferenced');
});
test('an id that does not resolve blames the project first', () => {
const [state, detail] = classify(null, true);
assert.equal(state, 'referenced-missing');
assert.match(detail, /project scoped/);
assert.match(repairLines(state)[0], /project/);
assert.equal(classify(undefined, false)[0], 'not-found');
});
test('completed files with no bytes is named rather than guessed', () => {
const odd = store({ total: 9, completed: 9, bytes: 0 });
assert.equal(emptiness(odd), 'zero-bytes');
const [state, detail] = classify(odd, true);
assert.equal(state, 'referenced-zero-bytes');
assert.match(detail, /disagree/);
assert.ok(repairLines(state).some((l) => l.includes('before deciding')));
});
test('configuredIds survives the trailing comma', () => {
assert.deepEqual(configuredIds('vs_a1,vs_b2,'), ['vs_a1', 'vs_b2']);
assert.deepEqual(configuredIds('vs_a1 vs_b2\nvs_a1'), ['vs_a1', 'vs_b2']);
assert.deepEqual(configuredIds(null, ['vs_c3'], 'vs_c3'), ['vs_c3']);
assert.deepEqual(configuredIds(''), []);
assert.deepEqual(configuredIds(), []);
});
test('a grounded store reports its size', () => {
const good = store({ total: 812, completed: 812, bytes: 43200512 });
const [state, detail] = classify(good, true);
assert.equal(state, 'grounded');
assert.match(detail, /41\.2 MiB/);
assert.deepEqual(repairLines(state), []);
assert.equal(usageBytes({ usage_bytes: 'nope' }), 0);
assert.equal(emptiness(null), 'no-files');
});
FAQ
A store can be empty because every attach failed. Which note owns that?
The attach-failure note, and the field that decides is file_counts.total. Zero means nothing was ever attached to this store, so there is no per-file error to look up and the repair is to run the ingest. Greater than zero with completed at zero means the ingest did run, produced nothing, and left a last_error.code on every child, so the repair is per error code and lives in the other note. This script prints the distinction as a cause line next to the finding rather than making you go and derive it, because the two states look identical from the retrieval side and are fixed by different people.
Why does the script need me to tell it which store ids I use?
Because emptiness on its own is not a fault. An empty store nobody references holds no bytes, bills nothing and grounds nothing; every team that has prototyped retrieval has a few and they are harmless. The finding is the intersection of empty and referenced, and the platform cannot see the second half: nothing in the API records which store ids your code passes in vector_store_ids. Pass them from the deployed configuration rather than from memory, because the point is to check what ships.
The store id 404s but I can see it in the dashboard. What is going on?
Almost always the project. Vector stores are project-scoped resources, so a project key issued in one project gets a clean 404 for a store that is alive and correct in another, and that 404 is byte-identical to the one a deleted store returns. Check which project the key belongs to before you re-ingest anything. The other two causes are a store that was deleted and a store that passed its expiration policy, and the expiry note covers the second.
Should the script just run a test query to see whether retrieval works?
No. A file_search query is a generation: it costs money, it goes through a model, and every script in this section is read-only in the strict sense that it only makes GET requests. It would also answer a different question. A query tells you what the index returned for that one query; file_counts.completed tells you whether the index contains anything at all, which is the failure this note is about and the one a test query would most easily mask.
What does the startup assertion actually look like?
For each id in your vector_store_ids configuration, GET /v1/vector_stores/{id} once during boot and refuse to start when the call 404s or when file_counts.completed is zero. It is one request per store per process start, it runs against the same key and the same project the application will use in production, and it converts a silent quality regression into a deploy that does not go out. That last property is the whole value: an ungrounded answer is invisible, and a failed boot is not.
Related field notes
- The store that has files in it and never indexed some of them
- The index that deletes itself on a schedule nobody diaried
- A tool wired into every request that never fires
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Vector stores — OpenAI API reference
- File search tool — OpenAI platform docs
- File search — OpenAI platform docs
- openai-python API surface
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.