Diagnostic LLM APIs
Files failed to index and file_search quietly returns less
Somebody in support says the assistant does not know about the September pricing change, and you know for a fact that the September pricing memo was in the ingest. You check, and it was: the upload succeeded, the attach call returned 200, the ingest job logged 812 files indexed and exited zero. The store's status says completed. Every one of those things is true, and the memo is a scanned PDF with no text layer, so it is not in the index and never was.
Two GETs per store with a project key. GET /v1/vector_stores?limit=100 for the stores, then for each one GET /v1/vector_stores/{vector_store_id}/files?filter=failed&limit=100, paged on after. Every returned child carries last_error, which is null on a healthy file and otherwise {"code": ..., "message": ...} where code is exactly one of server_error, unsupported_file or invalid_file.
The reason nobody sees this is the parent object. A vector store's status becomes completed when no file is still in_progress, which is true whether the files succeeded or failed, and the only aggregate signal is file_counts.failed sitting next to a large and reassuring completed. An ingest job that polls for status == "completed" and then declares the corpus ready is polling the wrong field.
Reconcile the two rather than trusting either. The script compares file_counts.failed against the number of children the filtered listing actually returns and reports a disagreement as its own finding, because a summary that counts failures the listing no longer contains means the failed children were detached and the repair was never finished.
Then the same listing with filter=in_progress, checking each child's created_at against the clock. A file still processing an hour after the ingest ended is pinned, not slow, and it keeps the parent's status at in_progress forever.
If a store has file_counts.total == 0, this is not the note. Nothing was ever attached, which is a different fault with a different repair, and the script says so rather than reporting a zero per cent failure rate.
The problem in plain words
Attaching a file to a vector store is asynchronous and the acknowledgement is not the outcome. The request is accepted, a vector_store.file comes back in in_progress, and parsing, chunking and embedding happen afterwards on the server. Whatever happens then is recorded on the child object and nowhere else you are looking.
The things that fail are boring and common: a scanned PDF with no text layer, a password-protected document, an empty file, an extension the parser does not handle, something corrupt at the source. Each ends as status: "failed" with last_error populated. Nothing raises. No webhook fires. The next retrieval call succeeds and returns fewer results, and the only downstream symptom is that the model does not know something it should.
What makes it survive is that every summary you would naturally check looks fine. The ingest exit code is fine. The store's status is fine. The completed count is large. The one number that is not fine is file_counts.failed, which is four digits to the right of a number that is doing an excellent job of reassuring you.
The stalled case is the same shape with the clock instead of an error. A very large file, or an ingest that pushed past the platform's attachment rate, can leave individual children pinned in in_progress indefinitely. The parent stays in_progress as long as any child is, which at least looks unfinished — but only if somebody reads it, and the retrieval path does not.
Why it happens
The finding lives on the child object, and every reflex points at the parent. GET /v1/vector_stores/{id} is the call people make, and it returns a status word and five integers. The three error codes, the human-readable message that says which page of the PDF broke, and the file id you need in order to fix anything are all on vector_store.file, which you only see by listing the store's files. A note that read only the parent could tell you 37 files failed and could not tell you what to do about any of them.
The three codes have three different repairs, so bucketing by code is the output. unsupported_file is a format problem and the fix is a conversion at the source: OCR the scan, export to text or markdown. invalid_file usually means empty, corrupt or encrypted, and the fix is upstream of the API entirely. server_error is transient and the fix is to attach it again. A report that says "37 failures" sends somebody to look at 37 files; a report bucketed by code sends them to three decisions.
A failed file with no last_error is a real state and must not be dropped. The field is nullable on every child, including failed ones. A reader that keys a dictionary on last_error["code"] either raises or silently discards those rows, and discarding them is worse: the failures with no stated reason are exactly the ones nobody has looked at. They get their own bucket here.
The summary and the listing can disagree, and the disagreement is information. file_counts.failed is a stored aggregate; the filtered listing is paged and enumerates live children. When the count is non-zero and the listing returns nothing, somebody detached the failed files and stopped there, which is a half-finished repair rather than a healthy store. The script grades that separately instead of averaging the two numbers into a single confident wrong one.
An empty store is not a zero per cent failure rate. Dividing failed by total when total is zero gives either an exception or a clean bill of health, and both are wrong. A store with nothing in it is a different fault, and the script routes it to the other note by name rather than grading it here.
The fix, as a flow
The attach call is the last thing in this chain that anybody watches, and it succeeds. Parsing, chunking and embedding happen afterwards on the server, and the only record that any of it went wrong is a per-file field on a listing nobody requests. The store's own summary does not help: its status turns to completed once no file is still in progress, which is true whether the files succeeded or failed.
How to fix it
Use a project key for the project that owns the stores
/v1/vector_stores is a project-scoped path, so an organization admin key is the wrong credential here and a project key from the wrong project simply will not see the store. The official client still sends OpenAI-Beta: assistants=v2 on every vector store call, so these scripts send it too rather than betting on where the listing is in its graduation out of that beta.
List the stores and read file_counts, not status
GET /v1/vector_stores?limit=100, paged on after with has_more and last_id. Read the five integers in file_counts. status: "completed" means no child is pending; it is not a statement that any child succeeded.
List the failed children and bucket them by last_error.code
GET /v1/vector_stores/{vector_store_id}/files?filter=failed&limit=100, paged on after. filter accepts in_progress, completed, failed and cancelled. Bucket on last_error.code and keep a bucket for children whose last_error is null.
Reconcile the bucket total against file_counts.failed
Equal numbers mean the summary and the children agree and the failure list is complete. A non-zero count with an empty listing means the failed children were removed and never re-attached, which the script reports as its own state rather than as zero failures.
Sweep for children pinned in_progress, and print the repair
filter=in_progress on the same path, comparing each child's created_at against the clock. Anything older than an hour is pinned rather than slow. The repair is printed per bucket — convert, fix at source, or re-attach — along with the durable one: make file_counts.failed == 0 the completion gate in the ingest job, not status == "completed".
How to check it worked
Fix one bucket and re-run. The failure count should fall by exactly the size of that bucket, and the state should move to complete only when both the summary and the listing agree on zero. A store that moves from attach-failed to counts-disagree has had its failed files detached rather than repaired, which is the state this script exists to stop you shipping.
python3 openai_vector_store_attach_failures.py
# 3 store(s) visible to this key
# attach-failed vs_a1 handbook-corpus: 37 of 849 file(s) failed (4.4%)
# unsupported_file 19 file(s) file-9k2, file-9k4, file-9m1 ...
# invalid_file 14 file(s) file-7b1, file-7b8, file-8c2 ...
# server_error 4 file(s) file-2d9, file-3a0, file-3a7, file-4b1
# repair: unsupported_file is a format the parser cannot read. OCR the scanned
# PDFs and export the rest to .md or .txt, then attach again.
# repair: invalid_file is usually empty, corrupt or password protected. Fix it
# at the source; re-attaching the same bytes fails the same way.
# repair: server_error is transient. Attach those 4 again and re-check.
# repair: gate the ingest job on file_counts.failed == 0, not on
# status == "completed", which only means nothing is pending.
# ingestion-stalled vs_b2 policies: 2 file(s) in_progress for over 1h
# no-files vs_c3 scratch: nothing has ever been attached, so this is
# the empty vector store note rather than this one.
# 2 finding(s)The full code
Two paged GETs per store and six pure functions. counts, which coerces the five file_counts integers so a missing key cannot become a string; bucket_errors, which groups the failed children by last_error.code and keeps a bucket for the ones with no error at all; stalled, which measures each in_progress child's age against the clock; failure_rate, which returns zero rather than raising on an empty store; reconcile, which reports the summary count and the listed count as two numbers instead of averaging them; and verdict, which routes an empty store to the other note by name before it grades anything.
"""Find files that never indexed in an OpenAI vector store.
Read only. Every request is a GET: /v1/vector_stores for the parents, then
/v1/vector_stores/{id}/files with filter=failed and filter=in_progress for the
children. No request body is ever constructed, and in particular no file_search
query is ever run. A retrieval query is a generation, it is billed, and a script
about a broken index has no business creating traffic against it.
The subject is the child object. A vector_store.file carries last_error.code
with one of exactly three values; the parent carries a failed count that its own
status field does not reflect, because status becomes "completed" when nothing
is still in progress whether or not anything succeeded.
A store with no files at all is not this note, and is reported as such.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_vector_store_attach_failures")
API = "https://api.openai.com/v1"
# The official client still sends this on every vector store call, so this
# script does too rather than betting on where the listing has got to in its
# graduation out of the Assistants beta. It is a GET either way.
BETA = {"OpenAI-Beta": "assistants=v2"}
# The complete set. last_error.code is documented as exactly these three, and a
# fourth arriving is worth reporting rather than bucketing into "other".
ERROR_CODES = ("server_error", "unsupported_file", "invalid_file")
# A failed child whose last_error is null. The field is nullable on every child
# including the failed ones, and a reader that keys on last_error["code"] either
# raises or drops these rows. Dropping them is worse: a failure with no stated
# reason is the one nobody has looked at.
UNREPORTED = "unreported"
REPAIRS = {
"unsupported_file":
"unsupported_file is a format the parser cannot read: a scan with no "
"text layer, or an extension it does not handle. OCR the scans and "
"export the rest to .md or .txt, then attach again.",
"invalid_file":
"invalid_file is usually empty, corrupt or password protected. Fix it "
"at the source; re-attaching the same bytes fails the same way.",
"server_error":
"server_error is transient. Attach those files again and re-check "
"before treating them as a content problem.",
UNREPORTED:
"these failed with no last_error at all. Fetch each one with GET "
"/v1/vector_stores/{vector_store_id}/files/{file_id} before deciding, "
"because a failure with no stated reason has not been looked at.",
}
FINDINGS = ("attach-failed", "ingestion-stalled", "counts-disagree")
def counts(store):
"""The five file_counts integers, coerced. Pure.
A missing key becomes 0 rather than None, so every caller can do arithmetic
without guarding, and a string that arrives where an integer was promised
does not propagate into a division.
"""
raw = (store or {}).get("file_counts") or {}
out = {}
for key in ("in_progress", "completed", "failed", "cancelled", "total"):
try:
out[key] = int(raw.get(key) or 0)
except (TypeError, ValueError):
out[key] = 0
return out
def bucket_errors(files):
"""{last_error.code: [file_id, ...]} over the failed children. Pure.
Only children whose status is actually "failed" are counted, because the
filtered listing is a request parameter rather than a guarantee, and a
caller that forgets the filter would otherwise bucket the whole store.
"""
out = {}
for entry in files or []:
row = entry or {}
if str(row.get("status") or "").strip().lower() != "failed":
continue
err = row.get("last_error") or {}
code = str(err.get("code") or "").strip().lower() or UNREPORTED
out.setdefault(code, []).append(str(row.get("id") or "?"))
for ids in out.values():
ids.sort()
return out
def stalled(files, now, max_age=3600):
"""[(file_id, age_seconds)] for children pinned in_progress. Pure.
Sorted oldest first. A child with no usable created_at is skipped rather
than treated as infinitely old, which would report every store as stalled
the first time the field shape changes.
"""
out = []
for entry in files or []:
row = entry or {}
if str(row.get("status") or "").strip().lower() != "in_progress":
continue
try:
created = int(row.get("created_at") or 0)
except (TypeError, ValueError):
continue
if created > 0 and (now - created) > max_age:
out.append((str(row.get("id") or "?"), int(now - created)))
out.sort(key=lambda r: (-r[1], r[0]))
return out
def failure_rate(c):
"""failed / total. Pure. Zero on an empty store rather than an exception."""
total = (c or {}).get("total") or 0
if total <= 0:
return 0.0
return float((c or {}).get("failed") or 0) / float(total)
def reconcile(c, buckets):
"""(claimed, listed) failure counts. Pure.
Two numbers, never one. file_counts.failed is a stored aggregate and the
filtered listing enumerates live children, so they can legitimately differ,
and averaging them into a single confident number destroys the only signal
that says a repair was started and abandoned.
"""
listed = sum(len(v) for v in (buckets or {}).values())
try:
claimed = int((c or {}).get("failed") or 0)
except (TypeError, ValueError):
claimed = 0
return (claimed, listed)
def verdict(c, buckets, stalled_rows):
"""Classify one store. Pure. Returns (state, detail).
The empty case is answered first and handed to the other note by name. A
store with nothing in it has a zero per cent failure rate, which is true and
useless, and its repair is re-running an ingest rather than fixing a format.
"""
c = c or {}
total = int(c.get("total") or 0)
claimed, listed = reconcile(c, buckets)
stalled_rows = list(stalled_rows or [])
if total <= 0:
return ("no-files",
"nothing has ever been attached, so this is the empty vector "
"store note rather than this one")
if listed > 0:
detail = ("%d of %d file(s) failed (%.1f%%)"
% (listed, total, failure_rate(c) * 100))
if claimed != listed:
detail += (" -- file_counts.failed says %d and the listing returns "
"%d, so read the listing" % (claimed, listed))
return ("attach-failed", detail)
if claimed > 0:
return ("counts-disagree",
"file_counts.failed is %d and the filtered listing returns "
"none, which is what a half-finished repair looks like: the "
"failed files were detached and never attached again"
% claimed)
if stalled_rows:
oldest = stalled_rows[0][1] // 3600
return ("ingestion-stalled",
"%d file(s) still in_progress, the oldest for over %dh. The "
"parent stays in_progress while any child is."
% (len(stalled_rows), max(oldest, 1)))
if int(c.get("in_progress") or 0) > 0:
return ("still-ingesting",
"%d file(s) in_progress and none of them old enough to call "
"pinned. Re-run after the ingest settles."
% int(c.get("in_progress") or 0))
return ("complete",
"%d file(s), all completed, and the summary agrees with the listing"
% total)
def repair_lines(state, buckets=None, stalled_rows=()):
"""The repair for one verdict. Pure. Printed, never performed."""
buckets = buckets or {}
if state == "attach-failed":
lines = [REPAIRS[code] for code in
sorted(buckets, key=lambda k: (-len(buckets[k]), k))
if code in REPAIRS]
unknown = sorted(set(buckets) - set(REPAIRS))
if unknown:
lines.append("last_error.code came back as %s, which is not one of "
"the three documented values. Read the message field "
"before acting on it." % ", ".join(unknown))
lines.append("gate the ingest job on file_counts.failed == 0, not on "
"status == \"completed\", which only means nothing is "
"pending.")
return lines
if state == "counts-disagree":
return [
"list the store's files without a filter and compare the ids "
"against your ingest manifest. The failures are gone from the "
"store and are still missing from retrieval.",
"re-attach the manifest entries that no longer appear, then assert "
"file_counts.failed == 0 and file_counts.completed == "
"file_counts.total before declaring the store ready.",
]
if state == "ingestion-stalled":
oldest = list(stalled_rows or [])[:5]
lines = ["detach and attach those files again rather than waiting. A "
"child pinned for hours is not going to finish on its own."]
if oldest:
lines.append("oldest pinned: " + ", ".join(
"%s (%dh)" % (fid, age // 3600) for fid, age in oldest))
lines.append("stagger large ingests, and poll file_counts.in_progress "
"down to zero with a timeout rather than assuming that "
"attach means indexed.")
return lines
if state == "no-files":
return ["an empty store fails differently and is repaired differently. "
"Re-run the ingest, or stop naming the store in "
"vector_store_ids."]
return []
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=90)
if r.status_code in (401, 403):
raise SystemExit("%d from OpenAI: /v1/vector_stores needs a project "
"key for the project that owns the stores"
% r.status_code)
r.raise_for_status()
return r.json()
def paged(session, path, max_pages=200, **params):
"""Walk an after/last_id cursor listing."""
params = dict(params)
for _ in range(max_pages):
page = get(session, path, **params)
data = page.get("data") or []
for item in data:
yield item
if not page.get("has_more") or not data:
return
params["after"] = page.get("last_id") or (data[-1] or {}).get("id")
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--store-id", action="append", default=[],
help="restrict to these store ids (repeatable)")
ap.add_argument("--stalled-hours", type=float, default=1.0,
help="age at which an in_progress file is called pinned")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key for the project that "
"owns the vector stores")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + key, **BETA})
stores = list(paged(s, "/vector_stores", limit=100))
wanted = set(args.store_id or [])
if wanted:
stores = [st for st in stores if (st or {}).get("id") in wanted]
log.info("%d store(s) visible to this key", len(stores))
now = int(time.time())
max_age = int(args.stalled_hours * 3600)
findings = 0
for store in stores:
sid = (store or {}).get("id") or "?"
name = (store or {}).get("name") or "(unnamed)"
c = counts(store)
failed = []
pending = []
if c["total"] > 0:
failed = list(paged(s, "/vector_stores/%s/files" % sid,
limit=100, filter="failed"))
if c["in_progress"] > 0:
pending = list(paged(s, "/vector_stores/%s/files" % sid,
limit=100, filter="in_progress"))
buckets = bucket_errors(failed)
stalled_rows = stalled(pending, now, max_age)
state, detail = verdict(c, buckets, stalled_rows)
emit = log.warning if state in FINDINGS else log.info
emit("%-20s %s %s: %s", state, sid, name, detail)
if state == "attach-failed":
for code in sorted(buckets, key=lambda k: (-len(buckets[k]), k)):
ids = buckets[code]
shown = ", ".join(ids[:3]) + (" ..." if len(ids) > 3 else "")
emit(" %-18s %d file(s) %s", code, len(ids), shown)
for line in repair_lines(state, buckets, stalled_rows):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find files that never indexed in an OpenAI vector store.
*
* Read only. Every request is a GET. No request body is constructed and no
* file_search query is ever run, because a retrieval query is a generation and
* a script about a broken index should not create traffic against it.
*
* The subject is the child object: a vector_store.file carries last_error.code
* with one of exactly three values, while the parent's status becomes
* "completed" when nothing is pending whether or not anything succeeded.
*/
const API = 'https://api.openai.com/v1';
// The official client still sends this on every vector store call.
const BETA = { 'OpenAI-Beta': 'assistants=v2' };
export const ERROR_CODES = ['server_error', 'unsupported_file', 'invalid_file'];
// A failed child whose last_error is null. Nullable on every child, and a
// reader that keys on last_error.code drops exactly the rows nobody has read.
export const UNREPORTED = 'unreported';
const REPAIRS = {
unsupported_file:
'unsupported_file is a format the parser cannot read: a scan with no text '
+ 'layer, or an extension it does not handle. OCR the scans and export the '
+ 'rest to .md or .txt, then attach again.',
invalid_file:
'invalid_file is usually empty, corrupt or password protected. Fix it at '
+ 'the source; re-attaching the same bytes fails the same way.',
server_error:
'server_error is transient. Attach those files again and re-check before '
+ 'treating them as a content problem.',
[UNREPORTED]:
'these failed with no last_error at all. Fetch each one with GET '
+ '/v1/vector_stores/{vector_store_id}/files/{file_id} before deciding, '
+ 'because a failure with no stated reason has not been looked at.',
};
const FINDINGS = new Set(['attach-failed', 'ingestion-stalled', 'counts-disagree']);
/** The five file_counts integers, coerced. Pure. */
export function counts(store) {
const raw = store?.file_counts ?? {};
const out = {};
for (const key of ['in_progress', 'completed', 'failed', 'cancelled', 'total']) {
const n = Number(raw[key] ?? 0);
out[key] = Number.isFinite(n) ? Math.trunc(n) : 0;
}
return out;
}
/** {code: [fileId]} over the failed children. Pure. */
export function bucketErrors(files) {
const out = {};
for (const entry of files ?? []) {
const row = entry ?? {};
if (String(row.status ?? '').trim().toLowerCase() !== 'failed') continue;
const code = String(row.last_error?.code ?? '').trim().toLowerCase() || UNREPORTED;
(out[code] ??= []).push(String(row.id ?? '?'));
}
for (const ids of Object.values(out)) ids.sort();
return out;
}
/** [[fileId, ageSeconds]] for children pinned in_progress. Pure. Oldest first. */
export function stalled(files, now, maxAge = 3600) {
const out = [];
for (const entry of files ?? []) {
const row = entry ?? {};
if (String(row.status ?? '').trim().toLowerCase() !== 'in_progress') continue;
const created = Number(row.created_at ?? 0);
if (!Number.isFinite(created) || created <= 0) continue;
if (now - created > maxAge) out.push([String(row.id ?? '?'), Math.trunc(now - created)]);
}
out.sort((a, b) => (b[1] - a[1]) || a[0].localeCompare(b[0]));
return out;
}
/** failed / total. Pure. Zero on an empty store rather than a division by zero. */
export function failureRate(c) {
const total = Number(c?.total ?? 0);
if (!(total > 0)) return 0;
return Number(c?.failed ?? 0) / total;
}
/** [claimed, listed] failure counts. Pure. Two numbers, never one. */
export function reconcile(c, buckets) {
const listed = Object.values(buckets ?? {}).reduce((a, v) => a + v.length, 0);
const claimed = Number(c?.failed ?? 0);
return [Number.isFinite(claimed) ? Math.trunc(claimed) : 0, listed];
}
/** Classify one store. Pure. Returns [state, detail]. */
export function verdict(c, buckets, stalledRows) {
const cc = c ?? {};
const total = Math.trunc(Number(cc.total ?? 0));
const [claimed, listed] = reconcile(cc, buckets);
const rows = [...(stalledRows ?? [])];
if (total <= 0) {
return ['no-files',
'nothing has ever been attached, so this is the empty vector store '
+ 'note rather than this one'];
}
if (listed > 0) {
let detail = `${listed} of ${total} file(s) failed `
+ `(${(failureRate(cc) * 100).toFixed(1)}%)`;
if (claimed !== listed) {
detail += ` -- file_counts.failed says ${claimed} and the listing returns `
+ `${listed}, so read the listing`;
}
return ['attach-failed', detail];
}
if (claimed > 0) {
return ['counts-disagree',
`file_counts.failed is ${claimed} and the filtered listing returns `
+ 'none, which is what a half-finished repair looks like: the failed '
+ 'files were detached and never attached again'];
}
if (rows.length) {
const oldest = Math.max(Math.trunc(rows[0][1] / 3600), 1);
return ['ingestion-stalled',
`${rows.length} file(s) still in_progress, the oldest for over `
+ `${oldest}h. The parent stays in_progress while any child is.`];
}
if (Math.trunc(Number(cc.in_progress ?? 0)) > 0) {
return ['still-ingesting',
`${Math.trunc(Number(cc.in_progress))} file(s) in_progress and none of `
+ 'them old enough to call pinned. Re-run after the ingest settles.'];
}
return ['complete',
`${total} file(s), all completed, and the summary agrees with the listing`];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, buckets = {}, stalledRows = []) {
const b = buckets ?? {};
if (state === 'attach-failed') {
const ordered = Object.keys(b).sort(
(x, y) => (b[y].length - b[x].length) || x.localeCompare(y));
const lines = ordered.filter((code) => REPAIRS[code]).map((code) => REPAIRS[code]);
const unknown = ordered.filter((code) => !REPAIRS[code]);
if (unknown.length) {
lines.push(`last_error.code came back as ${unknown.join(', ')}, which is not `
+ 'one of the three documented values. Read the message field before '
+ 'acting on it.');
}
lines.push('gate the ingest job on file_counts.failed == 0, not on '
+ 'status == "completed", which only means nothing is pending.');
return lines;
}
if (state === 'counts-disagree') {
return [
"list the store's files without a filter and compare the ids against your "
+ 'ingest manifest. The failures are gone from the store and are still '
+ 'missing from retrieval.',
're-attach the manifest entries that no longer appear, then assert '
+ 'file_counts.failed == 0 and file_counts.completed == file_counts.total '
+ 'before declaring the store ready.',
];
}
if (state === 'ingestion-stalled') {
const oldest = [...(stalledRows ?? [])].slice(0, 5);
const lines = ['detach and attach those files again rather than waiting. A '
+ 'child pinned for hours is not going to finish on its own.'];
if (oldest.length) {
lines.push('oldest pinned: ' + oldest
.map(([id, age]) => `${id} (${Math.trunc(age / 3600)}h)`).join(', '));
}
lines.push('stagger large ingests, and poll file_counts.in_progress down to '
+ 'zero with a timeout rather than assuming that attach means indexed.');
return lines;
}
if (state === 'no-files') {
return ['an empty store fails differently and is repaired differently. '
+ 'Re-run the ingest, or stop naming the store in vector_store_ids.'];
}
return [];
}
async function read(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
const r = await fetch(url, { headers: { Authorization: `Bearer ${key}`, ...BETA } });
if (r.status === 401 || r.status === 403) {
throw new Error(`${r.status} from OpenAI: /v1/vector_stores needs a project key `
+ 'for the project that owns the stores');
}
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
async function* paged(key, path, params, maxPages = 200) {
const q = { ...params };
for (let i = 0; i < maxPages; i += 1) {
const page = await read(key, path, q);
const data = page.data ?? [];
for (const item of data) yield item;
if (!page.has_more || data.length === 0) return;
q.after = page.last_id ?? data[data.length - 1]?.id;
}
}
async function collect(key, path, params) {
const out = [];
for await (const item of paged(key, path, params)) out.push(item);
return out;
}
async function main() {
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key for the project that owns '
+ 'the vector stores');
process.exitCode = 2;
return;
}
const maxAge = Math.trunc(Number(process.env.STALLED_HOURS ?? 1) * 3600);
const wanted = new Set((process.env.VECTOR_STORE_IDS ?? '')
.split(/[,\s]+/).filter(Boolean));
let stores = await collect(key, '/vector_stores', { limit: 100 });
if (wanted.size) stores = stores.filter((st) => wanted.has(st?.id));
console.log(`${stores.length} store(s) visible to this key`);
const now = Math.floor(Date.now() / 1000);
let findings = 0;
for (const store of stores) {
const sid = store?.id ?? '?';
const name = store?.name ?? '(unnamed)';
const c = counts(store);
let failed = [];
let pending = [];
if (c.total > 0) {
failed = await collect(key, `/vector_stores/${sid}/files`,
{ limit: 100, filter: 'failed' });
if (c.in_progress > 0) {
pending = await collect(key, `/vector_stores/${sid}/files`,
{ limit: 100, filter: 'in_progress' });
}
}
const buckets = bucketErrors(failed);
const stalledRows = stalled(pending, now, maxAge);
const [state, detail] = verdict(c, buckets, stalledRows);
console.log(`${state.padEnd(20)} ${sid} ${name}: ${detail}`);
if (state === 'attach-failed') {
const ordered = Object.keys(buckets).sort(
(x, y) => (buckets[y].length - buckets[x].length) || x.localeCompare(y));
for (const code of ordered) {
const ids = buckets[code];
const shown = ids.slice(0, 3).join(', ') + (ids.length > 3 ? ' ...' : '');
console.log(` ${code.padEnd(18)} ${ids.length} file(s) ${shown}`);
}
}
for (const line of repairLines(state, buckets, stalledRows)) {
console.log(` repair: ${line}`);
}
if (FINDINGS.has(state)) findings += 1;
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the note: a store whose status is completed and whose file_counts.failed is 37 has to come back as a finding, and the failures have to arrive bucketed by code rather than as a number. Next to it, the case that keeps this note out of its neighbour's territory — a store with total == 0 must be no-files, must not be a finding here, and must say which note owns it. Then the failed child with a null last_error, which a naive fold drops; the summary and the listing disagreeing, which is a half-finished repair rather than a healthy store; the pinned children measured against the clock; and failure_rate on an empty store, which must be zero and not an exception.
from openai_vector_store_attach_failures import (UNREPORTED, bucket_errors,
counts, failure_rate,
reconcile, repair_lines,
stalled, verdict)
def store(total=0, completed=0, failed=0, in_progress=0, cancelled=0,
status="completed"):
return {"id": "vs_a1", "name": "handbook", "status": status,
"file_counts": {"total": total, "completed": completed,
"failed": failed, "in_progress": in_progress,
"cancelled": cancelled}}
def child(fid, status, code=None, created_at=1_700_000_000):
row = {"id": fid, "object": "vector_store.file", "status": status,
"created_at": created_at, "vector_store_id": "vs_a1"}
row["last_error"] = {"code": code, "message": "..."} if code else None
return row
def test_a_completed_store_with_failed_children_is_the_finding():
# The whole note. status is "completed" because nothing is pending, which
# is exactly what an ingest job polls for before declaring the corpus ready.
c = counts(store(total=849, completed=812, failed=37))
buckets = bucket_errors(
[child("file-9k%d" % i, "failed", "unsupported_file") for i in range(19)]
+ [child("file-7b%d" % i, "failed", "invalid_file") for i in range(14)]
+ [child("file-2d%d" % i, "failed", "server_error") for i in range(4)])
state, detail = verdict(c, buckets, [])
assert state == "attach-failed"
assert "37 of 849" in detail
assert sorted(buckets) == ["invalid_file", "server_error", "unsupported_file"]
repairs = repair_lines(state, buckets)
assert any("OCR" in line for line in repairs)
assert any("file_counts.failed == 0" in line for line in repairs)
def test_an_empty_store_is_handed_to_the_other_note_by_name():
# The boundary between this note and its closest neighbour, asserted rather
# than described. total == 0 means nothing was ever attached; that is not a
# zero per cent failure rate and it is not repaired the same way.
c = counts(store(total=0))
state, detail = verdict(c, {}, [])
assert state == "no-files"
assert "empty vector store note" in detail
assert failure_rate(c) == 0.0
assert any("vector_store_ids" in line for line in repair_lines(state))
def test_a_failed_child_with_no_last_error_keeps_its_own_bucket():
buckets = bucket_errors([child("file-1", "failed", "invalid_file"),
child("file-2", "failed", None),
child("file-3", "completed", None)])
assert buckets[UNREPORTED] == ["file-2"]
assert buckets["invalid_file"] == ["file-1"]
assert "completed" not in buckets
assert any("has not been looked at" in line
for line in repair_lines("attach-failed", buckets))
def test_the_summary_and_the_listing_can_disagree():
# file_counts still counts 37 failures and the filtered listing returns
# none: somebody detached the failed files and stopped there.
state, detail = verdict(counts(store(total=812, completed=812, failed=37)),
{}, [])
assert state == "counts-disagree"
assert "half-finished repair" in detail
assert any("ingest manifest" in line for line in repair_lines(state))
assert reconcile({"failed": 37}, {}) == (37, 0)
assert reconcile({"failed": 2}, {"server_error": ["a", "b"]}) == (2, 2)
def test_children_pinned_in_progress_are_measured_against_the_clock():
now = 1_700_050_000
rows = stalled([child("file-slow", "in_progress", created_at=now - 40_000),
child("file-newer", "in_progress", created_at=now - 20_000),
child("file-fresh", "in_progress", created_at=now - 60),
child("file-bad", "in_progress", created_at=None),
child("file-done", "completed", created_at=now - 90_000)],
now)
assert [r[0] for r in rows] == ["file-slow", "file-newer"]
state, detail = verdict(counts(store(total=5, completed=3, in_progress=2)),
{}, rows)
assert state == "ingestion-stalled"
assert "parent stays in_progress" in detail
assert any("file-slow (11h)" in line
for line in repair_lines(state, {}, rows))
def test_a_healthy_store_and_a_still_settling_one_are_not_findings():
assert verdict(counts(store(total=40, completed=40)), {}, [])[0] == "complete"
state, _ = verdict(counts(store(total=40, completed=38, in_progress=2)),
{}, [])
assert state == "still-ingesting"
assert repair_lines("complete") == []
assert bucket_errors(None) == {} and stalled(None, 0) == []
assert counts(None)["total"] == 0
assert counts({"file_counts": {"total": "not-a-number"}})["total"] == 0
def test_an_unknown_error_code_is_reported_rather_than_bucketed_away():
buckets = bucket_errors([child("file-x", "failed", "quota_exceeded")])
lines = repair_lines("attach-failed", buckets)
assert any("quota_exceeded" in line for line in lines)
assert any("three documented values" in line for line in lines)
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { UNREPORTED, bucketErrors, counts, failureRate, reconcile, repairLines,
stalled, verdict } from './openai-vector-store-attach-failures.mjs';
const store = ({ total = 0, completed = 0, failed = 0, in_progress = 0,
cancelled = 0, status = 'completed' } = {}) =>
({ id: 'vs_a1', name: 'handbook', status,
file_counts: { total, completed, failed, in_progress, cancelled } });
const child = (id, status, code = null, createdAt = 1700000000) =>
({ id, object: 'vector_store.file', status, created_at: createdAt,
vector_store_id: 'vs_a1',
last_error: code ? { code, message: '...' } : null });
test('a completed store with failed children is the finding', () => {
const c = counts(store({ total: 849, completed: 812, failed: 37 }));
const children = [];
for (let i = 0; i < 19; i += 1) children.push(child(`file-9k${i}`, 'failed', 'unsupported_file'));
for (let i = 0; i < 14; i += 1) children.push(child(`file-7b${i}`, 'failed', 'invalid_file'));
for (let i = 0; i < 4; i += 1) children.push(child(`file-2d${i}`, 'failed', 'server_error'));
const buckets = bucketErrors(children);
const [state, detail] = verdict(c, buckets, []);
assert.equal(state, 'attach-failed');
assert.match(detail, /37 of 849/);
assert.deepEqual(Object.keys(buckets).sort(),
['invalid_file', 'server_error', 'unsupported_file']);
const repairs = repairLines(state, buckets);
assert.ok(repairs.some((l) => l.includes('OCR')));
assert.ok(repairs.some((l) => l.includes('file_counts.failed == 0')));
});
test('an empty store is handed to the other note by name', () => {
const c = counts(store({ total: 0 }));
const [state, detail] = verdict(c, {}, []);
assert.equal(state, 'no-files');
assert.match(detail, /empty vector store note/);
assert.equal(failureRate(c), 0);
assert.ok(repairLines(state).some((l) => l.includes('vector_store_ids')));
});
test('a failed child with no last_error keeps its own bucket', () => {
const buckets = bucketErrors([child('file-1', 'failed', 'invalid_file'),
child('file-2', 'failed', null),
child('file-3', 'completed', null)]);
assert.deepEqual(buckets[UNREPORTED], ['file-2']);
assert.deepEqual(buckets.invalid_file, ['file-1']);
assert.equal(buckets.completed, undefined);
assert.ok(repairLines('attach-failed', buckets)
.some((l) => l.includes('has not been looked at')));
});
test('the summary and the listing can disagree', () => {
const [state, detail] = verdict(
counts(store({ total: 812, completed: 812, failed: 37 })), {}, []);
assert.equal(state, 'counts-disagree');
assert.match(detail, /half-finished repair/);
assert.ok(repairLines(state).some((l) => l.includes('ingest manifest')));
assert.deepEqual(reconcile({ failed: 37 }, {}), [37, 0]);
assert.deepEqual(reconcile({ failed: 2 }, { server_error: ['a', 'b'] }), [2, 2]);
});
test('children pinned in_progress are measured against the clock', () => {
const now = 1700050000;
const rows = stalled([child('file-slow', 'in_progress', null, now - 40000),
child('file-newer', 'in_progress', null, now - 20000),
child('file-fresh', 'in_progress', null, now - 60),
child('file-bad', 'in_progress', null, null),
child('file-done', 'completed', null, now - 90000)], now);
assert.deepEqual(rows.map((r) => r[0]), ['file-slow', 'file-newer']);
const [state, detail] = verdict(
counts(store({ total: 5, completed: 3, in_progress: 2 })), {}, rows);
assert.equal(state, 'ingestion-stalled');
assert.match(detail, /parent stays in_progress/);
assert.ok(repairLines(state, {}, rows).some((l) => l.includes('file-slow (11h)')));
});
test('a healthy store and a still settling one are not findings', () => {
assert.equal(verdict(counts(store({ total: 40, completed: 40 })), {}, [])[0],
'complete');
assert.equal(verdict(counts(store({ total: 40, completed: 38, in_progress: 2 })),
{}, [])[0], 'still-ingesting');
assert.deepEqual(repairLines('complete'), []);
assert.deepEqual(bucketErrors(null), {});
assert.deepEqual(stalled(null, 0), []);
assert.equal(counts(null).total, 0);
assert.equal(counts({ file_counts: { total: 'not-a-number' } }).total, 0);
});
test('an unknown error code is reported rather than bucketed away', () => {
const buckets = bucketErrors([child('file-x', 'failed', 'quota_exceeded')]);
const lines = repairLines('attach-failed', buckets);
assert.ok(lines.some((l) => l.includes('quota_exceeded')));
assert.ok(lines.some((l) => l.includes('three documented values')));
});
FAQ
How do I tell this apart from an empty vector store? They both retrieve nothing.
One field. file_counts.total is zero on an empty store and non-zero here. If files were attached and some of them failed, this is the note and the repair is per error code: convert the format, fix the source file, or re-attach the transient failures. If nothing was ever attached, the store is empty, retrieval was never grounded at all, and the repair is to run the ingest or stop naming the store. The overlap case is a store that is empty because every attach failed, and the script separates that too: total greater than zero with completed at zero says the ingest ran and produced nothing, which is this note wearing the other note's symptoms.
Why not just check the store's status field?
Because status does not mean what it looks like it means. A vector store's status becomes completed when no file is still in_progress. That is a statement about pendingness, not about success, and it is true of a store where every single file failed. The only aggregate that carries the failure is file_counts.failed, which sits next to a large completed count and gets read as noise. This is why the repair the script prints is to gate the ingest job on file_counts.failed == 0 rather than on the status word.
Is this the same as the batch error file nobody reads?
No, and they are not even the same resource. That note is about a Batch object's error_file_id, a file id the platform hands you when a batch job produces per-line failures and which nothing forces you to fetch. This one reads vector_store.file objects, which are children of a vector store, carry a three-valued last_error.code, and have no downloadable error file at all. The only thing the two share is the shape of the mistake: a failure recorded on an object nobody lists.
Does the script ever run a search to check the store really works?
No, deliberately. A file_search query is a generation: it is billed, it goes through a model, and it is a write in every sense that matters to this section. Everything here is a GET against the vector store and its children. That is also why the script cannot tell you whether retrieval quality is good, only whether the documents you attached are in the index at all, which is the question this note is about.
Anthropic has file search too. Why is this OpenAI only?
Because Anthropic has no managed vector store object. There is a Files API, and there are search tools, but there is nothing on that side that corresponds to a persistent server-side index with an id, a file_counts summary and per-file ingestion errors you can list. Every note in this batch is OpenAI for that reason, and the honest version of the sentence is that there is no equivalent to read rather than that nobody has written it yet.
Related field notes
- The store that has nothing in it and is still named in your config
- The index that deletes itself on a schedule nobody diaried
- The other file id nobody fetched, on a completely different object
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Vector store files — OpenAI API reference
- Vector stores — OpenAI API reference
- File search — OpenAI platform docs
- openai-openapi — the published OpenAPI specification
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.