Diagnostic LLM APIs
The batch failed validation and named the broken line
The submitter returned 200 and the run log says the nightly enrichment fired. Sixteen hours later the enrichment table is exactly as long as it was yesterday. There was no exception, no alert and no retry, because from the client's point of view nothing went wrong: the batch was accepted. It failed forty seconds later, in a state your code never looks at, and it has been holding a list of line numbers ever since.
GET /v1/batches?limit=100 with a project key set to Read Only, paginate on after, and keep every object where status is failed. On the Batch API that status means one thing: the input file did not survive validation. The batch never reached the model.
The detail is in errors, which is a list object whose data[] entries carry code, message, param and line — and line is documented as the line number of the input file where the error occurred. That is a pointer straight into the .jsonl you uploaded. Group the entries by code and print the line numbers under each one.
Confirm nothing was billed while you are there. A failed batch has request_counts of {total: 0, completed: 0, failed: 0}, because validation happens before any request is dispatched. That is the reassuring half of the finding and it is worth printing, since the first question anybody asks is whether this cost money.
Then read the file list for the failure that leaves no batch behind. /v1/batches accepts only an input file whose purpose is exactly batch. A .jsonl uploaded under user_data, assistants, fine-tune or vision is refused at creation, so there is no batch object to find and no errors array to read. GET /v1/files shows it sitting there, taking up storage, referenced by nothing.
The problem in plain words
POST /v1/batches returning 200 means the batch object was created. It does not mean the batch will run, and the two are separated by a validation pass that reads every line of your input file. A malformed JSON line, a missing custom_id, a duplicate custom_id, a per-line url that does not match the batch's endpoint, or a body naming a model this project cannot reach will fail the entire batch. Not the line. The batch.
So the failure arrives after the caller has gone. A submitter that fires and forgets sees a 200, writes a success line to its log and exits. The batch enters validating, leaves it for failed within a minute or two, and stays there. Nothing calls back. There is no webhook in the default flow and no exception to catch, and the batch object will still be sitting in the list a week later holding the exact reason and the exact line.
What makes this expensive is not the failed batch, which cost nothing. It is that the downstream table is simply not updated, and a table that is not updated looks identical to a table that had no new rows. Nightly enrichment that quietly stops enriching is discovered by a person noticing a stale timestamp, and the median time for that is measured in weeks.
The wrong-purpose input file is the same failure one step earlier and it is harder to see, because it never produces a batch at all. The Files API namespaces uploads by purpose, and a shared upload helper with a hard-coded default happily accepts your batch input as user_data. The upload succeeds. The file appears in GET /v1/files with a sensible filename and a byte count. It can never be used as batch input, and it counts against project storage forever.
Why it happens
failed on this API is a specific claim, not a general one. The status enum is validating, failed, in_progress, finalizing, completed, expired, cancelling, cancelled. A batch whose requests failed individually does not land on failed; it lands on completed with a non-zero request_counts.failed, which is a different note. failed means the input file was rejected as a whole. That is why this script does no arithmetic on counts at all: on a failed batch there are none to do.
The line field is the entire value of the reading. An error object with a code and a message tells you what kind of thing is wrong. line tells you where, in a file that might be two hundred thousand rows long, and it is documented exactly that way: the line number of the input file where the error occurred, if applicable. Grouping by code and listing the lines under each turns a wall of repeated messages into a short repair list.
Nothing was billed, and saying so is part of the output. Validation runs before any request is dispatched to the model, so a failed batch has an all-zero request_counts and no usage worth reading. A finding that does not answer "did this cost us anything" gets escalated as a cost incident by somebody who does not know, which wastes more time than the bug did.
The batch list cannot be filtered server-side, so the script pages and filters locally. GET /v1/batches takes exactly two query parameters, limit (1 to 100, default 20) and after. There is no status filter and no date range. A run that reads only the default first page will miss every failure older than the last twenty submissions, which on a nightly pipeline is three weeks.
The wrong-purpose file is invisible from the batch side by construction. Because the creation call is rejected outright, no batch object exists to carry the error. The only read-only evidence is a file whose purpose is not batch, whose name looks like batch input, and whose id is not the input_file_id of any batch you have. All three conditions are needed: without the last one the check would flag every input file you ever used successfully.
The fix, as a flow
Two hundred is a receipt for a batch object, not a promise that the batch will run. Between creation and the first request there is a validation pass over every line of the input file, and a single bad row fails the whole job rather than the row. Nothing is billed and nothing is produced, so the only trace is a status and an errors array holding the line numbers, sitting on an object nobody polled.
How to fix it
Use a project key set to Read Only
Both calls are GETs on project-scoped data, so no admin key is involved. The script never uploads a file, never creates a batch and never re-submits a failed one. Re-submitting spends money on inference, and only you know whether the rows are still wanted.
Page the whole batch list
GET /v1/batches?limit=100, then follow after with the last id until has_more is false. There is no server-side status filter, so the filtering happens on your side. Use --max-pages to bound a very long history, and know that bounding it means the answer is bounded too.
Group the errors by code and print the lines
For every failed batch, read errors.data[]. Each entry gives you code, message, param and line. The script prints one row per code with the offending line numbers listed under it, truncated with a count when there are many, because a hundred identical messages is not more informative than one with a hundred line numbers.
Check the input files that never became a batch
GET /v1/files, paginate on after, and flag any .jsonl whose purpose is one of user_data, assistants, fine-tune or vision and whose id is not the input_file_id of any batch in the list. That is a file uploaded for a batch that was refused at creation.
Fix the file, then re-create the batch yourself
The repair is printed, never run. Correct the input at the reported lines, re-upload with purpose set to batch, and create the batch again. Then make the pipeline poll: a 200 from creation is a receipt, and the only honest success signal is a batch that has left validating.
How to check it worked
Re-run after the input is fixed and the batch is re-created. The old failed batches do not disappear — batch objects persist — so pass --since-days to scope the run to the window you care about, and expect the count of failures inside that window to go to zero. The check worth keeping is the assertion in the submitter, not this script.
python3 openai_batch_validation_audit.py --since-days 30
# batch_68f2a1c9 failed at 2026-08-27T02:14:09Z nothing billed (0 requests)
# invalid_json lines 41207, 41208, 41209 and 6 more
# duplicate_custom_id lines 903, 41207
# batch_68e0b7d4 failed at 2026-08-19T02:13:51Z nothing billed (0 requests)
# model_not_found line 1 param body.model
# orphan-input file_9de2 nightly-enrich.jsonl purpose=user_data 1.4 MB
# validation-failed 2 batch(es) failed input validation in the last 30 days
# and 1 .jsonl was uploaded under a purpose /v1/batches
# will not accept
# measured: status, errors.data[] and request_counts from the batch list,
# purpose from the file list
# inferred: that the pipeline never polled, since a failed batch is otherwise
# indistinguishable from one nobody re-ran on purpose
# repair: fix the input at the reported lines, then re-upload with
# purpose=batch and create the batch again. Assert that status has
# left "validating" before the submitter logs success.
# 3 finding(s)The full code
Two paged GETs and seven pure functions. failed_batches, which is the whole server-side filter the API does not offer; error_rows, which normalises errors.data[] and survives the object being absent, empty or a shape nobody expected; lines_by_code, which groups and sorts so a hundred identical messages become one row with a hundred line numbers; nothing_billed, which reads request_counts and answers the first question anyone asks; batch_input_ids, which collects every input_file_id in the account so a file that was used is never flagged; mispurposed_inputs, which needs all three conditions to fire; and verdict, which keeps the two halves separate because they have different repairs.
"""Report OpenAI batches that failed input validation, and the lines that broke.
Read only. Two GET endpoints, /v1/batches and /v1/files, and nothing else. No
file is uploaded, no batch is created, and no failed batch is re-submitted:
re-running rows spends money on inference and only you know whether the rows
are still wanted.
On the Batch API, status "failed" is a specific claim. It means the input file
did not survive validation, which happens before any request reaches the model.
So request_counts is all zeros and nothing was billed. Individual requests that
failed inside a batch that ran are a different status and a different note.
The second half reads the file list, because an input file uploaded under the
wrong purpose is rejected at creation and never becomes a batch object at all.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_batch_validation_audit")
BATCHES_URL = "https://api.openai.com/v1/batches"
FILES_URL = "https://api.openai.com/v1/files"
# The batch list takes limit (1-100) and after. There is no status filter and no
# date range, so every bit of selection below happens on this side of the wire.
PAGE = 100
# Purposes the Files API namespaces separately from batch input. A .jsonl parked
# under one of these was uploaded for a batch that /v1/batches refused, since it
# accepts an input file only when purpose is exactly "batch".
NOT_BATCH_INPUT = ("user_data", "assistants", "fine-tune", "vision")
# Error codes the docs and the endpoint's own validation produce often enough to
# be worth a specific repair line rather than the generic one.
KNOWN_CODES = {
"invalid_json": "a line is not valid JSON. Validate the file locally before "
"upload: every line must parse on its own.",
"duplicate_custom_id": "two lines share a custom_id. They must be unique "
"within the file, because results come back unordered "
"and custom_id is the only join key.",
"missing_required_parameter": "a line is missing a required field. Each row "
"needs custom_id, method, url and body.",
"invalid_url": "a line's url does not match the batch endpoint. The two must "
"agree for every row in the file.",
"model_not_found": "the body names a model this project cannot reach. Check "
"the id against GET /v1/models with the same key.",
"empty_file": "the input file has no lines in it. The upload succeeded and "
"the content did not.",
}
FINDINGS = ("validation-failed", "orphan-input-files")
def failed_batches(batches):
"""Batches whose input file was rejected. Pure.
The only status that means "validation refused this file". A batch that ran
and had rows fail inside it reports completed with a non-zero
request_counts.failed, which this script deliberately does not look at.
"""
return [b for b in batches or [] if (b or {}).get("status") == "failed"]
def error_rows(batch):
"""Normalised entries from errors.data[]. Pure. Never raises on a shape.
errors is optional, its data list is optional, and an entry's line is
documented as "if applicable", so every field here is allowed to be absent.
"""
errors = (batch or {}).get("errors") or {}
data = errors.get("data") if isinstance(errors, dict) else None
out = []
for item in data or []:
if not isinstance(item, dict):
continue
line = item.get("line")
try:
line = int(line)
except (TypeError, ValueError):
line = None
out.append({"code": str(item.get("code") or "unknown"),
"message": str(item.get("message") or ""),
"param": item.get("param"),
"line": line})
return out
def lines_by_code(rows):
"""{code: (sorted lines, count, one message, one param)}. Pure.
A file with 40,000 bad rows produces 40,000 near-identical messages. One row
per code carrying the line numbers is the same information and is readable.
"""
grouped = {}
for row in rows or []:
slot = grouped.setdefault(row["code"], {"lines": set(), "count": 0,
"message": "", "param": None})
slot["count"] += 1
if row.get("line") is not None:
slot["lines"].add(row["line"])
if not slot["message"]:
slot["message"] = row.get("message") or ""
if slot["param"] is None:
slot["param"] = row.get("param")
return {code: (sorted(v["lines"]), v["count"], v["message"], v["param"])
for code, v in sorted(grouped.items())}
def nothing_billed(batch):
"""True when the batch dispatched no requests at all. Pure.
Validation runs before dispatch, so a failed batch has an all-zero
request_counts. An absent counts object is treated as zero, which is what a
batch that never started actually looks like.
"""
counts = (batch or {}).get("request_counts") or {}
try:
return all(int(counts.get(k) or 0) == 0
for k in ("total", "completed", "failed"))
except (TypeError, ValueError):
return False
def batch_input_ids(batches):
"""Every input_file_id the account has ever handed to a batch. Pure."""
return {str(b.get("input_file_id")) for b in batches or []
if (b or {}).get("input_file_id")}
def mispurposed_inputs(files, used_ids):
""".jsonl files that can never be batch input and never were. Pure.
All three conditions matter. Drop the last one and every input file you ever
used successfully gets flagged, which is how a check gets switched off.
"""
out = []
for f in files or []:
if not isinstance(f, dict):
continue
name = str(f.get("filename") or "")
purpose = str(f.get("purpose") or "")
if not name.lower().endswith(".jsonl"):
continue
if purpose not in NOT_BATCH_INPUT:
continue
if str(f.get("id")) in (used_ids or set()):
continue
out.append({"id": str(f.get("id")), "filename": name,
"purpose": purpose, "bytes": int(f.get("bytes") or 0)})
return sorted(out, key=lambda r: r["id"])
def within_window(batch, now, days):
"""True when the batch was created inside the window. Pure. days<=0 is all."""
if not days or days <= 0:
return True
try:
created = int((batch or {}).get("created_at") or 0)
except (TypeError, ValueError):
return False
return created >= now - days * 86400
def verdict(failed, orphans, days):
"""Grade the run. Pure. Returns (state, detail)."""
failed = list(failed or [])
orphans = list(orphans or [])
window = ("in the last %d days" % days) if days and days > 0 else "in the account"
if failed and orphans:
return ("validation-failed",
"%d batch(es) failed input validation %s, and %d .jsonl was "
"uploaded under a purpose /v1/batches will not accept"
% (len(failed), window, len(orphans)))
if failed:
return ("validation-failed",
"%d batch(es) failed input validation %s and nothing polled "
"them to find out" % (len(failed), window))
if orphans:
return ("orphan-input-files",
"%d .jsonl file(s) sit under a purpose /v1/batches will not "
"accept, referenced by no batch" % len(orphans))
return ("validation-clean",
"no batch %s failed validation, and every .jsonl in the file list "
"either carries purpose=batch or was used by a batch" % window)
def repair_lines(state, codes):
"""The repair for one verdict. Pure. Printed, never performed."""
if state == "validation-clean":
return ["nothing to change. Keep the assertion that a submitter only "
"logs success once status has left \"validating\"."]
lines = []
for code in sorted(set(codes or [])):
if code in KNOWN_CODES:
lines.append("%s: %s" % (code, KNOWN_CODES[code]))
if state == "validation-failed":
lines.append("fix the input at the reported lines, then re-upload with "
"purpose=batch and create the batch again. Nothing was "
"billed, so nothing needs reconciling.")
lines.append("make the submitter poll. A 200 from batch creation is a "
"receipt, not a result: the only honest success signal is "
"a batch that has left \"validating\".")
if state == "orphan-input-files":
lines.append("re-upload each file with purpose=batch and delete the "
"mis-purposed copy, which counts against project storage "
"until you do.")
lines.append("assert in the upload helper that the purpose matches the "
"endpoint that will consume the file.")
return lines
def get_json(url, key, params=None, timeout=30):
"""One GET. Returns (payload, error). Read only, always."""
try:
r = requests.get(url, headers={"Authorization": "Bearer %s" % key},
params=params or {}, timeout=timeout)
except requests.RequestException as exc:
return (None, "request failed: %s" % exc)
if r.status_code != 200:
detail = ""
try:
detail = str((r.json().get("error") or {}).get("message") or "")
except ValueError:
detail = (r.text or "")[:160]
return (None, "HTTP %d %s" % (r.status_code, detail))
try:
return (r.json(), None)
except ValueError:
return (None, "response was not JSON")
def page_all(url, key, params, max_pages):
"""Follow the after cursor. Returns (rows, error). GETs only."""
rows = []
after = None
for _ in range(max(1, max_pages)):
query = dict(params or {})
if after:
query["after"] = after
payload, err = get_json(url, key, query)
if err:
return (rows, err)
data = payload.get("data") or []
rows.extend(data)
if not payload.get("has_more") or not data:
break
after = data[-1].get("id")
if not after:
break
return (rows, None)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--since-days", type=int, default=30,
help="only report batches created inside this window (0 = all)")
ap.add_argument("--max-pages", type=int, default=20,
help="cap on pages of 100 for each list; a bounded read "
"gives a bounded answer and the output says so")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key set to Read Only")
return 2
now = int(time.time())
batches, err = page_all(BATCHES_URL, key, {"limit": PAGE}, args.max_pages)
if err and not batches:
log.error("could not read the batch list: %s", err)
return 2
if err:
log.warning("batch list stopped early: %s", err)
scoped = [b for b in batches if within_window(b, now, args.since_days)]
failed = failed_batches(scoped)
seen_codes = []
for b in failed:
billed = "nothing billed (0 requests)" if nothing_billed(b) \
else "request_counts is not all zero, which is unusual for failed"
stamp = time.strftime("%Y-%m-%dT%H:%M:%SZ",
time.gmtime(int(b.get("failed_at")
or b.get("created_at") or 0)))
log.warning("%-16s failed at %s %s", b.get("id"), stamp, billed)
groups = lines_by_code(error_rows(b))
if not groups:
log.warning(" (the errors object is empty, so the reason is not "
"readable from the API)")
for code, (lines, count, message, param) in groups.items():
seen_codes.append(code)
shown = ", ".join(str(n) for n in lines[:6])
more = " and %d more" % (len(lines) - 6) if len(lines) > 6 else ""
where = ("lines %s%s" % (shown, more)) if lines else "no line given"
extra = " param %s" % param if param else ""
log.warning(" %-26s %s%s", code, where, extra)
if message:
log.info(" %-26s %s", "", message[:140])
files, ferr = page_all(FILES_URL, key, {"limit": 10000}, args.max_pages)
if ferr:
log.warning("file list stopped early: %s", ferr)
orphans = mispurposed_inputs(files, batch_input_ids(batches))
for row in orphans:
log.warning("orphan-input %s %s purpose=%s %.1f MB", row["id"],
row["filename"], row["purpose"], row["bytes"] / 1048576.0)
state, detail = verdict(failed, orphans, args.since_days)
emit = log.warning if state in FINDINGS else log.info
emit("%-20s %s", state, detail)
emit(" measured: status, errors.data[] and request_counts from the batch "
"list, purpose from the file list")
emit(" inferred: that the pipeline never polled, since a failed batch is "
"otherwise indistinguishable from one nobody re-ran on purpose")
for line in repair_lines(state, seen_codes):
emit(" repair: %s", line)
total = len(failed) + len(orphans)
log.info("%d finding(s)", total)
return 1 if total else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report OpenAI batches that failed input validation, and the lines that broke.
*
* Read only. Two GET endpoints, /v1/batches and /v1/files. No upload, no batch
* creation, no re-submission of a failed batch.
*
* Status "failed" on this API means the input file did not survive validation,
* which happens before any request reaches the model: request_counts is all
* zeros and nothing was billed. Rows that failed inside a batch that ran are a
* different status and a different note.
*/
const BATCHES_URL = 'https://api.openai.com/v1/batches';
const FILES_URL = 'https://api.openai.com/v1/files';
const PAGE = 100;
export const NOT_BATCH_INPUT = new Set(['user_data', 'assistants', 'fine-tune', 'vision']);
export const KNOWN_CODES = {
invalid_json: 'a line is not valid JSON. Validate the file locally before '
+ 'upload: every line must parse on its own.',
duplicate_custom_id: 'two lines share a custom_id. They must be unique within '
+ 'the file, because results come back unordered and custom_id is the only '
+ 'join key.',
missing_required_parameter: 'a line is missing a required field. Each row needs '
+ 'custom_id, method, url and body.',
invalid_url: "a line's url does not match the batch endpoint. The two must "
+ 'agree for every row in the file.',
model_not_found: 'the body names a model this project cannot reach. Check the '
+ 'id against GET /v1/models with the same key.',
empty_file: 'the input file has no lines in it. The upload succeeded and the '
+ 'content did not.',
};
const FINDINGS = new Set(['validation-failed', 'orphan-input-files']);
/** Batches whose input file was rejected. Pure. */
export function failedBatches(batches) {
return (batches ?? []).filter((b) => (b ?? {}).status === 'failed');
}
/** Normalised entries from errors.data[]. Pure. Never throws on a shape. */
export function errorRows(batch) {
const errors = (batch ?? {}).errors;
const data = errors && typeof errors === 'object' ? errors.data : null;
const out = [];
for (const item of data ?? []) {
if (!item || typeof item !== 'object') continue;
const parsed = Number.parseInt(item.line, 10);
out.push({
code: String(item.code ?? 'unknown'),
message: String(item.message ?? ''),
param: item.param ?? null,
line: Number.isFinite(parsed) ? parsed : null,
});
}
return out;
}
/** {code: [lines, count, message, param]}. Pure. Sorted by code, then line. */
export function linesByCode(rows) {
const grouped = new Map();
for (const row of rows ?? []) {
if (!grouped.has(row.code)) {
grouped.set(row.code, { lines: new Set(), count: 0, message: '', param: null });
}
const slot = grouped.get(row.code);
slot.count += 1;
if (row.line !== null && row.line !== undefined) slot.lines.add(row.line);
if (!slot.message) slot.message = row.message ?? '';
if (slot.param === null) slot.param = row.param ?? null;
}
const out = {};
for (const code of [...grouped.keys()].sort()) {
const slot = grouped.get(code);
out[code] = [[...slot.lines].sort((a, b) => a - b), slot.count, slot.message,
slot.param];
}
return out;
}
/** True when the batch dispatched no requests at all. Pure. */
export function nothingBilled(batch) {
const counts = (batch ?? {}).request_counts ?? {};
return ['total', 'completed', 'failed']
.every((k) => (Number(counts[k]) || 0) === 0);
}
/** Every input_file_id the account has handed to a batch. Pure. */
export function batchInputIds(batches) {
const out = new Set();
for (const b of batches ?? []) {
if ((b ?? {}).input_file_id) out.add(String(b.input_file_id));
}
return out;
}
/** .jsonl files that can never be batch input and never were. Pure. */
export function mispurposedInputs(files, usedIds) {
const used = usedIds ?? new Set();
return (files ?? [])
.filter((f) => f && typeof f === 'object')
.filter((f) => String(f.filename ?? '').toLowerCase().endsWith('.jsonl'))
.filter((f) => NOT_BATCH_INPUT.has(String(f.purpose ?? '')))
.filter((f) => !used.has(String(f.id)))
.map((f) => ({ id: String(f.id), filename: String(f.filename ?? ''),
purpose: String(f.purpose ?? ''), bytes: Number(f.bytes) || 0 }))
.sort((a, b) => a.id.localeCompare(b.id));
}
/** True when the batch was created inside the window. Pure. */
export function withinWindow(batch, now, days) {
if (!days || days <= 0) return true;
const created = Number((batch ?? {}).created_at) || 0;
return created >= now - days * 86400;
}
/** Grade the run. Pure. Returns [state, detail]. */
export function verdict(failed, orphans, days) {
const f = (failed ?? []).length;
const o = (orphans ?? []).length;
const window = days && days > 0 ? `in the last ${days} days` : 'in the account';
if (f && o) {
return ['validation-failed',
`${f} batch(es) failed input validation ${window}, and ${o} .jsonl was `
+ 'uploaded under a purpose /v1/batches will not accept'];
}
if (f) {
return ['validation-failed',
`${f} batch(es) failed input validation ${window} and nothing polled them `
+ 'to find out'];
}
if (o) {
return ['orphan-input-files',
`${o} .jsonl file(s) sit under a purpose /v1/batches will not accept, `
+ 'referenced by no batch'];
}
return ['validation-clean',
`no batch ${window} failed validation, and every .jsonl in the file list `
+ 'either carries purpose=batch or was used by a batch'];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, codes) {
if (state === 'validation-clean') {
return ['nothing to change. Keep the assertion that a submitter only logs '
+ 'success once status has left "validating".'];
}
const lines = [];
for (const code of [...new Set(codes ?? [])].sort()) {
if (KNOWN_CODES[code]) lines.push(`${code}: ${KNOWN_CODES[code]}`);
}
if (state === 'validation-failed') {
lines.push('fix the input at the reported lines, then re-upload with '
+ 'purpose=batch and create the batch again. Nothing was billed, so '
+ 'nothing needs reconciling.');
lines.push('make the submitter poll. A 200 from batch creation is a receipt, '
+ 'not a result: the only honest success signal is a batch that has left '
+ '"validating".');
}
if (state === 'orphan-input-files') {
lines.push('re-upload each file with purpose=batch and delete the '
+ 'mis-purposed copy, which counts against project storage until you do.');
lines.push('assert in the upload helper that the purpose matches the '
+ 'endpoint that will consume the file.');
}
return lines;
}
async function getJson(url, key, params) {
const target = new URL(url);
for (const [k, v] of Object.entries(params ?? {})) {
target.searchParams.set(k, String(v));
}
let res;
try {
res = await fetch(target, { headers: { Authorization: `Bearer ${key}` } });
} catch (err) {
return [null, `request failed: ${err.message}`];
}
if (res.status !== 200) {
let detail = '';
try { detail = String((await res.json())?.error?.message ?? ''); } catch { detail = ''; }
return [null, `HTTP ${res.status} ${detail}`];
}
try {
return [await res.json(), null];
} catch {
return [null, 'response was not JSON'];
}
}
async function pageAll(url, key, params, maxPages) {
const rows = [];
let after = null;
for (let i = 0; i < Math.max(1, maxPages); i += 1) {
const query = { ...(params ?? {}) };
if (after) query.after = after;
const [payload, err] = await getJson(url, key, query);
if (err) return [rows, err];
const data = payload.data ?? [];
rows.push(...data);
if (!payload.has_more || !data.length) break;
after = data[data.length - 1]?.id;
if (!after) break;
}
return [rows, null];
}
function args(argv) {
const out = { sinceDays: 30, maxPages: 20 };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--since-days') out.sinceDays = Number.parseInt(argv[i += 1], 10);
else if (argv[i] === '--max-pages') out.maxPages = Number.parseInt(argv[i += 1], 10);
}
return out;
}
async function main() {
const opts = args(process.argv.slice(2));
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key set to Read Only');
process.exitCode = 2;
return;
}
const now = Math.floor(Date.now() / 1000);
const [batches, err] = await pageAll(BATCHES_URL, key, { limit: PAGE }, opts.maxPages);
if (err && !batches.length) {
console.error(`could not read the batch list: ${err}`);
process.exitCode = 2;
return;
}
if (err) console.log(`batch list stopped early: ${err}`);
const scoped = batches.filter((b) => withinWindow(b, now, opts.sinceDays));
const failed = failedBatches(scoped);
const seenCodes = [];
for (const b of failed) {
const billed = nothingBilled(b)
? 'nothing billed (0 requests)'
: 'request_counts is not all zero, which is unusual for failed';
const stamp = new Date((Number(b.failed_at || b.created_at) || 0) * 1000)
.toISOString().replace(/\.\d+Z$/, 'Z');
console.log(`${String(b.id).padEnd(16)} failed at ${stamp} ${billed}`);
const groups = linesByCode(errorRows(b));
if (!Object.keys(groups).length) {
console.log(' (the errors object is empty, so the reason is not readable '
+ 'from the API)');
}
for (const [code, [lines, , message, param]] of Object.entries(groups)) {
seenCodes.push(code);
const shown = lines.slice(0, 6).join(', ');
const more = lines.length > 6 ? ` and ${lines.length - 6} more` : '';
const where = lines.length ? `lines ${shown}${more}` : 'no line given';
const extra = param ? ` param ${param}` : '';
console.log(` ${code.padEnd(26)} ${where}${extra}`);
if (message) console.log(` ${''.padEnd(26)} ${message.slice(0, 140)}`);
}
}
const [files, ferr] = await pageAll(FILES_URL, key, { limit: 10000 }, opts.maxPages);
if (ferr) console.log(`file list stopped early: ${ferr}`);
const orphans = mispurposedInputs(files, batchInputIds(batches));
for (const row of orphans) {
console.log(`orphan-input ${row.id} ${row.filename} purpose=${row.purpose} `
+ `${(row.bytes / 1048576).toFixed(1)} MB`);
}
const [state, detail] = verdict(failed, orphans, opts.sinceDays);
console.log(`${state.padEnd(20)} ${detail}`);
console.log(' measured: status, errors.data[] and request_counts from the '
+ 'batch list, purpose from the file list');
console.log(' inferred: that the pipeline never polled, since a failed batch '
+ 'is otherwise indistinguishable from one nobody re-ran on purpose');
for (const line of repairLines(state, seenCodes)) console.log(` repair: ${line}`);
const total = failed.length + orphans.length;
console.log(`${total} finding(s)`);
process.exitCode = total ? 1 : 0;
void FINDINGS;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the claim the whole note rests on: a batch whose status is failed is a batch that never ran, so nothing_billed must be true for it and the errors must group by code with their line numbers intact. The second is the shape defence — errors absent, errors.data null, an entry that is a string rather than an object, and a line of null all have to degrade to no rows instead of an exception, because a script that crashes on the second batch never reports the first. The third is the three-condition filter on mis-purposed files, with a case for each condition removed, since the version that flags every input file you ever used is the version somebody switches off. The fourth checks that a completed batch with failed rows is never picked up here, which is the boundary against the published partial-failure note. Then the window arithmetic, and last the repair text, where a duplicate custom_id gets the documented fix rather than a generic one.
from openai_batch_validation_audit import (batch_input_ids, error_rows,
failed_batches, lines_by_code,
mispurposed_inputs, nothing_billed,
repair_lines, verdict, within_window)
NOW = 1_800_000_000
FAILED = {
"id": "batch_aa",
"status": "failed",
"created_at": NOW - 3600,
"failed_at": NOW - 3560,
"input_file_id": "file_in1",
"request_counts": {"total": 0, "completed": 0, "failed": 0},
"errors": {"object": "list", "data": [
{"code": "invalid_json", "message": "not valid JSON", "param": None,
"line": 41207},
{"code": "invalid_json", "message": "not valid JSON", "param": None,
"line": 41208},
{"code": "duplicate_custom_id", "message": "custom_id repeated",
"param": "custom_id", "line": 903},
]},
}
RAN = {
"id": "batch_bb",
"status": "completed",
"created_at": NOW - 7200,
"input_file_id": "file_in2",
"request_counts": {"total": 900, "completed": 880, "failed": 20},
"output_file_id": "file_out2",
}
def test_a_failed_batch_never_ran_and_names_its_lines():
assert [b["id"] for b in failed_batches([FAILED, RAN])] == ["batch_aa"]
# Validation happens before dispatch, so the reassuring half is provable.
assert nothing_billed(FAILED)
assert not nothing_billed(RAN)
groups = lines_by_code(error_rows(FAILED))
assert groups["invalid_json"][0] == [41207, 41208]
assert groups["invalid_json"][1] == 2
assert groups["duplicate_custom_id"][0] == [903]
assert groups["duplicate_custom_id"][3] == "custom_id"
def test_every_field_in_the_errors_object_is_allowed_to_be_missing():
assert error_rows({"status": "failed"}) == []
assert error_rows({"errors": None}) == []
assert error_rows({"errors": {"data": None}}) == []
assert error_rows({"errors": {"data": ["not an object"]}}) == []
rows = error_rows({"errors": {"data": [{"code": None, "line": None}]}})
assert rows == [{"code": "unknown", "message": "", "param": None,
"line": None}]
# A code with no line still gets a row, worded so nobody hunts for line 0.
assert lines_by_code(rows)["unknown"][0] == []
# An absent request_counts is what a batch that never started looks like.
assert nothing_billed({"status": "failed"})
def test_a_mispurposed_input_needs_all_three_conditions():
files = [
{"id": "file_x", "filename": "nightly.jsonl", "purpose": "user_data",
"bytes": 1400000},
{"id": "file_ok", "filename": "nightly.jsonl", "purpose": "batch",
"bytes": 1400000},
{"id": "file_in2", "filename": "used.jsonl", "purpose": "user_data",
"bytes": 10},
{"id": "file_img", "filename": "photo.png", "purpose": "vision",
"bytes": 900},
{"id": "file_res", "filename": "out.jsonl", "purpose": "batch_output",
"bytes": 50},
]
used = batch_input_ids([FAILED, RAN])
assert used == {"file_in1", "file_in2"}
found = mispurposed_inputs(files, used)
assert [r["id"] for r in found] == ["file_x"]
assert found[0]["purpose"] == "user_data"
# Outputs are not inputs, and a file that was used is never a finding.
assert mispurposed_inputs(files, {"file_x"} | used) == []
def test_rows_that_failed_inside_a_batch_that_ran_belong_to_another_note():
# request_counts.failed > 0 on a completed batch is the published
# partial-failure note. This script must not claim it.
assert failed_batches([RAN]) == []
state, detail = verdict([], [], 30)
assert state == "validation-clean"
assert "no batch in the last 30 days" in detail
def test_the_window_is_arithmetic_on_created_at_and_zero_means_everything():
assert within_window(FAILED, NOW, 30)
assert not within_window(FAILED, NOW + 40 * 86400, 30)
assert within_window(FAILED, NOW + 40 * 86400, 0)
assert not within_window({"created_at": "nonsense"}, NOW, 30)
def test_the_repair_names_the_documented_fix_for_the_code_it_saw():
state, detail = verdict([FAILED], [{"id": "file_x"}], 30)
assert state == "validation-failed"
assert "will not accept" in detail
lines = repair_lines(state, ["duplicate_custom_id", "invalid_json", "made_up"])
assert any("custom_id is the only join key" in line for line in lines)
assert any("every line must parse on its own" in line for line in lines)
assert not any("made_up" in line for line in lines)
assert any("receipt, not a result" in line for line in lines)
assert repair_lines("validation-clean", [])[0].startswith("nothing to change")
orphan_only = verdict([], [{"id": "file_x"}], 0)
assert orphan_only[0] == "orphan-input-files"
assert any("purpose matches the endpoint"
in line for line in repair_lines(orphan_only[0], []))
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { batchInputIds, errorRows, failedBatches, linesByCode, mispurposedInputs,
nothingBilled, repairLines, verdict,
withinWindow } from './openai-batch-validation-audit.mjs';
const NOW = 1800000000;
const FAILED = {
id: 'batch_aa',
status: 'failed',
created_at: NOW - 3600,
failed_at: NOW - 3560,
input_file_id: 'file_in1',
request_counts: { total: 0, completed: 0, failed: 0 },
errors: { object: 'list', data: [
{ code: 'invalid_json', message: 'not valid JSON', param: null, line: 41207 },
{ code: 'invalid_json', message: 'not valid JSON', param: null, line: 41208 },
{ code: 'duplicate_custom_id', message: 'custom_id repeated',
param: 'custom_id', line: 903 },
] },
};
const RAN = {
id: 'batch_bb',
status: 'completed',
created_at: NOW - 7200,
input_file_id: 'file_in2',
request_counts: { total: 900, completed: 880, failed: 20 },
output_file_id: 'file_out2',
};
test('a failed batch never ran and names its lines', () => {
assert.deepEqual(failedBatches([FAILED, RAN]).map((b) => b.id), ['batch_aa']);
assert.ok(nothingBilled(FAILED));
assert.ok(!nothingBilled(RAN));
const groups = linesByCode(errorRows(FAILED));
assert.deepEqual(groups.invalid_json[0], [41207, 41208]);
assert.equal(groups.invalid_json[1], 2);
assert.deepEqual(groups.duplicate_custom_id[0], [903]);
assert.equal(groups.duplicate_custom_id[3], 'custom_id');
});
test('every field in the errors object is allowed to be missing', () => {
assert.deepEqual(errorRows({ status: 'failed' }), []);
assert.deepEqual(errorRows({ errors: null }), []);
assert.deepEqual(errorRows({ errors: { data: null } }), []);
assert.deepEqual(errorRows({ errors: { data: ['not an object'] } }), []);
const rows = errorRows({ errors: { data: [{ code: null, line: null }] } });
assert.deepEqual(rows, [{ code: 'unknown', message: '', param: null, line: null }]);
assert.deepEqual(linesByCode(rows).unknown[0], []);
assert.ok(nothingBilled({ status: 'failed' }));
});
test('a mispurposed input needs all three conditions', () => {
const files = [
{ id: 'file_x', filename: 'nightly.jsonl', purpose: 'user_data', bytes: 1400000 },
{ id: 'file_ok', filename: 'nightly.jsonl', purpose: 'batch', bytes: 1400000 },
{ id: 'file_in2', filename: 'used.jsonl', purpose: 'user_data', bytes: 10 },
{ id: 'file_img', filename: 'photo.png', purpose: 'vision', bytes: 900 },
{ id: 'file_res', filename: 'out.jsonl', purpose: 'batch_output', bytes: 50 },
];
const used = batchInputIds([FAILED, RAN]);
assert.deepEqual([...used].sort(), ['file_in1', 'file_in2']);
const found = mispurposedInputs(files, used);
assert.deepEqual(found.map((r) => r.id), ['file_x']);
assert.equal(found[0].purpose, 'user_data');
assert.deepEqual(mispurposedInputs(files, new Set([...used, 'file_x'])), []);
});
test('rows that failed inside a batch that ran belong to another note', () => {
assert.deepEqual(failedBatches([RAN]), []);
const [state, detail] = verdict([], [], 30);
assert.equal(state, 'validation-clean');
assert.ok(detail.includes('no batch in the last 30 days'));
});
test('the window is arithmetic on created_at and zero means everything', () => {
assert.ok(withinWindow(FAILED, NOW, 30));
assert.ok(!withinWindow(FAILED, NOW + 40 * 86400, 30));
assert.ok(withinWindow(FAILED, NOW + 40 * 86400, 0));
assert.ok(!withinWindow({ created_at: 'nonsense' }, NOW, 30));
});
test('the repair names the documented fix for the code it saw', () => {
const [state, detail] = verdict([FAILED], [{ id: 'file_x' }], 30);
assert.equal(state, 'validation-failed');
assert.ok(detail.includes('will not accept'));
const lines = repairLines(state, ['duplicate_custom_id', 'invalid_json', 'made_up']);
assert.ok(lines.some((l) => l.includes('custom_id is the only join key')));
assert.ok(lines.some((l) => l.includes('every line must parse on its own')));
assert.ok(!lines.some((l) => l.includes('made_up')));
assert.ok(lines.some((l) => l.includes('receipt, not a result')));
assert.ok(repairLines('validation-clean', [])[0].startsWith('nothing to change'));
const [orphanState] = verdict([], [{ id: 'file_x' }], 0);
assert.equal(orphanState, 'orphan-input-files');
assert.ok(repairLines(orphanState, []).some((l) => l.includes('purpose matches the endpoint')));
});
FAQ
Does a failed batch cost anything?
No. Validation runs while the batch is in the <code>validating</code> state, before any request is dispatched to a model, so a batch that lands on <code>failed</code> has a <code>request_counts</code> of all zeros and no tokens attached to it. The script prints that alongside every failure, because it is the first question anybody asks and getting it wrong in either direction wastes an afternoon. What the failure does cost you is the work that did not happen, which is usually the more expensive half.
What is the difference between this and a batch that reads completed with failed rows?
Everything, including the repair. <code>failed</code> is a statement about the input file: it was rejected as a whole and nothing ran. A batch that reads <code>completed</code> with a non-zero <code>request_counts.failed</code> did run, was billed for the rows that succeeded, and wrote its failures to an error file. The first is fixed by editing a <code>.jsonl</code> and re-creating the batch; the second is fixed by reading <code>error_file_id</code> and retrying the rows that are worth retrying. Two of the other notes in this section own that second case.
Why does the script read the file list as well as the batch list?
Because one version of this failure never produces a batch object at all. <code>/v1/batches</code> accepts an input file only when its <code>purpose</code> is exactly <code>batch</code>, and a file uploaded under <code>user_data</code> or <code>assistants</code> is refused at creation. There is no batch to list and no <code>errors</code> array to read: the only evidence anywhere in the API is a <code>.jsonl</code> in the file list that nothing references. That is why the check needs all three conditions — the extension, the wrong purpose, and no batch pointing at it.
Can I filter the batch list to failed batches server-side?
No. <code>GET /v1/batches</code> takes <code>limit</code>, which ranges from 1 to 100 and defaults to 20, and <code>after</code> as a cursor. There is no status filter, no date range and no metadata query, so every bit of selection happens on your side of the wire after you have paged the list. The practical consequence is that a run which reads only the default first page sees the last twenty batches, and on a nightly pipeline that is about three weeks of history missed.
The errors object is empty on one of my failed batches. What then?
Then the reason is not readable from the API and the script says so in those words rather than printing an empty list. <code>errors</code> is an optional field, its <code>data</code> array is optional, and <code>line</code> is documented as present only where applicable. When there is nothing there, the remaining evidence is the input file itself: validate it locally, checking that every line parses, that <code>custom_id</code> values are unique, that each row's <code>url</code> matches the batch endpoint, and that the model in each body is one this project can reach.
Related field notes
- A batch that ran, reads completed, and has failed rows inside it
- Work you paid for and never collected, on both providers
- The clock that closes on a batch before it finishes
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Batch — OpenAI API reference
- Batch API guide — OpenAI developer docs
- Files — OpenAI API reference
- Error codes — OpenAI platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.