Diagnostic LLM APIs
The batch queue is full, so the next submission is refused
Nothing has failed. Every batch in the account is healthy, every one of them is inside its window, and the Messages API limits are not being touched. The only symptom is that the submitter has started getting 429s, and it is getting them for a reason that has nothing to do with the requests it is sending: an unrelated job, in an unrelated workspace, has parked four hundred thousand requests in a queue that the whole organization shares.
Read the ceiling, then measure the depth. The ceiling comes from the Admin API: GET /v1/organizations/rate_limits?group_type=batch returns a group whose models is null and whose limits array holds a single entry, {\"type\": \"enqueued_batch_requests\", \"value\": N}. That is the number of batch requests your organization may have waiting at once, and it is shared across every model.
The depth comes from the data plane: GET /v1/messages/batches?limit=1000 with a workspace key, and the sum of request_counts.processing over every batch whose processing_status is in_progress or canceling. The documentation defines the queue precisely — a batch request is part of the processing queue when it has yet to be successfully processed by the model — and processing is that count.
Depth against ceiling is the whole reading, and it needs two different credentials. Admin keys cannot reach the Messages Batches endpoints and workspace keys are rejected by every Admin endpoint, so a run with only one of them gets half the answer. The script says which half it is missing rather than reporting a ratio it could not compute.
One honest caveat is printed with every result. The ceiling is organization wide; the batch list is workspace scoped. A single workspace key therefore measures that workspace's contribution, which is a lower bound on the number that matters. Pass more workspace keys and the bound tightens.
The problem in plain words
The Message Batches API has its own rate limits, and they are not the ones anybody watches. There is a requests-per-minute limit on the batch endpoints, a cap on how many batch requests may sit in the processing queue at once, and a cap on the size of a single batch. All of them are shared across every model, which is the part that surprises people: moving a job to a different model does not move it to a different queue.
The queue cap is the one that bites, because it is a property of the organization rather than of the job being submitted. Enqueued batch requests run 200,000 on Start, 300,000 on Build and 500,000 on Scale. One nightly backfill that submits four hundred thousand rows at once can hold most of that for hours, and every other team's submission is refused for the duration — a 429 on /v1/messages/batches with the Messages API sitting idle.
What makes it hard to diagnose is that the refusal and the cause are in different places. The team that gets the 429 is not the team holding the queue, the error names a limit rather than a batch, and there is no view anywhere that says who is using the capacity. Meanwhile the ordinary rate-limit dashboards look fine, because they are about tokens per minute on the Messages API and this has nothing to do with either.
The second-order effect is worse than the refusal. A queue held at the ceiling slows everything behind it, and the documentation notes plainly that when processing is slowed by demand and request volume, you may see more requests expiring after 24 hours. So a full queue does not only reject new work, it puts existing work at risk of running out of window — which is a different note, and a different symptom, produced by this cause.
Why it happens
The ceiling is readable, which is unusual and worth the extra credential. Most limits in this section have to be inferred from response headers on a live call. This one is a stored configuration you can GET: the Rate Limits API returns your organization's configured groups, and the batch group carries enqueued_batch_requests as a plain number. That turns a guess into arithmetic, and it is the reason this note reads an Admin endpoint at all.
The queue is defined by a field, not by a status. It would be natural to count requests in every unfinished batch, but the documented definition is narrower: a batch request is in the processing queue when it has yet to be successfully processed by the model. request_counts.processing is exactly that number, and it is documented as starting at the full request count and moving to the other buckets only once processing of the whole batch ends. So the sum of processing over live batches is the measurement, and adding the succeeded or errored counts would overstate it.
Two credentials, two scopes, and the script refuses to paper over the gap. Anthropic's Admin API rejects workspace-scoped keys outright, and the Messages Batches endpoints are unreachable from an Admin key. The consequence is structural: the ceiling you can read is organization wide and the depth you can read is per workspace. A single-workspace measurement is a lower bound, the output labels it as one, and passing additional workspace keys is how you tighten it rather than a nicety.
Occupancy is not expiry, and the two notes stay apart. A batch drifting toward its 24 hour expires_at with work still unprocessed is the published expiry note; this one never looks at expires_at and never grades a batch as late. The reading here is a single number about the organization, taken at a moment, and its finding is about submissions that will be refused rather than about rows that will be lost.
On OpenAI the equivalent ceiling is not readable at all, and the note says so instead of guessing. OpenAI's Batch API also has a per-model cap on prompt tokens queued for batch processing, plus a limit of 2,000 batch creations per hour and 50,000 requests per batch. None of those is exposed by any endpoint: the token cap lives on the platform limits page and the only signal a script gets is the error text on a submission it is not allowed to make. So this script is Anthropic only, deliberately, rather than half-implemented on both.
The fix, as a flow
Nothing in this reading is broken. Every batch is healthy and inside its window, and the failure lands on somebody else: a submission refused with a 429 while the Messages API sits idle. The ceiling is one number on an Admin endpoint and the depth is a sum over a workspace's live batches, which means the two halves need two credentials that cannot read each other's data.
How to fix it
Provision the two keys
ANTHROPIC_ADMIN_KEY for GET /v1/organizations/rate_limits, which needs an Admin key or a personal or service-account key that is not scoped to a workspace. ANTHROPIC_API_KEY as a workspace key for GET /v1/messages/batches. Neither endpoint accepts the other's credential, so both are needed for a complete answer.
Read the ceiling
?group_type=batch narrows the response to one group. Its models is null, because batch limits are shared across every model rather than attached to one, and its limits array holds enqueued_batch_requests. Follow next_page if it is ever non-null; today the response is a single page.
Sum the live queue
Page /v1/messages/batches on after_id at up to 1000 per page, keep processing_status of in_progress or canceling, and add up request_counts.processing. Batches that have ended hold nothing in the queue, whatever their other counts say.
Add the other workspaces
ANTHROPIC_EXTRA_WORKSPACE_KEYS, comma separated, adds more workspace keys to the same sum. Every key you leave out makes the measurement a looser lower bound against an organization-wide ceiling, and the output prints how many workspaces the number covers so nobody reads it as complete when it is not.
Name the biggest holder and drain before enqueuing
The output ranks the live batches by processing so the conversation has a subject. The repair is printed and it is a scheduling change, not a cancellation: hold at most a few batches in flight, wait for one to end before submitting the next, and split a single enormous submission rather than parking it whole.
How to check it worked
Re-run during the window when submissions were being refused, not afterwards. This is a live measurement of a queue that drains, so a clean result at ten in the morning says nothing about two o'clock. The useful form is a scheduled run every few minutes through the batch window, alarming when occupancy crosses the threshold rather than when a submission has already been rejected.
python3 anthropic_batch_queue_depth.py --threshold 80
# ceiling enqueued_batch_requests 300,000 (organization wide)
# msgbatch_01Rf in_progress 214,900 processing
# msgbatch_01Qa in_progress 58,400 processing
# msgbatch_01Zc canceling 9,600 processing
# queue-near-limit 282,900 of 300,000 enqueued batch requests are in the
# processing queue, which is 94% of the ceiling
# measured: enqueued_batch_requests from the Rate Limits API, and the sum of
# request_counts.processing over 3 live batches in 1 workspace
# inferred: nothing about other workspaces. The ceiling is organization wide
# and this depth is a lower bound on it
# repair: hold at most a few batches in flight and wait for one to end before
# submitting the next. A batch request leaves the queue only when the
# model has processed it.
# repair: msgbatch_01Rf alone holds 214,900 of the 300,000. Split submissions
# of that size: the per batch cap is 100,000 requests or 256 MB,
# whichever comes first.
# 1 finding(s)The full code
Two GETs against two different credentials, and seven pure functions. enqueued_limit, which digs the one number out of a nested group-and-limits structure and returns None rather than a zero when it is absent, because zero would read as a ceiling of nothing; queue_rows, which keeps only the two non-terminal statuses and takes processing rather than any other count, since that is the field the documented definition of the queue points at; queue_depth, which is the sum; headroom, which returns the remaining requests and the occupancy ratio and refuses to divide by a missing limit; top_holders, which gives the finding a subject; workspace_keys, which de-duplicates the credentials so the same workspace is never counted twice; and verdict, which treats an unreadable ceiling as a finding in its own right.
"""Measure live batch queue depth against the organization's enqueued ceiling.
Read only. Two GET endpoints on two different credentials:
GET /v1/organizations/rate_limits?group_type=batch with an Admin key for the
ceiling, and GET /v1/messages/batches with a workspace key for the depth.
Nothing is submitted and nothing is cancelled.
The Message Batches API has its own limits, shared across all models. The one
this measures is the number of batch requests allowed in the processing queue
at once. A batch request is part of that queue when it has yet to be
successfully processed by the model, which is exactly request_counts.processing.
Scope caveat, printed with every result: the ceiling is organization wide and
the batch list is workspace scoped, so a single workspace key produces a lower
bound. Extra workspace keys tighten it.
This is Anthropic only on purpose. OpenAI's equivalent enqueued-token cap is
not returned by any endpoint, so a read-only script cannot compute the ratio
there at all.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_batch_queue_depth")
RATE_LIMITS_URL = "https://api.anthropic.com/v1/organizations/rate_limits"
BATCHES_URL = "https://api.anthropic.com/v1/messages/batches"
# The two non-terminal processing_status values. A batch that has ended holds
# nothing in the queue whatever its other counts say.
LIVE_STATES = ("in_progress", "canceling")
# Documented at every tier, and the same at every tier, which is why it is safe
# to print as context rather than looked up per organization.
PER_BATCH_REQUESTS = 100000
PER_BATCH_MB = 256
FINDINGS = ("queue-exhausted", "queue-near-limit", "queue-limit-unknown")
def enqueued_limit(payload):
"""The enqueued_batch_requests value, or None. Pure.
None rather than zero when it is missing. Zero would read as a ceiling of
nothing and turn every run into a false alarm at infinite occupancy.
"""
for group in (payload or {}).get("data") or []:
if not isinstance(group, dict):
continue
if group.get("group_type") != "batch":
continue
for limit in group.get("limits") or []:
if isinstance(limit, dict) and limit.get("type") == "enqueued_batch_requests":
try:
return int(limit.get("value"))
except (TypeError, ValueError):
return None
return None
def queue_rows(batches, workspace=""):
"""Live batches and what each holds in the queue. Pure.
processing, and only processing. Adding succeeded or errored would count
requests the model has already finished with, which are not in the queue.
"""
out = []
for b in batches or []:
status = str((b or {}).get("processing_status") or "")
if status not in LIVE_STATES:
continue
counts = b.get("request_counts") or {}
try:
processing = int(counts.get("processing") or 0)
except (TypeError, ValueError):
processing = 0
out.append({"id": str(b.get("id")), "status": status,
"processing": processing, "workspace": workspace})
return sorted(out, key=lambda r: (-r["processing"], r["id"]))
def queue_depth(rows):
"""Total requests waiting on the model. Pure."""
return sum(int(r.get("processing") or 0) for r in rows or [])
def headroom(depth, limit):
"""(remaining, occupancy) or (None, None) when the ceiling is unknown. Pure."""
if limit is None or limit <= 0:
return (None, None)
return (max(0, limit - depth), depth / float(limit))
def top_holders(rows, n=3):
"""The n biggest contributors. Pure. Gives the finding a subject."""
return [r for r in (rows or [])[:max(0, n)] if int(r.get("processing") or 0) > 0]
def workspace_keys(primary, extra):
"""Deduplicated workspace credentials. Pure. Order kept.
The same key passed twice would double the measured depth, which is the one
error in this script that would look like a real finding.
"""
out, seen = [], set()
for candidate in [primary] + str(extra or "").split(","):
key = (candidate or "").strip()
if key and key not in seen:
seen.add(key)
out.append(key)
return out
def verdict(depth, limit, rows, workspaces, threshold):
"""Grade the run. Pure. Returns (state, detail)."""
remaining, occupancy = headroom(depth, limit)
if limit is None:
return ("queue-limit-unknown",
"%d batch requests are in the processing queue across %d "
"workspace(s), but the enqueued_batch_requests ceiling could "
"not be read, so there is no headroom to report"
% (depth, workspaces))
percent = int(round(occupancy * 100))
if depth >= limit:
return ("queue-exhausted",
"%d of %d enqueued batch requests are in the processing queue, "
"which is the whole ceiling. New submissions are being refused"
% (depth, limit))
if percent >= threshold:
return ("queue-near-limit",
"%d of %d enqueued batch requests are in the processing queue, "
"which is %d%% of the ceiling" % (depth, limit, percent))
return ("queue-clear",
"%d of %d enqueued batch requests are in the processing queue, "
"leaving %d requests of headroom across %d live batch(es)"
% (depth, limit, remaining, len(rows or [])))
def repair_lines(state, rows, limit):
"""The repair for one verdict. Pure. Printed, never performed."""
if state == "queue-clear":
return ["nothing to change. Keep the check running through the batch "
"window rather than once a day: this is a queue that drains."]
if state == "queue-limit-unknown":
return ["read the ceiling with an Admin key: GET "
"/v1/organizations/rate_limits?group_type=batch returns "
"enqueued_batch_requests for the organization. Workspace keys "
"are rejected by every Admin endpoint.",
"without the ceiling this run is a raw count. It cannot tell "
"you whether the next submission will be accepted."]
lines = ["hold at most a few batches in flight and wait for one to end "
"before submitting the next. A batch request leaves the queue "
"only when the model has processed it."]
biggest = top_holders(rows, 1)
if biggest and limit:
lines.append("%s alone holds %d of the %d. Split submissions of that "
"size: the per batch cap is %d requests or %d MB, "
"whichever comes first."
% (biggest[0]["id"], biggest[0]["processing"], limit,
PER_BATCH_REQUESTS, PER_BATCH_MB))
lines.append("a queue held at the ceiling also slows what is already in it, "
"and slowed batches are the ones that run out of their 24 hour "
"window. Draining is the fix for both.")
return lines
def get_json(url, headers, params=None, timeout=30):
"""One GET. Returns (payload, error). Read only, always."""
try:
r = requests.get(url, headers=headers, params=params or {}, timeout=timeout)
except requests.RequestException as exc:
return (None, "request failed: %s" % exc)
if r.status_code != 200:
return (None, "HTTP %d %s" % (r.status_code, (r.text or "")[:160]))
try:
return (r.json(), None)
except ValueError:
return (None, "response was not JSON")
def read_ceiling(admin_key, max_pages=5):
"""The organization's enqueued_batch_requests. Returns (limit, error)."""
headers = {"x-api-key": admin_key, "anthropic-version": "2023-06-01"}
params = {"group_type": "batch"}
for _ in range(max(1, max_pages)):
payload, err = get_json(RATE_LIMITS_URL, headers, params)
if err:
return (None, err)
found = enqueued_limit(payload)
if found is not None:
return (found, None)
nxt = payload.get("next_page")
if not nxt:
return (None, "no batch group in the rate limits response")
params = {"group_type": "batch", "page": nxt}
return (None, "the rate limits response never carried a batch group")
def read_batches(key, max_pages=20):
"""One workspace's batches. Returns (rows, error). GETs only."""
headers = {"x-api-key": key, "anthropic-version": "2023-06-01"}
rows, after = [], None
for _ in range(max(1, max_pages)):
params = {"limit": 1000}
if after:
params["after_id"] = after
payload, err = get_json(BATCHES_URL, headers, params)
if err:
return (rows, err)
data = payload.get("data") or []
rows.extend(data)
if not payload.get("has_more") or not data:
break
after = payload.get("last_id") or data[-1].get("id")
if not after:
break
return (rows, None)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--threshold", type=int, default=80,
help="percent occupancy at which the queue is a finding")
ap.add_argument("--max-pages", type=int, default=20)
args = ap.parse_args()
admin_key = os.environ.get("ANTHROPIC_ADMIN_KEY")
keys = workspace_keys(os.environ.get("ANTHROPIC_API_KEY"),
os.environ.get("ANTHROPIC_EXTRA_WORKSPACE_KEYS"))
if not keys:
log.error("set ANTHROPIC_API_KEY to a workspace key. Add "
"ANTHROPIC_EXTRA_WORKSPACE_KEYS as a comma separated list to "
"cover more of the organization")
return 2
limit = None
if admin_key:
limit, err = read_ceiling(admin_key)
if err:
log.warning("could not read the ceiling: %s", err)
else:
log.warning("no ANTHROPIC_ADMIN_KEY, so the enqueued_batch_requests "
"ceiling cannot be read and only the raw depth is available")
if limit is not None:
log.info("%-12s enqueued_batch_requests %s (organization wide)",
"ceiling", format(limit, ","))
rows = []
for index, key in enumerate(keys):
batches, err = read_batches(key, args.max_pages)
if err:
log.warning("workspace %d batch list stopped early: %s", index + 1, err)
rows.extend(queue_rows(batches, workspace="ws%d" % (index + 1)))
rows.sort(key=lambda r: (-r["processing"], r["id"]))
for row in rows:
log.info("%-16s %-13s %s processing", row["id"][:16], row["status"],
format(row["processing"], ","))
depth = queue_depth(rows)
state, detail = verdict(depth, limit, rows, len(keys), args.threshold)
emit = log.warning if state in FINDINGS else log.info
emit("%-20s %s", state, detail)
emit(" measured: enqueued_batch_requests from the Rate Limits API, and the "
"sum of request_counts.processing over %d live batch(es) in %d "
"workspace(s)", len(rows), len(keys))
emit(" inferred: nothing about workspaces whose keys were not supplied. "
"The ceiling is organization wide and this depth is a lower bound on it")
for line in repair_lines(state, rows, limit):
emit(" repair: %s", line)
log.info("%d finding(s)", 1 if state in FINDINGS else 0)
return 1 if state in FINDINGS else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Measure live batch queue depth against the organization's enqueued ceiling.
*
* Read only. GET /v1/organizations/rate_limits?group_type=batch with an Admin
* key for the ceiling, GET /v1/messages/batches with a workspace key for the
* depth. Nothing is submitted and nothing is cancelled.
*
* A batch request is part of the processing queue when it has yet to be
* successfully processed by the model, which is request_counts.processing.
*
* The ceiling is organization wide and the batch list is workspace scoped, so
* a single workspace key produces a lower bound. Extra keys tighten it.
*/
const RATE_LIMITS_URL = 'https://api.anthropic.com/v1/organizations/rate_limits';
const BATCHES_URL = 'https://api.anthropic.com/v1/messages/batches';
export const LIVE_STATES = new Set(['in_progress', 'canceling']);
export const PER_BATCH_REQUESTS = 100000;
export const PER_BATCH_MB = 256;
const FINDINGS = new Set(['queue-exhausted', 'queue-near-limit', 'queue-limit-unknown']);
/** The enqueued_batch_requests value, or null. Pure. */
export function enqueuedLimit(payload) {
for (const group of (payload ?? {}).data ?? []) {
if (!group || typeof group !== 'object') continue;
if (group.group_type !== 'batch') continue;
for (const limit of group.limits ?? []) {
if (limit && typeof limit === 'object' && limit.type === 'enqueued_batch_requests') {
const value = Number(limit.value);
return Number.isFinite(value) ? Math.trunc(value) : null;
}
}
}
return null;
}
/** Live batches and what each holds in the queue. Pure. */
export function queueRows(batches, workspace = '') {
return (batches ?? [])
.filter((b) => LIVE_STATES.has(String((b ?? {}).processing_status ?? '')))
.map((b) => ({
id: String(b.id),
status: String(b.processing_status),
processing: Number((b.request_counts ?? {}).processing) || 0,
workspace,
}))
.sort((a, b) => (b.processing - a.processing) || a.id.localeCompare(b.id));
}
/** Total requests waiting on the model. Pure. */
export function queueDepth(rows) {
return (rows ?? []).reduce((n, r) => n + (Number(r?.processing) || 0), 0);
}
/** [remaining, occupancy] or [null, null] when the ceiling is unknown. Pure. */
export function headroom(depth, limit) {
if (limit === null || limit === undefined || limit <= 0) return [null, null];
return [Math.max(0, limit - depth), depth / limit];
}
/** The n biggest contributors. Pure. */
export function topHolders(rows, n = 3) {
return (rows ?? []).slice(0, Math.max(0, n)).filter((r) => (Number(r.processing) || 0) > 0);
}
/** Deduplicated workspace credentials. Pure. Order kept. */
export function workspaceKeys(primary, extra) {
const out = [];
const seen = new Set();
for (const candidate of [primary, ...String(extra ?? '').split(',')]) {
const key = (candidate ?? '').trim();
if (key && !seen.has(key)) {
seen.add(key);
out.push(key);
}
}
return out;
}
/** Grade the run. Pure. Returns [state, detail]. */
export function verdict(depth, limit, rows, workspaces, threshold) {
const [remaining, occupancy] = headroom(depth, limit);
if (limit === null || limit === undefined) {
return ['queue-limit-unknown',
`${depth} batch requests are in the processing queue across ${workspaces} `
+ 'workspace(s), but the enqueued_batch_requests ceiling could not be read, '
+ 'so there is no headroom to report'];
}
const percent = Math.round(occupancy * 100);
if (depth >= limit) {
return ['queue-exhausted',
`${depth} of ${limit} enqueued batch requests are in the processing queue, `
+ 'which is the whole ceiling. New submissions are being refused'];
}
if (percent >= threshold) {
return ['queue-near-limit',
`${depth} of ${limit} enqueued batch requests are in the processing queue, `
+ `which is ${percent}% of the ceiling`];
}
return ['queue-clear',
`${depth} of ${limit} enqueued batch requests are in the processing queue, `
+ `leaving ${remaining} requests of headroom across ${(rows ?? []).length} `
+ 'live batch(es)'];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, rows, limit) {
if (state === 'queue-clear') {
return ['nothing to change. Keep the check running through the batch window '
+ 'rather than once a day: this is a queue that drains.'];
}
if (state === 'queue-limit-unknown') {
return ['read the ceiling with an Admin key: GET '
+ '/v1/organizations/rate_limits?group_type=batch returns '
+ 'enqueued_batch_requests for the organization. Workspace keys are '
+ 'rejected by every Admin endpoint.',
'without the ceiling this run is a raw count. It cannot tell you whether '
+ 'the next submission will be accepted.'];
}
const lines = ['hold at most a few batches in flight and wait for one to end '
+ 'before submitting the next. A batch request leaves the queue only when '
+ 'the model has processed it.'];
const biggest = topHolders(rows, 1);
if (biggest.length && limit) {
lines.push(`${biggest[0].id} alone holds ${biggest[0].processing} of the `
+ `${limit}. Split submissions of that size: the per batch cap is `
+ `${PER_BATCH_REQUESTS} requests or ${PER_BATCH_MB} MB, whichever comes first.`);
}
lines.push('a queue held at the ceiling also slows what is already in it, and '
+ 'slowed batches are the ones that run out of their 24 hour window. '
+ 'Draining is the fix for both.');
return lines;
}
async function getJson(url, headers, params) {
const target = new URL(url);
for (const [k, v] of Object.entries(params ?? {})) target.searchParams.set(k, String(v));
let res;
try {
res = await fetch(target, { headers });
} catch (err) {
return [null, `request failed: ${err.message}`];
}
if (res.status !== 200) return [null, `HTTP ${res.status}`];
try {
return [await res.json(), null];
} catch {
return [null, 'response was not JSON'];
}
}
async function readCeiling(adminKey, maxPages = 5) {
const headers = { 'x-api-key': adminKey, 'anthropic-version': '2023-06-01' };
let params = { group_type: 'batch' };
for (let i = 0; i < Math.max(1, maxPages); i += 1) {
const [payload, err] = await getJson(RATE_LIMITS_URL, headers, params);
if (err) return [null, err];
const found = enqueuedLimit(payload);
if (found !== null) return [found, null];
if (!payload.next_page) return [null, 'no batch group in the rate limits response'];
params = { group_type: 'batch', page: payload.next_page };
}
return [null, 'the rate limits response never carried a batch group'];
}
async function readBatches(key, maxPages = 20) {
const headers = { 'x-api-key': key, 'anthropic-version': '2023-06-01' };
const rows = [];
let after = null;
for (let i = 0; i < Math.max(1, maxPages); i += 1) {
const params = { limit: 1000 };
if (after) params.after_id = after;
const [payload, err] = await getJson(BATCHES_URL, headers, params);
if (err) return [rows, err];
const data = payload.data ?? [];
rows.push(...data);
if (!payload.has_more || !data.length) break;
after = payload.last_id ?? data[data.length - 1]?.id;
if (!after) break;
}
return [rows, null];
}
function args(argv) {
const out = { threshold: 80, maxPages: 20 };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--threshold') out.threshold = Number.parseInt(argv[i += 1], 10);
else if (argv[i] === '--max-pages') out.maxPages = Number.parseInt(argv[i += 1], 10);
}
return out;
}
async function main() {
const opts = args(process.argv.slice(2));
const adminKey = process.env.ANTHROPIC_ADMIN_KEY;
const keys = workspaceKeys(process.env.ANTHROPIC_API_KEY,
process.env.ANTHROPIC_EXTRA_WORKSPACE_KEYS);
if (!keys.length) {
console.error('set ANTHROPIC_API_KEY to a workspace key. Add '
+ 'ANTHROPIC_EXTRA_WORKSPACE_KEYS as a comma separated list to cover more '
+ 'of the organization');
process.exitCode = 2;
return;
}
let limit = null;
if (adminKey) {
const [found, err] = await readCeiling(adminKey);
if (err) console.log(`could not read the ceiling: ${err}`);
limit = found;
} else {
console.log('no ANTHROPIC_ADMIN_KEY, so the enqueued_batch_requests ceiling '
+ 'cannot be read and only the raw depth is available');
}
if (limit !== null) {
console.log(`ceiling enqueued_batch_requests ${limit} (organization wide)`);
}
const rows = [];
for (const [index, key] of keys.entries()) {
const [batches, err] = await readBatches(key, opts.maxPages);
if (err) console.log(`workspace ${index + 1} batch list stopped early: ${err}`);
rows.push(...queueRows(batches, `ws${index + 1}`));
}
rows.sort((a, b) => (b.processing - a.processing) || a.id.localeCompare(b.id));
for (const row of rows) {
console.log(`${row.id.slice(0, 16).padEnd(16)} ${row.status.padEnd(13)} `
+ `${row.processing} processing`);
}
const depth = queueDepth(rows);
const [state, detail] = verdict(depth, limit, rows, keys.length, opts.threshold);
console.log(`${state.padEnd(20)} ${detail}`);
console.log(' measured: enqueued_batch_requests from the Rate Limits API, and '
+ `the sum of request_counts.processing over ${rows.length} live batch(es) `
+ `in ${keys.length} workspace(s)`);
console.log(' inferred: nothing about workspaces whose keys were not '
+ 'supplied. The ceiling is organization wide and this depth is a lower '
+ 'bound on it');
for (const line of repairLines(state, rows, limit)) console.log(` repair: ${line}`);
console.log(`${FINDINGS.has(state) ? 1 : 0} finding(s)`);
process.exitCode = FINDINGS.has(state) ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test digs the ceiling out of the real response shape, where the batch group sits alongside several model groups, carries a models of null, and holds its single limit inside a nested array. It also pins the decision that a missing ceiling comes back as None rather than zero, because a zero ceiling reads as infinite occupancy and turns every run into a false alarm. The second is the definition of the queue: only in_progress and canceling count, and only the processing field, so a batch that has ended with fifty thousand successes contributes nothing. The third is the occupancy arithmetic against the threshold, including the exhausted case at exactly the ceiling. The fourth is the unreadable ceiling, which has to be a finding with its own repair rather than a clean run. The fifth checks that the same workspace key passed twice does not double the depth, which is the one mistake here that would look exactly like a real alarm. And the last checks the repair names the biggest holder and the per-batch cap.
from anthropic_batch_queue_depth import (enqueued_limit, headroom, queue_depth,
queue_rows, repair_lines, top_holders,
verdict, workspace_keys)
RATE_LIMITS = {
"data": [
{"type": "rate_limit", "group_type": "model_group",
"models": ["claude-opus-5"],
"limits": [{"type": "requests_per_minute", "value": 4000},
{"type": "input_tokens_per_minute", "value": 10000000}]},
{"type": "rate_limit", "group_type": "batch", "models": None,
"limits": [{"type": "enqueued_batch_requests", "value": 300000}]},
],
"next_page": None,
}
BATCHES = [
{"id": "msgbatch_01Rf", "processing_status": "in_progress",
"request_counts": {"processing": 214900, "succeeded": 0, "errored": 0,
"canceled": 0, "expired": 0}},
{"id": "msgbatch_01Qa", "processing_status": "in_progress",
"request_counts": {"processing": 58400, "succeeded": 0, "errored": 0,
"canceled": 0, "expired": 0}},
{"id": "msgbatch_01Zc", "processing_status": "canceling",
"request_counts": {"processing": 9600, "succeeded": 200, "errored": 0,
"canceled": 0, "expired": 0}},
{"id": "msgbatch_01Done", "processing_status": "ended",
"request_counts": {"processing": 0, "succeeded": 50000, "errored": 0,
"canceled": 0, "expired": 0}},
]
def test_the_ceiling_comes_out_of_the_batch_group_and_nowhere_else():
assert enqueued_limit(RATE_LIMITS) == 300000
# Absent means unknown, never zero: a zero ceiling reads as infinite
# occupancy and alarms on an empty queue.
assert enqueued_limit({"data": [RATE_LIMITS["data"][0]]}) is None
assert enqueued_limit({}) is None
assert enqueued_limit({"data": [{"group_type": "batch",
"limits": [{"type": "other", "value": 1}]}]}) is None
assert enqueued_limit({"data": [{"group_type": "batch",
"limits": [{"type": "enqueued_batch_requests",
"value": "lots"}]}]}) is None
def test_only_live_batches_and_only_the_processing_count_are_the_queue():
rows = queue_rows(BATCHES, "ws1")
assert [r["id"] for r in rows] == ["msgbatch_01Rf", "msgbatch_01Qa",
"msgbatch_01Zc"]
# An ended batch holds nothing in the queue however many it succeeded on.
assert all(r["id"] != "msgbatch_01Done" for r in rows)
# canceling is still live, because those requests have not been processed.
assert rows[2]["status"] == "canceling"
assert queue_depth(rows) == 282900
assert queue_depth([]) == 0
def test_occupancy_is_measured_against_the_threshold_that_was_passed_in():
rows = queue_rows(BATCHES)
depth = queue_depth(rows)
remaining, occupancy = headroom(depth, 300000)
assert remaining == 17100 and round(occupancy, 3) == 0.943
state, detail = verdict(depth, 300000, rows, 1, 80)
assert state == "queue-near-limit" and "94% of the ceiling" in detail
assert verdict(depth, 300000, rows, 1, 95)[0] == "queue-clear"
# At or over the ceiling, submissions are refused rather than slowed.
state, detail = verdict(300000, 300000, rows, 1, 80)
assert state == "queue-exhausted" and "being refused" in detail
assert headroom(10, None) == (None, None)
assert headroom(10, 0) == (None, None)
def test_an_unreadable_ceiling_is_a_finding_with_its_own_repair():
rows = queue_rows(BATCHES)
state, detail = verdict(queue_depth(rows), None, rows, 1, 80)
assert state == "queue-limit-unknown"
assert "could not be read" in detail and "282900" in detail
lines = repair_lines(state, rows, None)
assert any("Workspace keys are rejected by every Admin endpoint" in line
for line in lines)
assert any("raw count" in line for line in lines)
def test_the_same_workspace_key_twice_does_not_double_the_depth():
assert workspace_keys("k1", "k2,k3") == ["k1", "k2", "k3"]
assert workspace_keys("k1", "k1, k1 ,") == ["k1"]
assert workspace_keys("", None) == []
assert workspace_keys(None, "k9") == ["k9"]
def test_the_repair_names_the_biggest_holder_and_the_per_batch_cap():
rows = queue_rows(BATCHES)
lines = repair_lines("queue-near-limit", rows, 300000)
assert any("msgbatch_01Rf alone holds 214900 of the 300000" in line
for line in lines)
assert any("100000 requests or 256 MB" in line for line in lines)
assert any("24 hour window" in line for line in lines)
assert top_holders(rows, 1)[0]["id"] == "msgbatch_01Rf"
assert top_holders([], 3) == []
assert repair_lines("queue-clear", rows, 300000)[0].startswith("nothing to change")
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { enqueuedLimit, headroom, queueDepth, queueRows, repairLines, topHolders,
verdict, workspaceKeys } from './anthropic-batch-queue-depth.mjs';
const RATE_LIMITS = {
data: [
{ type: 'rate_limit', group_type: 'model_group', models: ['claude-opus-5'],
limits: [{ type: 'requests_per_minute', value: 4000 },
{ type: 'input_tokens_per_minute', value: 10000000 }] },
{ type: 'rate_limit', group_type: 'batch', models: null,
limits: [{ type: 'enqueued_batch_requests', value: 300000 }] },
],
next_page: null,
};
const BATCHES = [
{ id: 'msgbatch_01Rf', processing_status: 'in_progress',
request_counts: { processing: 214900, succeeded: 0, errored: 0, canceled: 0,
expired: 0 } },
{ id: 'msgbatch_01Qa', processing_status: 'in_progress',
request_counts: { processing: 58400, succeeded: 0, errored: 0, canceled: 0,
expired: 0 } },
{ id: 'msgbatch_01Zc', processing_status: 'canceling',
request_counts: { processing: 9600, succeeded: 200, errored: 0, canceled: 0,
expired: 0 } },
{ id: 'msgbatch_01Done', processing_status: 'ended',
request_counts: { processing: 0, succeeded: 50000, errored: 0, canceled: 0,
expired: 0 } },
];
test('the ceiling comes out of the batch group and nowhere else', () => {
assert.equal(enqueuedLimit(RATE_LIMITS), 300000);
assert.equal(enqueuedLimit({ data: [RATE_LIMITS.data[0]] }), null);
assert.equal(enqueuedLimit({}), null);
assert.equal(enqueuedLimit({ data: [{ group_type: 'batch',
limits: [{ type: 'other', value: 1 }] }] }), null);
assert.equal(enqueuedLimit({ data: [{ group_type: 'batch',
limits: [{ type: 'enqueued_batch_requests', value: 'lots' }] }] }), null);
});
test('only live batches and only the processing count are the queue', () => {
const rows = queueRows(BATCHES, 'ws1');
assert.deepEqual(rows.map((r) => r.id),
['msgbatch_01Rf', 'msgbatch_01Qa', 'msgbatch_01Zc']);
assert.ok(rows.every((r) => r.id !== 'msgbatch_01Done'));
assert.equal(rows[2].status, 'canceling');
assert.equal(queueDepth(rows), 282900);
assert.equal(queueDepth([]), 0);
});
test('occupancy is measured against the threshold that was passed in', () => {
const rows = queueRows(BATCHES);
const depth = queueDepth(rows);
const [remaining, occupancy] = headroom(depth, 300000);
assert.equal(remaining, 17100);
assert.equal(Number(occupancy.toFixed(3)), 0.943);
const [state, detail] = verdict(depth, 300000, rows, 1, 80);
assert.equal(state, 'queue-near-limit');
assert.ok(detail.includes('94% of the ceiling'));
assert.equal(verdict(depth, 300000, rows, 1, 95)[0], 'queue-clear');
const [state2, detail2] = verdict(300000, 300000, rows, 1, 80);
assert.equal(state2, 'queue-exhausted');
assert.ok(detail2.includes('being refused'));
assert.deepEqual(headroom(10, null), [null, null]);
assert.deepEqual(headroom(10, 0), [null, null]);
});
test('an unreadable ceiling is a finding with its own repair', () => {
const rows = queueRows(BATCHES);
const [state, detail] = verdict(queueDepth(rows), null, rows, 1, 80);
assert.equal(state, 'queue-limit-unknown');
assert.ok(detail.includes('could not be read') && detail.includes('282900'));
const lines = repairLines(state, rows, null);
assert.ok(lines.some((l) => l.includes('Workspace keys are rejected by every Admin endpoint')));
assert.ok(lines.some((l) => l.includes('raw count')));
});
test('the same workspace key twice does not double the depth', () => {
assert.deepEqual(workspaceKeys('k1', 'k2,k3'), ['k1', 'k2', 'k3']);
assert.deepEqual(workspaceKeys('k1', 'k1, k1 ,'), ['k1']);
assert.deepEqual(workspaceKeys('', null), []);
assert.deepEqual(workspaceKeys(null, 'k9'), ['k9']);
});
test('the repair names the biggest holder and the per batch cap', () => {
const rows = queueRows(BATCHES);
const lines = repairLines('queue-near-limit', rows, 300000);
assert.ok(lines.some((l) => l.includes('msgbatch_01Rf alone holds 214900 of the 300000')));
assert.ok(lines.some((l) => l.includes('100000 requests or 256 MB')));
assert.ok(lines.some((l) => l.includes('24 hour window')));
assert.equal(topHolders(rows, 1)[0].id, 'msgbatch_01Rf');
assert.deepEqual(topHolders([], 3), []);
assert.ok(repairLines('queue-clear', rows, 300000)[0].startsWith('nothing to change'));
});
FAQ
What exactly counts as a batch request in the processing queue?
The documentation defines it as a batch request that has yet to be successfully processed by the model, and that is precisely <code>request_counts.processing</code> on the batch object. Those counts start with everything in <code>processing</code> and move to <code>succeeded</code>, <code>errored</code>, <code>canceled</code> or <code>expired</code> only once processing of the whole batch ends, with the five values always summing to the batch's total. So the queue is the sum of <code>processing</code> over batches that are <code>in_progress</code> or <code>canceling</code>, and an <code>ended</code> batch contributes nothing however large it was.
What are the actual numbers?
The enqueued batch requests limit is 200,000 on Start, 300,000 on Build and 500,000 on Scale, and it is shared across every model rather than allocated per model. A single batch is capped at 100,000 requests or 256 MB, whichever is reached first, at every tier. The batch endpoints also carry their own requests-per-minute limit of 1,000, 2,000 or 4,000 by tier. Rather than hardcode any of that, the script reads your organization's actual configured value from the Rate Limits API, because the tables move and your organization may not be on a standard tier.
Why does this need two different API keys?
Because Anthropic's two APIs do not share credentials in either direction. The Rate Limits endpoints are part of the Admin API and reject workspace-scoped keys outright; the Messages Batches endpoints are data plane and cannot be reached with an Admin key. There is no single credential that can read both the ceiling and the depth, so a complete answer needs both. With only the workspace key you get a raw count and the script says so; with only the Admin key you get a limit and nothing to compare it against.
Does this measure the whole organization?
Not unless you give it every workspace's key, and the output says which it is. <code>GET /v1/messages/batches</code> lists batches within the workspace that the key resolves to, while <code>enqueued_batch_requests</code> is an organization-wide ceiling. So a one-key run produces a lower bound on the depth: real occupancy is at least what you measured and possibly much more. <code>ANTHROPIC_EXTRA_WORKSPACE_KEYS</code> takes a comma-separated list, and the script de-duplicates them so passing the same key twice cannot invent a queue that is not there.
Does OpenAI have the same limit, and can this find it?
It has an equivalent and no, this cannot read it. OpenAI's Batch API caps the prompt tokens that may be queued for batch processing per model, allows up to 2,000 batch creations per hour, and limits a single batch to 50,000 requests. None of those values is returned by any endpoint: the token cap is published on the platform limits page, and the only programmatic signal is the error body on a submission a read-only script is not permitted to make. Rather than half-implement the check there, this note is Anthropic only and says why.
Related field notes
- Which limiter emptied when the 429 was on the synchronous path
- The workspace override sitting under the organization ceiling
- The clock that closes on a batch before it finishes
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits, including the Message Batches API limits
- Rate Limits API — Claude platform docs
- Batch processing — Claude platform docs
- List Message Batches — Claude API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.