Diagnostic LLM APIs
A non-streaming request over 10 minutes times out with 504
The prompt is two thousand tokens. The context window is nowhere near full. Nothing is too large by any measure anybody in the room has checked, and the request still dies — sometimes as 504 with timeout_error, more often as nothing at all, because a load balancer somewhere between you and Anthropic closed an idle connection while the model was still writing. The ceiling this hit is a clock.
Estimate the time, not the size. Generation runs at roughly fifty to sixty output tokens a second, so max_tokens divided by that rate is how long the call takes. A non-streaming request is documented not to run past 10 minutes, and the current large models allow 128,000 output tokens — which at that rate is nearly forty minutes on a single call.
At about fifty-five tokens a second the largest max_tokens that can finish inside ten minutes is roughly 33,000. Anything above that on a non-streaming path is a timeout waiting for a verbose answer.
The fix is streaming, not a smaller prompt. .stream() followed by .get_final_message() hands you the identical Message object with no event handling, and the connection never goes idle. For anything latency-tolerant, the Message Batches API removes the clock entirely.
The problem in plain words
Every other ceiling in this section is a size. This one is a duration, and duration is the dimension nobody instruments, because it is not in the request and it is not in the usage report. The request that fails is not big. It asked for a long answer, the model obliged, and the answer took longer than the connection was allowed to stay open.
The failure mode is unusually unhelpful. Sometimes it is a clean 504 with "type": "timeout_error". Often it is worse: no response at all, because an intermediate hop — a corporate proxy, a cloud load balancer, an ingress with a sixty-second idle timeout — dropped the connection before Anthropic answered. Then your client raises a connection error, which reads like a network fault, which is triaged as a network fault, and the network is fine.
Then the wrong repair gets applied. Somebody raises the client timeout, which changes nothing, because the ceiling is not the client's. Somebody shortens the prompt, which changes nothing, because the input side was never the problem. The SDKs actually guard against this — they validate that a non-streaming Messages request is not expected to exceed ten minutes and refuse the combination — but a raw HTTP client, a proxy layer or a homegrown wrapper has no such check, and those are exactly the places long-running calls end up.
Why it happens
The clock is a property of the answer, not of the question. A two-thousand-token prompt with max_tokens: 64000 takes twenty minutes to generate and a sixty-thousand-token prompt with max_tokens: 1024 takes about twenty seconds. Every intuition built on prompt size points the wrong way here, which is why this note estimates seconds and reports seconds.
Thinking tokens are output tokens, so they are on the clock too. Extended thinking is generated, billed and timed like anything else the model writes. Raising an effort setting is a change to how long every request takes, made in a config file, with no diff anywhere near the timeout.
Client timeout units genuinely differ between SDKs, and the mistake is silent. Python and Ruby take seconds. The TypeScript client takes milliseconds. Go takes a time.Duration, Java a Duration, C# a TimeSpan. A 600 copied from a Python example into a TypeScript constructor is six hundred milliseconds, and it produces a timeout on nearly every call that gets blamed on the API.
Streaming does not make it faster. It makes the connection busy. The same tokens are generated at the same rate; the difference is that bytes are arriving continuously, so nothing between you and the API decides the connection is idle. That is the entire mechanism, and it is why the repair here is a transport change rather than a size change.
This is not the model's cap. A max_tokens the model refuses outright is a different note and a 400 during validation. Here the value is perfectly legal — that is the problem. The model will happily accept a request for 128,000 tokens and then spend forty minutes trying to deliver them.
The fix, as a flow
Nothing here is too large. The prompt is small, the window is empty and the request still dies, because the ceiling is a clock and 128,000 output tokens take about forty minutes to write. The fix estimates seconds, normalises the client timeout out of whatever unit its SDK uses, and then changes the transport rather than the size.
How to fix it
List the call paths with their transport, not just their parameters
The tuple that matters is (model, max_tokens, streams or does not, client timeout, SDK). The transport is the field people leave out of configuration entirely, because it is expressed in code as a different method call rather than as a setting, and it is the field that decides whether the ceiling applies.
Estimate generation time from max_tokens
max_tokens divided by your observed output rate. Fifty-five tokens a second is a reasonable starting figure; measure your own from a handful of real responses and use that instead. Add prefill, which is fast — thousands of tokens a second — and matters only on genuinely enormous inputs.
Size the input side for free
POST /v1/messages/count_tokens gives you the input token count at no cost, and here it is being used to convert into seconds rather than to check a window. Prefill is usually a small share of the total, and the check is worth doing precisely so you can prove that and stop shortening prompts to fix a timeout.
Convert the client timeout into seconds before comparing it
Python and Ruby take seconds; TypeScript takes milliseconds; Go, Java and C# take duration types. Normalise before comparing, and flag anything under a second on a millisecond SDK as a copied number rather than a chosen one. Then note that a client timeout above ten minutes on a non-streaming path buys nothing at all.
Print the transport change, do not make it
For each path over the line: stream it, with .stream() and .get_final_message() in Python or .finalMessage() in TypeScript, which returns the same Message object and needs no event handling. For anything nobody is waiting on, the Message Batches API. For direct HTTP integrations, TCP keep-alive so the intermediate hops stop killing the socket. Switching a production call path to streaming changes error handling and back-pressure, so it is printed.
How to check it worked
Re-run after the change. Streaming paths should report comfortably, and any remaining non-streaming path should sit well under ten minutes at your measured rate.
python3 anthropic_wall_clock_preflight.py --config call-paths.json --tps 55
# over-wall-clock-not-streaming report-writer 19m 23s of generation estimated on a non-streaming path, past the 10m 00s ceiling. ...
# at 55 tok/s the largest max_tokens that finishes inside the ceiling is 32,978
# this model allows 128000 output tokens, which is 38m 47s on one call
# timeout-unit-mistake report-writer timeout 600 on the typescript client is 0.6s, not 10 minutes
# 3 path(s) checked, 2 finding(s)The full code
Seconds throughout. One GET per model for the cap, and one free count_tokens call per path that names a payload — used here to turn the input into prefill seconds rather than to check it against anything. Seven pure functions and none of them compares a size to a ceiling: the two rate conversions, the timeout normaliser that knows Python takes seconds and TypeScript takes milliseconds, the suspicion check for a number copied between them, the largest max_tokens that still finishes in time, the duration formatter, and a verdict that puts the wall clock ahead of the client timeout because raising the client timeout is the repair that does not work.
"""Estimate whether a non-streaming Claude call can finish inside 10 minutes.
Read only, with one deliberate exception. Nothing here creates a completion:
where a call path names a payload file, that body goes to
/v1/messages/count_tokens, which is free, creates no object, generates no
output and is not billed. It is used to turn the input into prefill seconds.
Everything else is a GET, and /v1/messages is never called.
The repair is a transport change and it is printed. Moving a production call
path onto streaming changes error handling and back pressure, which is a
decision, not an audit's side effect.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_wall_clock_preflight")
API = "https://api.anthropic.com/v1"
VERSION = "2023-06-01"
# The documented ceiling for a single non-streaming Messages request.
WALL_CLOCK = 600.0
# Starting figures, both meant to be replaced with your own measurements.
# Generation is the one that decides the answer; prefill is fast enough that it
# only matters on very large inputs, and proving that is half the point.
DEFAULT_TPS = 55.0
DEFAULT_PREFILL_TPS = 6000.0
# Client timeouts are not expressed in the same unit across SDKs, and a number
# copied from one language's example into another's constructor is the quiet
# half of this note.
SDK_TIMEOUT_UNITS = {
"python": ("seconds", 1.0),
"ruby": ("seconds", 1.0),
"php": ("seconds", 1.0),
"typescript": ("milliseconds", 0.001),
"javascript": ("milliseconds", 0.001),
"node": ("milliseconds", 0.001),
"go": ("a time.Duration", 1.0),
"java": ("a Duration", 1.0),
"csharp": ("a TimeSpan", 1.0),
}
MILLISECOND_SDKS = ("typescript", "javascript", "node")
SAMPLING_ONLY = ("max_tokens", "stream", "temperature", "top_p", "top_k",
"stop_sequences", "metadata", "service_tier")
FINDINGS = ("over-wall-clock-not-streaming", "over-client-timeout",
"near-wall-clock-not-streaming")
def duration(seconds):
"""Seconds as minutes and seconds. Pure."""
total = int(max(0.0, float(seconds or 0)))
return "%dm %02ds" % (total // 60, total % 60)
def generation_seconds(max_tokens, tps=DEFAULT_TPS):
"""How long it takes to write max_tokens output tokens. Pure.
This is the number the whole note turns on, and it has nothing to do with
the size of the prompt. Thinking tokens are output tokens, so an effort
setting moves it too.
"""
rate = float(tps or 0)
if rate <= 0:
return 0.0
return max(0, int(max_tokens or 0)) / rate
def prefill_seconds(input_tokens, prefill_tps=DEFAULT_PREFILL_TPS):
"""How long it takes to read the input. Pure.
Kept separate and reported separately because it is almost always small,
and the point of measuring it is to stop people shortening prompts to fix a
problem that lives entirely on the output side.
"""
rate = float(prefill_tps or 0)
if rate <= 0:
return 0.0
return max(0, int(input_tokens or 0)) / rate
def timeout_seconds(sdk, value):
"""A client timeout in seconds, whatever unit the SDK takes. Pure.
None when the SDK is unknown, because guessing the unit is precisely the
mistake this function exists to catch.
"""
if value is None:
return None
unit = SDK_TIMEOUT_UNITS.get(str(sdk or "").strip().lower())
if unit is None:
return None
try:
return float(value) * unit[1]
except (TypeError, ValueError):
return None
def unit_suspicion(sdk, value):
"""True when a timeout looks written in the wrong unit. Pure.
600 in the TypeScript client is six hundred milliseconds, not ten minutes.
Nobody chooses a sub-second timeout for an LLM call on purpose, so anything
under a second on a millisecond SDK is a number copied from a seconds-based
example.
"""
seconds = timeout_seconds(sdk, value)
if seconds is None:
return False
return str(sdk or "").strip().lower() in MILLISECOND_SDKS and seconds < 1.0
def safe_max_tokens(tps=DEFAULT_TPS, wall_clock=WALL_CLOCK, prefill=0.0):
"""The largest max_tokens that still finishes inside the ceiling. Pure.
The number to put in the config, as opposed to the model's cap, which is
the number that fits in the request.
"""
rate = float(tps or 0)
room = max(0.0, float(wall_clock or 0) - max(0.0, float(prefill or 0)))
if rate <= 0:
return 0
return int(room * rate)
def verdict(seconds, streams, timeout_s=None, wall_clock=WALL_CLOCK, near=0.8):
"""Classify one call path against the clock. Pure. (state, detail).
Order matters. The wall clock is checked before the client timeout, because
a non-streaming request past ten minutes fails on the far side whatever the
client is configured to wait for, and raising the client timeout is both
the first repair people reach for and the one that does nothing.
"""
shape = "%s of generation estimated" % duration(seconds)
if not streams and seconds > wall_clock:
return ("over-wall-clock-not-streaming",
"%s on a non-streaming path, past the %s ceiling. That is a 504 "
"timeout_error, or no response at all when an intermediate hop "
"drops the idle connection first. Raising the client timeout "
"does not move it." % (shape, duration(wall_clock)))
if timeout_s is not None and seconds > timeout_s:
return ("over-client-timeout",
"%s against a client timeout of %s, so the client gives up "
"before the API is finished." % (shape, duration(timeout_s)))
if not streams and seconds >= wall_clock * near:
return ("near-wall-clock-not-streaming",
"%s on a non-streaming path, inside %.0f%% of the %s ceiling. "
"One unusually long answer crosses it."
% (shape, near * 100, duration(wall_clock)))
if streams and seconds > wall_clock:
return ("streams-past-ten-minutes",
"%s, and the path streams, so the connection never goes idle "
"and the ceiling does not apply. Worth the Message Batches API "
"if nobody is waiting on it." % shape)
return ("within-budget", "%s." % shape)
def get(session, path):
r = session.get(API + path, timeout=30)
if r.status_code in (401, 403):
raise SystemExit("%d from Anthropic: ANTHROPIC_API_KEY has to be a "
"workspace key" % r.status_code)
r.raise_for_status()
return r.json()
def count_input(session, payload_path):
"""The one non-GET call, and it neither creates nor bills anything.
The counting endpoint returns an input_tokens number for free. Here that
number is converted straight into seconds of prefill; it is not compared
against any ceiling, which is a different note.
"""
with open(payload_path, "r", encoding="utf-8") as fh:
body = json.load(fh)
trimmed = {k: v for k, v in body.items() if k not in SAMPLING_ONLY}
r = session.post(API + "/messages/count_tokens", json=trimmed, timeout=60)
r.raise_for_status()
return int((r.json() or {}).get("input_tokens") or 0)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--config", required=True,
help="JSON file of call paths: "
'{"name": {"model": ..., "max_tokens": ..., '
'"stream": false, "sdk": "typescript", '
'"timeout": 600, "payload": "body.json"}}')
ap.add_argument("--tps", type=float, default=DEFAULT_TPS,
help="observed output tokens per second (default 55)")
ap.add_argument("--prefill-tps", type=float, default=DEFAULT_PREFILL_TPS,
help="observed input tokens per second (default 6000)")
ap.add_argument("--show-all", action="store_true",
help="also print paths comfortably inside the clock")
args = ap.parse_args()
key = os.environ.get("ANTHROPIC_API_KEY")
if not key:
log.error("set ANTHROPIC_API_KEY to a workspace key")
return 2
with open(args.config, "r", encoding="utf-8") as fh:
paths = json.load(fh)
session = requests.Session()
session.headers.update({"x-api-key": key, "anthropic-version": VERSION,
"content-type": "application/json"})
caps = {}
bad = 0
for name in sorted(paths):
entry = paths[name] or {}
model_id = str(entry.get("model") or "")
streams = bool(entry.get("stream"))
sdk = entry.get("sdk")
if model_id and model_id not in caps:
caps[model_id] = get(session, "/models/" + model_id).get("max_tokens")
input_tokens = int(entry.get("input_tokens") or 0)
if entry.get("payload"):
input_tokens = count_input(session, entry["payload"])
prefill = prefill_seconds(input_tokens, args.prefill_tps)
seconds = prefill + generation_seconds(entry.get("max_tokens"), args.tps)
client = timeout_seconds(sdk, entry.get("timeout"))
state, detail = verdict(seconds, streams, client)
line = "%-30s %-16s %s" % (state, name, detail)
if state in FINDINGS:
bad += 1
log.warning(line)
elif state == "streams-past-ten-minutes":
log.info(line)
elif args.show_all:
log.info(line)
if unit_suspicion(sdk, entry.get("timeout")):
bad += 1
log.warning("%-30s %-16s timeout %s on the %s client is %.1fs, not "
"%s: that unit is milliseconds",
"timeout-unit-mistake", name, entry.get("timeout"), sdk,
client or 0.0, duration(entry.get("timeout") or 0))
if state in ("over-wall-clock-not-streaming",
"near-wall-clock-not-streaming"):
log.warning(" at %.0f tok/s the largest max_tokens that finishes "
"inside the ceiling is %d",
args.tps, safe_max_tokens(args.tps, WALL_CLOCK, prefill))
cap = caps.get(model_id)
if cap:
log.warning(" this model allows %d output tokens, which is %s "
"on one call", cap,
duration(generation_seconds(cap, args.tps)))
log.warning(" repair: stream it. .stream() plus "
".get_final_message() returns the identical Message "
"object with no event handling, and the connection "
"never goes idle. For latency tolerant work use the "
"Message Batches API, which has no such clock. Printed, "
"not applied.")
log.info("%d path(s) checked, %d finding(s)", len(paths), bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Estimate whether a non-streaming Claude call can finish inside 10 minutes.
*
* Read only, with one deliberate exception. Nothing here creates a completion:
* where a call path names a payload file, that body goes to
* /v1/messages/count_tokens, which is free, creates no object, generates no
* output and is not billed. It is used to turn the input into prefill seconds.
* Everything else is a GET, and /v1/messages is never called.
*
* The repair is a transport change and it is printed.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://api.anthropic.com/v1';
const VERSION = '2023-06-01';
const WALL_CLOCK = 600;
const DEFAULT_TPS = 55;
const DEFAULT_PREFILL_TPS = 6000;
const SDK_TIMEOUT_UNITS = {
python: ['seconds', 1],
ruby: ['seconds', 1],
php: ['seconds', 1],
typescript: ['milliseconds', 0.001],
javascript: ['milliseconds', 0.001],
node: ['milliseconds', 0.001],
go: ['a time.Duration', 1],
java: ['a Duration', 1],
csharp: ['a TimeSpan', 1],
};
const MILLISECOND_SDKS = new Set(['typescript', 'javascript', 'node']);
const SAMPLING_ONLY = new Set(['max_tokens', 'stream', 'temperature', 'top_p',
'top_k', 'stop_sequences', 'metadata', 'service_tier']);
const FINDINGS = new Set(['over-wall-clock-not-streaming', 'over-client-timeout',
'near-wall-clock-not-streaming']);
/** Seconds as minutes and seconds. Pure. */
export function duration(seconds) {
const total = Math.trunc(Math.max(0, Number(seconds || 0)));
return `${Math.floor(total / 60)}m ${String(total % 60).padStart(2, '0')}s`;
}
/** How long it takes to write maxTokens output tokens. Pure. */
export function generationSeconds(maxTokens, tps = DEFAULT_TPS) {
const rate = Number(tps || 0);
if (rate <= 0) return 0;
return Math.max(0, Math.trunc(maxTokens || 0)) / rate;
}
/** How long it takes to read the input. Pure. Reported separately on purpose. */
export function prefillSeconds(inputTokens, prefillTps = DEFAULT_PREFILL_TPS) {
const rate = Number(prefillTps || 0);
if (rate <= 0) return 0;
return Math.max(0, Math.trunc(inputTokens || 0)) / rate;
}
/**
* A client timeout in seconds, whatever unit the SDK takes. Pure.
* Null when the SDK is unknown, because guessing the unit is the mistake this
* function exists to catch.
*/
export function timeoutSeconds(sdk, value) {
if (value === null || value === undefined) return null;
const unit = SDK_TIMEOUT_UNITS[String(sdk ?? '').trim().toLowerCase()];
if (!unit) return null;
const n = Number(value);
return Number.isFinite(n) ? n * unit[1] : null;
}
/**
* True when a timeout looks written in the wrong unit. Pure.
* 600 in the TypeScript client is six hundred milliseconds, not ten minutes.
*/
export function unitSuspicion(sdk, value) {
const seconds = timeoutSeconds(sdk, value);
if (seconds === null) return false;
return MILLISECOND_SDKS.has(String(sdk ?? '').trim().toLowerCase()) && seconds < 1;
}
/** The largest max_tokens that still finishes inside the ceiling. Pure. */
export function safeMaxTokens(tps = DEFAULT_TPS, wallClock = WALL_CLOCK, prefill = 0) {
const rate = Number(tps || 0);
const room = Math.max(0, Number(wallClock || 0) - Math.max(0, Number(prefill || 0)));
if (rate <= 0) return 0;
return Math.trunc(room * rate);
}
/**
* Classify one call path against the clock. Pure. [state, detail].
* The wall clock is checked before the client timeout, because a non-streaming
* request past ten minutes fails on the far side whatever the client waits for.
*/
export function verdict(seconds, streams, timeoutS = null, wallClock = WALL_CLOCK, near = 0.8) {
const shape = `${duration(seconds)} of generation estimated`;
if (!streams && seconds > wallClock) {
return ['over-wall-clock-not-streaming',
`${shape} on a non-streaming path, past the ${duration(wallClock)} ` +
'ceiling. That is a 504 timeout_error, or no response at all when an ' +
'intermediate hop drops the idle connection first. Raising the client ' +
'timeout does not move it.'];
}
if (timeoutS !== null && timeoutS !== undefined && seconds > timeoutS) {
return ['over-client-timeout',
`${shape} against a client timeout of ${duration(timeoutS)}, so the ` +
'client gives up before the API is finished.'];
}
if (!streams && seconds >= wallClock * near) {
return ['near-wall-clock-not-streaming',
`${shape} on a non-streaming path, inside ${(near * 100).toFixed(0)}% of ` +
`the ${duration(wallClock)} ceiling. One unusually long answer crosses it.`];
}
if (streams && seconds > wallClock) {
return ['streams-past-ten-minutes',
`${shape}, and the path streams, so the connection never goes idle and ` +
'the ceiling does not apply. Worth the Message Batches API if nobody is ' +
'waiting on it.'];
}
return ['within-budget', `${shape}.`];
}
function headers(key) {
return { 'x-api-key': key, 'anthropic-version': VERSION,
'content-type': 'application/json' };
}
async function get(key, path) {
const res = await fetch(API + path, { headers: headers(key) });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Anthropic: ANTHROPIC_API_KEY has to be a workspace key`);
}
if (!res.ok) throw new Error(`${res.status} from ${path}`);
return res.json();
}
/** The one non-GET call, and it neither creates nor bills anything. */
async function countInput(key, payloadPath) {
const body = JSON.parse(await readFile(payloadPath, 'utf8'));
const trimmed = Object.fromEntries(
Object.entries(body).filter(([k]) => !SAMPLING_ONLY.has(k)));
const res = await fetch(`${API}/messages/count_tokens`, {
method: 'POST', // count_tokens creates nothing and bills nothing
headers: headers(key),
body: JSON.stringify(trimmed),
});
if (!res.ok) throw new Error(`${res.status} from /messages/count_tokens`);
return Math.trunc((await res.json())?.input_tokens ?? 0);
}
async function main() {
const key = process.env.ANTHROPIC_API_KEY;
if (!key) {
console.error('set ANTHROPIC_API_KEY to a workspace key');
process.exitCode = 2;
return;
}
const configPath = process.env.CONFIG ?? process.argv[2];
if (!configPath) {
console.error('set CONFIG, or pass the call-paths JSON file as an argument');
process.exitCode = 2;
return;
}
const paths = JSON.parse(await readFile(configPath, 'utf8'));
const tps = Number(process.env.TPS ?? DEFAULT_TPS);
const prefillTps = Number(process.env.PREFILL_TPS ?? DEFAULT_PREFILL_TPS);
const showAll = process.env.SHOW_ALL === '1';
const caps = new Map();
let bad = 0;
for (const name of Object.keys(paths).sort()) {
const entry = paths[name] ?? {};
const modelId = String(entry.model ?? '');
const streams = Boolean(entry.stream);
const sdk = entry.sdk;
if (modelId && !caps.has(modelId)) {
caps.set(modelId, (await get(key, `/models/${modelId}`)).max_tokens ?? null);
}
let inputTokens = Math.trunc(entry.input_tokens ?? 0);
if (entry.payload) inputTokens = await countInput(key, entry.payload);
const prefill = prefillSeconds(inputTokens, prefillTps);
const seconds = prefill + generationSeconds(entry.max_tokens, tps);
const client = timeoutSeconds(sdk, entry.timeout);
const [state, detail] = verdict(seconds, streams, client);
const line = `${state.padEnd(30)} ${name.padEnd(16)} ${detail}`;
if (FINDINGS.has(state)) { bad += 1; console.warn(line); }
else if (state === 'streams-past-ten-minutes') console.log(line);
else if (showAll) console.log(line);
if (unitSuspicion(sdk, entry.timeout)) {
bad += 1;
console.warn(`${'timeout-unit-mistake'.padEnd(30)} ${name.padEnd(16)} ` +
`timeout ${entry.timeout} on the ${sdk} client is ` +
`${(client ?? 0).toFixed(1)}s, not ${duration(entry.timeout ?? 0)}: ` +
'that unit is milliseconds');
}
if (state === 'over-wall-clock-not-streaming' || state === 'near-wall-clock-not-streaming') {
console.warn(` at ${tps.toFixed(0)} tok/s the largest max_tokens that ` +
`finishes inside the ceiling is ${safeMaxTokens(tps, WALL_CLOCK, prefill)}`);
const cap = caps.get(modelId);
if (cap) {
console.warn(` this model allows ${cap} output tokens, which is ` +
`${duration(generationSeconds(cap, tps))} on one call`);
}
console.warn(' repair: stream it. .stream() plus .finalMessage() returns the ' +
'identical Message object with no event handling, and the ' +
'connection never goes idle. For latency tolerant work use the ' +
'Message Batches API, which has no such clock. Printed, not applied.');
}
}
console.log(`${Object.keys(paths).length} path(s) checked, ${bad} finding(s)`);
process.exitCode = bad ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first test is the sentence the note exists for: the same two-thousand-token prompt with max_tokens: 64000 is nineteen minutes and a finding, and with stream: true it is nineteen minutes and fine. Nothing about the prompt changed. The second is its mirror — sixty thousand tokens of input with a small max_tokens takes about twenty seconds, so shortening the prompt was never going to help. The rest hold the units: 600 on the Python client is ten minutes and 600 on the TypeScript client is six hundred milliseconds, and the wall clock has to be reported ahead of the client timeout because raising the client timeout is the repair that does nothing.
from anthropic_wall_clock_preflight import (duration, generation_seconds,
prefill_seconds, safe_max_tokens,
timeout_seconds, unit_suspicion,
verdict)
def test_the_transport_decides_it_and_the_prompt_does_not():
# A two thousand token prompt asking for 64,000 tokens back.
seconds = prefill_seconds(2_000) + generation_seconds(64_000)
assert duration(seconds) == "19m 23s"
state, detail = verdict(seconds, streams=False)
assert state == "over-wall-clock-not-streaming"
assert "504" in detail
assert "Raising the client timeout does not move it" in detail
# Same seconds, same prompt, streaming: not a finding at all.
state, detail = verdict(seconds, streams=True)
assert state == "streams-past-ten-minutes"
assert "never goes idle" in detail
def test_an_enormous_prompt_with_a_small_answer_is_quick():
# The mirror image, and the reason this script reports prefill separately:
# thirty times the input, a twentieth of the time.
seconds = prefill_seconds(60_000) + generation_seconds(1_024)
assert duration(seconds) == "0m 28s"
assert verdict(seconds, streams=False)[0] == "within-budget"
def test_the_models_own_cap_is_forty_minutes_of_generation():
# Legal to request, impossible to deliver on a non-streaming call.
assert duration(generation_seconds(128_000)) == "38m 47s"
assert safe_max_tokens() == 33_000
assert safe_max_tokens(55.0, 600.0, prefill=100.0) == 27_500
assert safe_max_tokens(tps=0) == 0
def test_six_hundred_means_two_different_things_in_two_sdks():
assert timeout_seconds("python", 600) == 600.0
assert timeout_seconds("ruby", 600) == 600.0
assert timeout_seconds("typescript", 600) == 0.6
assert timeout_seconds("TypeScript", 600) == 0.6
assert unit_suspicion("typescript", 600) is True
assert unit_suspicion("node", 600) is True
assert unit_suspicion("python", 600) is False
# A deliberate ten minutes on the TypeScript client is not suspicious.
assert unit_suspicion("typescript", 600_000) is False
# An SDK this script does not know about gets no guess at all.
assert timeout_seconds("rust", 600) is None
assert unit_suspicion("rust", 600) is False
assert timeout_seconds("python", None) is None
def test_the_wall_clock_is_reported_ahead_of_the_client_timeout():
# Both are true for this path. The wall clock is the one that matters,
# because raising the client timeout leaves the request failing.
state, _ = verdict(1_200, streams=False, timeout_s=300.0)
assert state == "over-wall-clock-not-streaming"
# Streaming removes the wall clock, and then the client timeout is the
# binding number.
state, detail = verdict(1_200, streams=True, timeout_s=300.0)
assert state == "over-client-timeout"
assert "gives up before the API is finished" in detail
def test_a_path_close_to_the_ceiling_is_reported_before_it_crosses():
state, detail = verdict(540, streams=False)
assert state == "near-wall-clock-not-streaming"
assert "inside 80% of the 10m 00s ceiling" in detail
assert verdict(400, streams=False)[0] == "within-budget"
def test_durations_read_as_minutes_and_seconds():
assert duration(0) == "0m 00s"
assert duration(59.9) == "0m 59s"
assert duration(600) == "10m 00s"
assert duration(-5) == "0m 00s"
assert duration(None) == "0m 00s"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { duration, generationSeconds, prefillSeconds, safeMaxTokens,
timeoutSeconds, unitSuspicion, verdict }
from './anthropic-wall-clock-preflight.mjs';
test('the transport decides it and the prompt does not', () => {
const seconds = prefillSeconds(2000) + generationSeconds(64000);
assert.equal(duration(seconds), '19m 23s');
const [state, detail] = verdict(seconds, false);
assert.equal(state, 'over-wall-clock-not-streaming');
assert.match(detail, /504/);
assert.match(detail, /Raising the client timeout does not move it/);
const [streamState, streamDetail] = verdict(seconds, true);
assert.equal(streamState, 'streams-past-ten-minutes');
assert.match(streamDetail, /never goes idle/);
});
test('an enormous prompt with a small answer is quick', () => {
const seconds = prefillSeconds(60000) + generationSeconds(1024);
assert.equal(duration(seconds), '0m 28s');
assert.equal(verdict(seconds, false)[0], 'within-budget');
});
test('the models own cap is forty minutes of generation', () => {
assert.equal(duration(generationSeconds(128000)), '38m 47s');
assert.equal(safeMaxTokens(), 33000);
assert.equal(safeMaxTokens(55, 600, 100), 27500);
assert.equal(safeMaxTokens(0), 0);
});
test('six hundred means two different things in two sdks', () => {
assert.equal(timeoutSeconds('python', 600), 600);
assert.equal(timeoutSeconds('ruby', 600), 600);
assert.equal(timeoutSeconds('typescript', 600), 0.6);
assert.equal(timeoutSeconds('TypeScript', 600), 0.6);
assert.equal(unitSuspicion('typescript', 600), true);
assert.equal(unitSuspicion('node', 600), true);
assert.equal(unitSuspicion('python', 600), false);
assert.equal(unitSuspicion('typescript', 600000), false);
assert.equal(timeoutSeconds('rust', 600), null);
assert.equal(unitSuspicion('rust', 600), false);
assert.equal(timeoutSeconds('python', null), null);
});
test('the wall clock is reported ahead of the client timeout', () => {
assert.equal(verdict(1200, false, 300)[0], 'over-wall-clock-not-streaming');
const [state, detail] = verdict(1200, true, 300);
assert.equal(state, 'over-client-timeout');
assert.match(detail, /gives up before the API is finished/);
});
test('a path close to the ceiling is reported before it crosses', () => {
const [state, detail] = verdict(540, false);
assert.equal(state, 'near-wall-clock-not-streaming');
assert.match(detail, /inside 80% of the 10m 00s ceiling/);
assert.equal(verdict(400, false)[0], 'within-budget');
});
test('durations read as minutes and seconds', () => {
assert.equal(duration(0), '0m 00s');
assert.equal(duration(59.9), '0m 59s');
assert.equal(duration(600), '10m 00s');
assert.equal(duration(-5), '0m 00s');
assert.equal(duration(null), '0m 00s');
});
FAQ
Does streaming make the model faster?
No. The same tokens are generated at the same rate. What changes is that bytes arrive continuously, so nothing between you and the API concludes the connection is idle and closes it, and the documented ten-minute ceiling on non-streaming requests stops applying. It is a transport change, not a performance one, which is exactly why shortening the prompt does nothing.
I raised the client timeout to thirty minutes and it still fails.
It would. The ceiling is not yours to raise. A non-streaming Messages request is not expected to run past ten minutes on Anthropic's side, and the SDKs will refuse the combination outright rather than let you wait for something that is not coming. A raw HTTP client has no such guard, which is why this failure clusters in hand-rolled wrappers and proxy layers.
Why do I sometimes get no response at all instead of a 504?
Because something closer to you gave up first. Corporate proxies, cloud load balancers and ingress controllers routinely close connections that have been idle for sixty seconds, and a non-streaming request is idle by definition while the model writes. Your client then raises a connection error, which looks like a network fault and is triaged as one.
What is a safe max_tokens for a non-streaming path?
Measure your own output rate and divide ten minutes by it, then leave real margin. At around fifty-five tokens a second that puts the arithmetic ceiling near 33,000 and a sensible working value nearer 16,000. Remember thinking tokens count toward this, so raising an effort setting shortens the budget without touching the number.
Does the Batch API have the same limit?
No, and that is what makes it the other half of the repair. Batches are asynchronous by design, so there is no connection to hold open and no ten-minute clock; the trade is that results arrive when they arrive rather than in the request. For work nobody is waiting on it is the better answer than streaming, and it is cheaper.
Related field notes
- A max_tokens value the model object says is illegal
- Latency-tolerant work paying full price on the synchronous path
- Usage totals thrown away because the stream was read wrong
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Errors — Claude API
- Models — Claude API reference
- Models overview — Claude Docs
- Message Batches — Claude Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.