Diagnostic Twilio
REST concurrency exhausted, so bursts come back 20429
The fan-out worked in staging with fifty recipients and fell over in production with fifty thousand. Not slowly: a wall of HTTP 429s with 20429 in the body, all at once, from a client that was doing nothing wrong except doing it all at the same moment. Retrying fixed it, which is exactly why the underlying number went unmeasured for another six months.
Twilio limits how many REST requests an account can have in flight at once. Issue any cheap read — GET /2010-04-01/Accounts/{AccountSid}.json will do — and read the Twilio-Concurrent-Requests response header, which reports the account's current concurrency at the moment your request was served.
Sample it repeatedly during peak rather than once at midnight. A value that sits near your ceiling is the warning; an observed 20429 is the confirmation. The ceiling itself is not in any response field, so the script takes it as an argument and refuses to invent one.
The problem in plain words
Concurrency is not requests per second, and conflating the two is why this surprises people. A hundred requests a second that each return in 50 ms is five in flight. A hundred a second that each take two seconds — because a downstream carrier is slow, or because you are creating messages rather than reading them — is two hundred in flight, from a client whose request rate never changed. The thing that breaches the limit is often Twilio's own latency rising, not your traffic.
The failure mode compounds itself. When responses slow down, in-flight requests pile up; when the account hits the ceiling, Twilio starts returning 20429, and a naive client retries immediately, adding load to a system already at its limit. Every rejected request still occupied a slot to be rejected. Without backoff, the client's own retry storm is what keeps the account pinned there long after the original slow patch has cleared.
Why it happens
Nothing in the response body mentions the limit. The concurrency figure arrives in a header, and the ceiling it should be compared against is not published per account through the API at all. That is why this check samples an observable and asks you for the threshold, instead of pretending to read a limit that does not exist as a field.
Serverless multiplies concurrency by design. A queue consumer that scales to two hundred parallel invocations, each holding one Twilio request open, is two hundred concurrent requests by construction. The limit lives on the account; your scaling policy does not know that, and there is no shared counter between invocations to enforce it.
Subaccount concurrency does not roll up. Requests made against a subaccount SID count against that subaccount, so splitting a large tenant into its own subaccount genuinely gives it its own budget. This is the one architectural lever here, and it is worth knowing before you need it.
Retrying a 20429 is safe, and that is not permission to retry immediately. The request was rejected before it was processed, so nothing was half-created — but retries without exponential backoff and jitter reconverge on the same instant and reproduce the burst that caused the problem.
The fix, as a flow
There is no readable limit field, so the check samples what it can see and asks you for the ceiling. A peak with no threshold beside it is an observation, and the classifier says so rather than inventing one.
How to fix it
Pick a read that costs nothing and sample it
GET /2010-04-01/Accounts/{AccountSid}.json is a single small resource. The header is attached to the response regardless of which endpoint produced it, so there is no reason to sample with an expensive call.
Sample during peak, and sample repeatedly
One reading is a single instant of a quantity that moves constantly. A dozen samples spread across the busiest ten minutes of your day describes the shape of it; a reading at three in the morning describes an idle account and reassures you about nothing.
Remember the probe counts itself
The request that carries the header is itself in flight when the number is computed, so a completely quiet account reads 1 rather than 0. It also means a probe running in a tight loop is contributing to the very figure it reports, which is why the sampling interval here is seconds rather than milliseconds.
Compare against your actual ceiling, not a guess
The limit is an account property Twilio support can tell you, and it is not exposed as a readable field. Pass it in with --limit. Without it the script still reports the observed peak, but it labels the state unmeasured, because a number with no threshold beside it is not a finding.
Cap the client, then shard if capping is not enough
There is no console setting to fix. Bound your own concurrency below the ceiling with a semaphore or a worker pool, retry 20429 with exponential backoff and jitter, and if one tenant genuinely needs more headroom than the account has, move that tenant into its own subaccount so its budget is separate. Check first that the subaccount is not itself stopped: a suspended subaccount fails the same way for entirely different reasons.
How to check it worked
Re-run the probe during your busiest window with the ceiling your account actually has. The peak should sit under the warning ratio.
python3 twilio_concurrency_probe.py --samples 12 --interval 5 --limit 100
# 12 sample(s), peak 34, no 20429 observed
# headroom peak concurrency 34 of a 100 ceiling (34%).The full code
The probe issues one cheap GET per sample and reads a response header off each. It counts a 429 as data rather than as an error, because observing one is the strongest confirmation available. The classifier is pure and takes the samples, the ceiling and whether a 429 was seen, so the rules about what counts as near the limit can be argued with in a test rather than in production.
"""Sample Twilio's REST concurrency header and report how close to the ceiling.
Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. The repair is printed, never performed,
because this script holds a credential to an account that can send messages and
spend money.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_concurrency_probe")
HOST = "https://api.twilio.com"
BASE = HOST + "/2010-04-01"
HEADER = "Twilio-Concurrent-Requests"
def concurrency_of(headers):
"""The concurrency figure out of a response's headers, or None.
Pure, and case-insensitive by hand: requests hands back a case-insensitive
mapping, a plain dict in a test does not, and the difference should not
decide whether the check works.
"""
for name, value in (headers or {}).items():
if str(name).lower() != HEADER.lower():
continue
try:
return int(str(value).strip())
except (TypeError, ValueError):
return None
return None
def verdict(samples, limit=None, saw_429=False, warn_ratio=0.7):
"""Classify a run of concurrency samples. Pure, so the thresholds are
testable without waiting for a real peak.
samples: the per-request readings, with None for a response that carried no
header. Returns (state, detail).
"""
readings = [s for s in (samples or []) if s is not None]
if saw_429:
peak = max(readings) if readings else 0
return ("throttled",
"a 429 came back during the sample itself, at a peak concurrency "
"of %d: the account is at its ceiling right now, and every "
"rejected request still took a slot to be rejected." % peak)
if not readings:
return ("no-header",
"no %s header on any of the %d sample(s): with nothing to read, "
"this check cannot say anything about concurrency."
% (HEADER, len(samples or [])))
peak = max(readings)
if limit is None:
return ("unmeasured",
"peak concurrency %d over %d sample(s), and no ceiling to compare "
"it against: the limit is not a readable field, so pass the one "
"your account has with --limit." % (peak, len(readings)))
ratio = peak / float(limit)
if ratio >= 1.0:
return ("at-limit",
"peak concurrency %d against a %d ceiling: requests are being "
"refused with 20429 at the top of every burst." % (peak, limit))
if ratio >= warn_ratio:
return ("near-limit",
"peak concurrency %d of a %d ceiling (%.0f%%): one slow patch "
"downstream lengthens every in-flight request and closes that "
"gap without your traffic changing at all."
% (peak, limit, ratio * 100))
return ("headroom",
"peak concurrency %d of a %d ceiling (%.0f%%)."
% (peak, limit, ratio * 100))
def probe(session, account, samples, interval):
"""Take n samples of the concurrency header. Returns (readings, saw_429)."""
readings = []
saw_429 = False
url = "%s/Accounts/%s.json" % (BASE, account)
for i in range(samples):
r = session.get(url, timeout=30)
if r.status_code in (401, 403):
raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that "
"the API key belongs to that account with read access"
% r.status_code)
if r.status_code == 429:
saw_429 = True
value = concurrency_of(r.headers)
readings.append(value)
log.info(" sample %2d: %s", i + 1,
"no header" if value is None else "%d in flight" % value)
if i + 1 < samples:
time.sleep(interval)
return readings, saw_429
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--samples", type=int, default=12,
help="how many readings to take")
ap.add_argument("--interval", type=float, default=5.0,
help="seconds between readings; keep it above one so the "
"probe is not measuring itself")
ap.add_argument("--limit", type=int, default=None,
help="your account's concurrency ceiling, which is not a "
"readable field: get it from Twilio support")
ap.add_argument("--warn-ratio", type=float, default=0.7,
help="fraction of the ceiling that counts as too close")
args = ap.parse_args()
account = os.environ.get("TWILIO_ACCOUNT_SID")
key = os.environ.get("TWILIO_API_KEY")
secret = os.environ.get("TWILIO_API_SECRET")
if not (account and key and secret):
log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
"(an API Key with read access, not the auth token)")
return 2
session = requests.Session()
session.auth = (key, secret)
readings, saw_429 = probe(session, account, args.samples, args.interval)
seen = [r for r in readings if r is not None]
log.info("%d sample(s), peak %s, %s",
len(readings), max(seen) if seen else "unknown",
"a 20429 was observed" if saw_429 else "no 20429 observed")
state, detail = verdict(readings, args.limit, saw_429, args.warn_ratio)
if state == "headroom":
log.info("%-14s %s", state, detail)
return 0
log.warning("%-14s %s", state, detail)
log.warning(" no console setting fixes this: bound the client's own "
"concurrency below the ceiling with a semaphore or a fixed "
"worker pool")
log.warning(" retry 20429 with exponential backoff and jitter; the request "
"was rejected before processing, so retrying is safe")
log.warning(" a high-volume tenant can be moved into its own subaccount: "
"concurrency is counted per account and does not roll up")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Sample Twilio's REST concurrency header and report how close to the ceiling.
*
* Read only. GET requests and nothing else: give this an API Key with read
* access rather than the account auth token. The repair is printed, never
* performed.
*/
const HOST = 'https://api.twilio.com';
const BASE = `${HOST}/2010-04-01`;
const HEADER = 'twilio-concurrent-requests';
/**
* The concurrency figure out of a response's headers, or null. Pure, and
* case-insensitive by hand so a plain object in a test behaves like the real
* Headers instance.
*/
export function concurrencyOf(headers) {
const entries = headers instanceof Headers
? [...headers.entries()]
: Object.entries(headers ?? {});
for (const [name, value] of entries) {
if (String(name).toLowerCase() !== HEADER) continue;
const n = Number.parseInt(String(value).trim(), 10);
return Number.isFinite(n) ? n : null;
}
return null;
}
/**
* Classify a run of concurrency samples. Pure, so the thresholds are testable
* without waiting for a real peak. Returns [state, detail].
*/
export function verdict(samples, limit = null, saw429 = false, warnRatio = 0.7) {
const readings = (samples ?? []).filter((s) => s !== null && s !== undefined);
if (saw429) {
const peak = readings.length ? Math.max(...readings) : 0;
return ['throttled',
`a 429 came back during the sample itself, at a peak concurrency of ${peak}: ` +
'the account is at its ceiling right now, and every rejected request still ' +
'took a slot to be rejected.'];
}
if (!readings.length) {
return ['no-header',
`no Twilio-Concurrent-Requests header on any of the ${(samples ?? []).length} ` +
'sample(s): with nothing to read, this check cannot say anything about ' +
'concurrency.'];
}
const peak = Math.max(...readings);
if (limit === null || limit === undefined) {
return ['unmeasured',
`peak concurrency ${peak} over ${readings.length} sample(s), and no ceiling ` +
'to compare it against: the limit is not a readable field, so pass the one ' +
'your account has with --limit.'];
}
const ratio = peak / limit;
if (ratio >= 1) {
return ['at-limit',
`peak concurrency ${peak} against a ${limit} ceiling: requests are being ` +
'refused with 20429 at the top of every burst.'];
}
if (ratio >= warnRatio) {
return ['near-limit',
`peak concurrency ${peak} of a ${limit} ceiling (${Math.round(ratio * 100)}%): ` +
'one slow patch downstream lengthens every in-flight request and closes that ' +
'gap without your traffic changing at all.'];
}
return ['headroom',
`peak concurrency ${peak} of a ${limit} ceiling (${Math.round(ratio * 100)}%).`];
}
function authHeader(key, secret) {
return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
export async function probe(auth, account, samples, interval) {
const readings = [];
let saw429 = false;
const url = `${BASE}/Accounts/${account}.json`;
for (let i = 0; i < samples; i += 1) {
const res = await fetch(url, { headers: { Authorization: auth } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
'that the API key belongs to that account with read access');
}
if (res.status === 429) saw429 = true;
const value = concurrencyOf(res.headers);
readings.push(value);
console.log(` sample ${String(i + 1).padStart(2)}: ` +
`${value === null ? 'no header' : `${value} in flight`}`);
if (i + 1 < samples) await sleep(interval * 1000);
}
return [readings, saw429];
}
async function main() {
const account = process.env.TWILIO_ACCOUNT_SID;
const key = process.env.TWILIO_API_KEY;
const secret = process.env.TWILIO_API_SECRET;
if (!account || !key || !secret) {
console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
'(an API Key with read access, not the auth token)');
process.exitCode = 2;
return;
}
const auth = authHeader(key, secret);
const arg = (name, fallback) => {
const i = process.argv.indexOf(name);
return i === -1 ? fallback : Number.parseFloat(process.argv[i + 1]);
};
const samples = arg('--samples', 12);
const interval = arg('--interval', 5);
const limit = process.argv.includes('--limit') ? arg('--limit', null) : null;
const warnRatio = arg('--warn-ratio', 0.7);
const [readings, saw429] = await probe(auth, account, samples, interval);
const seen = readings.filter((r) => r !== null);
console.log(`${readings.length} sample(s), peak ` +
`${seen.length ? Math.max(...seen) : 'unknown'}, ` +
`${saw429 ? 'a 20429 was observed' : 'no 20429 observed'}`);
const [state, detail] = verdict(readings, limit, saw429, warnRatio);
if (state === 'headroom') {
console.log(`${state.padEnd(14)} ${detail}`);
return;
}
console.warn(`${state.padEnd(14)} ${detail}`);
console.warn(' no console setting fixes this: bound the client\'s own ' +
'concurrency below the ceiling with a semaphore or a fixed worker pool');
console.warn(' retry 20429 with exponential backoff and jitter; the request was ' +
'rejected before processing, so retrying is safe');
console.warn(' a high-volume tenant can be moved into its own subaccount: ' +
'concurrency is counted per account and does not roll up');
process.exitCode = 1;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The header lookup gets its own cases because header casing differs between a real response object and the dictionary anybody writes in a test, and a check that silently returns nothing is worse than one that fails. The rest fixes the boundaries: a run with no ceiling is not a finding, an observed 429 outranks every reading, and 70% of the limit is close enough to say so.
from twilio_concurrency_probe import concurrency_of, verdict
def test_the_header_is_read_whatever_its_casing():
assert concurrency_of({"Twilio-Concurrent-Requests": "7"}) == 7
assert concurrency_of({"twilio-concurrent-requests": " 12 "}) == 12
def test_a_missing_or_unusable_header_is_none_rather_than_zero():
assert concurrency_of({}) is None
assert concurrency_of({"Content-Type": "application/json"}) is None
assert concurrency_of({"Twilio-Concurrent-Requests": "many"}) is None
def test_no_header_anywhere_is_reported_as_unmeasurable():
state, detail = verdict([None, None, None])
assert state == "no-header"
assert "3 sample(s)" in detail
def test_samples_with_no_ceiling_are_an_observation_not_a_finding():
state, detail = verdict([3, 5, 4])
assert state == "unmeasured"
assert "peak concurrency 5" in detail
def test_a_quiet_account_against_a_real_ceiling_has_headroom():
state, _ = verdict([3, 5, 4], limit=100)
assert state == "headroom"
def test_seventy_percent_of_the_ceiling_is_close_enough_to_warn():
state, detail = verdict([40, 70, 55], limit=100)
assert state == "near-limit"
assert "70%" in detail
def test_touching_the_ceiling_is_the_20429():
state, detail = verdict([98, 100], limit=100)
assert state == "at-limit"
assert "20429" in detail
def test_an_observed_429_outranks_every_reading():
state, detail = verdict([2, 3], limit=100, saw_429=True)
assert state == "throttled"
assert "a peak concurrency of 3" in detail
def test_a_429_with_no_readings_still_reports_rather_than_crashing():
state, _ = verdict([None], limit=100, saw_429=True)
assert state == "throttled"
def test_the_warn_ratio_is_adjustable():
assert verdict([50], limit=100, warn_ratio=0.4)[0] == "near-limit"
assert verdict([50], limit=100, warn_ratio=0.9)[0] == "headroom"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { concurrencyOf, verdict } from './twilio-concurrency-probe.mjs';
test('the header is read whatever its casing', () => {
assert.equal(concurrencyOf({ 'Twilio-Concurrent-Requests': '7' }), 7);
assert.equal(concurrencyOf({ 'twilio-concurrent-requests': ' 12 ' }), 12);
});
test('a missing or unusable header is null rather than zero', () => {
assert.equal(concurrencyOf({}), null);
assert.equal(concurrencyOf({ 'Content-Type': 'application/json' }), null);
assert.equal(concurrencyOf({ 'Twilio-Concurrent-Requests': 'many' }), null);
});
test('a real Headers instance is read the same way', () => {
assert.equal(concurrencyOf(new Headers({ 'Twilio-Concurrent-Requests': '9' })), 9);
});
test('no header anywhere is reported as unmeasurable', () => {
const [state, detail] = verdict([null, null, null]);
assert.equal(state, 'no-header');
assert.match(detail, /3 sample\(s\)/);
});
test('samples with no ceiling are an observation, not a finding', () => {
const [state, detail] = verdict([3, 5, 4]);
assert.equal(state, 'unmeasured');
assert.match(detail, /peak concurrency 5/);
});
test('a quiet account against a real ceiling has headroom', () => {
assert.equal(verdict([3, 5, 4], 100)[0], 'headroom');
});
test('seventy percent of the ceiling is close enough to warn', () => {
const [state, detail] = verdict([40, 70, 55], 100);
assert.equal(state, 'near-limit');
assert.match(detail, /70%/);
});
test('touching the ceiling is the 20429', () => {
const [state, detail] = verdict([98, 100], 100);
assert.equal(state, 'at-limit');
assert.match(detail, /20429/);
});
test('an observed 429 outranks every reading', () => {
const [state, detail] = verdict([2, 3], 100, true);
assert.equal(state, 'throttled');
assert.match(detail, /a peak concurrency of 3/);
});
test('a 429 with no readings still reports rather than crashing', () => {
assert.equal(verdict([null], 100, true)[0], 'throttled');
});
test('the warn ratio is adjustable', () => {
assert.equal(verdict([50], 100, false, 0.4)[0], 'near-limit');
assert.equal(verdict([50], 100, false, 0.9)[0], 'headroom');
});
FAQ
What is my account's concurrency limit?
It is an account property rather than a documented constant, and it is not returned by any endpoint, which is why this script takes it as an argument. Twilio support will tell you what yours is and will discuss raising it for a documented workload. Until you have the number, the probe still reports your observed peak, which is the half of the comparison you can gather yourself.
How is this different from a rate limit?
A rate limit counts requests per unit of time; concurrency counts requests that have been sent and not yet answered. You can breach a concurrency limit without your request rate changing at all, simply because responses got slower and the in-flight pile grew. That is the usual cause, and it is why the fix is a bound on parallelism rather than a delay between calls.
Is it safe to retry a 20429?
Yes. The request was refused before it was processed, so nothing was partially created and no message was sent. What is not safe is retrying immediately: without exponential backoff and jitter, every rejected caller retries at the same moment and reproduces the burst. Retrying wrong is what turns a two-second problem into a two-minute one.
Does the probe itself affect the number?
Slightly, and honestly so. The request carrying the header is in flight while the figure is computed, so an idle account reads 1 rather than 0. At a sampling interval of seconds that contribution is one request and negligible; at an interval of milliseconds you would be measuring your own probe, which is why the default interval is five seconds.
Will splitting into subaccounts really help?
It genuinely separates the budgets, because concurrency is counted against the account SID the request is made under and subaccount usage does not roll up into the parent's tally. It is a real lever for isolating one loud tenant. It is also a structural change with its own operational cost, so cap the client first and shard only when a single client cannot fit under the ceiling.
Related field notes
- The other burst failure, where the queue rather than the API gives way
- Sharding across subaccounts, and the way one goes quiet
- The credential this probe should be holding
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Error 20429: too many requests — Twilio Docs
- REST API best practices — Twilio Docs
- Twilio REST API — Twilio Docs
- Subaccounts — Twilio Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.