Diagnostic Twilio
the balance is one busy hour from a 20005 suspension
The account had ninety dollars on it, which had been plenty for eight months. Then a product launch put four times the usual traffic through it in one evening, the balance crossed zero somewhere around nine o'clock, and every send after that came back 20005. Nobody had done anything wrong. The number in Balance.json had simply stopped being large enough, and nothing in the system's job was to notice that.
Read GET /2010-04-01/Accounts/{AccountSid}/Balance.json for balance and currency, then read GET /2010-04-01/Accounts/{AccountSid}/Usage/Records/Daily.json?Category=totalprice over the last month for usage_records[].price. Divide one by the other. That quotient is days of runway, and it is the number worth alerting on.
Twilio is prepay by default: when the balance reaches zero the account is suspended, not slowed. Compare the balance against the busiest day in the window as well as the median one, because the day that empties the account is never the median day.
The problem in plain words
A balance is a stock and your traffic is a flow, and a healthy-looking stock tells you nothing on its own. Ninety dollars is six months of runway on an account that sends a hundred messages a day and forty minutes of runway on an account running a campaign. The console shows you the stock. Nothing in the console or the API divides it by the flow, which is the only form of the number that means anything.
The failure is also discontinuous, and that is what makes it expensive. Twilio does not throttle you down as the balance thins; the account runs at full speed until the balance is gone and then stops entirely, with 20005 on every REST call and 30002 on everything already queued. There is no degraded mode in between where somebody notices something is off. The first symptom is total outage, and it arrives during peak traffic by construction, because peak traffic is what consumed the balance.
Why it happens
Auto Recharge fails silently, and its state is not readable. The card expires, the issuer declines an unusual amount, the billing email goes to a mailbox nobody has opened since the person who set it up left. There is no API field that says "auto recharge is enabled and last succeeded", so the only evidence you can gather through the API is the balance itself and the fact that it is not going back up.
The account average is not the risk. Burn rate computed as a monthly total divided by thirty flatters every account with spiky traffic, which is most of them. A launch day, a end-of-month billing run or an OTP surge can cost five times a normal day, and the balance has to survive that day rather than the average one.
Suspension cascades to subaccounts. The parent runs out and every tenant underneath it stops at the same instant, including the ones whose own usage was trivial. One balance is a shared single point of failure across every tenant you have.
Reactivation is not instant. Adding funds clears the suspension after a delay measured in minutes, and anything that was queued has already failed with 30002 and will not be retried for you. The recovery is not just the payment, it is the replay of everything the outage dropped.
The fix, as a flow
The classifier divides one number by another, and the whole judgement lives in which day it divides by. The median day gives the runway; the busiest day already seen is the one that ends the account.
How to fix it
Read the balance and the currency together
GET /2010-04-01/Accounts/{AccountSid}/Balance.json returns balance as a string and currency beside it. Keep the currency: an account billed in EUR compared against a USD threshold somebody hardcoded is a check that reports the wrong answer with total confidence.
Get a real burn rate, not a monthly average
GET /2010-04-01/Accounts/{AccountSid}/Usage/Records/Daily.json?Category=totalprice&StartDate={30d ago} gives one row per day with a price on it. totalprice is the category that captures spend wherever it happened, so a voice bill and a messaging bill both land in it.
Divide by the median day and check against the busiest one
The median day is the honest denominator for runway, because a single quiet weekend does not deserve to double your estimate. The maximum day is the one that answers the question in the title: if the whole balance is smaller than one day you have already had, the account is one repeat of that day away from a suspension.
Pick a floor and treat crossing it as an incident
Seven days is a reasonable floor: long enough that a failed card, a weekend and a purchasing process all fit inside it. The exact number matters less than having one, because a threshold you can compute is a threshold you can page on, and a balance you merely look at is not.
Fix the recharge, then set the alarm that catches the next one
Console → Billing → Manage billing, enable Auto Recharge with a trigger amount of at least a week of spend and a card that is not about to expire. Then set the spend alarm, which is a different check with a different failure mode: no Usage Trigger, so overspend runs with nothing watching.
How to check it worked
Re-run the script. The state should be ok and the runway should be comfortably past your floor.
python3 twilio_balance_runway.py --floor-days 7
# balance 412.80 USD over 30 day(s)
# ok balance 412.80 against a median day of 9.44: 43.7 days of runway.
The full code
Two GETs: the balance, and one page of daily usage records. A read-access API Key is enough and is what you should give it, because the credential this holds could otherwise spend the very balance it is measuring. The arithmetic — the median, the runway, and the comparison against the busiest day — is a pure function, since that is the part worth arguing about and the part worth testing.
"""Report a Twilio balance that will not survive the next busy day.
Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. The repair is printed, never performed,
because this script holds a credential to an account that can send messages and
spend money.
"""
import argparse
import datetime
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_balance_runway")
HOST = "https://api.twilio.com"
BASE = HOST + "/2010-04-01"
SPEND_CATEGORY = "totalprice"
def price_of(record):
"""One daily usage record as a float, or None when the field is unusable.
price arrives as a string. A negative day is a credit or an adjustment
rather than spend, and letting it through would drag the burn rate down and
report runway the account does not have, so it is clamped at zero.
"""
try:
value = float(record.get("price"))
except (TypeError, ValueError):
return None
return max(0.0, value)
def daily_prices(records):
"""The parseable daily prices out of a Usage/Records/Daily page. Pure."""
return [p for p in (price_of(r) for r in records or []) if p is not None]
def median(values):
"""Median of a list of floats, 0.0 when empty.
The median rather than the mean because one launch day in thirty should not
be allowed to flatten into a burn rate that looks survivable.
"""
ordered = sorted(values or [])
n = len(ordered)
if not n:
return 0.0
mid = n // 2
if n % 2:
return ordered[mid]
return (ordered[mid - 1] + ordered[mid]) / 2.0
def runway_days(balance, rate):
"""Days the balance covers at a given daily rate, or None at a zero rate."""
if not rate or rate <= 0:
return None
return balance / rate
def verdict(balance, prices, floor_days=7.0):
"""Classify a balance against the spend behind it. Pure, so the arithmetic
that decides whether somebody gets paged is testable without a network.
Returns (state, detail).
"""
if balance is None:
return ("unknown",
"Balance.json returned no usable balance: with no number there is "
"nothing to divide by a burn rate, and the check cannot answer.")
values = list(prices or [])
typical = median(values)
peak = max(values) if values else 0.0
if balance <= 0:
return ("empty",
"balance is %.2f: this is the state Twilio suspends on rather than "
"throttles, so REST calls come back 20005 and anything already "
"queued fails 30002." % balance)
if typical <= 0:
return ("idle",
"balance %.2f and no priced usage in the window: there is no burn "
"rate to divide by, so the floor has to come from the spend you "
"expect rather than the spend you have had." % balance)
days = balance / typical
if days < 1.0:
return ("critical",
"balance %.2f against a median day of %.2f: under one ordinary day "
"of runway left." % (balance, typical))
if days < floor_days:
return ("low",
"balance %.2f against a median day of %.2f: %.1f days of runway, "
"below the %.0f-day floor." % (balance, typical, days, floor_days))
if balance < peak:
return ("burst-exposed",
"%.1f days of runway at the median day of %.2f, but the busiest day "
"in the window cost %.2f, more than the entire balance: one repeat "
"of that day ends in a suspension." % (days, typical, peak))
return ("ok",
"balance %.2f against a median day of %.2f: %.1f days of runway."
% (balance, typical, days))
def get(session, url, **params):
r = session.get(url, params=params, timeout=30)
if r.status_code in (401, 403):
raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that the "
"API key belongs to that account with read access"
% r.status_code)
r.raise_for_status()
return r.json()
def read_balance(session, account):
"""Returns (balance as float or None, currency)."""
page = get(session, "%s/Accounts/%s/Balance.json" % (BASE, account))
try:
return (float(page.get("balance")), page.get("currency") or "")
except (TypeError, ValueError):
return (None, page.get("currency") or "")
def read_daily(session, account, days):
"""One page of daily totalprice records covering the requested window."""
start = (datetime.date.today() - datetime.timedelta(days=days)).isoformat()
page = get(session, "%s/Accounts/%s/Usage/Records/Daily.json" % (BASE, account),
Category=SPEND_CATEGORY, StartDate=start, PageSize=100)
return page.get("usage_records", [])
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30,
help="how many days of usage the burn rate is taken from")
ap.add_argument("--floor-days", type=float, default=7.0,
help="days of runway below which the balance is a finding")
args = ap.parse_args()
account = os.environ.get("TWILIO_ACCOUNT_SID")
key = os.environ.get("TWILIO_API_KEY")
secret = os.environ.get("TWILIO_API_SECRET")
if not (account and key and secret):
log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
"(an API Key with read access, not the auth token)")
return 2
session = requests.Session()
session.auth = (key, secret)
balance, currency = read_balance(session, account)
prices = daily_prices(read_daily(session, account, args.days))
log.info("balance %s %s over %d day(s)",
"unreadable" if balance is None else "%.2f" % balance,
currency, args.days)
state, detail = verdict(balance, prices, args.floor_days)
if state == "ok":
log.info("%-14s %s", state, detail)
return 0
log.warning("%-14s %s", state, detail)
if prices:
needed = median(prices) * args.floor_days
log.warning(" %.0f days at the median day is %.2f %s: keep the recharge "
"trigger at or above that", args.floor_days, needed, currency)
log.warning(" repair: Console > Billing > Manage billing > Auto Recharge, with "
"a trigger amount of at least %.0f days of spend and a card that is "
"not about to expire", args.floor_days)
log.warning(" auto recharge state is not exposed by the API: the only evidence "
"it is working is this balance going back up")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Report a Twilio balance that will not survive the next busy day.
*
* Read only. GET requests and nothing else: give this an API Key with read
* access rather than the account auth token. The repair is printed, never
* performed.
*/
const HOST = 'https://api.twilio.com';
const BASE = `${HOST}/2010-04-01`;
const SPEND_CATEGORY = 'totalprice';
/**
* One daily usage record as a number, or null when the field is unusable.
* A negative day is a credit rather than spend and is clamped at zero, because
* letting it through reports runway the account does not have.
*/
export function priceOf(record) {
const value = Number.parseFloat(record?.price ?? '');
if (!Number.isFinite(value)) return null;
return Math.max(0, value);
}
/** The parseable daily prices out of a Usage/Records/Daily page. Pure. */
export function dailyPrices(records) {
return (records ?? []).map(priceOf).filter((p) => p !== null);
}
/** Median of a list of numbers, 0 when empty. Pure. */
export function median(values) {
const ordered = [...(values ?? [])].sort((a, b) => a - b);
if (!ordered.length) return 0;
const mid = Math.floor(ordered.length / 2);
return ordered.length % 2 ? ordered[mid] : (ordered[mid - 1] + ordered[mid]) / 2;
}
/** Days the balance covers at a given daily rate, or null at a zero rate. */
export function runwayDays(balance, rate) {
if (!rate || rate <= 0) return null;
return balance / rate;
}
/**
* Classify a balance against the spend behind it. Pure, so the arithmetic that
* decides whether somebody gets paged is testable without a network.
* Returns [state, detail].
*/
export function verdict(balance, prices, floorDays = 7.0) {
if (balance === null || balance === undefined || !Number.isFinite(balance)) {
return ['unknown',
'Balance.json returned no usable balance: with no number there is nothing ' +
'to divide by a burn rate, and the check cannot answer.'];
}
const values = [...(prices ?? [])];
const typical = median(values);
const peak = values.length ? Math.max(...values) : 0;
if (balance <= 0) {
return ['empty',
`balance is ${balance.toFixed(2)}: this is the state Twilio suspends on ` +
'rather than throttles, so REST calls come back 20005 and anything already ' +
'queued fails 30002.'];
}
if (typical <= 0) {
return ['idle',
`balance ${balance.toFixed(2)} and no priced usage in the window: there is ` +
'no burn rate to divide by, so the floor has to come from the spend you ' +
'expect rather than the spend you have had.'];
}
const days = balance / typical;
if (days < 1) {
return ['critical',
`balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
'under one ordinary day of runway left.'];
}
if (days < floorDays) {
return ['low',
`balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
`${days.toFixed(1)} days of runway, below the ${floorDays.toFixed(0)}-day floor.`];
}
if (balance < peak) {
return ['burst-exposed',
`${days.toFixed(1)} days of runway at the median day of ${typical.toFixed(2)}, ` +
`but the busiest day in the window cost ${peak.toFixed(2)}, more than the ` +
'entire balance: one repeat of that day ends in a suspension.'];
}
return ['ok',
`balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
`${days.toFixed(1)} days of runway.`];
}
function authHeader(key, secret) {
return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}
async function get(auth, url, params = {}) {
const u = new URL(url);
for (const [k, v] of Object.entries(params)) u.searchParams.set(k, v);
const res = await fetch(u, { headers: { Authorization: auth } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
'that the API key belongs to that account with read access');
}
if (!res.ok) throw new Error(`${res.status} from ${u.pathname}`);
return res.json();
}
export async function readBalance(auth, account) {
const page = await get(auth, `${BASE}/Accounts/${account}/Balance.json`);
const value = Number.parseFloat(page.balance ?? '');
return [Number.isFinite(value) ? value : null, page.currency ?? ''];
}
export async function readDaily(auth, account, days) {
const start = new Date(Date.now() - days * 86400000).toISOString().slice(0, 10);
const page = await get(auth, `${BASE}/Accounts/${account}/Usage/Records/Daily.json`,
{ Category: SPEND_CATEGORY, StartDate: start, PageSize: 100 });
return page.usage_records ?? [];
}
async function main() {
const account = process.env.TWILIO_ACCOUNT_SID;
const key = process.env.TWILIO_API_KEY;
const secret = process.env.TWILIO_API_SECRET;
if (!account || !key || !secret) {
console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
'(an API Key with read access, not the auth token)');
process.exitCode = 2;
return;
}
const auth = authHeader(key, secret);
const arg = (name, fallback) => {
const i = process.argv.indexOf(name);
return i === -1 ? fallback : Number.parseFloat(process.argv[i + 1]);
};
const days = arg('--days', 30);
const floorDays = arg('--floor-days', 7);
const [balance, currency] = await readBalance(auth, account);
const prices = dailyPrices(await readDaily(auth, account, days));
console.log(`balance ${balance === null ? 'unreadable' : balance.toFixed(2)} ` +
`${currency} over ${days} day(s)`);
const [state, detail] = verdict(balance, prices, floorDays);
if (state === 'ok') {
console.log(`${state.padEnd(14)} ${detail}`);
return;
}
console.warn(`${state.padEnd(14)} ${detail}`);
if (prices.length) {
const needed = median(prices) * floorDays;
console.warn(` ${floorDays.toFixed(0)} days at the median day is ` +
`${needed.toFixed(2)} ${currency}: keep the recharge trigger at ` +
'or above that');
}
console.warn(' repair: Console > Billing > Manage billing > Auto Recharge, with ' +
`a trigger amount of at least ${floorDays.toFixed(0)} days of spend ` +
'and a card that is not about to expire');
console.warn(' auto recharge state is not exposed by the API: the only evidence ' +
'it is working is this balance going back up');
process.exitCode = 1;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The cases that matter are the ones where a healthy-looking balance is not healthy: a month of quiet days hiding one enormous one, a negative usage row that would otherwise flatter the burn rate, and an account with no spend at all, where dividing by zero is not an answer and saying so is. The rest pins the boundary either side of the floor.
from twilio_balance_runway import (daily_prices, median, price_of, runway_days,
verdict)
def test_median_of_an_even_run_is_the_middle_pair():
assert median([1.0, 2.0, 3.0, 4.0]) == 2.5
assert median([3.0, 1.0, 2.0]) == 2.0
assert median([]) == 0.0
def test_a_credit_day_is_clamped_rather_than_subtracted():
assert price_of({"price": "-42.00"}) == 0.0
assert price_of({"price": "12.50"}) == 12.5
def test_unparseable_prices_are_dropped_not_guessed():
assert price_of({"price": None}) is None
assert price_of({"price": "n/a"}) is None
assert daily_prices([{"price": "4"}, {"price": "x"}, {}]) == [4.0]
def test_runway_is_undefined_at_a_zero_burn_rate():
assert runway_days(100.0, 0.0) is None
assert runway_days(100.0, 10.0) == 10.0
def test_a_missing_balance_is_reported_rather_than_assumed_healthy():
state, _ = verdict(None, [{"price": "10"}])
assert state == "unknown"
def test_a_zero_balance_is_already_the_suspension():
state, detail = verdict(0.0, [10.0, 10.0])
assert state == "empty"
assert "20005" in detail
def test_an_account_with_no_spend_has_no_runway_to_compute():
state, _ = verdict(500.0, [])
assert state == "idle"
def test_under_one_median_day_is_critical():
state, _ = verdict(5.0, [10.0, 10.0, 10.0])
assert state == "critical"
def test_four_days_of_runway_is_below_a_seven_day_floor():
state, detail = verdict(40.0, [10.0, 10.0, 10.0])
assert state == "low"
assert "4.0 days" in detail
def test_a_quiet_median_hides_a_day_bigger_than_the_whole_balance():
state, detail = verdict(500.0, [1.0, 1.0, 900.0])
assert state == "burst-exposed"
assert "900.00" in detail
def test_a_balance_past_the_floor_and_past_the_busiest_day_is_fine():
state, _ = verdict(10000.0, [10.0, 10.0, 12.0])
assert state == "ok"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { dailyPrices, median, priceOf, runwayDays, verdict }
from './twilio-balance-runway.mjs';
test('median of an even run is the middle pair', () => {
assert.equal(median([1, 2, 3, 4]), 2.5);
assert.equal(median([3, 1, 2]), 2);
assert.equal(median([]), 0);
});
test('a credit day is clamped rather than subtracted', () => {
assert.equal(priceOf({ price: '-42.00' }), 0);
assert.equal(priceOf({ price: '12.50' }), 12.5);
});
test('unparseable prices are dropped, not guessed', () => {
assert.equal(priceOf({ price: null }), null);
assert.equal(priceOf({ price: 'n/a' }), null);
assert.deepEqual(dailyPrices([{ price: '4' }, { price: 'x' }, {}]), [4]);
});
test('runway is undefined at a zero burn rate', () => {
assert.equal(runwayDays(100, 0), null);
assert.equal(runwayDays(100, 10), 10);
});
test('a missing balance is reported rather than assumed healthy', () => {
assert.equal(verdict(null, [10])[0], 'unknown');
});
test('a zero balance is already the suspension', () => {
const [state, detail] = verdict(0, [10, 10]);
assert.equal(state, 'empty');
assert.match(detail, /20005/);
});
test('an account with no spend has no runway to compute', () => {
assert.equal(verdict(500, [])[0], 'idle');
});
test('under one median day is critical', () => {
assert.equal(verdict(5, [10, 10, 10])[0], 'critical');
});
test('four days of runway is below a seven-day floor', () => {
const [state, detail] = verdict(40, [10, 10, 10]);
assert.equal(state, 'low');
assert.match(detail, /4\.0 days/);
});
test('a quiet median hides a day bigger than the whole balance', () => {
const [state, detail] = verdict(500, [1, 1, 900]);
assert.equal(state, 'burst-exposed');
assert.match(detail, /900\.00/);
});
test('a balance past the floor and past the busiest day is fine', () => {
assert.equal(verdict(10000, [10, 10, 12])[0], 'ok');
});
FAQ
Does Twilio not email me before the balance runs out?
There are balance notification emails to the account owner, and they are a mailbox rather than a pager. They are also not something you can assert on: no API field tells you whether the notification is enabled, where it goes, or whether it was delivered. The balance and the daily usage records are readable, which is why the check is built on those two numbers and not on your inbox.
Why the median day rather than the average?
Because the mean is dragged around by exactly the days you care about. One launch day at twenty times normal pushes the mean up, which makes the runway estimate look shorter than the account's ordinary behaviour warrants, and a fortnight of quiet weekends pushes it down. The median describes an ordinary day, and the maximum, checked separately, describes the day that will actually empty the account.
What should the floor be?
Seven days is a defensible starting point because it spans a weekend, a declined card and a purchase approval. If your recharge is automatic and reliable, three days is fine; if funding the account involves a finance team and a purchase order, fourteen is not paranoid. The value of the floor is that it is a number the script can compare against, not that it is the right number.
Will adding funds bring the account straight back?
The suspension clears a few minutes after the payment, but that is not the whole recovery. Everything that was queued during the outage has already failed with 30002, and Twilio does not replay it. Whatever your application does about undelivered messages has to run afterwards, so the outage is longer than the suspension was.
Should this run per subaccount?
Read the balance on the parent, because that is where the money is: subaccounts bill up to the parent rather than holding their own balance. What is worth running per subaccount is the status check, since a parent suspension cascades and one tenant can also be suspended on its own. That is covered separately in the subaccount note.
Related field notes
- The spend alarm that would have caught the burn
- One tenant suspended while the parent looks healthy
- Numbers billed every month for carrying nothing
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Account resource — Twilio Docs
- UsageRecord resource — Twilio Docs
- Error 20005: account not active — Twilio Docs
- UsageTrigger resource — Twilio Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.