Skip to content

Diagnostic Twilio

the balance is one busy hour from a 20005 suspension

The account had ninety dollars on it, which had been plenty for eight months. Then a product launch put four times the usual traffic through it in one evening, the balance crossed zero somewhere around nine o'clock, and every send after that came back 20005. Nobody had done anything wrong. The number in Balance.json had simply stopped being large enough, and nothing in the system's job was to notice that.

Read-only key Python and Node.js Tests included
Person using calculator at desk with coffee mug
Photo by Towfiqu barbhuiya on Unsplash
The short answer

Read GET /2010-04-01/Accounts/{AccountSid}/Balance.json for balance and currency, then read GET /2010-04-01/Accounts/{AccountSid}/Usage/Records/Daily.json?Category=totalprice over the last month for usage_records[].price. Divide one by the other. That quotient is days of runway, and it is the number worth alerting on.

Twilio is prepay by default: when the balance reaches zero the account is suspended, not slowed. Compare the balance against the busiest day in the window as well as the median one, because the day that empties the account is never the median day.

The problem in plain words

A balance is a stock and your traffic is a flow, and a healthy-looking stock tells you nothing on its own. Ninety dollars is six months of runway on an account that sends a hundred messages a day and forty minutes of runway on an account running a campaign. The console shows you the stock. Nothing in the console or the API divides it by the flow, which is the only form of the number that means anything.

The failure is also discontinuous, and that is what makes it expensive. Twilio does not throttle you down as the balance thins; the account runs at full speed until the balance is gone and then stops entirely, with 20005 on every REST call and 30002 on everything already queued. There is no degraded mode in between where somebody notices something is off. The first symptom is total outage, and it arrives during peak traffic by construction, because peak traffic is what consumed the balance.

Balance looksfinemonths at the oldrateTrafficmultipliesa launch, acampaignBalance crosseszerono throttle, nowarningAccountsuspended20005 on everycallQueue fails too30002, nothingreplayed
Nothing degrades on the way down. The account runs at full speed until the balance is gone, and then refuses everything at once.

Why it happens

Auto Recharge fails silently, and its state is not readable. The card expires, the issuer declines an unusual amount, the billing email goes to a mailbox nobody has opened since the person who set it up left. There is no API field that says "auto recharge is enabled and last succeeded", so the only evidence you can gather through the API is the balance itself and the fact that it is not going back up.

The account average is not the risk. Burn rate computed as a monthly total divided by thirty flatters every account with spiky traffic, which is most of them. A launch day, a end-of-month billing run or an OTP surge can cost five times a normal day, and the balance has to survive that day rather than the average one.

Suspension cascades to subaccounts. The parent runs out and every tenant underneath it stops at the same instant, including the ones whose own usage was trivial. One balance is a shared single point of failure across every tenant you have.

Reactivation is not instant. Adding funds clears the suspension after a delay measured in minutes, and anything that was queued has already failed with 30002 and will not be retried for you. The recovery is not just the payment, it is the replay of everything the outage dropped.

The fix, as a flow

The classifier divides one number by another, and the whole judgement lives in which day it divides by. The median day gives the runway; the busiest day already seen is the one that ends the account.

Balance.json and daily usagebalance over the median dayPast the floor and the peakleave itNo priced usage at allno rate to divide byUnder the seven day floora card, a weekend, goneSmaller than one busy dayone repeat suspends it
A stock divided by a flow. Ninety dollars is six months on one account and forty minutes on another, and only the quotient says which.

How to fix it

Read the balance and the currency together

GET /2010-04-01/Accounts/{AccountSid}/Balance.json returns balance as a string and currency beside it. Keep the currency: an account billed in EUR compared against a USD threshold somebody hardcoded is a check that reports the wrong answer with total confidence.

Get a real burn rate, not a monthly average

GET /2010-04-01/Accounts/{AccountSid}/Usage/Records/Daily.json?Category=totalprice&StartDate={30d ago} gives one row per day with a price on it. totalprice is the category that captures spend wherever it happened, so a voice bill and a messaging bill both land in it.

Divide by the median day and check against the busiest one

The median day is the honest denominator for runway, because a single quiet weekend does not deserve to double your estimate. The maximum day is the one that answers the question in the title: if the whole balance is smaller than one day you have already had, the account is one repeat of that day away from a suspension.

Pick a floor and treat crossing it as an incident

Seven days is a reasonable floor: long enough that a failed card, a weekend and a purchasing process all fit inside it. The exact number matters less than having one, because a threshold you can compute is a threshold you can page on, and a balance you merely look at is not.

Fix the recharge, then set the alarm that catches the next one

Console → Billing → Manage billing, enable Auto Recharge with a trigger amount of at least a week of spend and a card that is not about to expire. Then set the spend alarm, which is a different check with a different failure mode: no Usage Trigger, so overspend runs with nothing watching.

How to check it worked

Re-run the script. The state should be ok and the runway should be comfortably past your floor.

python3 twilio_balance_runway.py --floor-days 7
# balance 412.80 USD over 30 day(s)
# ok             balance 412.80 against a median day of 9.44: 43.7 days of runway.

The full code

Two GETs: the balance, and one page of daily usage records. A read-access API Key is enough and is what you should give it, because the credential this holds could otherwise spend the very balance it is measuring. The arithmetic — the median, the runway, and the comparison against the busiest day — is a pure function, since that is the part worth arguing about and the part worth testing.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Read only, it never writes. One of 117 Twilio fixes, free and open source.
twilio_balance_runway.py
"""Report a Twilio balance that will not survive the next busy day.

Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. The repair is printed, never performed,
because this script holds a credential to an account that can send messages and
spend money.
"""
import argparse
import datetime
import logging
import os
import sys

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_balance_runway")

HOST = "https://api.twilio.com"
BASE = HOST + "/2010-04-01"

SPEND_CATEGORY = "totalprice"


def price_of(record):
    """One daily usage record as a float, or None when the field is unusable.

    price arrives as a string. A negative day is a credit or an adjustment
    rather than spend, and letting it through would drag the burn rate down and
    report runway the account does not have, so it is clamped at zero.
    """
    try:
        value = float(record.get("price"))
    except (TypeError, ValueError):
        return None
    return max(0.0, value)


def daily_prices(records):
    """The parseable daily prices out of a Usage/Records/Daily page. Pure."""
    return [p for p in (price_of(r) for r in records or []) if p is not None]


def median(values):
    """Median of a list of floats, 0.0 when empty.

    The median rather than the mean because one launch day in thirty should not
    be allowed to flatten into a burn rate that looks survivable.
    """
    ordered = sorted(values or [])
    n = len(ordered)
    if not n:
        return 0.0
    mid = n // 2
    if n % 2:
        return ordered[mid]
    return (ordered[mid - 1] + ordered[mid]) / 2.0


def runway_days(balance, rate):
    """Days the balance covers at a given daily rate, or None at a zero rate."""
    if not rate or rate <= 0:
        return None
    return balance / rate


def verdict(balance, prices, floor_days=7.0):
    """Classify a balance against the spend behind it. Pure, so the arithmetic
    that decides whether somebody gets paged is testable without a network.

    Returns (state, detail).
    """
    if balance is None:
        return ("unknown",
                "Balance.json returned no usable balance: with no number there is "
                "nothing to divide by a burn rate, and the check cannot answer.")

    values = list(prices or [])
    typical = median(values)
    peak = max(values) if values else 0.0

    if balance <= 0:
        return ("empty",
                "balance is %.2f: this is the state Twilio suspends on rather than "
                "throttles, so REST calls come back 20005 and anything already "
                "queued fails 30002." % balance)

    if typical <= 0:
        return ("idle",
                "balance %.2f and no priced usage in the window: there is no burn "
                "rate to divide by, so the floor has to come from the spend you "
                "expect rather than the spend you have had." % balance)

    days = balance / typical
    if days < 1.0:
        return ("critical",
                "balance %.2f against a median day of %.2f: under one ordinary day "
                "of runway left." % (balance, typical))
    if days < floor_days:
        return ("low",
                "balance %.2f against a median day of %.2f: %.1f days of runway, "
                "below the %.0f-day floor." % (balance, typical, days, floor_days))
    if balance < peak:
        return ("burst-exposed",
                "%.1f days of runway at the median day of %.2f, but the busiest day "
                "in the window cost %.2f, more than the entire balance: one repeat "
                "of that day ends in a suspension." % (days, typical, peak))
    return ("ok",
            "balance %.2f against a median day of %.2f: %.1f days of runway."
            % (balance, typical, days))


def get(session, url, **params):
    r = session.get(url, params=params, timeout=30)
    if r.status_code in (401, 403):
        raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that the "
                         "API key belongs to that account with read access"
                         % r.status_code)
    r.raise_for_status()
    return r.json()


def read_balance(session, account):
    """Returns (balance as float or None, currency)."""
    page = get(session, "%s/Accounts/%s/Balance.json" % (BASE, account))
    try:
        return (float(page.get("balance")), page.get("currency") or "")
    except (TypeError, ValueError):
        return (None, page.get("currency") or "")


def read_daily(session, account, days):
    """One page of daily totalprice records covering the requested window."""
    start = (datetime.date.today() - datetime.timedelta(days=days)).isoformat()
    page = get(session, "%s/Accounts/%s/Usage/Records/Daily.json" % (BASE, account),
               Category=SPEND_CATEGORY, StartDate=start, PageSize=100)
    return page.get("usage_records", [])


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--days", type=int, default=30,
                    help="how many days of usage the burn rate is taken from")
    ap.add_argument("--floor-days", type=float, default=7.0,
                    help="days of runway below which the balance is a finding")
    args = ap.parse_args()

    account = os.environ.get("TWILIO_ACCOUNT_SID")
    key = os.environ.get("TWILIO_API_KEY")
    secret = os.environ.get("TWILIO_API_SECRET")
    if not (account and key and secret):
        log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
                  "(an API Key with read access, not the auth token)")
        return 2

    session = requests.Session()
    session.auth = (key, secret)

    balance, currency = read_balance(session, account)
    prices = daily_prices(read_daily(session, account, args.days))
    log.info("balance %s %s over %d day(s)",
             "unreadable" if balance is None else "%.2f" % balance,
             currency, args.days)

    state, detail = verdict(balance, prices, args.floor_days)
    if state == "ok":
        log.info("%-14s %s", state, detail)
        return 0

    log.warning("%-14s %s", state, detail)
    if prices:
        needed = median(prices) * args.floor_days
        log.warning("  %.0f days at the median day is %.2f %s: keep the recharge "
                    "trigger at or above that", args.floor_days, needed, currency)
    log.warning("  repair: Console > Billing > Manage billing > Auto Recharge, with "
                "a trigger amount of at least %.0f days of spend and a card that is "
                "not about to expire", args.floor_days)
    log.warning("  auto recharge state is not exposed by the API: the only evidence "
                "it is working is this balance going back up")
    return 1


if __name__ == "__main__":
    sys.exit(main())
twilio-balance-runway.mjs
/**
 * Report a Twilio balance that will not survive the next busy day.
 *
 * Read only. GET requests and nothing else: give this an API Key with read
 * access rather than the account auth token. The repair is printed, never
 * performed.
 */
const HOST = 'https://api.twilio.com';
const BASE = `${HOST}/2010-04-01`;

const SPEND_CATEGORY = 'totalprice';

/**
 * One daily usage record as a number, or null when the field is unusable.
 * A negative day is a credit rather than spend and is clamped at zero, because
 * letting it through reports runway the account does not have.
 */
export function priceOf(record) {
  const value = Number.parseFloat(record?.price ?? '');
  if (!Number.isFinite(value)) return null;
  return Math.max(0, value);
}

/** The parseable daily prices out of a Usage/Records/Daily page. Pure. */
export function dailyPrices(records) {
  return (records ?? []).map(priceOf).filter((p) => p !== null);
}

/** Median of a list of numbers, 0 when empty. Pure. */
export function median(values) {
  const ordered = [...(values ?? [])].sort((a, b) => a - b);
  if (!ordered.length) return 0;
  const mid = Math.floor(ordered.length / 2);
  return ordered.length % 2 ? ordered[mid] : (ordered[mid - 1] + ordered[mid]) / 2;
}

/** Days the balance covers at a given daily rate, or null at a zero rate. */
export function runwayDays(balance, rate) {
  if (!rate || rate <= 0) return null;
  return balance / rate;
}

/**
 * Classify a balance against the spend behind it. Pure, so the arithmetic that
 * decides whether somebody gets paged is testable without a network.
 * Returns [state, detail].
 */
export function verdict(balance, prices, floorDays = 7.0) {
  if (balance === null || balance === undefined || !Number.isFinite(balance)) {
    return ['unknown',
      'Balance.json returned no usable balance: with no number there is nothing ' +
      'to divide by a burn rate, and the check cannot answer.'];
  }

  const values = [...(prices ?? [])];
  const typical = median(values);
  const peak = values.length ? Math.max(...values) : 0;

  if (balance <= 0) {
    return ['empty',
      `balance is ${balance.toFixed(2)}: this is the state Twilio suspends on ` +
      'rather than throttles, so REST calls come back 20005 and anything already ' +
      'queued fails 30002.'];
  }

  if (typical <= 0) {
    return ['idle',
      `balance ${balance.toFixed(2)} and no priced usage in the window: there is ` +
      'no burn rate to divide by, so the floor has to come from the spend you ' +
      'expect rather than the spend you have had.'];
  }

  const days = balance / typical;
  if (days < 1) {
    return ['critical',
      `balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
      'under one ordinary day of runway left.'];
  }
  if (days < floorDays) {
    return ['low',
      `balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
      `${days.toFixed(1)} days of runway, below the ${floorDays.toFixed(0)}-day floor.`];
  }
  if (balance < peak) {
    return ['burst-exposed',
      `${days.toFixed(1)} days of runway at the median day of ${typical.toFixed(2)}, ` +
      `but the busiest day in the window cost ${peak.toFixed(2)}, more than the ` +
      'entire balance: one repeat of that day ends in a suspension.'];
  }
  return ['ok',
    `balance ${balance.toFixed(2)} against a median day of ${typical.toFixed(2)}: ` +
    `${days.toFixed(1)} days of runway.`];
}

function authHeader(key, secret) {
  return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}

async function get(auth, url, params = {}) {
  const u = new URL(url);
  for (const [k, v] of Object.entries(params)) u.searchParams.set(k, v);
  const res = await fetch(u, { headers: { Authorization: auth } });
  if (res.status === 401 || res.status === 403) {
    throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
                    'that the API key belongs to that account with read access');
  }
  if (!res.ok) throw new Error(`${res.status} from ${u.pathname}`);
  return res.json();
}

export async function readBalance(auth, account) {
  const page = await get(auth, `${BASE}/Accounts/${account}/Balance.json`);
  const value = Number.parseFloat(page.balance ?? '');
  return [Number.isFinite(value) ? value : null, page.currency ?? ''];
}

export async function readDaily(auth, account, days) {
  const start = new Date(Date.now() - days * 86400000).toISOString().slice(0, 10);
  const page = await get(auth, `${BASE}/Accounts/${account}/Usage/Records/Daily.json`,
                         { Category: SPEND_CATEGORY, StartDate: start, PageSize: 100 });
  return page.usage_records ?? [];
}

async function main() {
  const account = process.env.TWILIO_ACCOUNT_SID;
  const key = process.env.TWILIO_API_KEY;
  const secret = process.env.TWILIO_API_SECRET;
  if (!account || !key || !secret) {
    console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
                  '(an API Key with read access, not the auth token)');
    process.exitCode = 2;
    return;
  }
  const auth = authHeader(key, secret);

  const arg = (name, fallback) => {
    const i = process.argv.indexOf(name);
    return i === -1 ? fallback : Number.parseFloat(process.argv[i + 1]);
  };
  const days = arg('--days', 30);
  const floorDays = arg('--floor-days', 7);

  const [balance, currency] = await readBalance(auth, account);
  const prices = dailyPrices(await readDaily(auth, account, days));
  console.log(`balance ${balance === null ? 'unreadable' : balance.toFixed(2)} ` +
              `${currency} over ${days} day(s)`);

  const [state, detail] = verdict(balance, prices, floorDays);
  if (state === 'ok') {
    console.log(`${state.padEnd(14)} ${detail}`);
    return;
  }
  console.warn(`${state.padEnd(14)} ${detail}`);
  if (prices.length) {
    const needed = median(prices) * floorDays;
    console.warn(`  ${floorDays.toFixed(0)} days at the median day is ` +
                 `${needed.toFixed(2)} ${currency}: keep the recharge trigger at ` +
                 'or above that');
  }
  console.warn('  repair: Console > Billing > Manage billing > Auto Recharge, with ' +
               `a trigger amount of at least ${floorDays.toFixed(0)} days of spend ` +
               'and a card that is not about to expire');
  console.warn('  auto recharge state is not exposed by the API: the only evidence ' +
               'it is working is this balance going back up');
  process.exitCode = 1;
}

// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}

Add a test

The cases that matter are the ones where a healthy-looking balance is not healthy: a month of quiet days hiding one enormous one, a negative usage row that would otherwise flatter the burn rate, and an account with no spend at all, where dividing by zero is not an answer and saying so is. The rest pins the boundary either side of the floor.

test_twilio_balance_runway.py
from twilio_balance_runway import (daily_prices, median, price_of, runway_days,
                                   verdict)


def test_median_of_an_even_run_is_the_middle_pair():
    assert median([1.0, 2.0, 3.0, 4.0]) == 2.5
    assert median([3.0, 1.0, 2.0]) == 2.0
    assert median([]) == 0.0


def test_a_credit_day_is_clamped_rather_than_subtracted():
    assert price_of({"price": "-42.00"}) == 0.0
    assert price_of({"price": "12.50"}) == 12.5


def test_unparseable_prices_are_dropped_not_guessed():
    assert price_of({"price": None}) is None
    assert price_of({"price": "n/a"}) is None
    assert daily_prices([{"price": "4"}, {"price": "x"}, {}]) == [4.0]


def test_runway_is_undefined_at_a_zero_burn_rate():
    assert runway_days(100.0, 0.0) is None
    assert runway_days(100.0, 10.0) == 10.0


def test_a_missing_balance_is_reported_rather_than_assumed_healthy():
    state, _ = verdict(None, [{"price": "10"}])
    assert state == "unknown"


def test_a_zero_balance_is_already_the_suspension():
    state, detail = verdict(0.0, [10.0, 10.0])
    assert state == "empty"
    assert "20005" in detail


def test_an_account_with_no_spend_has_no_runway_to_compute():
    state, _ = verdict(500.0, [])
    assert state == "idle"


def test_under_one_median_day_is_critical():
    state, _ = verdict(5.0, [10.0, 10.0, 10.0])
    assert state == "critical"


def test_four_days_of_runway_is_below_a_seven_day_floor():
    state, detail = verdict(40.0, [10.0, 10.0, 10.0])
    assert state == "low"
    assert "4.0 days" in detail


def test_a_quiet_median_hides_a_day_bigger_than_the_whole_balance():
    state, detail = verdict(500.0, [1.0, 1.0, 900.0])
    assert state == "burst-exposed"
    assert "900.00" in detail


def test_a_balance_past_the_floor_and_past_the_busiest_day_is_fine():
    state, _ = verdict(10000.0, [10.0, 10.0, 12.0])
    assert state == "ok"
twilio-balance-runway.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { dailyPrices, median, priceOf, runwayDays, verdict }
  from './twilio-balance-runway.mjs';

test('median of an even run is the middle pair', () => {
  assert.equal(median([1, 2, 3, 4]), 2.5);
  assert.equal(median([3, 1, 2]), 2);
  assert.equal(median([]), 0);
});

test('a credit day is clamped rather than subtracted', () => {
  assert.equal(priceOf({ price: '-42.00' }), 0);
  assert.equal(priceOf({ price: '12.50' }), 12.5);
});

test('unparseable prices are dropped, not guessed', () => {
  assert.equal(priceOf({ price: null }), null);
  assert.equal(priceOf({ price: 'n/a' }), null);
  assert.deepEqual(dailyPrices([{ price: '4' }, { price: 'x' }, {}]), [4]);
});

test('runway is undefined at a zero burn rate', () => {
  assert.equal(runwayDays(100, 0), null);
  assert.equal(runwayDays(100, 10), 10);
});

test('a missing balance is reported rather than assumed healthy', () => {
  assert.equal(verdict(null, [10])[0], 'unknown');
});

test('a zero balance is already the suspension', () => {
  const [state, detail] = verdict(0, [10, 10]);
  assert.equal(state, 'empty');
  assert.match(detail, /20005/);
});

test('an account with no spend has no runway to compute', () => {
  assert.equal(verdict(500, [])[0], 'idle');
});

test('under one median day is critical', () => {
  assert.equal(verdict(5, [10, 10, 10])[0], 'critical');
});

test('four days of runway is below a seven-day floor', () => {
  const [state, detail] = verdict(40, [10, 10, 10]);
  assert.equal(state, 'low');
  assert.match(detail, /4\.0 days/);
});

test('a quiet median hides a day bigger than the whole balance', () => {
  const [state, detail] = verdict(500, [1, 1, 900]);
  assert.equal(state, 'burst-exposed');
  assert.match(detail, /900\.00/);
});

test('a balance past the floor and past the busiest day is fine', () => {
  assert.equal(verdict(10000, [10, 10, 12])[0], 'ok');
});

FAQ

Does Twilio not email me before the balance runs out?

There are balance notification emails to the account owner, and they are a mailbox rather than a pager. They are also not something you can assert on: no API field tells you whether the notification is enabled, where it goes, or whether it was delivered. The balance and the daily usage records are readable, which is why the check is built on those two numbers and not on your inbox.

Why the median day rather than the average?

Because the mean is dragged around by exactly the days you care about. One launch day at twenty times normal pushes the mean up, which makes the runway estimate look shorter than the account's ordinary behaviour warrants, and a fortnight of quiet weekends pushes it down. The median describes an ordinary day, and the maximum, checked separately, describes the day that will actually empty the account.

What should the floor be?

Seven days is a defensible starting point because it spans a weekend, a declined card and a purchase approval. If your recharge is automatic and reliable, three days is fine; if funding the account involves a finance team and a purchase order, fourteen is not paranoid. The value of the floor is that it is a number the script can compare against, not that it is the right number.

Will adding funds bring the account straight back?

The suspension clears a few minutes after the payment, but that is not the whole recovery. Everything that was queued during the outage has already failed with 30002, and Twilio does not replay it. Whatever your application does about undelivered messages has to run afterwards, so the outage is longer than the suspension was.

Should this run per subaccount?

Read the balance on the parent, because that is where the money is: subaccounts bill up to the parent rather than holding their own balance. What is worth running per subaccount is the status check, since a parent suspension cascades and one tenant can also be suspended on its own. That is covered separately in the subaccount note.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.