Diagnostic Twilio
a suspended subaccount, so one tenant's traffic 20005s
One customer opens a ticket saying none of their messages have gone out since Thursday. Every other customer is fine. The parent account dashboard is green, the balance is healthy, the Debugger is quiet, and the only thing wrong is a three-letter field on a resource nobody on the team has read since the tenant was provisioned: that subaccount's status is suspended, and Twilio told nobody.
With parent credentials, read GET /2010-04-01/Accounts.json?Status=suspended and then GET /2010-04-01/Accounts.json?Status=closed. Any row whose owner_account_sid is your parent SID is one of your tenants, and it is not sending anything.
Reactivating is a write on the subaccount with parent credentials, so this script prints it rather than doing it. closed is terminal — there is no path back from it, which is why the two statuses are reported separately instead of as one count of "not active".
The problem in plain words
Multi-tenant Twilio integrations put each customer in a subaccount, which is the right shape: separate SIDs, separate usage records, separate numbers. What the shape does not give you is a single place that says whether every tenant can still send. Your application authenticates as the parent, or with an API key on the parent, and every health check you wrote asks the parent how it is doing. The parent is fine. The parent is always fine.
Meanwhile the suspended tenant fails uniformly and quietly. Their REST calls come back 20005, their queued messages fail 30002, and because their traffic is a small slice of the account total, no aggregate graph moves enough to notice. The finding arrives through support, days late, from the customer — which is the most expensive possible route for a fact that one paginated GET could have delivered every morning.
Why it happens
Nothing notifies you. A subaccount suspended through the API or by an internal process generates no email, no Debugger alert and no webhook. The status field changes and the API starts refusing that SID's traffic. Reading the field is the entire detection surface.
Suspension cascades downward, and only downward. If the parent is suspended every subaccount stops with it. If a subaccount is suspended the parent carries on undisturbed, which is exactly the direction that makes the parent a useless place to watch from.
Teams suspend tenants deliberately and then forget. Suspending a subaccount is a reasonable response to a non-paying customer or an abuse investigation. It becomes a bug the moment the customer pays, the ticket closes, and nobody flips the status back — and there is no expiry, so it stays suspended forever.
Closed is not a worse kind of suspended, it is permanent. A closed subaccount cannot be reopened. Its numbers are released. Reporting the two states as one number invites somebody to treat a closure as a thing they can undo on Monday.
The fix, as a flow
The parent is not a useful place to watch from, so the script asks the account list for the stopped ones by status and then checks ownership, because the parent's own row lists itself as its owner.
How to fix it
List by status rather than listing everything
GET /2010-04-01/Accounts.json?Status=suspended&PageSize=50, following next_page_uri, is a much cheaper question than paging every account on a large parent. Repeat it for Status=closed. On a parent with a handful of tenants, listing all of them is fine too and gives you a denominator.
Confirm ownership with owner_account_sid
Each row carries sid, friendly_name, status, type and owner_account_sid. The parent's own row appears in the list with owner_account_sid equal to its own sid; a genuine tenant is a row where the two differ and the owner is your parent. Keying on that field rather than on position keeps the check honest when credentials get swapped around.
Keep suspended and closed apart
Suspended is recoverable by one write with parent credentials. Closed is not recoverable at all: the subaccount is gone, its numbers have been released, and the tenant needs a new subaccount and new numbers. These are different incidents with different response times and they should never be summed into a single count.
Cross-check against the tenant's traffic
GET /2010-04-01/Accounts/{SubAccountSid}/Messages.json?PageSize=1 with the sub SID in the path tells you when that tenant last sent anything. A suspension dated to the last message is the confirmation, and it also gives support the exact window of dropped traffic to talk to the customer about.
Print the reactivation, run the audit on a schedule
The repair is a POST to /2010-04-01/Accounts/{SubAccountSid}.json with Status=active, authenticated as the parent. This script prints that and stops. Then run it every morning: the check is two GETs, and the whole point is that this failure has no other way to reach you.
How to check it worked
Re-run the script. Every tenant should classify as active and the roll-up should be clean.
python3 twilio_subaccount_status_audit.py --all
# ACxxxxxxxx Acme Corp (prod) active
# ACyyyyyyyy Northwind (prod) active
# clean 14 subaccount(s), all active.The full code
Two GETs by default, one per status, or a single paginated listing with --all so the report has a denominator. A read-access API Key on the parent is enough. The classifier is pure and takes one account row plus the parent SID, because the interesting judgements — the parent's own row is not a finding, a row owned by somebody else is a different finding, closed is not suspended — are all decisions about fields rather than about HTTP.
"""Report Twilio subaccounts that are suspended or closed under this parent.
Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. The repair is printed, never performed,
because this script holds a credential to an account that can send messages and
spend money.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_subaccount_status_audit")
HOST = "https://api.twilio.com"
BASE = HOST + "/2010-04-01"
STATUSES = ("suspended", "closed")
def verdict(account, parent_sid):
"""Classify one row from Accounts.json against the parent it should belong
to. Pure, so the ownership and status rules can be tested without a network.
Returns (state, detail).
"""
sid = str(account.get("sid") or "").strip()
owner = str(account.get("owner_account_sid") or "").strip()
status = str(account.get("status") or "").strip().lower()
kind = str(account.get("type") or "").strip().lower()
name = str(account.get("friendly_name") or "").strip() or "(no friendly name)"
if sid and sid == parent_sid:
return ("parent",
"%s is the parent account itself, not a tenant: its own row always "
"lists it as its owner." % name)
if owner and parent_sid and owner != parent_sid:
return ("foreign",
"%s is owned by %s rather than by this parent: the credential in "
"use is not the one that can change it." % (name, owner))
if status == "suspended":
return ("suspended",
"%s is suspended: every REST call on that SID returns 20005 and "
"anything queued fails 30002, and nothing was sent to tell you."
% name)
if status == "closed":
return ("closed",
"%s is closed, which is terminal: the subaccount cannot be "
"reopened and its numbers have been released." % name)
if kind == "trial":
return ("trial",
"%s is active but still of type Trial: sends are restricted to "
"verified numbers and carry the trial prefix." % name)
if status == "active":
return ("active", "%s is active." % name)
return ("unknown",
"%s has status %r, which is not one of active, suspended or closed."
% (name, status or ""))
def summary(states):
"""Roll a run of per-account states into one answer. Pure.
Suspended outranks closed in the report only because it is the one you can
still do something about this morning; both are printed either way.
"""
states = list(states or [])
tenants = [s for s in states if s != "parent"]
suspended = states.count("suspended")
closed = states.count("closed")
if suspended:
return ("suspended",
"%d suspended subaccount(s): that tenant's traffic is failing now "
"and can be restored with one write." % suspended)
if closed:
return ("closed",
"%d closed subaccount(s) and none suspended: closures are "
"permanent, so this is a record rather than a repair." % closed)
if not tenants:
return ("single",
"no subaccounts under this parent: there is nothing here to "
"suspend, and this check has nothing to watch.")
return ("clean", "%d subaccount(s), all active." % len(tenants))
def get(session, url, **params):
r = session.get(url, params=params, timeout=30)
if r.status_code in (401, 403):
raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that the "
"API key belongs to that account with read access"
% r.status_code)
r.raise_for_status()
return r.json()
def list_accounts(session, status=None, limit=500):
"""Page Accounts.json. next_page_uri is a path, not an absolute URL."""
url = "%s/Accounts.json" % BASE
params = {"PageSize": 50}
if status:
params["Status"] = status
out = []
while url and len(out) < limit:
page = get(session, url, **params)
out.extend(page.get("accounts", []))
nxt = page.get("next_page_uri")
url, params = (HOST + nxt) if nxt else None, {}
return out[:limit]
def last_message(session, account_sid):
"""When this tenant last sent anything, or None. One GET, one row."""
page = get(session, "%s/Accounts/%s/Messages.json" % (BASE, account_sid),
PageSize=1)
rows = page.get("messages", [])
return rows[0].get("date_sent") if rows else None
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--all", action="store_true",
help="list every subaccount rather than only the stopped ones")
ap.add_argument("--check-traffic", action="store_true",
help="for each finding, read when that tenant last sent")
args = ap.parse_args()
parent = os.environ.get("TWILIO_ACCOUNT_SID")
key = os.environ.get("TWILIO_API_KEY")
secret = os.environ.get("TWILIO_API_SECRET")
if not (parent and key and secret):
log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
"(an API Key with read access on the parent, not the auth token)")
return 2
session = requests.Session()
session.auth = (key, secret)
if args.all:
rows = list_accounts(session)
else:
rows = []
for status in STATUSES:
rows.extend(list_accounts(session, status=status))
states = []
findings = []
for row in rows:
state, detail = verdict(row, parent)
states.append(state)
line = "%-34s %s" % (row.get("sid", "?"), state)
if state in ("suspended", "closed", "foreign", "unknown"):
findings.append((row, state, detail))
log.warning("%s %s", line, detail)
else:
log.info("%s", line)
state, detail = summary(states)
if state in ("clean", "single"):
log.info("%-14s %s", state, detail)
return 0
log.warning("%-14s %s", state, detail)
for row, kind, _ in findings:
sid = row.get("sid", "{SubAccountSid}")
if args.check_traffic:
log.warning(" %s last sent: %s", sid, last_message(session, sid) or "never")
if kind == "suspended":
log.warning(" repair: POST %s/Accounts/%s.json Status=active, "
"authenticated as the parent account", BASE, sid)
elif kind == "closed":
log.warning(" %s is closed and cannot be reopened: provision a new "
"subaccount and new numbers for that tenant", sid)
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Report Twilio subaccounts that are suspended or closed under this parent.
*
* Read only. GET requests and nothing else: give this an API Key with read
* access rather than the account auth token. The repair is printed, never
* performed.
*/
const HOST = 'https://api.twilio.com';
const BASE = `${HOST}/2010-04-01`;
const STATUSES = ['suspended', 'closed'];
/**
* Classify one row from Accounts.json against the parent it should belong to.
* Pure, so the ownership and status rules can be tested without a network.
* Returns [state, detail].
*/
export function verdict(account, parentSid) {
const sid = String(account?.sid ?? '').trim();
const owner = String(account?.owner_account_sid ?? '').trim();
const status = String(account?.status ?? '').trim().toLowerCase();
const kind = String(account?.type ?? '').trim().toLowerCase();
const name = String(account?.friendly_name ?? '').trim() || '(no friendly name)';
if (sid && sid === parentSid) {
return ['parent',
`${name} is the parent account itself, not a tenant: its own row always ` +
'lists it as its owner.'];
}
if (owner && parentSid && owner !== parentSid) {
return ['foreign',
`${name} is owned by ${owner} rather than by this parent: the credential ` +
'in use is not the one that can change it.'];
}
if (status === 'suspended') {
return ['suspended',
`${name} is suspended: every REST call on that SID returns 20005 and ` +
'anything queued fails 30002, and nothing was sent to tell you.'];
}
if (status === 'closed') {
return ['closed',
`${name} is closed, which is terminal: the subaccount cannot be reopened ` +
'and its numbers have been released.'];
}
if (kind === 'trial') {
return ['trial',
`${name} is active but still of type Trial: sends are restricted to ` +
'verified numbers and carry the trial prefix.'];
}
if (status === 'active') return ['active', `${name} is active.`];
return ['unknown',
`${name} has status "${status}", which is not one of active, suspended or closed.`];
}
/** Roll a run of per-account states into one answer. Pure. */
export function summary(states) {
const all = [...(states ?? [])];
const tenants = all.filter((s) => s !== 'parent');
const suspended = all.filter((s) => s === 'suspended').length;
const closed = all.filter((s) => s === 'closed').length;
if (suspended) {
return ['suspended',
`${suspended} suspended subaccount(s): that tenant's traffic is failing now ` +
'and can be restored with one write.'];
}
if (closed) {
return ['closed',
`${closed} closed subaccount(s) and none suspended: closures are permanent, ` +
'so this is a record rather than a repair.'];
}
if (!tenants.length) {
return ['single',
'no subaccounts under this parent: there is nothing here to suspend, and ' +
'this check has nothing to watch.'];
}
return ['clean', `${tenants.length} subaccount(s), all active.`];
}
function authHeader(key, secret) {
return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}
async function get(auth, url, params = {}) {
const u = new URL(url);
for (const [k, v] of Object.entries(params)) u.searchParams.set(k, v);
const res = await fetch(u, { headers: { Authorization: auth } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
'that the API key belongs to that account with read access');
}
if (!res.ok) throw new Error(`${res.status} from ${u.pathname}`);
return res.json();
}
export async function listAccounts(auth, status = null, limit = 500) {
let url = `${BASE}/Accounts.json`;
let params = status ? { PageSize: 50, Status: status } : { PageSize: 50 };
const out = [];
while (url && out.length < limit) {
const page = await get(auth, url, params);
out.push(...(page.accounts ?? []));
url = page.next_page_uri ? HOST + page.next_page_uri : null;
params = {};
}
return out.slice(0, limit);
}
async function lastMessage(auth, accountSid) {
const page = await get(auth, `${BASE}/Accounts/${accountSid}/Messages.json`,
{ PageSize: 1 });
const rows = page.messages ?? [];
return rows.length ? rows[0].date_sent : null;
}
async function main() {
const parent = process.env.TWILIO_ACCOUNT_SID;
const key = process.env.TWILIO_API_KEY;
const secret = process.env.TWILIO_API_SECRET;
if (!parent || !key || !secret) {
console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
'(an API Key with read access on the parent, not the auth token)');
process.exitCode = 2;
return;
}
const auth = authHeader(key, secret);
const listAll = process.argv.includes('--all');
const checkTraffic = process.argv.includes('--check-traffic');
let rows = [];
if (listAll) {
rows = await listAccounts(auth);
} else {
for (const status of STATUSES) rows.push(...await listAccounts(auth, status));
}
const states = [];
const findings = [];
for (const row of rows) {
const [state, detail] = verdict(row, parent);
states.push(state);
const line = `${String(row.sid ?? '?').padEnd(34)} ${state}`;
if (['suspended', 'closed', 'foreign', 'unknown'].includes(state)) {
findings.push([row, state]);
console.warn(`${line} ${detail}`);
} else {
console.log(line);
}
}
const [state, detail] = summary(states);
if (state === 'clean' || state === 'single') {
console.log(`${state.padEnd(14)} ${detail}`);
return;
}
console.warn(`${state.padEnd(14)} ${detail}`);
for (const [row, kind] of findings) {
const sid = row.sid ?? '{SubAccountSid}';
if (checkTraffic) {
console.warn(` ${sid} last sent: ${await lastMessage(auth, sid) ?? 'never'}`);
}
if (kind === 'suspended') {
console.warn(` repair: POST ${BASE}/Accounts/${sid}.json Status=active, ` +
'authenticated as the parent account');
} else if (kind === 'closed') {
console.warn(` ${sid} is closed and cannot be reopened: provision a new ` +
'subaccount and new numbers for that tenant');
}
}
process.exitCode = 1;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Three of these cases are the ones that turn a useful report into a misleading one: the parent's own row, which lists itself as its owner and is not a finding; a row belonging to a different parent, which this credential cannot fix; and a closed subaccount, which must never be counted alongside a suspended one because only one of the two can be undone.
from twilio_subaccount_status_audit import summary, verdict
PARENT = "ACparent0000000000000000000000000"
def make(**kw):
row = {"sid": "ACtenant000000000000000000000001",
"owner_account_sid": PARENT,
"friendly_name": "Acme Corp (prod)",
"status": "active",
"type": "Full"}
row.update(kw)
return row
def test_the_parents_own_row_is_not_a_tenant():
state, detail = verdict(make(sid=PARENT, owner_account_sid=PARENT), PARENT)
assert state == "parent"
assert "owner" in detail
def test_a_suspended_tenant_is_the_finding():
state, detail = verdict(make(status="suspended"), PARENT)
assert state == "suspended"
assert "20005" in detail
def test_a_closed_tenant_is_reported_as_terminal():
state, detail = verdict(make(status="closed"), PARENT)
assert state == "closed"
assert "cannot be reopened" in detail
def test_a_row_owned_by_another_parent_is_not_ours_to_fix():
state, _ = verdict(make(owner_account_sid="ACsomeoneelse"), PARENT)
assert state == "foreign"
def test_an_active_trial_subaccount_is_still_worth_saying():
state, _ = verdict(make(type="Trial"), PARENT)
assert state == "trial"
def test_status_casing_from_the_api_does_not_change_the_answer():
assert verdict(make(status="SUSPENDED"), PARENT)[0] == "suspended"
def test_an_unrecognised_status_is_not_quietly_called_active():
state, _ = verdict(make(status="pending"), PARENT)
assert state == "unknown"
def test_summary_reports_the_recoverable_failure_first():
state, detail = summary(["parent", "active", "suspended", "closed"])
assert state == "suspended"
assert "one write" in detail
def test_summary_keeps_closures_separate_from_suspensions():
state, detail = summary(["parent", "active", "closed"])
assert state == "closed"
assert "permanent" in detail
def test_a_parent_with_no_subaccounts_has_nothing_to_watch():
assert summary(["parent"])[0] == "single"
def test_all_active_tenants_are_clean():
state, detail = summary(["parent", "active", "active", "trial"])
assert state == "clean"
assert "3 subaccount(s)" in detail
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { summary, verdict } from './twilio-subaccount-status-audit.mjs';
const PARENT = 'ACparent0000000000000000000000000';
const make = (over = {}) => ({
sid: 'ACtenant000000000000000000000001',
owner_account_sid: PARENT,
friendly_name: 'Acme Corp (prod)',
status: 'active',
type: 'Full',
...over,
});
test("the parent's own row is not a tenant", () => {
const [state, detail] = verdict(
make({ sid: PARENT, owner_account_sid: PARENT }), PARENT);
assert.equal(state, 'parent');
assert.match(detail, /owner/);
});
test('a suspended tenant is the finding', () => {
const [state, detail] = verdict(make({ status: 'suspended' }), PARENT);
assert.equal(state, 'suspended');
assert.match(detail, /20005/);
});
test('a closed tenant is reported as terminal', () => {
const [state, detail] = verdict(make({ status: 'closed' }), PARENT);
assert.equal(state, 'closed');
assert.match(detail, /cannot be reopened/);
});
test('a row owned by another parent is not ours to fix', () => {
assert.equal(verdict(make({ owner_account_sid: 'ACsomeoneelse' }), PARENT)[0],
'foreign');
});
test('an active trial subaccount is still worth saying', () => {
assert.equal(verdict(make({ type: 'Trial' }), PARENT)[0], 'trial');
});
test('status casing from the API does not change the answer', () => {
assert.equal(verdict(make({ status: 'SUSPENDED' }), PARENT)[0], 'suspended');
});
test('an unrecognised status is not quietly called active', () => {
assert.equal(verdict(make({ status: 'pending' }), PARENT)[0], 'unknown');
});
test('summary reports the recoverable failure first', () => {
const [state, detail] = summary(['parent', 'active', 'suspended', 'closed']);
assert.equal(state, 'suspended');
assert.match(detail, /one write/);
});
test('summary keeps closures separate from suspensions', () => {
const [state, detail] = summary(['parent', 'active', 'closed']);
assert.equal(state, 'closed');
assert.match(detail, /permanent/);
});
test('a parent with no subaccounts has nothing to watch', () => {
assert.equal(summary(['parent'])[0], 'single');
});
test('all active tenants are clean', () => {
const [state, detail] = summary(['parent', 'active', 'active', 'trial']);
assert.equal(state, 'clean');
assert.match(detail, /3 subaccount\(s\)/);
});
FAQ
Who suspends a subaccount, if nobody on my team did?
Three routes. Somebody on your side called the API or used the console, usually during a billing or abuse process. The parent account was suspended, which cascades to every subaccount underneath it. Or Twilio acted on the account for policy or payment reasons. The status field does not record which, so the audit tells you a tenant is stopped and your billing and abuse history tells you why.
Can I reactivate it from this script?
No, and deliberately so. Every script in this section holds a credential to an account that can send messages and spend money, so none of them writes. Reactivation is one POST to the subaccount with parent credentials and Status=active; the script prints exactly that line so a human can run it after deciding the suspension should not have happened.
Is a closed subaccount really unrecoverable?
Yes. Closing is permanent, the numbers attached to it are released back to the pool, and there is no reopen. If a tenant's subaccount is closed, that tenant needs a new subaccount and new numbers, and any A2P registration or toll-free verification attached to the old numbers has to be done again. That is why the report never merges the two states.
Should I run this with the parent auth token instead?
No. An API Key created on the parent with read access lists subaccounts perfectly well, and it is the credential that cannot do damage if the audit host is compromised. If you have no keys at all, the auth token is doing this work everywhere, which is its own finding, covered in the API key note.
How often should it run?
Daily is enough for most, and hourly is cheap: it is two GETs on a small collection. The value is bounded by how long you are willing for a tenant to be down before you hear it from them, so pick an interval shorter than your support response time and let it run.
Related field notes
- The parent balance whose exhaustion cascades to every tenant
- A subaccount that cannot send even while active
- Spend alarms, which every subaccount needs its own of
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Subaccounts — Twilio Docs
- Account resource (IAM) — Twilio Docs
- Error 20005: account not active — Twilio Docs
- Error 30002: account suspended — Twilio Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.