Diagnostic Twilio
nothing subscribes to error logs, so failures age out
The postmortem needs the hour before the incident, and the incident was seven weeks ago. That hour does not exist. It is not in the warehouse, not in the logging stack, not in a bucket somewhere: the Debugger holds its alerts for thirty days and pushes them nowhere unless something asks it to, and on this account nothing ever asked.
Read GET https://events.twilio.com/v1/Subscriptions, then GET https://events.twilio.com/v1/Subscriptions/{Sid}/SubscribedEvents for each one, and look for a type beginning com.twilio.error-logs. Subscriptions that carry only message or call events are not coverage, however many of them there are.
Pair the survivors with GET https://events.twilio.com/v1/Sinks on sink_sid: a subscription feeding a sink that is not active is subscribed and not delivering. Note the blind spot in the same breath — the Debugger webhook is console-only and has no read API, so this check can prove coverage exists and cannot prove it does not.
The problem in plain words
Twilio records what went wrong on your account and keeps it for thirty days. That retention is not a detail, it is the boundary of every diagnostic in this section: any check that reads the Alerts API can only see back that far, so a failure that started six weeks ago is not a hard question, it is an unanswerable one. The record existed, and then it stopped existing, and nothing anywhere noticed the transition.
Getting the data out is opt-in and there is exactly nothing on by default. No subscription, no Debugger webhook, no copy. That absence produces no symptom of its own — an account with nothing subscribed behaves identically to one with everything subscribed, right up until the day somebody needs last quarter's errors and finds out they were never kept. It is the same shape as having no backups: the cost is zero every day and then it is the whole thing at once.
Why it happens
Nothing streams by default. Event Streams exists, error-log event types exist, and an account is created with no subscription to either. As with Usage Triggers, the absence is not a misconfiguration anyone made; it is what every account looks like until somebody deliberately sets one up.
Having subscriptions is not having error coverage. A team that streams message and call events into a warehouse has a populated Subscriptions list, a working pipeline and a dashboard, and still keeps no errors at all. The list response cannot tell you the difference, because the types live on a subresource: you have to fetch SubscribedEvents per subscription to see what is actually being carried.
Event types are versioned, so exact matching rots. Pinning the full com.twilio.error-logs.error-log.logged string works until the day it does not, and the failure is silent in the worst direction: the check reports no coverage, or worse, reports coverage missing on an account that has it. Matching the product prefix survives the version bump.
Subscribed is not delivering. A subscription names a sink_sid, and a sink can be failed, validating or initialized instead of active. Errors subscribed into a dead sink leave you exactly as blind as errors nobody subscribed to, with more moving parts to reassure you. That failure has its own note; this check only needs to know the sink is not carrying anything.
The other route out is invisible to the API. A Debugger webhook is configured in the console and there is no read endpoint for it. So a clean report from this script means no Event Streams subscription, not no coverage, and the script has to say that out loud rather than let a green line imply something it cannot see.
The fix, as a flow
The event types live on a subresource, so the script fetches SubscribedEvents for every subscription: the list response cannot tell a busy pipeline from a useful one.
How to fix it
List the subscriptions
GET https://events.twilio.com/v1/Subscriptions?PageSize=50, following meta.next_page_url. An empty list is the finding on its own and needs no further calls: nothing is being streamed anywhere, so the Debugger is the only copy of anything.
Fetch the types each one actually carries
GET https://events.twilio.com/v1/Subscriptions/{Sid}/SubscribedEvents, one call per subscription. This is the step that separates a busy pipeline from a useful one, and it cannot be skipped, because the subscription list gives you a description and a sink and no indication of what is on the wire.
Match the product prefix, not the whole type
Compare against com.twilio.error-logs rather than the full com.twilio.error-logs.error-log.logged. The suffix carries the resource and the verb and can gain a variant; a prefix comparison keeps matching when it does and costs nothing today.
Resolve every sink_sid before calling it covered
GET https://events.twilio.com/v1/Sinks and look each sink_sid up by sid. Anything other than active means the pipe is not carrying, and a sink_sid that is not in the list at all should be reported as unresolved rather than assumed fine.
Subscribe errors somewhere durable, and check the console route too
Create a sink, validate it, and POST https://events.twilio.com/v1/Subscriptions with an error-log type and that SinkSid. Or set a Debugger webhook under Console → Monitor → Debugger. Since the script cannot see the second one, confirm it by hand while you are in there.
How to check it worked
Re-run once a subscription is attached to a validated sink.
python3 twilio_error_log_subscription_audit.py
# covered 1 subscription(s) carrying error-log events into an active sink.The full code
One paginated GET over the subscriptions, one per subscription for its types, and one over the sinks — read access throughout. The classifier takes the three results as plain data rather than as responses, because the judgement is a join across all of them and the join is the thing worth testing: an account with six subscriptions and no error-log type is as blind as an account with none, and the report has to say that in those words.
"""Report a Twilio account with nothing subscribed to its own error logs.
Debugger alerts are retained for thirty days and pushed nowhere unless an Event
Streams subscription or a Debugger webhook exists. That retention window is the
boundary of every other diagnostic on the account, so this is the check about
the window itself.
Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. The repair is printed, never performed,
because this script holds a credential to an account that can send messages and
spend money.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_error_log_subscription_audit")
EVENTS = "https://events.twilio.com/v1"
ERROR_LOG_PREFIX = "com.twilio.error-logs"
ACTIVE = "active"
RETENTION_DAYS = 30
# Printed, never sent. Kept as a literal so the repair line is the exact shape
# the API expects rather than a paraphrase of it.
REPAIR_TYPES = '{"type":"com.twilio.error-logs.error-log.logged"}'
def is_error_log_type(event_type):
"""True for any error-log event type. Pure.
Matched on the product prefix rather than the full
com.twilio.error-logs.error-log.logged string, because the suffix carries a
resource and a verb and can gain a variant. A pinned full string stops
matching on the day that happens, and reports an account that has coverage
as an account that does not.
"""
return str(event_type or "").strip().lower().startswith(ERROR_LOG_PREFIX)
def verdict(subscriptions, types_by_subscription, sink_status):
"""Classify what this account keeps of its own errors. Pure, so the join can
be tested without a network.
types_by_subscription maps a subscription sid to the event types it carries.
sink_status maps a sink sid to its status. Both are plain data rather than
responses, because the judgement here is the join across all three and that
is the part worth pinning.
Returns (state, detail).
"""
subs = list(subscriptions or [])
types = types_by_subscription or {}
sinks = sink_status or {}
if not subs:
return ("none",
"no Event Streams subscriptions on this account: nothing carries "
"errors anywhere, so the Debugger is the only copy and it is kept "
"for %d days." % RETENTION_DAYS)
carrying = [s for s in subs
if any(is_error_log_type(t) for t in types.get(s.get("sid"), []))]
if not carrying:
return ("no-error-logs",
"%d subscription(s), none of them carrying a %s type: whatever "
"else is being streamed, the errors are not, and they age out "
"after %d days." % (len(subs), ERROR_LOG_PREFIX, RETENTION_DAYS))
live = [s for s in carrying
if str(sinks.get(s.get("sink_sid")) or "").strip().lower() == ACTIVE]
if not live:
states = sorted({str(sinks.get(s.get("sink_sid")) or "unresolved").strip().lower()
for s in carrying})
return ("sink-not-active",
"%d subscription(s) carry error logs and every sink behind them "
"is %s rather than active: subscribed and not delivering is the "
"same blind spot with more moving parts."
% (len(carrying), ", ".join(states)))
return ("covered",
"%d subscription(s) carrying error-log events into an active sink."
% len(live))
def get(session, url, **params):
r = session.get(url, params=params, timeout=30)
if r.status_code in (401, 403):
raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that the "
"API key belongs to that account with read access"
% r.status_code)
r.raise_for_status()
return r.json()
def list_all(session, url, key, limit=200):
"""Page a newer-domain list. meta.next_page_url is absolute here, unlike the
next_page_uri path the 2010-04-01 API returns."""
params = {"PageSize": 50}
out = []
while url and len(out) < limit:
page = get(session, url, **params)
out.extend(page.get(key, []))
url = (page.get("meta") or {}).get("next_page_url")
params = {}
return out[:limit]
def subscribed_types(session, subscriptions):
"""One call per subscription. The types live on a subresource, so the list
response cannot tell a busy pipeline from a useful one."""
types = {}
for subscription in subscriptions:
sid = subscription.get("sid")
events = list_all(session, "%s/Subscriptions/%s/SubscribedEvents" % (EVENTS, sid),
"types")
types[sid] = [e.get("type") for e in events]
return types
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--max-subscriptions", type=int, default=200,
help="stop after this many subscriptions")
args = ap.parse_args()
account = os.environ.get("TWILIO_ACCOUNT_SID")
key = os.environ.get("TWILIO_API_KEY")
secret = os.environ.get("TWILIO_API_SECRET")
if not (account and key and secret):
log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
"(an API Key with read access, not the auth token)")
return 2
session = requests.Session()
session.auth = (key, secret)
subscriptions = list_all(session, EVENTS + "/Subscriptions", "subscriptions",
args.max_subscriptions)
types = subscribed_types(session, subscriptions) if subscriptions else {}
sinks = {s.get("sid"): s.get("status")
for s in list_all(session, EVENTS + "/Sinks", "sinks")}
for subscription in subscriptions:
sid = subscription.get("sid")
log.info(" %s sink=%s status=%s types=%s", sid,
subscription.get("sink_sid", "?"),
sinks.get(subscription.get("sink_sid"), "unresolved"),
", ".join(t for t in types.get(sid, []) if t) or "none")
state, detail = verdict(subscriptions, types, sinks)
if state == "covered":
log.info("%-16s %s", state, detail)
return 0
log.warning("%-16s %s", state, detail)
log.warning(" repair: create and validate a sink, then POST %s/Subscriptions "
"Description=error-logs SinkSid={SinkSid} Types=%s",
EVENTS, REPAIR_TYPES)
log.warning(" or set a Debugger webhook: Console > Monitor > Debugger > Webhook")
log.warning(" note: the Debugger webhook has no read API, so this check can "
"prove coverage exists and cannot prove it does not")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Report a Twilio account with nothing subscribed to its own error logs.
*
* Debugger alerts are retained for thirty days and pushed nowhere unless an
* Event Streams subscription or a Debugger webhook exists. That window is the
* boundary of every other diagnostic on the account.
*
* Read only. GET requests and nothing else: give this an API Key with read
* access rather than the account auth token. The repair is printed, never
* performed.
*/
const EVENTS = 'https://events.twilio.com/v1';
const ERROR_LOG_PREFIX = 'com.twilio.error-logs';
const ACTIVE = 'active';
const RETENTION_DAYS = 30;
// Printed, never sent. Kept as a literal so the repair line is the exact shape
// the API expects rather than a paraphrase of it.
const REPAIR_TYPES = '{"type":"com.twilio.error-logs.error-log.logged"}';
/**
* True for any error-log event type. Pure. Matched on the product prefix rather
* than the full type string, because the suffix carries a resource and a verb
* and can gain a variant.
*/
export function isErrorLogType(eventType) {
return String(eventType ?? '').trim().toLowerCase().startsWith(ERROR_LOG_PREFIX);
}
/**
* Classify what this account keeps of its own errors. Pure, so the join can be
* tested without a network. Returns [state, detail].
*/
export function verdict(subscriptions, typesBySubscription, sinkStatus) {
const subs = [...(subscriptions ?? [])];
const types = typesBySubscription ?? {};
const sinks = sinkStatus ?? {};
if (subs.length === 0) {
return ['none',
'no Event Streams subscriptions on this account: nothing carries errors ' +
`anywhere, so the Debugger is the only copy and it is kept for ` +
`${RETENTION_DAYS} days.`];
}
const carrying = subs.filter(
(s) => (types[s.sid] ?? []).some(isErrorLogType));
if (carrying.length === 0) {
return ['no-error-logs',
`${subs.length} subscription(s), none of them carrying a ` +
`${ERROR_LOG_PREFIX} type: whatever else is being streamed, the errors ` +
`are not, and they age out after ${RETENTION_DAYS} days.`];
}
const live = carrying.filter(
(s) => String(sinks[s.sink_sid] ?? '').trim().toLowerCase() === ACTIVE);
if (live.length === 0) {
const states = [...new Set(carrying.map(
(s) => String(sinks[s.sink_sid] ?? 'unresolved').trim().toLowerCase()))].sort();
return ['sink-not-active',
`${carrying.length} subscription(s) carry error logs and every sink behind ` +
`them is ${states.join(', ')} rather than active: subscribed and not ` +
'delivering is the same blind spot with more moving parts.'];
}
return ['covered',
`${live.length} subscription(s) carrying error-log events into an active sink.`];
}
function authHeader(key, secret) {
return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}
async function get(auth, url, params = {}) {
const u = new URL(url);
for (const [k, v] of Object.entries(params)) u.searchParams.set(k, v);
const res = await fetch(u, { headers: { Authorization: auth } });
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
'that the API key belongs to that account with read access');
}
if (!res.ok) throw new Error(`${res.status} from ${u.pathname}`);
return res.json();
}
export async function listAll(auth, url, key, limit = 200) {
let next = url;
let params = { PageSize: 50 };
const out = [];
while (next && out.length < limit) {
const page = await get(auth, next, params);
out.push(...(page[key] ?? []));
next = page.meta?.next_page_url ?? null;
params = {};
}
return out.slice(0, limit);
}
async function main() {
const account = process.env.TWILIO_ACCOUNT_SID;
const key = process.env.TWILIO_API_KEY;
const secret = process.env.TWILIO_API_SECRET;
if (!account || !key || !secret) {
console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
'(an API Key with read access, not the auth token)');
process.exitCode = 2;
return;
}
const auth = authHeader(key, secret);
const subscriptions = await listAll(auth, `${EVENTS}/Subscriptions`, 'subscriptions');
const types = {};
for (const s of subscriptions) {
const events = await listAll(
auth, `${EVENTS}/Subscriptions/${s.sid}/SubscribedEvents`, 'types');
types[s.sid] = events.map((e) => e.type);
}
const sinks = Object.fromEntries(
(await listAll(auth, `${EVENTS}/Sinks`, 'sinks')).map((s) => [s.sid, s.status]));
for (const s of subscriptions) {
const carried = (types[s.sid] ?? []).filter(Boolean).join(', ') || 'none';
console.log(` ${s.sid} sink=${s.sink_sid ?? '?'} ` +
`status=${sinks[s.sink_sid] ?? 'unresolved'} types=${carried}`);
}
const [state, detail] = verdict(subscriptions, types, sinks);
if (state === 'covered') {
console.log(`${state.padEnd(16)} ${detail}`);
return;
}
console.warn(`${state.padEnd(16)} ${detail}`);
console.warn(` repair: create and validate a sink, then POST ${EVENTS}/Subscriptions ` +
`Description=error-logs SinkSid={SinkSid} Types=${REPAIR_TYPES}`);
console.warn(' or set a Debugger webhook: Console > Monitor > Debugger > Webhook');
console.warn(' note: the Debugger webhook has no read API, so this check can ' +
'prove coverage exists and cannot prove it does not');
process.exitCode = 1;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The account worth writing a test for is the one with a working pipeline and no error coverage: six subscriptions, an active sink, a warehouse full of message events, and not one error kept past thirty days. It looks like the most instrumented account in the report and it is exactly as blind as the empty one. The other case pinned here is the unresolvable sink, because a sink_sid missing from the list must not fall through to covered.
from twilio_error_log_subscription_audit import is_error_log_type, verdict
ERRORS = "com.twilio.error-logs.error-log.logged"
MESSAGES = "com.twilio.messaging.message.delivered"
def sub(sid="DF01", sink="DG01"):
return {"sid": sid, "sink_sid": sink, "description": "warehouse"}
def test_an_account_with_no_subscriptions_keeps_nothing_past_the_window():
state, detail = verdict([], {}, {})
assert state == "none"
assert "30 days" in detail
def test_a_busy_pipeline_with_no_error_types_is_just_as_blind():
state, detail = verdict([sub("DF01"), sub("DF02")],
{"DF01": [MESSAGES], "DF02": [MESSAGES]},
{"DG01": "active"})
assert state == "no-error-logs"
assert "whatever else is being streamed" in detail
def test_error_logs_into_an_active_sink_is_coverage():
state, _ = verdict([sub()], {"DF01": [MESSAGES, ERRORS]}, {"DG01": "active"})
assert state == "covered"
def test_error_logs_into_a_failed_sink_is_subscribed_and_not_delivering():
state, detail = verdict([sub()], {"DF01": [ERRORS]}, {"DG01": "failed"})
assert state == "sink-not-active"
assert "failed" in detail
def test_a_sink_sid_that_is_not_in_the_list_is_unresolved_rather_than_fine():
state, detail = verdict([sub(sink="DG99")], {"DF01": [ERRORS]}, {"DG01": "active"})
assert state == "sink-not-active"
assert "unresolved" in detail
def test_one_live_error_subscription_outweighs_the_dead_ones_beside_it():
state, _ = verdict([sub("DF01", "DG_DEAD"), sub("DF02", "DG01")],
{"DF01": [ERRORS], "DF02": [ERRORS]},
{"DG01": "active", "DG_DEAD": "failed"})
assert state == "covered"
def test_the_type_is_matched_on_the_product_prefix_not_the_whole_string():
assert is_error_log_type(ERRORS) is True
assert is_error_log_type("com.twilio.error-logs.error-log.logged.v2") is True
assert is_error_log_type("COM.TWILIO.ERROR-LOGS.ERROR-LOG.LOGGED") is True
assert is_error_log_type(MESSAGES) is False
assert is_error_log_type(None) is False
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { isErrorLogType, verdict } from './twilio-error-log-subscription-audit.mjs';
const ERRORS = 'com.twilio.error-logs.error-log.logged';
const MESSAGES = 'com.twilio.messaging.message.delivered';
const sub = (sid = 'DF01', sink = 'DG01') => ({
sid, sink_sid: sink, description: 'warehouse',
});
test('an account with no subscriptions keeps nothing past the window', () => {
const [state, detail] = verdict([], {}, {});
assert.equal(state, 'none');
assert.match(detail, /30 days/);
});
test('a busy pipeline with no error types is just as blind', () => {
const [state, detail] = verdict([sub('DF01'), sub('DF02')],
{ DF01: [MESSAGES], DF02: [MESSAGES] },
{ DG01: 'active' });
assert.equal(state, 'no-error-logs');
assert.match(detail, /whatever else is being streamed/);
});
test('error logs into an active sink is coverage', () => {
assert.equal(
verdict([sub()], { DF01: [MESSAGES, ERRORS] }, { DG01: 'active' })[0],
'covered');
});
test('error logs into a failed sink is subscribed and not delivering', () => {
const [state, detail] = verdict([sub()], { DF01: [ERRORS] }, { DG01: 'failed' });
assert.equal(state, 'sink-not-active');
assert.match(detail, /failed/);
});
test('a sink_sid that is not in the list is unresolved rather than fine', () => {
const [state, detail] = verdict([sub('DF01', 'DG99')], { DF01: [ERRORS] },
{ DG01: 'active' });
assert.equal(state, 'sink-not-active');
assert.match(detail, /unresolved/);
});
test('one live error subscription outweighs the dead ones beside it', () => {
const [state] = verdict([sub('DF01', 'DG_DEAD'), sub('DF02', 'DG01')],
{ DF01: [ERRORS], DF02: [ERRORS] },
{ DG01: 'active', DG_DEAD: 'failed' });
assert.equal(state, 'covered');
});
test('the type is matched on the product prefix not the whole string', () => {
assert.equal(isErrorLogType(ERRORS), true);
assert.equal(isErrorLogType('com.twilio.error-logs.error-log.logged.v2'), true);
assert.equal(isErrorLogType('COM.TWILIO.ERROR-LOGS.ERROR-LOG.LOGGED'), true);
assert.equal(isErrorLogType(MESSAGES), false);
assert.equal(isErrorLogType(null), false);
});
FAQ
The Debugger shows my errors. Is that not enough?
It is enough for thirty days and for one person looking at a screen. It is not enough for a postmortem written seven weeks later, for an alert that pages somebody, or for a query that counts how often something happened. Those all need the events somewhere you control, and getting them there is a subscription you have to create.
We already stream events into our warehouse. Are we covered?
Only if one of those subscriptions carries an error-log type, which the subscription list will not tell you. Streaming message and call events is a different subscription with different types, and it is by far the most common way an account ends up with a real pipeline, a real dashboard and no errors in either. The subresource is the only place the answer lives.
Why match a prefix instead of the exact event type?
Because the exact type carries a resource and a verb on the end and those can change. A check pinned to the full string keeps working until the day a variant appears, and then it reports an account with perfectly good coverage as having none. Matching com.twilio.error-logs costs nothing now and survives that.
The script says none. Does that definitely mean nothing is kept?
No, and it says so in its own output. A Debugger webhook is configured in the console and has no read endpoint, so an account can be pushing every alert somewhere without leaving a trace this script can find. Treat a none as prove-it-by-hand rather than as a confirmed finding, and go look at the Debugger settings page.
What about the sink itself failing later?
That is a different check and it has its own note, because it is a different question: this one asks whether anything is subscribed, that one asks whether the pipe is carrying. They overlap in one state, where an error-log subscription points at a sink that is not active, and the report names it separately so you know which of the two things to go and fix.
Related field notes
- The pipe that was carrying errors and stopped
- The other alarm an account starts without
- Delivery state you are also not keeping
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Error log event types — Twilio Docs
- Debugging your application — Twilio Docs
- Event Streams Subscription resource — Twilio Docs
- Event Streams Sink resource — Twilio Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.