Skip to content

Diagnostic Twilio

no API keys exist, so the auth token is the credential

Every other note in this section tells you to run its script with an API Key that has read access rather than the account auth token. This is the one that explains why, and how to find out whether anything you run is still holding the token. Nothing is failing right now. That is the shape of the problem: it costs you nothing until the day you have to rotate, and then it costs you everything at once.

Read-only key Python and Node.js Tests included
Black imgix server system
Photo by imgix on Unsplash
The short answer

Read GET /2010-04-01/Accounts/{AccountSid}/Keys.json. An empty keys[] is conclusive: there is no API key on the account, so whatever authenticates to Twilio is presenting the account auth token.

For the harder question — which of your deployments still holds it — look at the basic-auth username rather than the password. A username beginning SK is an API Key SID. A username that is the account SID, beginning AC, means the password beside it is the auth token. There is no field anywhere in the API that reports this; it is in your configuration, and the script reads the one instance it can see for certain, which is its own.

The problem in plain words

The account auth token is a single secret that does two unrelated jobs. It is the password for every REST call your services make, and it is the HMAC key Twilio uses to sign inbound webhooks in X-Twilio-Signature. Those two jobs have completely different lifecycles and completely different exposure, and one string covers both.

So the failure is not an outage today, it is the shape of the outage later. A token that appears in a log line, a screenshot, a support ticket or a departing contractor's shell history has to be rotated, and rotating it revokes access for every service simultaneously and changes the signing key for every webhook receiver at the same moment. Anything you miss starts returning 20003 immediately, with no grace period and no per-service fallback. The blast radius of the fix is the whole integration, which is exactly why the fix keeps getting postponed and the token keeps living in more places.

One secretauth tokeneverywhereIt leaksa log line, ascreenshotRotate itaccount wide,instantEvery service20003no per servicefallbackSignatures failtoosame value signswebhooks
Nothing was broken before the rotation. The blast radius is the whole integration, which is why the rotation keeps getting postponed.

Why it happens

The auth token is the credential the console hands you. It sits on the dashboard next to the account SID, it is what every quickstart pastes into a .env, and it works immediately for everything. Creating an API key is a deliberate extra step that nothing prompts you to take, so the default state of a working integration is one secret shared everywhere.

One secret, two jobs, one lifecycle. You cannot rotate the REST credential without also rotating the webhook signing key, because they are the same string. A rotation therefore has to be coordinated across the services that call Twilio and the services Twilio calls, in the same window, which is a change nobody wants to schedule.

Revocation has no granularity. An API key can be deleted on its own: one service loses access and everything else carries on. The auth token cannot. Killing a compromised credential and killing your production traffic are the same action.

The API cannot tell you where a credential is used. Keys.json lists which keys exist. There is no last-used timestamp, no per-key request log, and nothing that maps a key to a deployment. That is a real blind spot, not a gap in the script, and it is why the useful signal is the username your own clients present rather than anything you can query.

Fewer keys than services means sharing. If one key is spread across six deployments, deleting it takes down six things, which puts you back in exactly the position the key was supposed to get you out of. Counting keys against separately deployed workloads is a rough measure and a useful one.

The fix, as a flow

The script reads which keys exist, and then reads the one thing the API never reports: the basic-auth username it authenticated with. An account SID in that position means the password beside it was the auth token.

Keys, services and theusernamewhat exists, and what this run usedA key per workloadrevocable one at a timeFewer keys than servicesa shared credentialNo keys on the accountthe token is doing the workThis run used AC plus tokenproof, not inference
The username is the tell. SK is a key you can revoke on its own; AC is the account SID, and the only password that pairs with it is the token.

How to fix it

Read the keys that exist

GET /2010-04-01/Accounts/{AccountSid}/Keys.json?PageSize=50, following next_page_uri. Each entry has sid, friendly_name, date_created and date_updated. An empty list ends the question: nothing but the auth token is in use.

Look at the username, not the password

Every client that authenticates to the Twilio REST API sends HTTP basic auth. The username is either an SK key SID or the AC account SID, and there is no third option. The script classifies the credential it was given, which proves at minimum that this one shell is or is not on a key. For the rest, audit configuration: grep your secret store and deployment manifests for TWILIO_AUTH_TOKEN and for any Twilio client constructed with the account SID as its username.

Count the workloads that should have their own key

There is no exact number, so use an upper bound you can read: GET https://messaging.twilio.com/v1/Services and GET /2010-04-01/Accounts/{AccountSid}/Applications.json. If you have eight of those between them and two keys, at least some deployments are sharing a credential. Override the count with --workloads when you know the real answer.

Leave the auth token exactly one job

Signature validation. The service that receives Twilio's webhooks needs the auth token to verify X-Twilio-Signature, and that is the only place it should exist. Everything making outbound calls moves to a key of its own, named after the thing that holds it.

Migrate one service at a time, and verify by deleting

POST /2010-04-01/Accounts/{AccountSid}/Keys.json with FriendlyName set to the service name; store the returned sid and secret as the basic-auth pair. The proof that a service moved is that deleting its old key breaks only that service. That is a test you can run deliberately in a maintenance window, which is the whole difference from rotating an auth token.

How to check it worked

Re-run the script with a key rather than the token. It should report keyed and exit zero.

python3 twilio_credential_audit.py
# keyed          6 API key(s) for 5 workload(s).

The full code

Three GETs — the keys, the Messaging Services and the TwiML Applications — plus one fact the API will never give you, which this script gets for free: the username it authenticated with. If that username is the account SID, the password beside it was the auth token, and the report says so before it says anything else. The classifier is pure and takes the credential kind as an argument, so all four outcomes can be exercised offline.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Read only, it never writes. One of 117 Twilio fixes, free and open source.
twilio_credential_audit.py
"""Report whether a Twilio account is still running on its auth token.

Read only. GET requests and nothing else: give this an API Key with read access
rather than the account auth token. If you give it the auth token, it will tell
you so, which is the point.
"""
import argparse
import logging
import os
import sys

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("twilio_credential_audit")

HOST = "https://api.twilio.com"
BASE = HOST + "/2010-04-01"
MESSAGING = "https://messaging.twilio.com/v1"


def credential_kind(username):
    """Which Twilio credential a basic-auth username implies.

    Every client that talks to the REST API sends basic auth, and the username is
    either an SK API Key SID or the AC account SID. If it is the account SID then
    the password beside it is the account auth token: there is no other password
    that pairs with it. No API field reports this, so the username is the only
    read-only tell there is.
    """
    u = str(username or "").strip().upper()
    if u.startswith("SK"):
        return "api-key"
    if u.startswith("AC"):
        return "auth-token"
    return "unknown"


def verdict(keys, workloads=0, running_as="unknown"):
    """Classify the account's credential posture. Pure, so all four outcomes can
    be tested without a network.

    Order matters: running under the auth token is proof, while a key count is
    inference, and the report should lead with whichever it actually knows.

    Returns (state, detail).
    """
    keys = list(keys or [])
    if running_as == "auth-token":
        return ("auth-token",
                "this run authenticated with the account SID as its basic-auth "
                "username, so the password was the account auth token. That is "
                "proof rather than inference: at least one deployment, this one, "
                "holds the account-wide secret. %d API key(s) exist." % len(keys))

    if not keys:
        return ("no-keys",
                "the account has no API keys, so every service that talks to "
                "Twilio is presenting the auth token: one secret, no per-service "
                "revocation, and the same value that signs your webhooks.")

    if workloads and len(keys) < workloads:
        return ("under-keyed",
                "%d API key(s) for %d separately deployed thing(s): some of them "
                "share a credential, and a shared credential cannot be revoked "
                "for one service without breaking the others."
                % (len(keys), workloads))

    return ("keyed", "%d API key(s) for %d workload(s)." % (len(keys), workloads))


def get(session, url, **params):
    r = session.get(url, params=params, timeout=30)
    if r.status_code in (401, 403):
        raise SystemExit("%d from Twilio: check TWILIO_ACCOUNT_SID and that the "
                         "credential belongs to that account with read access"
                         % r.status_code)
    r.raise_for_status()
    return r.json()


def list_keys(session, account, limit=200):
    """Page Keys.json. next_page_uri is a path, not an absolute URL."""
    url = "%s/Accounts/%s/Keys.json" % (BASE, account)
    params = {"PageSize": 50}
    out = []
    while url and len(out) < limit:
        page = get(session, url, **params)
        out.extend(page.get("keys", []))
        nxt = page.get("next_page_uri")
        url, params = (HOST + nxt) if nxt else None, {}
    return out[:limit]


def count_workloads(session, account):
    """An upper bound on how many separate credentials the account should have.

    Messaging Services and TwiML Applications are a proxy for deployed things,
    not a census of them. It is a rough number and the flag exists to replace it.
    """
    services = get(session, "%s/Services" % MESSAGING, PageSize=50)
    apps = get(session, "%s/Accounts/%s/Applications.json" % (BASE, account),
               PageSize=50)
    return len(services.get("services", [])) + len(apps.get("applications", []))


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--workloads", type=int, default=None,
                    help="how many services should have their own key; "
                         "skips the Messaging Service and Application count")
    args = ap.parse_args()

    account = os.environ.get("TWILIO_ACCOUNT_SID")
    key = os.environ.get("TWILIO_API_KEY")
    secret = os.environ.get("TWILIO_API_SECRET")
    if not (account and key and secret):
        log.error("set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET "
                  "(an API Key with read access, not the auth token)")
        return 2

    running_as = credential_kind(key)
    log.info("this run is authenticating as: %s", running_as)

    session = requests.Session()
    session.auth = (key, secret)

    keys = list_keys(session, account)
    for entry in keys:
        log.info("  %s  %s  created %s", entry.get("sid", "?"),
                 entry.get("friendly_name") or "(unnamed)",
                 entry.get("date_created", "?"))

    workloads = args.workloads
    if workloads is None:
        workloads = count_workloads(session, account)

    state, detail = verdict(keys, workloads, running_as)
    if state == "keyed":
        log.info("%-12s %s", state, detail)
        return 0

    log.warning("%-12s %s", state, detail)
    log.warning("  repair: POST %s/Accounts/%s/Keys.json FriendlyName={service-name}, "
                "then store the returned sid and secret as the basic-auth pair",
                BASE, account)
    log.warning("  keep the auth token for X-Twilio-Signature validation and "
                "nowhere else")
    return 1


if __name__ == "__main__":
    sys.exit(main())
twilio-credential-audit.mjs
/**
 * Report whether a Twilio account is still running on its auth token.
 *
 * Read only. GET requests and nothing else: give this an API Key with read
 * access rather than the account auth token. If you give it the auth token, it
 * will tell you so, which is the point.
 */
const HOST = 'https://api.twilio.com';
const BASE = `${HOST}/2010-04-01`;
const MESSAGING = 'https://messaging.twilio.com/v1';

/**
 * Which Twilio credential a basic-auth username implies. An SK username is an
 * API Key SID; the AC account SID as a username means the password beside it is
 * the account auth token. No API field reports this, so the username is the only
 * read-only tell there is.
 */
export function credentialKind(username) {
  const u = String(username ?? '').trim().toUpperCase();
  if (u.startsWith('SK')) return 'api-key';
  if (u.startsWith('AC')) return 'auth-token';
  return 'unknown';
}

/**
 * Classify the account's credential posture. Pure, so all four outcomes can be
 * tested without a network. Returns [state, detail].
 */
export function verdict(keys, workloads = 0, runningAs = 'unknown') {
  const all = [...(keys ?? [])];
  if (runningAs === 'auth-token') {
    return ['auth-token',
      'this run authenticated with the account SID as its basic-auth username, ' +
      'so the password was the account auth token. That is proof rather than ' +
      'inference: at least one deployment, this one, holds the account-wide ' +
      `secret. ${all.length} API key(s) exist.`];
  }

  if (all.length === 0) {
    return ['no-keys',
      'the account has no API keys, so every service that talks to Twilio is ' +
      'presenting the auth token: one secret, no per-service revocation, and the ' +
      'same value that signs your webhooks.'];
  }

  if (workloads && all.length < workloads) {
    return ['under-keyed',
      `${all.length} API key(s) for ${workloads} separately deployed thing(s): ` +
      'some of them share a credential, and a shared credential cannot be revoked ' +
      'for one service without breaking the others.'];
  }

  return ['keyed', `${all.length} API key(s) for ${workloads} workload(s).`];
}

function authHeader(key, secret) {
  return `Basic ${Buffer.from(`${key}:${secret}`).toString('base64')}`;
}

async function get(auth, url, params = {}) {
  const u = new URL(url);
  for (const [k, v] of Object.entries(params)) u.searchParams.set(k, v);
  const res = await fetch(u, { headers: { Authorization: auth } });
  if (res.status === 401 || res.status === 403) {
    throw new Error(`${res.status} from Twilio: check TWILIO_ACCOUNT_SID and ` +
                    'that the credential belongs to that account with read access');
  }
  if (!res.ok) throw new Error(`${res.status} from ${u.pathname}`);
  return res.json();
}

export async function listKeys(auth, account, limit = 200) {
  let url = `${BASE}/Accounts/${account}/Keys.json`;
  let params = { PageSize: 50 };
  const out = [];
  while (url && out.length < limit) {
    const page = await get(auth, url, params);
    out.push(...(page.keys ?? []));
    url = page.next_page_uri ? HOST + page.next_page_uri : null;
    params = {};
  }
  return out.slice(0, limit);
}

async function countWorkloads(auth, account) {
  const services = await get(auth, `${MESSAGING}/Services`, { PageSize: 50 });
  const apps = await get(auth, `${BASE}/Accounts/${account}/Applications.json`,
                         { PageSize: 50 });
  return (services.services ?? []).length + (apps.applications ?? []).length;
}

async function main() {
  const account = process.env.TWILIO_ACCOUNT_SID;
  const key = process.env.TWILIO_API_KEY;
  const secret = process.env.TWILIO_API_SECRET;
  if (!account || !key || !secret) {
    console.error('set TWILIO_ACCOUNT_SID, TWILIO_API_KEY and TWILIO_API_SECRET ' +
                  '(an API Key with read access, not the auth token)');
    process.exitCode = 2;
    return;
  }

  const runningAs = credentialKind(key);
  console.log(`this run is authenticating as: ${runningAs}`);

  const auth = authHeader(key, secret);
  const keys = await listKeys(auth, account);
  for (const entry of keys) {
    console.log(`  ${entry.sid ?? '?'}  ${entry.friendly_name || '(unnamed)'}  ` +
                `created ${entry.date_created ?? '?'}`);
  }

  const flag = process.argv.indexOf('--workloads');
  const workloads = flag === -1
    ? await countWorkloads(auth, account)
    : Number.parseInt(process.argv[flag + 1], 10);

  const [state, detail] = verdict(keys, workloads, runningAs);
  if (state === 'keyed') {
    console.log(`${state.padEnd(12)} ${detail}`);
    return;
  }

  console.warn(`${state.padEnd(12)} ${detail}`);
  console.warn(`  repair: POST ${BASE}/Accounts/${account}/Keys.json ` +
               'FriendlyName={service-name}, then store the returned sid and ' +
               'secret as the basic-auth pair');
  console.warn('  keep the auth token for X-Twilio-Signature validation and ' +
               'nowhere else');
  process.exitCode = 1;
}

// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing credentials and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}

Add a test

Two things are worth pinning here. The username classification, because everything else in the note rests on SK against AC and on it surviving whatever casing and stray whitespace an environment variable arrives with. And the precedence: an account with keys is still running on the auth token if this process authenticated with one, and the report has to lead with the thing it knows rather than the thing it inferred.

test_twilio_credential_audit.py
from twilio_credential_audit import credential_kind, verdict

KEY = {"sid": "SK00000000000000000000000000000001", "friendly_name": "billing-worker"}


def test_an_sk_username_is_an_api_key():
    assert credential_kind("SK00000000000000000000000000000001") == "api-key"


def test_an_account_sid_as_the_username_means_the_auth_token():
    assert credential_kind("AC00000000000000000000000000000001") == "auth-token"


def test_case_and_whitespace_do_not_change_the_answer():
    assert credential_kind("  sk0123456789  ") == "api-key"
    assert credential_kind("ac0123456789") == "auth-token"


def test_an_empty_or_odd_username_is_not_guessed_at():
    assert credential_kind("") == "unknown"
    assert credential_kind(None) == "unknown"
    assert credential_kind("username") == "unknown"


def test_no_keys_at_all_is_the_headline_finding():
    state, detail = verdict([], workloads=4)
    assert state == "no-keys"
    assert "signs your webhooks" in detail


def test_running_under_the_auth_token_outranks_a_healthy_key_count():
    # Six keys and a tidy account, and this shell still holds the account secret.
    state, detail = verdict([KEY] * 6, workloads=3, running_as="auth-token")
    assert state == "auth-token"
    assert "proof" in detail


def test_fewer_keys_than_workloads_means_a_shared_credential():
    state, detail = verdict([KEY, KEY], workloads=7, running_as="api-key")
    assert state == "under-keyed"
    assert "share a credential" in detail


def test_a_key_per_workload_passes():
    state, _ = verdict([KEY] * 5, workloads=5, running_as="api-key")
    assert state == "keyed"


def test_an_unknown_workload_count_does_not_manufacture_a_finding():
    state, _ = verdict([KEY], workloads=0, running_as="api-key")
    assert state == "keyed"
twilio-credential-audit.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { credentialKind, verdict } from './twilio-credential-audit.mjs';

const KEY = { sid: 'SK00000000000000000000000000000001', friendly_name: 'billing-worker' };
const keys = (n) => Array.from({ length: n }, () => KEY);

test('an SK username is an api key', () => {
  assert.equal(credentialKind('SK00000000000000000000000000000001'), 'api-key');
});

test('an account sid as the username means the auth token', () => {
  assert.equal(credentialKind('AC00000000000000000000000000000001'), 'auth-token');
});

test('case and whitespace do not change the answer', () => {
  assert.equal(credentialKind('  sk0123456789  '), 'api-key');
  assert.equal(credentialKind('ac0123456789'), 'auth-token');
});

test('an empty or odd username is not guessed at', () => {
  assert.equal(credentialKind(''), 'unknown');
  assert.equal(credentialKind(null), 'unknown');
  assert.equal(credentialKind('username'), 'unknown');
});

test('no keys at all is the headline finding', () => {
  const [state, detail] = verdict([], 4);
  assert.equal(state, 'no-keys');
  assert.match(detail, /signs your webhooks/);
});

test('running under the auth token outranks a healthy key count', () => {
  const [state, detail] = verdict(keys(6), 3, 'auth-token');
  assert.equal(state, 'auth-token');
  assert.match(detail, /proof/);
});

test('fewer keys than workloads means a shared credential', () => {
  const [state, detail] = verdict(keys(2), 7, 'api-key');
  assert.equal(state, 'under-keyed');
  assert.match(detail, /share a credential/);
});

test('a key per workload passes', () => {
  assert.equal(verdict(keys(5), 5, 'api-key')[0], 'keyed');
});

test('an unknown workload count does not manufacture a finding', () => {
  assert.equal(verdict(keys(1), 0, 'api-key')[0], 'keyed');
});

FAQ

Can I not just rotate the auth token and be done?

You can rotate it, and it is instant and account-wide. Every deployment that was not updated in the same window starts returning 20003 with no fallback, and your webhook receivers begin rejecting Twilio's signatures at the same moment because the signing key changed underneath them. Twilio provides a secondary auth token so the change can be staged rather than done as a cutover, but staged or not, the unit of change is the entire account. That is the property an API key removes.

What is the auth token still for, then?

Validating X-Twilio-Signature on inbound webhooks. Twilio signs those with the account auth token, so the service that receives them needs it and nothing else does. Keeping it in exactly one place, held by exactly one service, turns a shared password into a signing secret with a single consumer.

How do I find every place it is still used, given there is no last-used field?

By reading configuration rather than the API, and the tell is the username. Any Twilio client constructed with the account SID as its username is sending the auth token as the password. Grep the secret store, the deployment manifests and the CI variables for TWILIO_AUTH_TOKEN, and for account SIDs appearing where a credential is expected. The script proves the case for one process, its own; the rest is an audit of your own repos.

Are API keys actually less privileged than the auth token?

A standard key carries the same REST privileges the auth token does, so the win is not least privilege by default. The win is revocation scope: deleting one key takes down one service, at a moment you chose. Twilio also offers restricted keys with per-resource permissions where the product supports them, and a read-scoped key is what every script in this section asks for, because a leaked read key exposes your configuration rather than your ability to send.

Should this fail a CI job?

It exits non-zero on anything but keyed, which makes it usable as a gate. The more valuable placement is a scheduled run rather than a pipeline step, because this state does not change when you deploy. It changes when somebody adds a service in a hurry, and that is a Tuesday, not a release.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.