Diagnostic Slack
The value in SLACK_APP_TOKEN is not an app-level token
The deploy went out on a Tuesday and the service has been green ever since. The pod is running, the readiness probe passes, the log says connecting to Slack and then says it again a few seconds later, and again, and has been saying it for nine days. Nobody reads that line because nothing about it is an error.
In the channel, eleven people have typed @releasebot status since Tuesday and none of them has had an answer. The bot has not posted a single message since the deploy. It is not slow, not rate limited and not stuck — it never opened a connection at all, because the string in SLACK_APP_TOKEN begins with xoxb-.
apps.connections.open accepts one credential class: an app-level token, minted on Basic Information, always prefixed xapp-, always sent in the Authorization header. Hand it a bot token and it answers not_allowed_token_type. Hand it a token from a different app and it answers with an auth mismatch. Hand it nothing usable and no WebSocket is ever minted, so no event is ever delivered, and none of that raises anything your monitoring understands.
You can settle the commonest form of this from the environment alone, with no call at all. A Slack credential announces its class in its first characters: xapp- app-level, xoxb- bot, xoxp- user, xoxe- a rotation refresh token. Read what is in each variable, compare it against the class that variable's consumer requires, and the wrong-slot and same-string-in-both cases fall out immediately. Nothing is sent anywhere, so the credential under suspicion is never exercised and no connection is consumed.
The workspace confirms it in a shape no other Socket Mode fault produces: a step. Page conversations.history for a channel the bot is in and the app's last message sits just before a deploy, with every mention after it unanswered. Not a fraction of them. All of them, from one timestamp onward, for as long as you look.
The problem in plain words
The reason this survives a code review is that every individual piece of it is correct. There is an app-level token in the app configuration. There is a variable called SLACK_APP_TOKEN in the deployment. Somebody filled it in. The failure is a pairing error between two strings that look interchangeable at a glance, and the shell will happily hold either one in either name.
Four shapes account for nearly all of it. The first is the swap: SLACK_BOT_TOKEN and SLACK_APP_TOKEN set from each other, usually while copying a secret block between two environments. The second is the duplicate: the same xoxb- value pasted into both, because somebody was looking for “the Slack token” and there was only one in the password manager. The third is the quotes: a value copied out of a .env file with its surrounding quotation marks still attached, so the token Slack receives has a literal " at each end and matches nothing. The fourth is the wrong app: two Slack apps exist because Slack has no notion of environments and duplicating the app is the supported way to get one — and the staging app's xapp- token ends up beside the production app's xoxb-.
What turns a five-minute fix into nine days is the reconnect loop. Socket Mode clients are built to survive network trouble, so a failed connection attempt is retried by design. not_allowed_token_type is not network trouble — it is a permanent statement about a credential, and it will be true on the ten thousandth attempt too — but the client cannot tell those apart without special-casing the error, and several of them historically did not. The process therefore stays up, exits nonzero never, and passes every liveness check you have, while doing nothing whatsoever.
And the failure happens before your application code exists. There is no handler to instrument, no middleware to log from, no event to trace. The first line of your business logic has never run. That is why the only two places this is visible are the environment your process was handed and the silence in the channel — and why the check below reads exactly those two things.
Why it happens
The prefix is the class, and the class is the whole question here. Slack's token types are distinguishable by their first characters by design, which means a script can classify a credential without transmitting it. That matters more than convenience: the audit never sends the suspect value to Slack, never risks a lockout on a bad string, and never touches the connection budget. The check that costs nothing is also the check that cannot do harm.
This is not the missing-scope note, and the difference is which question gets to be asked. If the token in the slot is a genuine xapp- token belonging to this app, then the question becomes which app-level scopes it was minted with, and that is a different note with a different probe. This note ends where that one begins: it settles the credential's class and slot, and hands over the moment those are right.
The error strings from a real attempt are readable without making one. Most teams already have the failure in a log. not_allowed_token_type means the wrong class of token. invalid_auth means an app-level token that has been revoked, or an app that was deleted and recreated. An auth mismatch means the token belongs to another app. missing_scope means the class and app are right and the scopes are not. Exactly one of the answers Slack can give here is worth retrying, and it is ratelimited.
Total silence is a different fingerprint from partial loss, and the script insists on the distinction. A connection that was never opened loses everything from a single timestamp. A leaked connection pool loses a steady fraction. A refresh mishandled loses a cluster every few hours. Several replicas produce duplicates and misses together. Reporting “the bot missed some messages” is useless; reporting “the bot has answered nothing at all since 14:02 on the 9th, across 11 mentions” names this note and no other.
The script prints no token, ever. Not truncated, not masked, not the last four characters. It prints the variable name, the class it detected, and the length. A diagnostic that leaks the credential it is diagnosing into a terminal buffer, a CI log and a screenshot has made the situation worse.
Opening a connection to test the connection is not available to us, and would not help. apps.connections.open mints a URL and consumes one of the app's ten permitted connections. A read-only audit does not call it, and an audit that did would be answering a question the prefix already answered, at a price.
The fix, as a flow
The red arrow is the first one, which is the shape of a failure that happens before your application exists. There is no handler to instrument and no event to trace, because the credential was refused at the door and the WebSocket was never minted. Every box after that arrow is an absence wearing the costume of a healthy service: a process that stays up, a probe that passes, a log line that repeats. The fix branch sorts the contents of an environment rather than any answer from Slack, and it does so offline, because a prefix settles the question and sending the token would consume one of ten connections to learn the same thing.
How to fix it
Classify what is in each variable without sending any of it
credential_class maps a value to app-level, bot, user, refresh, quoted, absent or unknown from its opening characters. It is a pure function over a string; no request leaves the machine, so the suspect credential is never exercised and no connection is consumed by the act of asking.
Hold each class against the slot it was put in
slot_verdict knows that SLACK_APP_TOKEN feeds apps.connections.open and therefore needs an app-level token, and that SLACK_BOT_TOKEN feeds the Web API and needs a bot token. The output is per-variable and names the consumer, because “wrong token” is unhelpful and “this one goes to the WebSocket opener, which takes only xapp-” is not.
Check for the same string in two slots
duplicate_slots compares the values by identity rather than by class, because two variables holding one xoxb- token is a distinct mistake with a distinct cause — somebody looked for the Slack token — and it is worth saying out loud rather than reporting twice as a wrong class.
Read the error you already have in the log
open_error turns a logged apps.connections.open failure into a cause and a repair, and marks whether it is worth retrying. Pass it with --logged-error. This is the one place a real attempt informs the check, and the attempt was made by your app rather than by this script.
Confirm the step in the channel
silence_profile pages conversations.history, counts mentions of the bot, counts the app's own messages, and returns never-answered, stopped with the timestamp it stopped at, answering, or no-traffic. A step from a single moment, with nothing after it, is this note. A fraction lost is somebody else's.
Make the connection failure fatal
The repair the script prints is two lines of startup code, not a token change: assert the xapp- prefix before constructing the client, and treat a non-ratelimited failure from apps.connections.open as a reason to exit rather than to reconnect. A process that crashes on a permanent credential fault is discovered in minutes; one that retries it is discovered by a customer.
How to check it worked
Fix the variable, redeploy, and run it again against the same channel. The line to watch is the last one: the step should have an end.
python3 slack_socket_credential.py --channel C05REL9QT --logged-error not_allowed_token_type
# identity U07BOT9QD (releasebot) in Northwind
# slot SLACK_APP_TOKEN wrong-class holds a bot token (56 chars); this variable
# feeds apps.connections.open, which takes only xapp-
# slot SLACK_BOT_TOKEN ok holds a bot token (56 chars)
# slot duplicate SLACK_APP_TOKEN and SLACK_BOT_TOKEN hold the same string
# logged wrong-class not_allowed_token_type: a bot or user token was sent to a
# method that accepts only an app-level token. Not a retry
# history 412 message(s) from C05REL9QT
# silence never-answered 11 mention(s) of U07BOT9QD and 0 reply from the app
# verdict Socket Mode has never connected with this environment
# repair: put the xapp- token in SLACK_APP_TOKEN and restart
# repair: assert the xapp- prefix at startup, and exit rather than reconnect on
# not_allowed_token_type, invalid_auth or an auth mismatchThe full code
Two reads, and neither of them touches the credential under suspicion. credential_class and slot_verdict are pure string work over the environment, which is the point: the app-level token is classified, never transmitted, so the audit costs nothing and consumes none of the app's ten permitted connections. open_error exists so the error already sitting in your log becomes a sentence. silence_profile is the corroboration, and it is written to return a shape rather than a count, because the shape is what tells this note apart from the other three.
"""Decide whether Socket Mode can connect at all, without connecting.
Read only, and deliberately more careful than that. apps.connections.open mints
a WebSocket URL: it is a write, it consumes one of the ten connections an app is
allowed, and this script never calls it. The app-level token is classified from
its first characters and is never sent anywhere, so the value under suspicion is
not exercised by the act of auditing it.
Two GETs are made, both with the bot token: auth.test to learn the app's own
user id, and conversations.history to see whether the workspace has been talking
to a bot that stopped answering. Nothing is written, and no token value is ever
printed - only its variable, its detected class and its length.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_socket_credential")
API = "https://slack.com/api/"
# Longest first, so xoxe.xoxp- is not read as a user token. Slack's token types
# are distinguishable by prefix by design, which is what lets this script answer
# the question without transmitting the credential.
PREFIXES = (
("xoxe.xoxp-", "refresh"),
("xoxe-", "refresh"),
("xapp-", "app-level"),
("xoxb-", "bot"),
("xoxp-", "user"),
("xoxa-", "legacy-workspace"),
("xoxr-", "legacy-refresh"),
("xoxc-", "browser"),
)
# What each variable's consumer requires. The consumer is named in the output
# because "wrong token" is not a finding and "this one is handed to the method
# that opens the WebSocket" is.
SLOTS = {
"SLACK_APP_TOKEN": ("app-level", "apps.connections.open, which accepts only an "
"app-level token"),
"SLACK_BOT_TOKEN": ("bot", "the Web API as your app's bot user"),
"SLACK_USER_TOKEN": ("user", "the Web API as the installing person"),
}
# One of these is transient. The other five are permanent statements about a
# credential, and a client that reconnects on them will reconnect forever.
OPEN_ERRORS = {
"not_allowed_token_type": (
"wrong-class", False,
"a bot or user token was sent to a method that accepts only an app-level "
"token. The value in the app token slot is not an xapp- token"),
"invalid_auth": (
"dead-credential", False,
"the app-level token is not valid: revoked from Basic Information, or "
"belonging to an app that was deleted and recreated"),
"auth_mismatch": (
"wrong-app", False,
"this app-level token was minted by a different app. Duplicating an app is "
"how Slack does environments, and this is how the halves get crossed"),
"token_revoked": (
"dead-credential", False,
"the token was revoked. Generate a new one and redeploy"),
"missing_scope": (
"wrong-scopes", False,
"the class and the app are right and the scope set is not. That is a "
"different problem with a different check"),
"ratelimited": (
"transient", True,
"the only answer on this list worth retrying. Back off and try again"),
}
def credential_class(value):
"""What kind of Slack credential is this string? Pure, and offline.
Returns (klass, detail). Nothing is transmitted: the class is read from the
prefix, so the token under suspicion is never exercised, never rate limited
and never able to consume one of the app's ten Socket Mode connections just
by being checked.
"quoted" is its own answer because a value copied out of a .env file with
its quotation marks attached is a real and common failure that would
otherwise be reported as an unrecognisable string.
"""
if value is None:
return ("absent", "the variable is not set at all")
raw = str(value)
if not raw.strip():
return ("absent", "the variable is set to an empty value")
text = raw.strip()
if len(text) >= 2 and text[0] in ('"', "'") and text[-1] == text[0]:
return ("quoted", "the value still carries the quotation marks it was copied "
"with, so Slack receives a string that matches no token")
for prefix, klass in PREFIXES:
if text.startswith(prefix):
return (klass, "prefix %s" % prefix)
return ("unknown", "no recognised Slack token prefix; this is not a Slack "
"credential, or it was truncated on the way in")
def slot_verdict(name, value):
"""Does this variable hold the class its consumer requires? Pure.
Returns (state, detail). States: ok, empty, quoted, wrong-class,
unrecognised, unknown-slot. The length is reported and the value never is.
"""
want = SLOTS.get(name)
klass, why = credential_class(value)
size = len(str(value or "").strip())
if want is None:
return ("unknown-slot", "no requirement is known for %s" % name)
need, consumer = want
if klass == "absent":
return ("empty", "nothing is set, and it is handed to %s" % consumer)
if klass == "quoted":
return ("quoted", "%s (%d chars)" % (why, size))
if klass == "unknown":
return ("unrecognised", "%s (%d chars)" % (why, size))
if klass != need:
return ("wrong-class", "holds a %s token (%d chars); this variable feeds %s"
% (klass, size, consumer))
return ("ok", "holds a %s token (%d chars)" % (klass, size))
def duplicate_slots(env):
"""Which variables hold the identical string? Pure.
Compared by value rather than by class, because one token pasted into two
variables has a different cause from two wrong tokens - somebody went
looking for "the Slack token" and there was only one - and saying so is
more use than reporting the same wrong class twice.
"""
seen = {}
for name in sorted(env):
value = str(env.get(name) or "").strip()
if not value:
continue
seen.setdefault(value, []).append(name)
return sorted(tuple(names) for names in seen.values() if len(names) > 1)
def open_error(error):
"""Read a logged apps.connections.open failure. Pure, and no call is made.
Returns (cause, retryable, detail). The script never produces one of these
errors itself; it reads the one your app already logged, which is the only
way to have the evidence without minting a connection to get it.
"""
text = str(error or "").strip()
if not text:
return ("unreported", False, "no error was supplied; the environment check "
"above stands on its own")
if text in OPEN_ERRORS:
cause, retryable, detail = OPEN_ERRORS[text]
return (cause, retryable, "%s: %s" % (text, detail))
return ("unclassified", False, "%s is not one of the documented refusals; read it "
"against the method reference" % text)
def silence_profile(messages, bot_user_id, min_mentions=1):
"""What shape is the app's absence in this channel? Pure.
Returns (state, detail, counts). The states exist to keep this note apart
from its neighbours, because every Socket Mode fault looks like "the bot
missed something" from a distance:
never-answered mentions exist and the app has posted nothing at all. A
connection that was never opened. This is the note.
stopped the app posted, then stopped, and every mention after its
last message went unanswered. A step, with a timestamp.
answering some mentions after the last app message, some before. A
fraction lost is a different failure entirely.
no-traffic nobody addressed the bot in this window, so there is
nothing to conclude and the script says so.
"""
mentions, own = [], []
marker = "<@%s>" % bot_user_id if bot_user_id else None
for m in messages or []:
ts = str((m or {}).get("ts") or "")
if not ts:
continue
author = (m or {}).get("user") or ""
if (bot_user_id and author == bot_user_id) or (m or {}).get("bot_id"):
own.append(ts)
continue
if marker and marker in str((m or {}).get("text") or ""):
mentions.append(ts)
counts = {"mentions": len(mentions), "app_messages": len(own)}
if len(mentions) < max(1, min_mentions):
return ("no-traffic", "nobody addressed this bot in the messages read, so its "
"silence proves nothing", counts)
if not own:
return ("never-answered", "%d mention(s) of the bot and not one message from "
"the app. Nothing has ever been delivered to it in "
"this window" % len(mentions), counts)
last_own = max(float(t) for t in own)
after = [t for t in mentions if float(t) > last_own]
counts["mentions_after_last_reply"] = len(after)
if len(after) >= max(1, min_mentions) and len(after) == len(mentions):
return ("never-answered", "%d mention(s), all of them after the app's last "
"message at %s" % (len(mentions), min(after)),
counts)
if after:
return ("stopped", "the app last posted at %.0f and %d mention(s) since then "
"have gone unanswered" % (last_own, len(after)), counts)
return ("answering", "the app has replied more recently than the last mention; "
"this is not a connection that never opened", counts)
def page_history(session, channel, limit, max_pages):
"""Page conversations.history. A read, and one of two calls made."""
out, cursor, pages = [], "", 0
while pages < max_pages:
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.error("history unavailable %s", body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = ((body.get("response_metadata") or {}).get("next_cursor") or "")
pages += 1
if not cursor:
break
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding the bot token used to read")
ap.add_argument("--slots", default="SLACK_APP_TOKEN,SLACK_BOT_TOKEN",
help="comma separated variable names to classify")
ap.add_argument("--logged-error", default="",
help="an apps.connections.open error your app already logged")
ap.add_argument("--channel", default="",
help="a channel the bot is in, to confirm the silence")
ap.add_argument("--limit", type=int, default=200)
ap.add_argument("--max-pages", type=int, default=3)
args = ap.parse_args()
names = [n.strip() for n in args.slots.split(",") if n.strip()]
env = {n: os.environ.get(n) for n in names}
faults = 0
for name in names:
state, detail = slot_verdict(name, env.get(name))
line = ("slot %-19s %-12s %s", name, state, detail)
if state == "ok":
log.info(*line)
else:
log.warning(*line)
faults += 1
for pair in duplicate_slots(env):
log.warning("slot duplicate %s hold the same string",
" and ".join(pair))
faults += 1
cause, retryable, detail = open_error(args.logged_error)
if cause != "unreported":
log.warning("logged %-19s %s", cause, detail)
log.info("logged retryable %s", "yes" if retryable else "no")
if not retryable:
faults += 1
if args.channel:
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token with channels:history", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s (%s) in %s", who.get("user_id"), who.get("user"),
who.get("team"))
messages = page_history(s, args.channel, args.limit, args.max_pages)
log.info("history %d message(s) from %s", len(messages), args.channel)
state, detail, counts = silence_profile(messages, who.get("user_id"))
log.warning("silence %-19s %s", state, detail)
log.info("silence counts %s", counts)
if state in ("never-answered", "stopped"):
faults += 1
if not faults:
log.info("verdict clean every slot holds the class its "
"consumer requires")
return 0
log.warning("verdict Socket Mode cannot connect with this environment")
log.warning(" repair: put the xapp- token from Basic Information in "
"SLACK_APP_TOKEN and the xoxb- token in SLACK_BOT_TOKEN, then restart")
log.warning(" repair: assert the xapp- prefix at startup, and exit rather than "
"reconnect on not_allowed_token_type, invalid_auth or auth_mismatch")
log.warning(" note: no connection was opened to establish this; "
"apps.connections.open is a write and would consume one of ten")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Decide whether Socket Mode can connect at all, without connecting.
*
* Read only, and deliberately more careful than that. apps.connections.open
* mints a WebSocket URL: it is a write, it consumes one of the ten connections
* an app is allowed, and this script never calls it. The app-level token is
* classified from its first characters and is never sent anywhere.
*
* Two GETs, both with the bot token: auth.test for the app's own user id, and
* conversations.history for the silence. No token value is ever printed.
*/
const API = 'https://slack.com/api/';
// Longest first, so xoxe.xoxp- is not read as a user token.
export const PREFIXES = [
['xoxe.xoxp-', 'refresh'],
['xoxe-', 'refresh'],
['xapp-', 'app-level'],
['xoxb-', 'bot'],
['xoxp-', 'user'],
['xoxa-', 'legacy-workspace'],
['xoxr-', 'legacy-refresh'],
['xoxc-', 'browser'],
];
// What each variable's consumer requires, named so the output can say which
// method is being handed the wrong thing.
export const SLOTS = {
SLACK_APP_TOKEN: ['app-level',
'apps.connections.open, which accepts only an app-level token'],
SLACK_BOT_TOKEN: ['bot', 'the Web API as your app bot user'],
SLACK_USER_TOKEN: ['user', 'the Web API as the installing person'],
};
// One of these is transient. The other five are permanent statements about a
// credential, and a client that reconnects on them reconnects forever.
export const OPEN_ERRORS = {
not_allowed_token_type: ['wrong-class', false,
'a bot or user token was sent to a method that accepts only an app-level token. '
+ 'The value in the app token slot is not an xapp- token'],
invalid_auth: ['dead-credential', false,
'the app-level token is not valid: revoked from Basic Information, or belonging '
+ 'to an app that was deleted and recreated'],
auth_mismatch: ['wrong-app', false,
'this app-level token was minted by a different app. Duplicating an app is how '
+ 'Slack does environments, and this is how the halves get crossed'],
token_revoked: ['dead-credential', false,
'the token was revoked. Generate a new one and redeploy'],
missing_scope: ['wrong-scopes', false,
'the class and the app are right and the scope set is not. That is a different '
+ 'problem with a different check'],
ratelimited: ['transient', true,
'the only answer on this list worth retrying. Back off and try again'],
};
/**
* What kind of Slack credential is this string? Pure, and offline.
* Returns [klass, detail]. Nothing is transmitted.
*/
export function credentialClass(value) {
if (value === null || value === undefined) {
return ['absent', 'the variable is not set at all'];
}
const raw = String(value);
if (!raw.trim()) return ['absent', 'the variable is set to an empty value'];
const text = raw.trim();
const first = text[0];
if (text.length >= 2 && (first === '"' || first === "'") && text.endsWith(first)) {
return ['quoted', 'the value still carries the quotation marks it was copied with, '
+ 'so Slack receives a string that matches no token'];
}
for (const [prefix, klass] of PREFIXES) {
if (text.startsWith(prefix)) return [klass, `prefix ${prefix}`];
}
return ['unknown', 'no recognised Slack token prefix; this is not a Slack credential, '
+ 'or it was truncated on the way in'];
}
/** Does this variable hold the class its consumer requires? Pure. */
export function slotVerdict(name, value) {
const want = SLOTS[name];
const [klass, why] = credentialClass(value);
const size = String(value ?? '').trim().length;
if (!want) return ['unknown-slot', `no requirement is known for ${name}`];
const [need, consumer] = want;
if (klass === 'absent') {
return ['empty', `nothing is set, and it is handed to ${consumer}`];
}
if (klass === 'quoted') return ['quoted', `${why} (${size} chars)`];
if (klass === 'unknown') return ['unrecognised', `${why} (${size} chars)`];
if (klass !== need) {
return ['wrong-class',
`holds a ${klass} token (${size} chars); this variable feeds ${consumer}`];
}
return ['ok', `holds a ${klass} token (${size} chars)`];
}
/** Which variables hold the identical string? Pure. */
export function duplicateSlots(env) {
const seen = new Map();
for (const name of Object.keys(env ?? {}).sort()) {
const value = String(env[name] ?? '').trim();
if (!value) continue;
if (!seen.has(value)) seen.set(value, []);
seen.get(value).push(name);
}
return [...seen.values()].filter((names) => names.length > 1);
}
/**
* Read a logged apps.connections.open failure. Pure, and no call is made.
* Returns [cause, retryable, detail].
*/
export function openError(error) {
const text = String(error ?? '').trim();
if (!text) {
return ['unreported', false,
'no error was supplied; the environment check above stands on its own'];
}
if (Object.prototype.hasOwnProperty.call(OPEN_ERRORS, text)) {
const [cause, retryable, detail] = OPEN_ERRORS[text];
return [cause, retryable, `${text}: ${detail}`];
}
return ['unclassified', false,
`${text} is not one of the documented refusals; read it against the method `
+ 'reference'];
}
/**
* What shape is the app absence in this channel? Pure.
* Returns [state, detail, counts]; states never-answered, stopped, answering,
* no-traffic. The shape is what separates this note from its neighbours.
*/
export function silenceProfile(messages, botUserId, minMentions = 1) {
const mentions = [];
const own = [];
const marker = botUserId ? `<@${botUserId}>` : null;
for (const m of messages ?? []) {
const ts = String((m ?? {}).ts ?? '');
if (!ts) continue;
const author = (m ?? {}).user ?? '';
if ((botUserId && author === botUserId) || (m ?? {}).bot_id) {
own.push(ts);
continue;
}
if (marker && String((m ?? {}).text ?? '').includes(marker)) mentions.push(ts);
}
const counts = { mentions: mentions.length, app_messages: own.length };
const floor = Math.max(1, minMentions);
if (mentions.length < floor) {
return ['no-traffic', 'nobody addressed this bot in the messages read, so its '
+ 'silence proves nothing', counts];
}
if (!own.length) {
return ['never-answered', `${mentions.length} mention(s) of the bot and not one `
+ 'message from the app. Nothing has ever been delivered to it in this window',
counts];
}
const lastOwn = Math.max(...own.map(Number));
const after = mentions.filter((t) => Number(t) > lastOwn);
counts.mentions_after_last_reply = after.length;
if (after.length >= floor && after.length === mentions.length) {
return ['never-answered', `${mentions.length} mention(s), all of them after the `
+ `app last message at ${after.slice().sort()[0]}`, counts];
}
if (after.length) {
return ['stopped', `the app last posted at ${lastOwn.toFixed(0)} and `
+ `${after.length} mention(s) since then have gone unanswered`, counts];
}
return ['answering', 'the app has replied more recently than the last mention; this '
+ 'is not a connection that never opened', counts];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function pageHistory(headers, channel, limit, maxPages) {
const out = [];
let cursor = '';
for (let page = 0; page < maxPages; page += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.error(`history unavailable ${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = (body.response_metadata ?? {}).next_cursor ?? '';
if (!cursor) break;
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const names = String(arg(args, '--slots', 'SLACK_APP_TOKEN,SLACK_BOT_TOKEN'))
.split(',').map((n) => n.trim()).filter(Boolean);
const env = {};
for (const n of names) env[n] = process.env[n];
let faults = 0;
for (const name of names) {
const [state, detail] = slotVerdict(name, env[name]);
const line = `slot ${name.padEnd(19)} ${state.padEnd(12)} ${detail}`;
if (state === 'ok') console.log(line);
else { console.warn(line); faults += 1; }
}
for (const pair of duplicateSlots(env)) {
console.warn(`slot duplicate ${pair.join(' and ')} hold the same `
+ 'string');
faults += 1;
}
const [cause, retryable, detail] = openError(arg(args, '--logged-error', ''));
if (cause !== 'unreported') {
console.warn(`logged ${cause.padEnd(19)} ${detail}`);
console.log(`logged retryable ${retryable ? 'yes' : 'no'}`);
if (!retryable) faults += 1;
}
const channel = arg(args, '--channel', '');
if (channel) {
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} to a bot token with channels:history`);
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} (${who.user}) in ${who.team}`);
const messages = await pageHistory(headers, channel,
Number(arg(args, '--limit', '200')), Number(arg(args, '--max-pages', '3')));
console.log(`history ${messages.length} message(s) from ${channel}`);
const [state, why, counts] = silenceProfile(messages, who.user_id);
console.warn(`silence ${state.padEnd(19)} ${why}`);
console.log(`silence counts ${JSON.stringify(counts)}`);
if (state === 'never-answered' || state === 'stopped') faults += 1;
}
if (!faults) {
console.log('verdict clean every slot holds the class its '
+ 'consumer requires');
return;
}
console.warn('verdict Socket Mode cannot connect with this environment');
console.warn(' repair: put the xapp- token from Basic Information in '
+ 'SLACK_APP_TOKEN and the xoxb- token in SLACK_BOT_TOKEN, then restart');
console.warn(' repair: assert the xapp- prefix at startup, and exit rather than '
+ 'reconnect on not_allowed_token_type, invalid_auth or auth_mismatch');
console.warn(' note: no connection was opened to establish this; '
+ 'apps.connections.open is a write and would consume one of ten');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Every fixture token here is obviously fake and shorter than any real one, because a test file is the last place a working credential should end up. The assertions worth reading are the ones about restraint: credential_class is tested for the quoted value and the truncated value, which are the two shapes a prefix check gets wrong if it only looks for xapp-; open_error is tested for the single retryable answer among six; and silence_profile is tested for the case where the bot is still replying, because reporting a step that is not there would send somebody to rotate a token that was never wrong.
from slack_socket_credential import (
credential_class, duplicate_slots, open_error, silence_profile, slot_verdict,
)
APP = "xapp-1-A-fake"
BOT = "xoxb-fake"
USER = "xoxp-fake"
def test_the_prefix_names_the_class_without_any_call():
assert credential_class(APP)[0] == "app-level"
assert credential_class(BOT)[0] == "bot"
assert credential_class(USER)[0] == "user"
def test_a_rotation_refresh_token_is_not_read_as_a_user_token():
assert credential_class("xoxe.xoxp-fake")[0] == "refresh"
assert credential_class("xoxe-fake")[0] == "refresh"
def test_a_value_that_kept_its_quotes_is_its_own_answer():
klass, detail = credential_class('"xoxb-fake"')
assert klass == "quoted"
assert "quotation marks" in detail
def test_an_unset_or_empty_variable_is_absent_rather_than_unknown():
assert credential_class(None)[0] == "absent"
assert credential_class(" ")[0] == "absent"
def test_a_truncated_value_is_unknown_and_says_so():
assert credential_class("1-A-fake")[0] == "unknown"
def test_a_bot_token_in_the_app_slot_is_the_finding():
state, detail = slot_verdict("SLACK_APP_TOKEN", BOT)
assert state == "wrong-class"
assert "apps.connections.open" in detail
def test_the_right_class_in_the_right_slot_passes():
assert slot_verdict("SLACK_APP_TOKEN", APP)[0] == "ok"
assert slot_verdict("SLACK_BOT_TOKEN", BOT)[0] == "ok"
def test_the_length_is_reported_and_the_value_is_not():
_state, detail = slot_verdict("SLACK_BOT_TOKEN", BOT)
assert str(len(BOT)) in detail
assert BOT not in detail
def test_an_empty_slot_names_the_consumer_that_will_go_hungry():
state, detail = slot_verdict("SLACK_APP_TOKEN", "")
assert state == "empty"
assert "apps.connections.open" in detail
def test_one_token_in_two_variables_is_reported_once_as_itself():
assert duplicate_slots({"SLACK_APP_TOKEN": BOT, "SLACK_BOT_TOKEN": BOT}) == [
("SLACK_APP_TOKEN", "SLACK_BOT_TOKEN")]
def test_different_tokens_are_not_duplicates_and_blanks_are_ignored():
assert duplicate_slots({"SLACK_APP_TOKEN": APP, "SLACK_BOT_TOKEN": BOT}) == []
assert duplicate_slots({"A": "", "B": ""}) == []
def test_the_wrong_token_type_error_is_permanent():
cause, retryable, _detail = open_error("not_allowed_token_type")
assert cause == "wrong-class"
assert retryable is False
def test_a_token_from_another_app_is_named_as_such():
assert open_error("auth_mismatch")[0] == "wrong-app"
def test_ratelimited_is_the_only_answer_here_worth_retrying():
assert open_error("ratelimited")[1] is True
for err in ("not_allowed_token_type", "invalid_auth", "auth_mismatch",
"missing_scope", "token_revoked"):
assert open_error(err)[1] is False
def test_a_missing_scope_is_handed_to_the_other_check():
cause, _retryable, detail = open_error("missing_scope")
assert cause == "wrong-scopes"
assert "scope set" in detail
def test_no_logged_error_is_not_a_finding():
assert open_error("")[0] == "unreported"
assert open_error(None)[0] == "unreported"
def test_mentions_with_no_reply_at_all_is_the_step_this_note_owns():
msgs = [{"ts": "10", "user": "U1", "text": "hello <@UBOT>"},
{"ts": "20", "user": "U2", "text": "<@UBOT> status"}]
state, detail, counts = silence_profile(msgs, "UBOT")
assert state == "never-answered"
assert counts == {"mentions": 2, "app_messages": 0}
assert "not one message" in detail
def test_a_bot_that_replied_before_the_deploy_and_not_after_is_a_stop():
msgs = [{"ts": "10", "user": "U1", "text": "<@UBOT> hi"},
{"ts": "11", "user": "UBOT", "text": "hello"},
{"ts": "20", "user": "U2", "text": "<@UBOT> status"}]
assert silence_profile(msgs, "UBOT")[0] == "stopped"
def test_a_bot_that_is_still_answering_is_not_this_note():
msgs = [{"ts": "10", "user": "U1", "text": "<@UBOT> hi"},
{"ts": "11", "user": "UBOT", "text": "hello"}]
assert silence_profile(msgs, "UBOT")[0] == "answering"
def test_a_message_posted_by_a_bot_id_counts_as_the_app_speaking():
msgs = [{"ts": "10", "user": "U1", "text": "<@UBOT> hi"},
{"ts": "11", "bot_id": "B1", "text": "hello"}]
assert silence_profile(msgs, "UBOT")[2]["app_messages"] == 1
def test_a_quiet_channel_proves_nothing_and_says_so():
assert silence_profile([{"ts": "10", "user": "U1", "text": "morning"}],
"UBOT")[0] == "no-traffic"
assert silence_profile([], "UBOT")[0] == "no-traffic"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
credentialClass, duplicateSlots, openError, silenceProfile, slotVerdict,
} from './slack-socket-credential.mjs';
const APP = 'xapp-1-A-fake';
const BOT = 'xoxb-fake';
const USER = 'xoxp-fake';
test('the prefix names the class without any call', () => {
assert.equal(credentialClass(APP)[0], 'app-level');
assert.equal(credentialClass(BOT)[0], 'bot');
assert.equal(credentialClass(USER)[0], 'user');
});
test('a rotation refresh token is not read as a user token', () => {
assert.equal(credentialClass('xoxe.xoxp-fake')[0], 'refresh');
assert.equal(credentialClass('xoxe-fake')[0], 'refresh');
});
test('a value that kept its quotes is its own answer', () => {
const [klass, detail] = credentialClass('"xoxb-fake"');
assert.equal(klass, 'quoted');
assert.match(detail, /quotation marks/);
});
test('an unset or empty variable is absent rather than unknown', () => {
assert.equal(credentialClass(null)[0], 'absent');
assert.equal(credentialClass(' ')[0], 'absent');
});
test('a truncated value is unknown and says so', () => {
assert.equal(credentialClass('1-A-fake')[0], 'unknown');
});
test('a bot token in the app slot is the finding', () => {
const [state, detail] = slotVerdict('SLACK_APP_TOKEN', BOT);
assert.equal(state, 'wrong-class');
assert.match(detail, /apps.connections.open/);
});
test('the right class in the right slot passes', () => {
assert.equal(slotVerdict('SLACK_APP_TOKEN', APP)[0], 'ok');
assert.equal(slotVerdict('SLACK_BOT_TOKEN', BOT)[0], 'ok');
});
test('the length is reported and the value is not', () => {
const [, detail] = slotVerdict('SLACK_BOT_TOKEN', BOT);
assert.match(detail, new RegExp(String(BOT.length)));
assert.equal(detail.includes(BOT), false);
});
test('an empty slot names the consumer that will go hungry', () => {
const [state, detail] = slotVerdict('SLACK_APP_TOKEN', '');
assert.equal(state, 'empty');
assert.match(detail, /apps.connections.open/);
});
test('one token in two variables is reported once as itself', () => {
assert.deepEqual(duplicateSlots({ SLACK_APP_TOKEN: BOT, SLACK_BOT_TOKEN: BOT }),
[['SLACK_APP_TOKEN', 'SLACK_BOT_TOKEN']]);
});
test('different tokens are not duplicates and blanks are ignored', () => {
assert.deepEqual(duplicateSlots({ SLACK_APP_TOKEN: APP, SLACK_BOT_TOKEN: BOT }), []);
assert.deepEqual(duplicateSlots({ A: '', B: '' }), []);
});
test('the wrong token type error is permanent', () => {
const [cause, retryable] = openError('not_allowed_token_type');
assert.equal(cause, 'wrong-class');
assert.equal(retryable, false);
});
test('a token from another app is named as such', () => {
assert.equal(openError('auth_mismatch')[0], 'wrong-app');
});
test('ratelimited is the only answer here worth retrying', () => {
assert.equal(openError('ratelimited')[1], true);
for (const err of ['not_allowed_token_type', 'invalid_auth', 'auth_mismatch',
'missing_scope', 'token_revoked']) {
assert.equal(openError(err)[1], false);
}
});
test('a missing scope is handed to the other check', () => {
const [cause, , detail] = openError('missing_scope');
assert.equal(cause, 'wrong-scopes');
assert.match(detail, /scope set/);
});
test('no logged error is not a finding', () => {
assert.equal(openError('')[0], 'unreported');
assert.equal(openError(null)[0], 'unreported');
});
test('mentions with no reply at all is the step this note owns', () => {
const msgs = [{ ts: '10', user: 'U1', text: 'hello <@UBOT>' },
{ ts: '20', user: 'U2', text: '<@UBOT> status' }];
const [state, detail, counts] = silenceProfile(msgs, 'UBOT');
assert.equal(state, 'never-answered');
assert.deepEqual(counts, { mentions: 2, app_messages: 0 });
assert.match(detail, /not one message/);
});
test('a bot that replied before the deploy and not after is a stop', () => {
const msgs = [{ ts: '10', user: 'U1', text: '<@UBOT> hi' },
{ ts: '11', user: 'UBOT', text: 'hello' },
{ ts: '20', user: 'U2', text: '<@UBOT> status' }];
assert.equal(silenceProfile(msgs, 'UBOT')[0], 'stopped');
});
test('a bot that is still answering is not this note', () => {
const msgs = [{ ts: '10', user: 'U1', text: '<@UBOT> hi' },
{ ts: '11', user: 'UBOT', text: 'hello' }];
assert.equal(silenceProfile(msgs, 'UBOT')[0], 'answering');
});
test('a message posted by a bot_id counts as the app speaking', () => {
const msgs = [{ ts: '10', user: 'U1', text: '<@UBOT> hi' },
{ ts: '11', bot_id: 'B1', text: 'hello' }];
assert.equal(silenceProfile(msgs, 'UBOT')[2].app_messages, 1);
});
test('a quiet channel proves nothing and says so', () => {
assert.equal(silenceProfile([{ ts: '10', user: 'U1', text: 'morning' }],
'UBOT')[0], 'no-traffic');
assert.equal(silenceProfile([], 'UBOT')[0], 'no-traffic');
});
FAQ
Why not just call apps.connections.open and read the error?
Because it is a write. The method mints a WebSocket URL, which changes state on Slack's side and consumes one of the ten concurrent connections your app is allowed. On a healthy app that is rude; on an app that is already near the cap it can displace a live connection and cause the outage you were investigating. The prefix in the environment answers the same question for free, and the error you are curious about is almost certainly already in your log, where open_error will read it for you.
The token starts with xapp- and it still does not connect. Now what?
Then this note is finished and the next question is the scope set. An app-level token is minted per scope on the Basic Information page and cannot be edited afterwards, so a token created for authorizations:read does not carry connections:write. That gap has its own check, which probes the one read method an app-level token can call rather than opening a connection. The note on the underscoped xapp- token is linked below.
Our health check passes. How can it not be connected?
Because nothing crashed. Socket Mode clients treat a failed connection as something to retry, and a credential error is not distinguishable from a network error without special-casing it. The process is alive, the port is open, the readiness probe answers, and the connection has never been established once. This is why the repair is a startup assertion rather than a monitoring change: the only reliable signal is the one you create yourself, by refusing to boot with a token that cannot possibly work.
We have two Slack apps, one for staging and one for production. Is that the problem?
It is a very common cause of the wrong-app variant, and it is not a mistake in itself. Slack has no concept of environments, so duplicating the app is the supported way to get one, and the cost is two app ids and two complete sets of credentials that look identical to the eye. When the xapp- token from one app meets the xoxb- token from the other, apps.connections.open answers with an auth mismatch, and everything else about the deployment looks fine. Label the secrets by app id, not by purpose.
Could the silence in the channel have some other cause?
Yes, and that is why the script reports a shape rather than a number. Total silence from a single timestamp with nothing after it is what a connection that never opened looks like. A steady fraction of unanswered mentions points at leaked connections filling the cap. Losses that cluster every few hours point at a refresh that was not overlapped. Duplicates and misses together point at several replicas each holding a socket. All four are linked from this section, and the script tells you which shape it actually found instead of assuming yours.
Related field notes
- the same token, right class and wrong scopes
- the error itself, across the rest of the API
- a transport that works and nothing subscribed to it
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- apps.connections.open method reference - Slack Docs
- Token types - Slack Docs
- Using Socket Mode - Slack Docs
- bolt-js #1748: Socket Mode retries a permanent auth failure forever
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.