Diagnostic Slack
Socket Mode off, no Request URL, and nowhere to deliver
Somebody turned Socket Mode off. It was a reasonable thing to do — the team is preparing to submit the app, and a listed app has to receive events over HTTPS, so the switch had to move eventually. It moved on a Thursday afternoon and nothing happened, which everyone took as a good sign.
Nine days later a support ticket says the release bot has not announced a release since the 12th. It has not. It has not done anything since the 12th. There is no error in the logs, no failed delivery, no alert, no red banner on the configuration screen. The app is running, healthy, subscribed to fourteen events, and Slack has had nowhere to send a single one of them since the switch moved.
Event delivery needs exactly one transport, and Slack does not fall back from one to the other. Either Socket Mode is on and your process holds a WebSocket, or an HTTPS Request URL is configured and verified and Slack posts to it. Turning Socket Mode off does not promote a Request URL, because there may not be one — and when there is not, Slack has no destination for your app's events and drops them where they are.
The finding is two fields held against each other. apps.manifest.export?app_id=A... returns settings.socket_mode_enabled and settings.event_subscriptions.request_url. Both false and empty, with a populated bot_events[], is the whole diagnosis: the app is asking Slack for fourteen kinds of traffic and has given Slack no way to hand any of it over.
The consequence is not uniform, which is why it gets misread. Interactivity carries its own request_url, and every slash command carries a url of its own, so a command configured back when the app spoke HTTP keeps working while events go dark. The app is half alive, and the half that still answers is the half people test with.
The workspace dates it for you. The app's own message series does not thin out or start missing a fraction — it ends, at a timestamp you can hand to whoever reads the audit log. The script below measures that as a rate change rather than as unanswered mentions, because a transport that was removed produces a cliff and a transport that is struggling produces a slope.
The problem in plain words
The reason this survives for days is that every part of it reports success. The process starts. The manifest is valid. The subscriptions are still listed on the Event Subscriptions screen, all fourteen of them, in a box that looks exactly as it did last week. No call fails, because your app is not making any — it is waiting to be called, and nothing is calling.
If the app was running a Socket Mode client when the switch moved, there is one signal and it is easy to throw away. The client's next connection attempt comes back with a disconnect whose reason is link_disabled, which means precisely “Socket Mode is off for this app and reconnecting will never work.” A client that treats every disconnect as weather retries it forever at one line of INFO per attempt. The signal exists, it is correct, it is specific, and it is indistinguishable in a log viewer from the reconnect chatter of a perfectly healthy service.
The half-alive shape does the rest of the damage. Slash commands each store their own URL, so /deploy still opens its dialog and the on-call engineer concludes that the app is up. Interactivity stores its own URL too, so if that one was set the buttons keep working. What has actually stopped is one specific surface — the events subscription, the only surface with no URL of its own to fall back on — and the events surface is the one nobody clicks.
Then there is the direction of the mistake. Teams arrive at this state while doing the right thing: moving off Socket Mode because a Marketplace listing requires it, or turning the switch off to test the HTTP path, or copying a manifest between apps and dropping a field. Nobody sets out to configure zero transports. It happens because the two settings live on two screens and neither screen mentions the other, so it is entirely possible to remove the last route into your application without any single click looking wrong.
Why it happens
Zero is a state, and it needs to be counted rather than inferred. The check is not “is Socket Mode on” and it is not “is there a Request URL”. It is how many transports there are, and the answer has four values: none, socket, http, both. One is healthy. Two is the note on both delivery paths at once, which names this state in passing and is about the opposite problem. Zero is this note, and it is the only one of the four that produces silence.
Surfaces do not all fail together, so the script reports them one at a time. The events subscription has no URL of its own. Interactivity has one. Every slash command has one. That asymmetry is the difference between an engineer concluding “the app is down” and concluding “the app is up, therefore this is not a configuration problem” — and the second conclusion is the one that costs a week. Reporting per surface makes the half-alive shape visible instead of confusing.
The declaration count is the part that makes it undeniable. An empty bot_events[] beside no transport is an app that does nothing and was never meant to; that is the nothing subscribed note, and it is the exact inverse of this one. A populated bot_events[] beside no transport is a contradiction the app is carrying in writing: it has told Slack which events it wants and given Slack nowhere to put them. The script prints the number and the names.
A cliff is a different statistic from a gap, and only one of them dates the change. Counting unanswered mentions tells you the app is not replying. Measuring the app's own posting rate before and after tells you when it stopped and how confident you can be that it stopped rather than paused: an app that posted four times a day for a month and has posted nothing for nine days is not slow. The output is a timestamp somebody can take to the audit log and match against who moved which switch.
Nothing here dials your Request URL, because there is no Request URL to dial. That is the point of the note. Where a URL does exist and is broken, the question is whether it verified (a handshake that failed) or whether it points at a laptop that closed (a tunnel that died). Both of those notes are about the quality of a string. This one is about the absence of the string.
The repair is an assertion, not a setting. Setting the switch back takes ten seconds and does not stop it happening again, because the same two screens will still be there next quarter. An app that reads its own manifest at startup and refuses to boot with zero transports turns a nine-day silence into a failed deploy, which is the only version of this failure anybody notices in time.
The fix, as a flow
The red arrow is in the middle, and everything to the left of it is correct. Turning Socket Mode off was a reasonable thing to do, there was no Request URL underneath it because there had never needed to be one, and Slack is behaving exactly as documented when it drops what it cannot deliver. The failure only exists at the join. The fix branch sorts a count rather than an error, because the four possible counts are four different notes, and the two rows in the middle are the ones that keep an investigation honest: commands that still answer are not evidence that delivery works, and a series that simply ends is not the same statistic as a fraction going missing.
How to fix it
Count the transports rather than checking either one
transport_state reads settings.socket_mode_enabled and settings.event_subscriptions.request_url together and returns none, socket, http or both. Four states, one of which is healthy, one of which is this note and one of which is a different note. A boolean check on either field alone cannot tell those apart.
Say which surfaces went dark and which kept a route
dark_surfaces walks the events subscription, interactivity and every slash command, and marks each dark, routed or absent. Slash commands and interactivity carry URLs of their own; the events subscription does not. This is why the app looks up while being deaf, and printing it per surface is what stops the investigation going the wrong way.
Count what the app has declared it can never receive
undeliverable_declarations counts bot_events[] plus user_events[] and names them. A populated subscription list beside a transport count of zero is the app contradicting itself in its own configuration, and it is the sentence to put in the incident notes.
Date the cliff from the app's own messages
posting_cliff takes the timestamps of the app's own messages and the timestamp of the newest message in the channel, and returns cliff with a date, steady, never-posted or sparse. The test is the quiet period measured in the app's own average gap, so a chatty app and a once-a-week app are judged on their own habits rather than against a fixed threshold.
Read the disconnect reason if a client was running
If a Socket Mode client was live when the switch moved, its log holds link_disabled, which means the setting was turned off rather than the network having a bad day. Pass it with --disconnect-reason and the script grades the finding as confirmed rather than inferred. Absent, the manifest still settles it.
Make a transport count of zero a failed deploy
The printed repair is a startup assertion: read your own manifest with a configuration token, count the transports, and refuse to boot on zero. It costs one call at startup and converts the entire failure mode from a silence somebody reports in nine days into a crash loop somebody sees in nine seconds.
How to check it worked
Turn the switch back on, or configure and verify a Request URL, and run it again. The two lines to watch are transport, which should read socket or http, and cliff, which should have stopped being a cliff.
python3 slack_event_transport.py --app-id A05RELB0T --channel C05REL9QT \
--disconnect-reason link_disabled
# manifest socket mode off, event Request URL: not set
# transport none neither transport is configured, so Slack has no
# destination for this app's events and drops them
# declared 14 subscription(s) that cannot arrive: app_mention, message.channels,
# reaction_added, team_join, ...
# surface event subscriptions dark 14 subscription(s) and no route of its own
# surface interactivity dark enabled with an empty request_url
# surface /deploy routed carries its own URL, so it still answers
# surface /rollback routed carries its own URL, so it still answers
# disconnect link_disabled Socket Mode was switched off underneath a running client
# identity U07BOT9QD (releasebot) in Northwind
# history 486 message(s) from C05REL9QT
# cliff cliff 4.1 message(s) a day for 31 days, then nothing for 9.2
# days; the series ends at 1755012340.000200
# verdict the app declares 14 event(s) and has nowhere to receive them
# repair: switch Socket Mode back on, or configure and verify an HTTPS Request URL
# repair: assert the transport count at startup and refuse to boot on zeroThe full code
The whole diagnosis is transport_state, and it is deliberately a count rather than a pair of booleans, because the four values are four different notes. Everything after it exists to make the count believable to somebody who did not run it: dark_surfaces explains why the app still answers /deploy, undeliverable_declarations turns the contradiction into a number, and posting_cliff puts a date on it. Three reads, all GETs, none of them aimed at your own endpoint.
"""Decide whether Slack has anywhere to deliver this app's events.
Read only. apps.manifest.export, auth.test and conversations.history are reads.
Nothing here writes a manifest, opens a socket, or sends anything at all to a
Request URL - there is no Request URL to send to, which is the finding.
Delivery needs exactly one transport: Socket Mode on and a WebSocket held, or
an HTTPS Request URL configured and verified. There is no fallback between
them. Switching Socket Mode off does not promote the Request URL, because there
may not be one, and Slack drops what it cannot deliver.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_event_transport")
API = "https://slack.com/api/"
DAY = 86400.0
# How much quiet, measured in the app's own average gap between messages, counts
# as the series having ended rather than merely being slow. Six is generous on
# purpose: an app that posts four times a day has to be silent for a day and a
# half before this script will call it a cliff.
QUIET_MULTIPLE = 6.0
# ...and never less than an hour, so a chatty app that paused over lunch is not
# reported as dead.
QUIET_FLOOR = 3600.0
# The one disconnect reason that means the setting moved rather than the network
# wobbled. A client that treats it as transient reconnects forever.
DISCONNECT_REASONS = {
"link_disabled": ("confirmed",
"Socket Mode was switched off for this app while a client was "
"connected. Reconnecting will never succeed"),
"refresh_requested": ("unrelated",
"a routine scheduled refresh, which every long-lived "
"connection gets. Not this note"),
"too_many_websockets": ("unrelated",
"the ten connection cap, which is a different note. "
"Socket Mode is on in that case, not off"),
"warning": ("unrelated",
"the ten second notice before a scheduled refresh. Not a failure"),
}
def transport_state(manifest):
"""How many transports can deliver an event to this app? Pure.
Returns (state, detail, facts). The four states are four different notes,
which is why this is a count and not a pair of boolean checks:
none neither is configured. Slack has no destination and drops events.
socket Socket Mode only. One of the two supported shapes.
http a Request URL only. The other supported shape.
both the switch is on and a URL is still stored underneath it, so two
paths exist and duplicates follow. A different note entirely.
unknown no manifest was available to read.
"""
if manifest is None:
return ("unknown", "no manifest was read, so the transports cannot be "
"counted; an app configuration token is what reads them",
{})
settings = manifest.get("settings") or {}
socket = bool(settings.get("socket_mode_enabled"))
events = settings.get("event_subscriptions") or {}
url = str(events.get("request_url") or "").strip()
facts = {"socket_mode": socket, "request_url": url}
if socket and url:
return ("both", "Socket Mode is on and a Request URL is still stored "
"underneath it; that is two delivery paths, not none", facts)
if socket:
return ("socket", "Socket Mode carries every event, interaction and command",
facts)
if url:
return ("http", "events are delivered to the stored Request URL", facts)
return ("none", "neither transport is configured, so Slack has no destination "
"for this app's events and drops them", facts)
def dark_surfaces(manifest):
"""Which inbound surfaces still have a route, and which do not? Pure.
Returns [(surface, state, detail), ...] with state dark, routed or absent.
They do not fail together, and that is the whole reason this failure is
misread. The events subscription has no URL of its own; interactivity keeps
one; every slash command keeps one. Turn Socket Mode off and the commands
carry on answering while events stop, so the app looks up and is deaf.
"""
if manifest is None:
return []
settings = manifest.get("settings") or {}
features = manifest.get("features") or {}
socket = bool(settings.get("socket_mode_enabled"))
out = []
events = settings.get("event_subscriptions") or {}
declared = list(events.get("bot_events") or []) + list(events.get("user_events") or [])
events_url = str(events.get("request_url") or "").strip()
if not declared:
out.append(("event subscriptions", "absent", "nothing is subscribed"))
elif socket:
out.append(("event subscriptions", "routed", "the socket carries them"))
elif events_url:
out.append(("event subscriptions", "routed", "a Request URL of its own"))
else:
out.append(("event subscriptions", "dark",
"%d subscription(s) and no route of its own" % len(declared)))
inter = settings.get("interactivity") or {}
inter_url = str(inter.get("request_url") or "").strip()
if not inter.get("is_enabled"):
out.append(("interactivity", "absent", "the switch is off"))
elif socket:
out.append(("interactivity", "routed", "the socket carries clicks too"))
elif inter_url:
out.append(("interactivity", "routed", "carries its own URL"))
else:
out.append(("interactivity", "dark", "enabled with an empty request_url"))
for cmd in features.get("slash_commands") or []:
name = str((cmd or {}).get("command") or "?")
cmd_url = str((cmd or {}).get("url") or "").strip()
if socket:
out.append((name, "routed", "the socket carries commands too"))
elif cmd_url:
out.append((name, "routed", "carries its own URL, so it still answers"))
else:
out.append((name, "dark", "no URL of its own and no socket"))
return out
def undeliverable_declarations(manifest, state):
"""What has the app asked for that it cannot receive? Pure.
Returns (count, names). Only meaningful when the transport count is zero: a
populated subscription list beside no transport is the app contradicting
itself in writing, and an empty one is a different note about an app that
subscribes to nothing at all.
"""
if manifest is None or state != "none":
return (0, [])
events = (manifest.get("settings") or {}).get("event_subscriptions") or {}
names = sorted({str(e) for e in (list(events.get("bot_events") or [])
+ list(events.get("user_events") or []))})
return (len(names), names)
def posting_cliff(app_timestamps, newest_ts, min_messages=4):
"""Did the app's own message series end, and when? Pure.
Returns (state, detail, facts). States:
cliff a steady rate, then quiet for far longer than that rate
predicts. A transport that was removed ends the series.
steady the app is still posting near the end of the window.
never-posted the app has posted nothing in what was read.
sparse too few messages to measure a rate honestly.
A rate change rather than a count of unanswered mentions, on purpose. A
transport that struggles produces a slope and loses a fraction; a transport
that was removed produces a step, and the step has a date on it that can be
matched against whoever moved the switch.
"""
stamps = sorted(float(t) for t in (app_timestamps or []))
now = float(newest_ts or 0.0)
facts = {"messages": len(stamps)}
if not stamps:
return ("never-posted", "the app has posted nothing in the messages read, so "
"there is no series to have ended", facts)
if len(stamps) < max(2, min_messages):
return ("sparse", "%d message(s) from the app is too few to measure a rate "
"against" % len(stamps), facts)
span = stamps[-1] - stamps[0]
if span <= 0:
return ("sparse", "every message from the app carries the same timestamp",
facts)
mean_gap = span / (len(stamps) - 1)
quiet = max(0.0, now - stamps[-1])
rate = len(stamps) / (span / DAY)
facts.update({"per_day": round(rate, 2), "quiet_days": round(quiet / DAY, 2),
"last_ts": stamps[-1]})
if quiet > max(QUIET_FLOOR, QUIET_MULTIPLE * mean_gap):
return ("cliff", "%.1f message(s) a day for %.0f days, then nothing for %.1f "
"days; the series ends at %.6f"
% (rate, span / DAY, quiet / DAY, stamps[-1]), facts)
return ("steady", "%.1f message(s) a day and still posting near the end of the "
"window" % rate, facts)
def disconnect_meaning(reason):
"""Read a Socket Mode disconnect reason your client already logged. Pure.
Returns (grade, detail). No connection is opened to produce one: the reason
is in your log if a client was running when the switch moved, and absent if
one was not, in which case the manifest settles it alone.
"""
text = str(reason or "").strip()
if not text:
return ("not-supplied", "no disconnect reason was given; the manifest read "
"stands on its own")
if text in DISCONNECT_REASONS:
grade, detail = DISCONNECT_REASONS[text]
return (grade, "%s: %s" % (text, detail))
return ("unrecognised", "%s is not a documented disconnect reason" % text)
def load_manifest(args):
"""Read the live manifest. A read: export returns it and changes nothing."""
if args.manifest:
return json.loads(open(args.manifest, encoding="utf-8").read())
token = os.environ.get(args.config_token_env)
if not token or not args.app_id:
log.warning("manifest unavailable set %s and --app-id, or pass "
"--manifest", args.config_token_env)
return None
body = requests.get(API + "apps.manifest.export",
headers={"Authorization": "Bearer " + token},
params={"app_id": args.app_id}, timeout=30).json()
if body.get("ok") is not True:
log.error("manifest unavailable %s", body.get("error"))
return None
return body.get("manifest") or {}
def page_history(session, channel, limit, max_pages):
"""Page conversations.history. A read."""
out, cursor, pages = [], "", 0
while pages < max_pages:
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.error("history unavailable %s", body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = ((body.get("response_metadata") or {}).get("next_cursor") or "")
pages += 1
if not cursor:
break
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--manifest", default="", help="path to an exported manifest")
ap.add_argument("--config-token-env", default="SLACK_CONFIG_TOKEN",
help="environment variable holding an app configuration token")
ap.add_argument("--app-id", default="", help="app id, for the manifest read")
ap.add_argument("--channel", default="",
help="a channel the app posts into, to date the silence")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding a bot token, for the history")
ap.add_argument("--disconnect-reason", default="",
help="a Socket Mode disconnect reason your client already logged")
ap.add_argument("--limit", type=int, default=200)
ap.add_argument("--max-pages", type=int, default=4)
args = ap.parse_args()
manifest = load_manifest(args)
state, detail, facts = transport_state(manifest)
if facts:
log.info("manifest socket mode %s, event Request URL: %s",
"on" if facts.get("socket_mode") else "off",
facts.get("request_url") or "not set")
(log.warning if state in ("none", "both") else log.info)(
"transport %-15s %s", state, detail)
count, names = undeliverable_declarations(manifest, state)
if count:
log.warning("declared %d subscription(s) that cannot arrive: %s", count,
", ".join(names[:6]) + (", ..." if count > 6 else ""))
for surface, surface_state, surface_detail in dark_surfaces(manifest):
(log.warning if surface_state == "dark" else log.info)(
"surface %-20s %-7s %s", surface, surface_state, surface_detail)
grade, grade_detail = disconnect_meaning(args.disconnect_reason)
if grade != "not-supplied":
(log.warning if grade == "confirmed" else log.info)(
"disconnect %-15s %s", grade, grade_detail)
cliff = "unmeasured"
if args.channel:
token = os.environ.get(args.token_env)
if not token:
log.warning("history set %s to date the silence", args.token_env)
else:
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s (%s) in %s", who.get("user_id"), who.get("user"),
who.get("team"))
messages = page_history(s, args.channel, args.limit, args.max_pages)
log.info("history %d message(s) from %s", len(messages), args.channel)
mine = [m.get("ts") for m in messages
if m.get("bot_id") or m.get("user") == who.get("user_id")]
newest = max([float(m.get("ts") or 0) for m in messages] or [0.0])
cliff, cliff_detail, cliff_facts = posting_cliff(mine, newest)
(log.warning if cliff == "cliff" else log.info)(
"cliff %-15s %s", cliff, cliff_detail)
log.info("cliff facts %s", cliff_facts)
if state == "none":
log.warning("verdict the app declares %d event(s) and has nowhere to "
"receive them", count)
log.warning(" repair: switch Socket Mode back on, or configure and verify an "
"HTTPS Request URL under Event Subscriptions")
log.warning(" repair: set the Interactivity Request URL too if clicks are "
"expected; it is a separate field on a separate screen")
log.warning(" repair: assert the transport count at startup and refuse to "
"boot on zero, so this fails at deploy time rather than in a week")
return 1
if state == "both":
log.warning("verdict two transports, not none; read the note on both "
"delivery paths being live at once")
return 1
if cliff == "cliff":
log.warning("verdict one transport is configured and the app still stopped "
"posting; look past the manifest")
return 1
log.info("verdict clean exactly one transport is configured")
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Decide whether Slack has anywhere to deliver this app's events.
*
* Read only. apps.manifest.export, auth.test and conversations.history are
* reads. Nothing here writes a manifest, opens a socket, or sends anything to a
* Request URL - there is no Request URL to send to, which is the finding.
*
* Delivery needs exactly one transport: Socket Mode on and a WebSocket held, or
* an HTTPS Request URL configured and verified. There is no fallback between
* them, and Slack drops what it cannot deliver.
*/
import { readFileSync } from 'node:fs';
const API = 'https://slack.com/api/';
export const DAY = 86400.0;
// How much quiet, measured in the app's own average gap, counts as the series
// having ended rather than merely being slow.
export const QUIET_MULTIPLE = 6.0;
export const QUIET_FLOOR = 3600.0;
// The one disconnect reason that means the setting moved rather than the
// network wobbling.
export const DISCONNECT_REASONS = {
link_disabled: ['confirmed',
'Socket Mode was switched off for this app while a client was connected. '
+ 'Reconnecting will never succeed'],
refresh_requested: ['unrelated',
'a routine scheduled refresh, which every long-lived connection gets. Not this '
+ 'note'],
too_many_websockets: ['unrelated',
'the ten connection cap, which is a different note. Socket Mode is on in that '
+ 'case, not off'],
warning: ['unrelated',
'the ten second notice before a scheduled refresh. Not a failure'],
};
/**
* How many transports can deliver an event to this app? Pure.
* Returns [state, detail, facts]; states none, socket, http, both, unknown.
*/
export function transportState(manifest) {
if (manifest === null || manifest === undefined) {
return ['unknown', 'no manifest was read, so the transports cannot be counted; an '
+ 'app configuration token is what reads them', {}];
}
const settings = manifest.settings ?? {};
const socket = Boolean(settings.socket_mode_enabled);
const events = settings.event_subscriptions ?? {};
const url = String(events.request_url ?? '').trim();
const facts = { socket_mode: socket, request_url: url };
if (socket && url) {
return ['both', 'Socket Mode is on and a Request URL is still stored underneath '
+ 'it; that is two delivery paths, not none', facts];
}
if (socket) {
return ['socket', 'Socket Mode carries every event, interaction and command',
facts];
}
if (url) return ['http', 'events are delivered to the stored Request URL', facts];
return ['none', 'neither transport is configured, so Slack has no destination for '
+ "this app's events and drops them", facts];
}
/**
* Which inbound surfaces still have a route, and which do not? Pure.
* Returns [[surface, state, detail], ...] with state dark, routed or absent.
*/
export function darkSurfaces(manifest) {
if (manifest === null || manifest === undefined) return [];
const settings = manifest.settings ?? {};
const features = manifest.features ?? {};
const socket = Boolean(settings.socket_mode_enabled);
const out = [];
const events = settings.event_subscriptions ?? {};
const declared = [...(events.bot_events ?? []), ...(events.user_events ?? [])];
const eventsUrl = String(events.request_url ?? '').trim();
if (!declared.length) out.push(['event subscriptions', 'absent', 'nothing is subscribed']);
else if (socket) out.push(['event subscriptions', 'routed', 'the socket carries them']);
else if (eventsUrl) out.push(['event subscriptions', 'routed', 'a Request URL of its own']);
else {
out.push(['event subscriptions', 'dark',
`${declared.length} subscription(s) and no route of its own`]);
}
const inter = settings.interactivity ?? {};
const interUrl = String(inter.request_url ?? '').trim();
if (!inter.is_enabled) out.push(['interactivity', 'absent', 'the switch is off']);
else if (socket) out.push(['interactivity', 'routed', 'the socket carries clicks too']);
else if (interUrl) out.push(['interactivity', 'routed', 'carries its own URL']);
else out.push(['interactivity', 'dark', 'enabled with an empty request_url']);
for (const cmd of features.slash_commands ?? []) {
const name = String((cmd ?? {}).command ?? '?');
const cmdUrl = String((cmd ?? {}).url ?? '').trim();
if (socket) out.push([name, 'routed', 'the socket carries commands too']);
else if (cmdUrl) out.push([name, 'routed', 'carries its own URL, so it still answers']);
else out.push([name, 'dark', 'no URL of its own and no socket']);
}
return out;
}
/**
* What has the app asked for that it cannot receive? Pure.
* Returns [count, names].
*/
export function undeliverableDeclarations(manifest, state) {
if (manifest === null || manifest === undefined || state !== 'none') return [0, []];
const events = (manifest.settings ?? {}).event_subscriptions ?? {};
const names = [...new Set([...(events.bot_events ?? []), ...(events.user_events ?? [])]
.map(String))].sort();
return [names.length, names];
}
/**
* Did the app's own message series end, and when? Pure.
* Returns [state, detail, facts]; states cliff, steady, never-posted, sparse.
*/
export function postingCliff(appTimestamps, newestTs, minMessages = 4) {
const stamps = [...(appTimestamps ?? [])].map(Number).sort((a, b) => a - b);
const now = Number(newestTs ?? 0);
const facts = { messages: stamps.length };
if (!stamps.length) {
return ['never-posted', 'the app has posted nothing in the messages read, so there '
+ 'is no series to have ended', facts];
}
if (stamps.length < Math.max(2, minMessages)) {
return ['sparse', `${stamps.length} message(s) from the app is too few to measure `
+ 'a rate against', facts];
}
const span = stamps[stamps.length - 1] - stamps[0];
if (span <= 0) {
return ['sparse', 'every message from the app carries the same timestamp', facts];
}
const meanGap = span / (stamps.length - 1);
const quiet = Math.max(0, now - stamps[stamps.length - 1]);
const rate = stamps.length / (span / DAY);
facts.per_day = Math.round(rate * 100) / 100;
facts.quiet_days = Math.round((quiet / DAY) * 100) / 100;
facts.last_ts = stamps[stamps.length - 1];
if (quiet > Math.max(QUIET_FLOOR, QUIET_MULTIPLE * meanGap)) {
return ['cliff', `${rate.toFixed(1)} message(s) a day for ${(span / DAY).toFixed(0)} `
+ `days, then nothing for ${(quiet / DAY).toFixed(1)} days; the series ends at `
+ `${stamps[stamps.length - 1].toFixed(6)}`, facts];
}
return ['steady', `${rate.toFixed(1)} message(s) a day and still posting near the end `
+ 'of the window', facts];
}
/**
* Read a Socket Mode disconnect reason your client already logged. Pure.
* Returns [grade, detail]. No connection is opened to produce one.
*/
export function disconnectMeaning(reason) {
const text = String(reason ?? '').trim();
if (!text) {
return ['not-supplied', 'no disconnect reason was given; the manifest read stands '
+ 'on its own'];
}
if (Object.prototype.hasOwnProperty.call(DISCONNECT_REASONS, text)) {
const [grade, detail] = DISCONNECT_REASONS[text];
return [grade, `${text}: ${detail}`];
}
return ['unrecognised', `${text} is not a documented disconnect reason`];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function loadManifest(args) {
const path = arg(args, '--manifest', '');
if (path) return JSON.parse(readFileSync(path, 'utf8'));
const tokenEnv = arg(args, '--config-token-env', 'SLACK_CONFIG_TOKEN');
const appId = arg(args, '--app-id', '');
const token = process.env[tokenEnv];
if (!token || !appId) {
console.warn(`manifest unavailable set ${tokenEnv} and --app-id, or pass `
+ '--manifest');
return null;
}
const params = new URLSearchParams({ app_id: appId });
const body = await (await fetch(`${API}apps.manifest.export?${params}`,
{ headers: { Authorization: `Bearer ${token}` } })).json();
if (body.ok !== true) {
console.error(`manifest unavailable ${body.error}`);
return null;
}
return body.manifest ?? {};
}
async function pageHistory(headers, channel, limit, maxPages) {
const out = [];
let cursor = '';
for (let page = 0; page < maxPages; page += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.error(`history unavailable ${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = (body.response_metadata ?? {}).next_cursor ?? '';
if (!cursor) break;
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const manifest = await loadManifest(args);
const [state, detail, facts] = transportState(manifest);
if (Object.keys(facts).length) {
console.log(`manifest socket mode ${facts.socket_mode ? 'on' : 'off'}, event `
+ `Request URL: ${facts.request_url || 'not set'}`);
}
const line = `transport ${state.padEnd(15)} ${detail}`;
if (state === 'none' || state === 'both') console.warn(line); else console.log(line);
const [count, names] = undeliverableDeclarations(manifest, state);
if (count) {
console.warn(`declared ${count} subscription(s) that cannot arrive: `
+ names.slice(0, 6).join(', ') + (count > 6 ? ', ...' : ''));
}
for (const [surface, surfaceState, surfaceDetail] of darkSurfaces(manifest)) {
const row = `surface ${surface.padEnd(20)} ${surfaceState.padEnd(7)} ${surfaceDetail}`;
if (surfaceState === 'dark') console.warn(row); else console.log(row);
}
const [grade, gradeDetail] = disconnectMeaning(arg(args, '--disconnect-reason', ''));
if (grade !== 'not-supplied') {
const row = `disconnect ${grade.padEnd(15)} ${gradeDetail}`;
if (grade === 'confirmed') console.warn(row); else console.log(row);
}
let cliff = 'unmeasured';
const channel = arg(args, '--channel', '');
if (channel) {
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) console.warn(`history set ${tokenEnv} to date the silence`);
else {
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} (${who.user}) in ${who.team}`);
const messages = await pageHistory(headers, channel,
Number(arg(args, '--limit', '200')), Number(arg(args, '--max-pages', '4')));
console.log(`history ${messages.length} message(s) from ${channel}`);
const mine = messages.filter((m) => m.bot_id || m.user === who.user_id)
.map((m) => m.ts);
const newest = Math.max(0, ...messages.map((m) => Number(m.ts ?? 0)));
const [cliffState, cliffDetail, cliffFacts] = postingCliff(mine, newest);
cliff = cliffState;
const row = `cliff ${cliffState.padEnd(15)} ${cliffDetail}`;
if (cliffState === 'cliff') console.warn(row); else console.log(row);
console.log(`cliff facts ${JSON.stringify(cliffFacts)}`);
}
}
if (state === 'none') {
console.warn(`verdict the app declares ${count} event(s) and has nowhere to `
+ 'receive them');
console.warn(' repair: switch Socket Mode back on, or configure and verify an '
+ 'HTTPS Request URL under Event Subscriptions');
console.warn(' repair: set the Interactivity Request URL too if clicks are '
+ 'expected; it is a separate field on a separate screen');
console.warn(' repair: assert the transport count at startup and refuse to boot '
+ 'on zero, so this fails at deploy time rather than in a week');
process.exitCode = 1;
return;
}
if (state === 'both') {
console.warn('verdict two transports, not none; read the note on both delivery '
+ 'paths being live at once');
process.exitCode = 1;
return;
}
if (cliff === 'cliff') {
console.warn('verdict one transport is configured and the app still stopped '
+ 'posting; look past the manifest');
process.exitCode = 1;
return;
}
console.log('verdict clean exactly one transport is configured');
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main().
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The assertions that matter are the ones about the four transport states, because the whole note turns on none being a different finding from both and from either healthy value. After that, the tests are about restraint: dark_surfaces is checked for the slash command that keeps working, which is the fact that misdirects the investigation; posting_cliff is checked for the app that is still posting and for the app that posted twice, because declaring a cliff on two data points would send somebody to the configuration screen for no reason.
from slack_event_transport import (
dark_surfaces, disconnect_meaning, posting_cliff, transport_state,
undeliverable_declarations,
)
DAY = 86400.0
def manifest(socket=False, url="", events=("app_mention",), inter=None, commands=()):
return {
"settings": {
"socket_mode_enabled": socket,
"event_subscriptions": {"request_url": url, "bot_events": list(events)},
"interactivity": inter or {},
},
"features": {"slash_commands": list(commands)},
}
def test_neither_transport_is_its_own_state_and_not_a_boolean():
state, detail, facts = transport_state(manifest())
assert state == "none"
assert facts == {"socket_mode": False, "request_url": ""}
assert "no destination" in detail
def test_socket_only_and_url_only_are_both_healthy():
assert transport_state(manifest(socket=True))[0] == "socket"
assert transport_state(manifest(url="https://ops.example.com/slack"))[0] == "http"
def test_both_at_once_is_the_other_note_rather_than_this_one():
state, detail, _facts = transport_state(
manifest(socket=True, url="https://ops.example.com/slack"))
assert state == "both"
assert "not none" in detail
def test_a_whitespace_url_is_not_a_transport():
assert transport_state(manifest(url=" "))[0] == "none"
def test_no_manifest_is_unknown_rather_than_a_finding():
assert transport_state(None)[0] == "unknown"
def test_the_events_surface_goes_dark_because_it_has_no_url_of_its_own():
rows = dict((r[0], r[1]) for r in dark_surfaces(manifest()))
assert rows["event subscriptions"] == "dark"
def test_a_slash_command_keeps_answering_and_that_is_the_misdirection():
rows = dark_surfaces(manifest(commands=[{"command": "/deploy",
"url": "https://ops.example.com/cmd"}]))
assert ("/deploy", "routed", "carries its own URL, so it still answers") in rows
def test_a_command_with_no_url_of_its_own_goes_dark_too():
rows = dict((r[0], r[1]) for r in dark_surfaces(manifest(commands=[
{"command": "/rollback", "url": ""}])))
assert rows["/rollback"] == "dark"
def test_the_socket_routes_every_surface_at_once():
rows = dark_surfaces(manifest(socket=True, inter={"is_enabled": True},
commands=[{"command": "/deploy", "url": ""}]))
assert {r[1] for r in rows} == {"routed"}
def test_interactivity_enabled_with_an_empty_url_is_dark():
rows = dict((r[0], r[1]) for r in dark_surfaces(manifest(inter={"is_enabled": True})))
assert rows["interactivity"] == "dark"
def test_interactivity_switched_off_is_absent_rather_than_dark():
rows = dict((r[0], r[1]) for r in dark_surfaces(manifest(inter={"is_enabled": False})))
assert rows["interactivity"] == "absent"
def test_nothing_subscribed_is_absent_which_is_a_different_note():
rows = dict((r[0], r[1]) for r in dark_surfaces(manifest(events=())))
assert rows["event subscriptions"] == "absent"
def test_the_declarations_are_counted_only_when_there_is_no_transport():
count, names = undeliverable_declarations(
manifest(events=("app_mention", "team_join")), "none")
assert count == 2
assert names == ["app_mention", "team_join"]
assert undeliverable_declarations(manifest(socket=True), "socket") == (0, [])
def test_user_events_count_too_and_duplicates_do_not():
m = manifest(events=("app_mention",))
m["settings"]["event_subscriptions"]["user_events"] = ["app_mention", "file_shared"]
assert undeliverable_declarations(m, "none")[0] == 2
def test_a_series_that_ends_is_a_cliff_with_a_date_on_it():
stamps = [1000.0 + i * DAY for i in range(30)]
state, detail, facts = posting_cliff(stamps, stamps[-1] + 9 * DAY)
assert state == "cliff"
assert facts["quiet_days"] == 9.0
assert "series ends at" in detail
def test_an_app_still_posting_near_the_end_is_not_a_cliff():
stamps = [1000.0 + i * DAY for i in range(30)]
assert posting_cliff(stamps, stamps[-1] + 600.0)[0] == "steady"
def test_two_messages_are_too_few_to_declare_anything():
assert posting_cliff([1000.0, 2000.0], 900000.0)[0] == "sparse"
def test_an_app_that_never_posted_is_not_a_cliff():
state, detail, _facts = posting_cliff([], 5000.0)
assert state == "never-posted"
assert "no series" in detail
def test_a_chatty_app_that_paused_for_an_hour_is_still_steady():
stamps = [1000.0 + i * 60.0 for i in range(40)]
assert posting_cliff(stamps, stamps[-1] + 1800.0)[0] == "steady"
def test_link_disabled_is_the_reason_that_confirms_this_note():
grade, detail = disconnect_meaning("link_disabled")
assert grade == "confirmed"
assert "switched off" in detail
def test_the_other_documented_reasons_belong_to_other_notes():
for reason in ("refresh_requested", "too_many_websockets", "warning"):
assert disconnect_meaning(reason)[0] == "unrelated"
def test_no_reason_supplied_is_not_a_finding():
assert disconnect_meaning("")[0] == "not-supplied"
assert disconnect_meaning(None)[0] == "not-supplied"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
DAY, darkSurfaces, disconnectMeaning, postingCliff, transportState,
undeliverableDeclarations,
} from './slack-event-transport.mjs';
function manifest({
socket = false, url = '', events = ['app_mention'], inter = {}, commands = [],
} = {}) {
return {
settings: {
socket_mode_enabled: socket,
event_subscriptions: { request_url: url, bot_events: [...events] },
interactivity: inter,
},
features: { slash_commands: [...commands] },
};
}
test('neither transport is its own state and not a boolean', () => {
const [state, detail, facts] = transportState(manifest());
assert.equal(state, 'none');
assert.deepEqual(facts, { socket_mode: false, request_url: '' });
assert.match(detail, /no destination/);
});
test('socket only and url only are both healthy', () => {
assert.equal(transportState(manifest({ socket: true }))[0], 'socket');
assert.equal(transportState(manifest({ url: 'https://ops.example.com/slack' }))[0],
'http');
});
test('both at once is the other note rather than this one', () => {
const [state, detail] = transportState(
manifest({ socket: true, url: 'https://ops.example.com/slack' }));
assert.equal(state, 'both');
assert.match(detail, /not none/);
});
test('a whitespace url is not a transport', () => {
assert.equal(transportState(manifest({ url: ' ' }))[0], 'none');
});
test('no manifest is unknown rather than a finding', () => {
assert.equal(transportState(null)[0], 'unknown');
});
test('the events surface goes dark because it has no url of its own', () => {
const rows = Object.fromEntries(darkSurfaces(manifest()).map((r) => [r[0], r[1]]));
assert.equal(rows['event subscriptions'], 'dark');
});
test('a slash command keeps answering and that is the misdirection', () => {
const rows = darkSurfaces(manifest({
commands: [{ command: '/deploy', url: 'https://ops.example.com/cmd' }],
}));
assert.deepEqual(rows.find((r) => r[0] === '/deploy'),
['/deploy', 'routed', 'carries its own URL, so it still answers']);
});
test('a command with no url of its own goes dark too', () => {
const rows = Object.fromEntries(darkSurfaces(manifest({
commands: [{ command: '/rollback', url: '' }],
})).map((r) => [r[0], r[1]]));
assert.equal(rows['/rollback'], 'dark');
});
test('the socket routes every surface at once', () => {
const rows = darkSurfaces(manifest({
socket: true, inter: { is_enabled: true }, commands: [{ command: '/deploy', url: '' }],
}));
assert.deepEqual([...new Set(rows.map((r) => r[1]))], ['routed']);
});
test('interactivity enabled with an empty url is dark', () => {
const rows = Object.fromEntries(darkSurfaces(manifest({ inter: { is_enabled: true } }))
.map((r) => [r[0], r[1]]));
assert.equal(rows.interactivity, 'dark');
});
test('interactivity switched off is absent rather than dark', () => {
const rows = Object.fromEntries(darkSurfaces(manifest({ inter: { is_enabled: false } }))
.map((r) => [r[0], r[1]]));
assert.equal(rows.interactivity, 'absent');
});
test('nothing subscribed is absent which is a different note', () => {
const rows = Object.fromEntries(darkSurfaces(manifest({ events: [] }))
.map((r) => [r[0], r[1]]));
assert.equal(rows['event subscriptions'], 'absent');
});
test('the declarations are counted only when there is no transport', () => {
const [count, names] = undeliverableDeclarations(
manifest({ events: ['app_mention', 'team_join'] }), 'none');
assert.equal(count, 2);
assert.deepEqual(names, ['app_mention', 'team_join']);
assert.deepEqual(undeliverableDeclarations(manifest({ socket: true }), 'socket'),
[0, []]);
});
test('user events count too and duplicates do not', () => {
const m = manifest({ events: ['app_mention'] });
m.settings.event_subscriptions.user_events = ['app_mention', 'file_shared'];
assert.equal(undeliverableDeclarations(m, 'none')[0], 2);
});
test('a series that ends is a cliff with a date on it', () => {
const stamps = Array.from({ length: 30 }, (_v, i) => 1000 + i * DAY);
const [state, detail, facts] = postingCliff(stamps, stamps[29] + 9 * DAY);
assert.equal(state, 'cliff');
assert.equal(facts.quiet_days, 9);
assert.match(detail, /series ends at/);
});
test('an app still posting near the end is not a cliff', () => {
const stamps = Array.from({ length: 30 }, (_v, i) => 1000 + i * DAY);
assert.equal(postingCliff(stamps, stamps[29] + 600)[0], 'steady');
});
test('two messages are too few to declare anything', () => {
assert.equal(postingCliff([1000, 2000], 900000)[0], 'sparse');
});
test('an app that never posted is not a cliff', () => {
const [state, detail] = postingCliff([], 5000);
assert.equal(state, 'never-posted');
assert.match(detail, /no series/);
});
test('a chatty app that paused for an hour is still steady', () => {
const stamps = Array.from({ length: 40 }, (_v, i) => 1000 + i * 60);
assert.equal(postingCliff(stamps, stamps[39] + 1800)[0], 'steady');
});
test('link_disabled is the reason that confirms this note', () => {
const [grade, detail] = disconnectMeaning('link_disabled');
assert.equal(grade, 'confirmed');
assert.match(detail, /switched off/);
});
test('the other documented reasons belong to other notes', () => {
for (const reason of ['refresh_requested', 'too_many_websockets', 'warning']) {
assert.equal(disconnectMeaning(reason)[0], 'unrelated');
}
});
test('no reason supplied is not a finding', () => {
assert.equal(disconnectMeaning('')[0], 'not-supplied');
assert.equal(disconnectMeaning(null)[0], 'not-supplied');
});
FAQ
If Socket Mode is off, doesn't Slack just use the Request URL again?
Only if there is one. Socket Mode and the Request URL are two independent settings, not two positions of one switch, and turning one off does not turn the other on. An app that was built on Socket Mode from the beginning has never had a Request URL configured, so switching the socket off leaves the events subscription with no destination at all. Slack does not queue those events for later and does not report them anywhere: it has nowhere to put them, so it drops them.
Our slash commands still work, so delivery is clearly fine. Isn't it?
Slash commands each store their own URL in the app configuration, which is why they survive. So does interactivity, if its URL was ever filled in. The events subscription is the one surface with no URL of its own, and it is the one that goes dark. That is exactly why this failure lasts so long: the surfaces people poke at to check whether the app is up are the surfaces that kept a route, and the surface that carries mentions, joins and reactions is the one that did not.
There is a Request URL configured, it just never verified. Is that this note?
No, and the distinction is worth keeping because the repairs are different. A URL that exists and failed the challenge is a handshake problem: your endpoint returned the wrong thing, or middleware stood in front of it, and the fix is in your code. A URL that exists and points at a tunnel that closed is a hostname problem, and the fix is in the configuration. This note is the case where the string is simply not there, and no amount of debugging your endpoint will help because Slack is not calling it.
Why does reading the manifest need a different token from the one the app runs on?
Because app configuration is a different surface from workspace data. A bot token reads channels, messages and users; it cannot read your app's own settings. The manifest is read with an app configuration token, which is minted on the app management page, lives twelve hours, and rotates through tooling.tokens.rotate rather than the OAuth endpoints. If that token is dead, this script reports the manifest as unavailable rather than reporting a clean transport count, because a check that could not run is not a check that passed.
Can this happen without anyone touching the switch?
Yes, in two ways. Copying a manifest between apps drops whatever fields the source did not have, so an app created from a manifest that never carried a Request URL starts life with zero transports the moment Socket Mode is not also copied. And an app created through the API from a partial manifest has the same shape. Both cases look like a brand new app that has never worked rather than an app that stopped, which is why the script reports never-posted as a distinct state from cliff.
Related field notes
- the mirror image: two transports rather than none
- a transport that works with nothing subscribed to it
- the credential that reads the manifest, and its twelve hours
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using Socket Mode - Slack Docs
- Events API - Slack Docs
- apps.manifest.export method reference - Slack Docs
- App manifests - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.