Diagnostic Slack
Socket Mode and a Request URL both switched on
Two people are debugging the same incident from opposite ends. One says the bot replied twice to a single mention. The other, running the app on her laptop, says her local console is full of production events that have nothing to do with what she is working on, and that staging has been silent since Tuesday.
Both are describing the same thing. There is one Slack app here, not three, and one app has one configuration: one Socket Mode switch, one Request URL, one set of subscriptions, shared by every environment that happens to be running.
Slack has no notion of environments. An app is a single configuration with a single delivery path, and turning Socket Mode on moves delivery to the WebSocket. The Request URL you configured before that is still stored in the manifest — the screen stops showing it, nothing clears it, and it becomes live again the instant somebody switches Socket Mode off.
The read is two fields held together. apps.manifest.export?app_id=A... gives you settings.socket_mode_enabled alongside settings.event_subscriptions.request_url, settings.interactivity.request_url and the url on every slash command. Socket Mode on plus a stored URL is the finding, and the stored URL is usually a tunnel that stopped existing months ago.
The damage is not really the stale string, though. It is that a developer running the app locally with the app-level token opens a connection for the app, and the app is the production app, so events belonging to the live workspace arrive on a laptop. When duplicates appear, the spacing tells you which kind you have: copies a second or two apart are two live handlers, and copies about a minute or five minutes apart are Slack retrying, which is a different note with a different fix.
The problem in plain words
The route in is always the same and always reasonable. The app is built for production with an HTTPS Request URL. Then somebody needs to develop against it, a public endpoint on a laptop is inconvenient, and Socket Mode is the documented answer — it is genuinely the right tool for developing behind a firewall. The switch gets flipped on the app that already exists, because creating a second app means a second set of tokens, a second manifest, a second install, and that is twenty minutes nobody has on a Tuesday.
From that moment the workspace's events go wherever the socket is. Production stops receiving anything and does not know it: no error, no failed delivery, no auto-disable, just a Request URL that nothing calls any more. Meanwhile the laptop receives real user interactions, and if the local build happens to post messages, it posts them into the real workspace with the real bot's name on them.
Then the switch gets flipped back before a release, or somebody else flips it, and everything inverts. The stored Request URL comes back to life, still pointing at whatever it pointed at when it was last edited, which for a lot of apps is a tunnel hostname from a laptop that has since been reimaged. Slack does not re-verify it and does not need to: it verified once, long ago, and that tick has never been re-examined since. That failure mode is its own note; what this one owns is the configuration that makes flipping between them possible at all.
The duplicate messages are what usually starts the investigation, and they send people to the wrong place. Slack does retry event deliveries, on a documented ladder, and there is a well-worn note about handling those idempotently. But two copies of a message arriving a second apart are not a retry — nothing in Slack's ladder is that fast — they are two processes that both received the same event and both acted on it. The gap between the copies is the discriminator, and it costs one conversations.history call to read.
Why it happens
One app is one configuration, and that is the whole root cause. There is no environment dimension anywhere in the app model: not in the manifest, not in the tokens, not in the subscriptions. Every mitigation people invent — feature flags, environment checks in the handler, only running the dev process sometimes — is working around the absence of something Slack does not have.
Socket Mode hides the Request URL rather than clearing it. The field disappears from the configuration screen, which reads as removed, and the manifest still carries the string. That gap between what the screen shows and what the configuration holds is exactly why this has to be read from the manifest and cannot be checked by looking at the app page.
A socket is opened for the app, not for an environment. The app-level token belongs to the app, so any process holding it can connect and start consuming the workspace's events. A laptop is not a subscriber to a subset; it is a peer of production, and Slack will happily deliver to it.
The spacing between duplicates is the discriminator. Two live handlers produce copies seconds apart with different message ids. Slack's own retries arrive on a ladder measured in minutes. Reading the gap decides which note you are in, and the script names the retry case explicitly so it can be handed off rather than half-fixed here.
A stored URL under Socket Mode is dormant rather than broken today. It delivers nothing right now, which is why nothing alerts on it, and it is one switch away from being the only delivery path again. The script reports it as a finding for that reason and says plainly that it is not currently receiving anything.
The repair is two apps, and that is the supported pattern. A development app with Socket Mode on and no Request URL, a production app with an HTTPS Request URL and Socket Mode off, two manifests, two app ids, two token sets. It is more setup than a switch and it is the only configuration where flipping one does not change the other.
The fix, as a flow
This chain stays black until the fourth arrow, which is the point: every step in it is a reasonable decision made by a competent person, and the configuration is not wrong at any moment along the way. It only becomes wrong when two processes are running at once, which is a Tuesday rather than a deploy. The fix branch sorts delivery paths rather than symptoms, and the two middle rows exist so this note can hand a reader to the retry note instead of half-solving somebody else's problem.
How to fix it
Export the manifest, because the app page will not show you this
The Request URL fields are hidden while Socket Mode is on, so the screen cannot answer the question. apps.manifest.export with an app configuration token can, and so can a manifest JSON you downloaded yourself and pass with --manifest. A bot token cannot read app configuration at all.
Walk every surface that stores a URL, not just the events one
configured_urls returns the event subscriptions URL, the interactivity URL, the menu options URL and the URL on each slash command. Slash commands are where the forgotten tunnel hostnames live, because nobody thinks of a slash command as part of the event configuration.
Hold the socket switch and the stored URLs in one verdict
delivery_state returns both, socket-only, http-only, neither or unreadable. both is the finding. neither is worth its own name: an app with no socket and no URL is inert and nothing has been delivered to it since it was created.
Say which app id each environment is actually using
environment_shape takes the app ids your environments run against, which you pass with --apps dev=A01...,prod=A02.... shared-app is the root cause in one word: the environments are not separate systems, they are the same registration seen from different machines.
Measure the gap between duplicates before blaming retries
gap_shape takes the timestamps of identical app-authored messages and returns simultaneous, retry-shaped, unexplained or single. Anything at a second or two is two live handlers; a minute or five minutes is Slack's retry ladder and belongs to the retry note.
Split the app in two rather than coordinating the switch
The printed repair is a second Slack app for development, with Socket Mode on and no Request URL, and production left on HTTPS with Socket Mode off. Creating apps and editing manifests are writes; this script prints the plan and performs none of it.
How to check it worked
After the split, the production manifest should have a URL and no socket, and the development manifest a socket and no URL. Run it against both.
python3 slack_delivery_paths.py --manifest prod-app.json --apps dev=A01DEV9QT,prod=A01DEV9QT \
--channel C01ENG9QT
# manifest socket mode on, 3 stored Request URL(s)
# url event subscriptions https://a1b2c3.ngrok.io/slack/events
# url interactivity https://a1b2c3.ngrok.io/slack/interact
# url slash /deploy https://hooks.acme.example/slack/deploy
# paths both Socket Mode is on and 3 Request URL(s) are still stored.
# They deliver nothing today and become live the moment the
# socket is switched off
# apps shared-app dev and prod run against the same app id, so they share one
# configuration and one delivery path
# duplicate simultaneous two copies 1.4s apart: Slack's retry ladder is never this
# fast, so two handlers both acted on one event
# verdict collision one configuration, both delivery paths, every environment
# repair: create a second Slack app for development, Socket Mode on, no Request URL
# repair: leave production on HTTPS with Socket Mode off, and never flip eitherThe full code
One manifest read, one optional history read, and five pure functions. configured_urls exists because the slash commands are the surfaces everyone forgets. delivery_state is the two-field cross that is the whole finding. environment_shape names the root cause rather than the symptom. gap_shape is the measurement that keeps this note out of the retry note's territory, and finding is the small matrix that combines what is configured with what was observed.
"""Find the app that has both delivery mechanisms configured at once.
Read only. apps.manifest.export and conversations.history are reads; nothing
here edits a manifest, creates an app, opens a socket or sends anything to a
Request URL. Every repair is printed.
Slack has no environments. One app is one configuration: one Socket Mode
switch, one Request URL, one set of subscriptions, shared by every process that
holds the app's credentials. Turning Socket Mode on moves delivery to the
socket and leaves the Request URL stored, hidden on the configuration screen
and still present in the manifest.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_delivery_paths")
API = "https://slack.com/api/"
# Slack's event retry ladder, roughly: a minute, five minutes, an hour. Bands
# rather than points, because the delay is approximate and the only thing that
# matters is telling a retry apart from a second live handler.
RETRY_BANDS = ((25.0, 95.0), (240.0, 360.0), (3300.0, 3900.0))
# Above this, two copies are too far apart to be two handlers reacting to one
# delivery. Below it, no retry ladder is ever that fast.
TOGETHER = 2.0
def configured_urls(manifest):
"""Every Request URL stored in this app's configuration. Pure.
Returns [(surface, url), ...]. The events URL is the one everybody checks
and the slash commands are the ones that keep a hostname from somebody's
laptop for years, because nobody thinks of a slash command as part of the
event configuration.
"""
m = manifest if isinstance(manifest, dict) else {}
settings = m.get("settings") if isinstance(m.get("settings"), dict) else {}
events = settings.get("event_subscriptions")
inter = settings.get("interactivity")
out = []
if isinstance(events, dict) and events.get("request_url"):
out.append(("event subscriptions", str(events["request_url"])))
if isinstance(inter, dict):
if inter.get("request_url"):
out.append(("interactivity", str(inter["request_url"])))
if inter.get("message_menu_options_url"):
out.append(("menu options", str(inter["message_menu_options_url"])))
features = m.get("features") if isinstance(m.get("features"), dict) else {}
for cmd in features.get("slash_commands") or []:
if isinstance(cmd, dict) and cmd.get("url"):
out.append(("slash %s" % (cmd.get("command") or "?"), str(cmd["url"])))
return out
def delivery_state(manifest):
"""How many ways can events reach this app? Pure.
Returns (state, detail, urls). both is the finding: Socket Mode carries the
traffic today and the stored URLs become the delivery path again the moment
somebody switches it off, without anything being edited or re-verified.
"""
if not isinstance(manifest, dict) or not manifest:
return ("unreadable", "no manifest was supplied, and a bot token cannot read "
"app configuration", [])
settings = manifest.get("settings") if isinstance(manifest.get("settings"),
dict) else {}
socket = bool(settings.get("socket_mode_enabled"))
urls = configured_urls(manifest)
if socket and urls:
return ("both", "Socket Mode is on and %d Request URL(s) are still stored. "
"They deliver nothing today and become live the moment the "
"socket is switched off" % len(urls), urls)
if socket:
return ("socket-only", "Socket Mode is on and no Request URL is stored, which "
"is one delivery path and no ambiguity", urls)
if urls:
return ("http-only", "delivery is over HTTPS to %d configured URL(s) and "
"Socket Mode is off" % len(urls), urls)
return ("neither", "no Socket Mode and no Request URL. Nothing has ever been "
"delivered to this app", urls)
def environment_shape(app_ids):
"""Do your environments share one app registration? Pure.
app_ids maps an environment name to the app id it runs against. shared-app
is the root cause of this note stated in one word: the environments are not
separate systems, they are one registration seen from different machines.
"""
rows = {k: str(v or "").strip() for k, v in (app_ids or {}).items()}
rows = {k: v for k, v in rows.items() if v}
if not rows:
return ("unknown", "no app ids were supplied, so the environments cannot be "
"compared")
if len(rows) < 2:
return ("single-environment", "only one environment was named, so there is "
"nothing to share")
distinct = sorted(set(rows.values()))
if len(distinct) == 1:
return ("shared-app", "%s run against the same app id, so they share one "
"configuration and one delivery path"
% " and ".join(sorted(rows)))
if len(distinct) == len(rows):
return ("separate-apps", "%d environment(s), %d app id(s): each one has its own "
"configuration" % (len(rows), len(distinct)))
return ("partly-shared", "%d environment(s) across %d app id(s), so at least two of "
"them share a configuration" % (len(rows), len(distinct)))
def gap_shape(timestamps):
"""Two copies of one message: what does the spacing say? Pure.
Returns (shape, seconds, detail). This is the boundary against the retry
note. Slack's retry ladder is measured in minutes, so anything at a second
or two is two processes that both received the same event, and that is a
delivery-path problem rather than an idempotency one.
"""
values = []
for t in timestamps or []:
try:
values.append(float(t))
except (TypeError, ValueError):
continue
values.sort()
if len(values) < 2:
return ("single", 0.0, "one copy, so there is nothing to compare")
gap = round(min(b - a for a, b in zip(values, values[1:])), 1)
if gap <= TOGETHER:
return ("simultaneous", gap, "two copies %.1fs apart. Slack's retry ladder is "
"never this fast, so two handlers both acted on "
"one event" % gap)
for low, high in RETRY_BANDS:
if low <= gap <= high:
return ("retry-shaped", gap, "two copies %.1fs apart, which is Slack "
"retrying a delivery it thinks failed. That "
"is an idempotency problem, not this one"
% gap)
return ("unexplained", gap, "two copies %.1fs apart, which matches neither two live "
"handlers nor the retry ladder. Look at your own "
"scheduling" % gap)
def finding(state, env, shape):
"""Cross what is configured with what was observed. Pure.
Returns (verdict, detail). collision is the strongest form and the only one
where all three inputs agree; two-listeners is the same conclusion reached
from evidence when the manifest was not available.
"""
if state == "unreadable":
return ("unreadable", "without the manifest the configuration half of this "
"check is missing, and only the duplicates can speak")
if state == "both" and env == "shared-app":
return ("collision", "one configuration, both delivery paths, and every "
"environment running against it. This is the whole note "
"in one app")
if state == "both":
return ("stale-url", "Socket Mode carries the traffic and the stored Request "
"URL is dormant. It is one switch away from being the "
"only delivery path again")
if state == "neither":
return ("inert", "no delivery path is configured at all, so nothing reaches "
"this app by any route")
if shape == "simultaneous":
return ("two-listeners", "the manifest looks clean and two processes are still "
"acting on the same event. Two instances are sharing "
"one app's credentials")
if shape == "retry-shaped":
return ("retries", "the duplicates are on Slack's retry ladder, which is a "
"different note and a different repair")
if env == "shared-app":
return ("shared-config", "one delivery path today, and every environment on one "
"app id. Nothing is duplicated yet and one switch "
"changes that")
return ("clear", "one delivery path, and nothing suggesting a second listener")
def load_manifest(args):
"""From a file, or from apps.manifest.export with a configuration token."""
if args.manifest:
raw = json.loads(open(args.manifest, encoding="utf-8").read())
return raw.get("manifest", raw)
token = os.environ.get(args.config_token_env)
if not token or not args.app_id:
log.warning("manifest supply --manifest, or set %s and pass --app-id; a bot "
"token cannot read app configuration", args.config_token_env)
return None
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
body = s.get(API + "apps.manifest.export", timeout=30,
params={"app_id": args.app_id}).json()
if body.get("ok") is not True:
log.error("manifest unavailable %s", body.get("error"))
return None
return body.get("manifest") or {}
def duplicate_gaps(session, channel, limit=200):
"""Timestamps of identical app-authored messages in one channel. A read."""
body = session.get(API + "conversations.history", timeout=30,
params={"channel": channel, "limit": str(limit)}).json()
if body.get("ok") is not True:
log.warning("history unavailable %s", body.get("error"))
return []
groups = {}
for m in body.get("messages") or []:
if not m.get("bot_id") and not m.get("app_id"):
continue
key = (m.get("bot_id") or m.get("app_id"), (m.get("text") or "").strip())
if not key[1]:
continue
groups.setdefault(key, []).append(m.get("ts"))
return [ts for ts in groups.values() if len(ts) > 1]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--manifest", default="", help="path to an exported manifest")
ap.add_argument("--config-token-env", default="SLACK_CONFIG_TOKEN",
help="environment variable holding an app configuration token")
ap.add_argument("--app-id", default="", help="app id, for the manifest read")
ap.add_argument("--apps", default="",
help="environment=app_id pairs, comma separated")
ap.add_argument("--channel", default="",
help="a channel to read for duplicate app-authored messages")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding a bot token, for the history read")
args = ap.parse_args()
manifest = load_manifest(args)
state, detail, urls = delivery_state(manifest)
if manifest is not None:
settings = (manifest or {}).get("settings") or {}
log.info("manifest socket mode %s, %d stored Request URL(s)",
"on" if settings.get("socket_mode_enabled") else "off", len(urls))
for surface, url in urls:
log.info("url %-21s %s", surface, url)
(log.warning if state in ("both", "neither") else log.info)(
"paths %-14s %s", state, detail)
pairs = {}
for chunk in args.apps.split(","):
if "=" in chunk:
name, value = chunk.split("=", 1)
pairs[name.strip()] = value.strip()
env, env_detail = environment_shape(pairs)
(log.warning if env in ("shared-app", "partly-shared") else log.info)(
"apps %-14s %s", env, env_detail)
shape = "single"
if args.channel:
token = os.environ.get(args.token_env)
if not token:
log.warning("history set %s to read duplicates", args.token_env)
else:
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
for group in duplicate_gaps(s, args.channel):
shape, _gap, gap_detail = gap_shape(group)
(log.warning if shape == "simultaneous" else log.info)(
"duplicate %-14s %s", shape, gap_detail)
if shape == "simultaneous":
break
verdict, why = finding(state, env, shape)
if verdict in ("collision", "two-listeners", "stale-url", "inert"):
log.warning("verdict %-14s %s", verdict, why)
log.warning(" repair: create a second Slack app for development, with Socket "
"Mode on and no Request URL of any kind")
log.warning(" repair: leave production on HTTPS with Socket Mode off, so no "
"switch on one app can move where events are delivered")
log.warning(" repair: give each app its own bot token, app-level token and "
"signing secret; Slack has no environments, so two apps is the "
"separation")
return 1
log.info("verdict %-14s %s", verdict, why)
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find the app that has both delivery mechanisms configured at once.
*
* Read only. apps.manifest.export and conversations.history are reads; nothing
* here edits a manifest, creates an app, opens a socket or sends anything to a
* Request URL.
*
* Slack has no environments. One app is one configuration, shared by every
* process holding the app's credentials, and turning Socket Mode on moves
* delivery to the socket while leaving the Request URL stored in the manifest.
*/
import { readFileSync } from 'node:fs';
const API = 'https://slack.com/api/';
/** Slack's retry ladder, roughly: a minute, five minutes, an hour. */
export const RETRY_BANDS = [[25, 95], [240, 360], [3300, 3900]];
/** Below this, no retry ladder is ever that fast: it is two live handlers. */
export const TOGETHER = 2;
/**
* Every Request URL stored in this app's configuration. Pure.
* The slash commands are the surfaces that keep a laptop hostname for years,
* because nobody thinks of a slash command as part of event configuration.
*/
export function configuredUrls(manifest) {
const m = (manifest && typeof manifest === 'object') ? manifest : {};
const settings = (m.settings && typeof m.settings === 'object') ? m.settings : {};
const events = settings.event_subscriptions;
const inter = settings.interactivity;
const out = [];
if (events && typeof events === 'object' && events.request_url) {
out.push(['event subscriptions', String(events.request_url)]);
}
if (inter && typeof inter === 'object') {
if (inter.request_url) out.push(['interactivity', String(inter.request_url)]);
if (inter.message_menu_options_url) {
out.push(['menu options', String(inter.message_menu_options_url)]);
}
}
const features = (m.features && typeof m.features === 'object') ? m.features : {};
for (const cmd of features.slash_commands ?? []) {
if (cmd && typeof cmd === 'object' && cmd.url) {
out.push([`slash ${cmd.command ?? '?'}`, String(cmd.url)]);
}
}
return out;
}
/**
* How many ways can events reach this app? Pure. Returns [state, detail, urls].
* both is the finding: the stored URLs deliver nothing today and become the
* delivery path again the moment the socket is switched off.
*/
export function deliveryState(manifest) {
if (!manifest || typeof manifest !== 'object' || !Object.keys(manifest).length) {
return ['unreadable', 'no manifest was supplied, and a bot token cannot read app '
+ 'configuration', []];
}
const settings = (manifest.settings && typeof manifest.settings === 'object')
? manifest.settings : {};
const socket = Boolean(settings.socket_mode_enabled);
const urls = configuredUrls(manifest);
if (socket && urls.length) {
return ['both', `Socket Mode is on and ${urls.length} Request URL(s) are still `
+ 'stored. They deliver nothing today and become live the moment the socket is '
+ 'switched off', urls];
}
if (socket) {
return ['socket-only', 'Socket Mode is on and no Request URL is stored, which is '
+ 'one delivery path and no ambiguity', urls];
}
if (urls.length) {
return ['http-only', `delivery is over HTTPS to ${urls.length} configured URL(s) `
+ 'and Socket Mode is off', urls];
}
return ['neither', 'no Socket Mode and no Request URL. Nothing has ever been '
+ 'delivered to this app', urls];
}
/**
* Do your environments share one app registration? Pure.
* shared-app is the root cause in one word: the environments are not separate
* systems, they are one registration seen from different machines.
*/
export function environmentShape(appIds) {
const rows = {};
for (const [k, v] of Object.entries(appIds ?? {})) {
const id = String(v ?? '').trim();
if (id) rows[k] = id;
}
const names = Object.keys(rows);
if (!names.length) {
return ['unknown', 'no app ids were supplied, so the environments cannot be '
+ 'compared'];
}
if (names.length < 2) {
return ['single-environment', 'only one environment was named, so there is nothing '
+ 'to share'];
}
const distinct = [...new Set(Object.values(rows))];
if (distinct.length === 1) {
return ['shared-app', `${names.sort().join(' and ')} run against the same app id, `
+ 'so they share one configuration and one delivery path'];
}
if (distinct.length === names.length) {
return ['separate-apps', `${names.length} environment(s), ${distinct.length} app `
+ 'id(s): each one has its own configuration'];
}
return ['partly-shared', `${names.length} environment(s) across ${distinct.length} `
+ 'app id(s), so at least two of them share a configuration'];
}
/**
* Two copies of one message: what does the spacing say? Pure.
* The boundary against the retry note: Slack's ladder is measured in minutes,
* so a gap of a second or two is two processes that both got the same event.
*/
export function gapShape(timestamps) {
const values = (timestamps ?? []).map(Number).filter(Number.isFinite)
.sort((a, b) => a - b);
if (values.length < 2) {
return ['single', 0, 'one copy, so there is nothing to compare'];
}
let smallest = Infinity;
for (let i = 1; i < values.length; i += 1) {
smallest = Math.min(smallest, values[i] - values[i - 1]);
}
const gap = Math.round(smallest * 10) / 10;
if (gap <= TOGETHER) {
return ['simultaneous', gap, `two copies ${gap.toFixed(1)}s apart. Slack's retry `
+ 'ladder is never this fast, so two handlers both acted on one event'];
}
for (const [low, high] of RETRY_BANDS) {
if (gap >= low && gap <= high) {
return ['retry-shaped', gap, `two copies ${gap.toFixed(1)}s apart, which is Slack `
+ 'retrying a delivery it thinks failed. That is an idempotency problem, not '
+ 'this one'];
}
}
return ['unexplained', gap, `two copies ${gap.toFixed(1)}s apart, which matches `
+ 'neither two live handlers nor the retry ladder. Look at your own scheduling'];
}
/**
* Cross what is configured with what was observed. Pure.
* collision is the strongest form; two-listeners is the same conclusion
* reached from evidence alone when the manifest was not available.
*/
export function finding(state, env, shape) {
if (state === 'unreadable') {
return ['unreadable', 'without the manifest the configuration half of this check is '
+ 'missing, and only the duplicates can speak'];
}
if (state === 'both' && env === 'shared-app') {
return ['collision', 'one configuration, both delivery paths, and every environment '
+ 'running against it. This is the whole note in one app'];
}
if (state === 'both') {
return ['stale-url', 'Socket Mode carries the traffic and the stored Request URL is '
+ 'dormant. It is one switch away from being the only delivery path again'];
}
if (state === 'neither') {
return ['inert', 'no delivery path is configured at all, so nothing reaches this '
+ 'app by any route'];
}
if (shape === 'simultaneous') {
return ['two-listeners', 'the manifest looks clean and two processes are still '
+ 'acting on the same event. Two instances are sharing one app credentials'];
}
if (shape === 'retry-shaped') {
return ['retries', 'the duplicates are on Slack retry ladder, which is a different '
+ 'note and a different repair'];
}
if (env === 'shared-app') {
return ['shared-config', 'one delivery path today, and every environment on one app '
+ 'id. Nothing is duplicated yet and one switch changes that'];
}
return ['clear', 'one delivery path, and nothing suggesting a second listener'];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function loadManifest(args) {
const path = arg(args, '--manifest', '');
if (path) {
const raw = JSON.parse(readFileSync(path, 'utf-8'));
return raw.manifest ?? raw;
}
const tokenEnv = arg(args, '--config-token-env', 'SLACK_CONFIG_TOKEN');
const token = process.env[tokenEnv];
const appId = arg(args, '--app-id', '');
if (!token || !appId) {
console.warn(`manifest supply --manifest, or set ${tokenEnv} and pass --app-id; a `
+ 'bot token cannot read app configuration');
return null;
}
const params = new URLSearchParams({ app_id: appId });
const body = await (await fetch(`${API}apps.manifest.export?${params}`,
{ headers: { Authorization: `Bearer ${token}` } })).json();
if (body.ok !== true) {
console.error(`manifest unavailable ${body.error}`);
return null;
}
return body.manifest ?? {};
}
async function duplicateGaps(headers, channel, limit = 200) {
const params = new URLSearchParams({ channel, limit: String(limit) });
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.warn(`history unavailable ${body.error}`);
return [];
}
const groups = new Map();
for (const m of body.messages ?? []) {
if (!m.bot_id && !m.app_id) continue;
const text = (m.text ?? '').trim();
if (!text) continue;
const key = `${m.bot_id ?? m.app_id}::${text}`;
groups.set(key, [...(groups.get(key) ?? []), m.ts]);
}
return [...groups.values()].filter((ts) => ts.length > 1);
}
async function main() {
const args = process.argv.slice(2);
const manifest = await loadManifest(args);
const [state, detail, urls] = deliveryState(manifest);
if (manifest) {
const settings = manifest.settings ?? {};
console.log(`manifest socket mode ${settings.socket_mode_enabled ? 'on' : 'off'}, `
+ `${urls.length} stored Request URL(s)`);
}
for (const [surface, url] of urls) {
console.log(`url ${surface.padEnd(21)} ${url}`);
}
const pathLine = `paths ${state.padEnd(14)} ${detail}`;
if (state === 'both' || state === 'neither') console.warn(pathLine);
else console.log(pathLine);
const pairs = {};
for (const chunk of (arg(args, '--apps', '') ?? '').split(',')) {
const at = chunk.indexOf('=');
if (at !== -1) pairs[chunk.slice(0, at).trim()] = chunk.slice(at + 1).trim();
}
const [env, envDetail] = environmentShape(pairs);
const envLine = `apps ${env.padEnd(14)} ${envDetail}`;
if (env === 'shared-app' || env === 'partly-shared') console.warn(envLine);
else console.log(envLine);
let shape = 'single';
const channel = arg(args, '--channel', '');
if (channel) {
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.warn(`history set ${tokenEnv} to read duplicates`);
} else {
const headers = { Authorization: `Bearer ${token}` };
for (const group of await duplicateGaps(headers, channel)) {
const [got, , gapDetail] = gapShape(group);
shape = got;
const dupLine = `duplicate ${got.padEnd(14)} ${gapDetail}`;
if (got === 'simultaneous') { console.warn(dupLine); break; }
console.log(dupLine);
}
}
}
const [verdict, why] = finding(state, env, shape);
if (['collision', 'two-listeners', 'stale-url', 'inert'].includes(verdict)) {
console.warn(`verdict ${verdict.padEnd(14)} ${why}`);
console.warn(' repair: create a second Slack app for development, with Socket Mode '
+ 'on and no Request URL of any kind');
console.warn(' repair: leave production on HTTPS with Socket Mode off, so no '
+ 'switch on one app can move where events are delivered');
process.exitCode = 1;
return;
}
console.log(`verdict ${verdict.padEnd(14)} ${why}`);
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing manifest.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The tests are mostly about the two boundaries this note has with its neighbours. gap_shape is asserted at a second and a half and again at sixty seconds, because that difference is the entire line between this note and the retry one, and getting it wrong sends somebody to write idempotency code for a problem that is really two processes. The other set is about the manifest shape: a slash command URL has to count as a stored delivery path, since it is the surface that keeps a dead tunnel hostname longest, and an empty manifest has to come back unreadable rather than neither.
from slack_delivery_paths import (configured_urls, delivery_state,
environment_shape, finding, gap_shape)
SOCKET_AND_URL = {
"settings": {
"socket_mode_enabled": True,
"event_subscriptions": {"request_url": "https://a1.ngrok.io/slack/events"},
},
}
def test_every_surface_that_stores_a_url_is_walked():
got = configured_urls({
"settings": {
"event_subscriptions": {"request_url": "https://acme.dev/e"},
"interactivity": {"request_url": "https://acme.dev/i",
"message_menu_options_url": "https://acme.dev/o"},
},
"features": {"slash_commands": [{"command": "/deploy",
"url": "https://acme.dev/d"}]},
})
assert [name for name, _url in got] == ["event subscriptions", "interactivity",
"menu options", "slash /deploy"]
def test_a_slash_command_url_counts_as_a_stored_delivery_path():
m = {"settings": {"socket_mode_enabled": True},
"features": {"slash_commands": [{"command": "/x", "url": "https://acme.dev/x"}]}}
assert delivery_state(m)[0] == "both"
def test_an_empty_manifest_is_unreadable_rather_than_empty_of_paths():
assert delivery_state({})[0] == "unreadable"
assert delivery_state(None)[0] == "unreadable"
def test_socket_mode_with_a_stored_url_is_the_finding():
state, detail, urls = delivery_state(SOCKET_AND_URL)
assert state == "both"
assert len(urls) == 1
assert "switched off" in detail
def test_socket_mode_alone_is_one_path_and_no_ambiguity():
assert delivery_state({"settings": {"socket_mode_enabled": True}})[0] == \
"socket-only"
def test_an_https_url_without_socket_mode_is_the_ordinary_shape():
m = {"settings": {"event_subscriptions": {"request_url": "https://acme.dev/e"}}}
assert delivery_state(m)[0] == "http-only"
def test_no_socket_and_no_url_means_nothing_is_delivered_at_all():
assert delivery_state({"settings": {}})[0] == "neither"
def test_two_environments_on_one_app_id_is_the_root_cause():
verdict, detail = environment_shape({"dev": "A01", "prod": "A01"})
assert verdict == "shared-app"
assert "one configuration" in detail
def test_two_environments_with_their_own_apps_are_separate():
assert environment_shape({"dev": "A01", "prod": "A02"})[0] == "separate-apps"
def test_three_environments_sharing_two_apps_are_partly_shared():
assert environment_shape({"dev": "A01", "stg": "A01",
"prod": "A02"})[0] == "partly-shared"
def test_one_environment_or_none_decides_nothing():
assert environment_shape({"prod": "A01"})[0] == "single-environment"
assert environment_shape({})[0] == "unknown"
assert environment_shape({"prod": ""})[0] == "unknown"
def test_copies_a_second_apart_are_two_handlers_not_a_retry():
shape, gap, detail = gap_shape(["1000.000100", "1001.400100"])
assert shape == "simultaneous"
assert gap == 1.4
assert "two handlers" in detail
def test_copies_a_minute_apart_belong_to_the_retry_note():
shape, _gap, detail = gap_shape(["1000.000100", "1060.000100"])
assert shape == "retry-shaped"
assert "idempotency" in detail
def test_the_five_minute_rung_is_also_a_retry():
assert gap_shape(["1000.0", "1300.0"])[0] == "retry-shaped"
def test_a_gap_matching_nothing_is_left_unexplained():
assert gap_shape(["1000.0", "1010.0"])[0] == "unexplained"
def test_one_copy_is_not_a_duplicate():
assert gap_shape(["1000.0"])[0] == "single"
assert gap_shape([])[0] == "single"
assert gap_shape(["not-a-ts", "1000.0"])[0] == "single"
def test_the_smallest_gap_in_a_group_is_the_one_that_decides():
assert gap_shape(["1000.0", "1060.0", "1061.0"])[0] == "simultaneous"
def test_both_paths_on_one_shared_app_is_the_whole_note():
verdict, detail = finding("both", "shared-app", "simultaneous")
assert verdict == "collision"
assert "every environment" in detail
def test_both_paths_without_the_environment_evidence_is_still_a_finding():
assert finding("both", "unknown", "single")[0] == "stale-url"
def test_duplicates_alone_still_reach_the_same_conclusion():
assert finding("socket-only", "unknown", "simultaneous")[0] == "two-listeners"
def test_a_retry_shaped_duplicate_is_handed_to_the_other_note():
assert finding("http-only", "separate-apps", "retry-shaped")[0] == "retries"
def test_a_clean_app_says_so():
assert finding("http-only", "separate-apps", "single")[0] == "clear"
assert finding("unreadable", "unknown", "single")[0] == "unreadable"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
configuredUrls, deliveryState, environmentShape, finding, gapShape,
} from './slack-delivery-paths.mjs';
const SOCKET_AND_URL = {
settings: {
socket_mode_enabled: true,
event_subscriptions: { request_url: 'https://a1.ngrok.io/slack/events' },
},
};
test('every surface that stores a url is walked', () => {
const got = configuredUrls({
settings: {
event_subscriptions: { request_url: 'https://acme.dev/e' },
interactivity: {
request_url: 'https://acme.dev/i',
message_menu_options_url: 'https://acme.dev/o',
},
},
features: { slash_commands: [{ command: '/deploy', url: 'https://acme.dev/d' }] },
});
assert.deepEqual(got.map(([name]) => name),
['event subscriptions', 'interactivity', 'menu options', 'slash /deploy']);
});
test('a slash command url counts as a stored delivery path', () => {
const m = {
settings: { socket_mode_enabled: true },
features: { slash_commands: [{ command: '/x', url: 'https://acme.dev/x' }] },
};
assert.equal(deliveryState(m)[0], 'both');
});
test('an empty manifest is unreadable rather than empty of paths', () => {
assert.equal(deliveryState({})[0], 'unreadable');
assert.equal(deliveryState(null)[0], 'unreadable');
});
test('socket mode with a stored url is the finding', () => {
const [state, detail, urls] = deliveryState(SOCKET_AND_URL);
assert.equal(state, 'both');
assert.equal(urls.length, 1);
assert.match(detail, /switched off/);
});
test('socket mode alone is one path and no ambiguity', () => {
assert.equal(deliveryState({ settings: { socket_mode_enabled: true } })[0],
'socket-only');
});
test('an https url without socket mode is the ordinary shape', () => {
const m = { settings: { event_subscriptions: { request_url: 'https://acme.dev/e' } } };
assert.equal(deliveryState(m)[0], 'http-only');
});
test('no socket and no url means nothing is delivered at all', () => {
assert.equal(deliveryState({ settings: {} })[0], 'neither');
});
test('two environments on one app id is the root cause', () => {
const [verdict, detail] = environmentShape({ dev: 'A01', prod: 'A01' });
assert.equal(verdict, 'shared-app');
assert.match(detail, /one configuration/);
});
test('two environments with their own apps are separate', () => {
assert.equal(environmentShape({ dev: 'A01', prod: 'A02' })[0], 'separate-apps');
});
test('three environments sharing two apps are partly shared', () => {
assert.equal(environmentShape({ dev: 'A01', stg: 'A01', prod: 'A02' })[0],
'partly-shared');
});
test('one environment or none decides nothing', () => {
assert.equal(environmentShape({ prod: 'A01' })[0], 'single-environment');
assert.equal(environmentShape({})[0], 'unknown');
assert.equal(environmentShape({ prod: '' })[0], 'unknown');
});
test('copies a second apart are two handlers not a retry', () => {
const [shape, gap, detail] = gapShape(['1000.000100', '1001.400100']);
assert.equal(shape, 'simultaneous');
assert.equal(gap, 1.4);
assert.match(detail, /two handlers/);
});
test('copies a minute apart belong to the retry note', () => {
const [shape, , detail] = gapShape(['1000.000100', '1060.000100']);
assert.equal(shape, 'retry-shaped');
assert.match(detail, /idempotency/);
});
test('the five minute rung is also a retry', () => {
assert.equal(gapShape(['1000.0', '1300.0'])[0], 'retry-shaped');
});
test('a gap matching nothing is left unexplained', () => {
assert.equal(gapShape(['1000.0', '1010.0'])[0], 'unexplained');
});
test('one copy is not a duplicate', () => {
assert.equal(gapShape(['1000.0'])[0], 'single');
assert.equal(gapShape([])[0], 'single');
assert.equal(gapShape(['not-a-ts', '1000.0'])[0], 'single');
});
test('the smallest gap in a group is the one that decides', () => {
assert.equal(gapShape(['1000.0', '1060.0', '1061.0'])[0], 'simultaneous');
});
test('both paths on one shared app is the whole note', () => {
const [verdict, detail] = finding('both', 'shared-app', 'simultaneous');
assert.equal(verdict, 'collision');
assert.match(detail, /every environment/);
});
test('both paths without the environment evidence is still a finding', () => {
assert.equal(finding('both', 'unknown', 'single')[0], 'stale-url');
});
test('duplicates alone still reach the same conclusion', () => {
assert.equal(finding('socket-only', 'unknown', 'simultaneous')[0], 'two-listeners');
});
test('a retry shaped duplicate is handed to the other note', () => {
assert.equal(finding('http-only', 'separate-apps', 'retry-shaped')[0], 'retries');
});
test('a clean app says so', () => {
assert.equal(finding('http-only', 'separate-apps', 'single')[0], 'clear');
assert.equal(finding('unreadable', 'unknown', 'single')[0], 'unreadable');
});
FAQ
If Socket Mode is on, does Slack really still call my Request URL?
No, and that is the trap rather than the relief. While Socket Mode is on, delivery goes down the WebSocket and the stored URL receives nothing, which is why nothing alerts and why the configuration can sit like this for a year. The URL is not gone though: it is still in the manifest, still verified from whenever it was last checked, and it becomes the only delivery path again the moment somebody turns the switch off. Dormant is not the same as removed.
Why not just add an environment check in the handler?
Because the event has already been delivered to the wrong place by then. If a laptop holds the app-level token and opens a socket, Slack sends it real workspace events, and a check inside the handler can only decide to ignore them - production never receives them at all. The problem is at the delivery layer and no amount of code in the consumer moves it.
We see the bot replying twice. Is this the same as the retry problem?
Measure the gap and you will know. Slack retries a delivery it believes failed on a ladder measured in minutes, so two copies about sixty seconds or five minutes apart are retries and want an idempotency key. Two copies a second or two apart cannot be retries - nothing in the ladder is that fast - and mean two processes both received the same event. The script reads the timestamps of identical app-authored messages and names which one you have.
Can I keep one app and just be careful about who runs what?
You can, and it holds until the first time two people are working at once. The failure is not carelessness, it is that the configuration has no room to express two environments: one switch, one URL, one set of subscriptions, and every process holding the credentials is a peer. Two apps costs a second install and a second set of tokens, and it makes the failure structurally impossible rather than a matter of discipline.
Which app should keep the Request URL?
Production, always, with Socket Mode off. The development app gets Socket Mode on and no Request URL at all, so there is nothing stored to become live later. Keeping a URL on the development app is how this comes back: somebody turns Socket Mode off to test something, a tunnel hostname from months ago wakes up, and the whole cycle starts again.
Related field notes
- duplicates from Slack's retry ladder, minutes apart rather than seconds
- the tunnel hostname the stored URL usually turns out to be
- the other cost of the same delivery choice
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using Socket Mode - Slack Docs
- apps.manifest.export method reference - Slack Docs
- Stack Overflow: Socket Mode app also has a Request URL configured
- Events API - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.