Diagnostic Slack
Socket Mode does not load balance: replicas duplicate and drop
The app was scaled from one pod to three on Thursday afternoon, for headroom before a launch. By Friday morning the support channel had two complaints that the bot had answered the same request twice and one that it had not answered at all. Restarting a pod changes which of those happens. Rolling back to one pod makes both stop.
Nothing is wrong with the code. Three pods, three Socket Mode connections, and Slack sends each payload to one of them — with, in its own words, no predictable distribution pattern. That is not a work queue. There is no consumer group, no acknowledgement-based redelivery to a different consumer, and no ordering guarantee anywhere in it.
Socket Mode is not a load balancer and it is not a message queue. When several connections are open for one app, Slack routes each payload to one of them and does not promise which. Add replicas and you do not get work sharing — you get a lottery.
The losses come from the lottery: a payload routed to a pod that is restarting, draining or wedged is gone, because there is no redelivery to a different consumer. The duplicates come from the retry landing somewhere else. When a delivery is not acknowledged in time Slack retries it, and with three connections open the retry can arrive on a different pod from the original. Each pod's dedupe cache is perfectly correct and completely blind: neither saw both copies. A per-process cache cannot deduplicate across processes, so the work is done twice and every individual component behaves exactly as designed.
The measurement is the fraction of triggers that got exactly one answer. Page conversations.history, count the app's replies inside a window after each mention, and sort them into none, one and many. Duplicates and misses in the same channel is the fingerprint no other Socket Mode failure produces: retries alone give duplicates without misses, a leaked connection pool gives misses without duplicates.
The problem in plain words
The reason teams walk into this is that every other part of their stack rewards adding replicas. HTTP behind a load balancer shares work. A queue consumer group shares work. A Kafka partition assignment shares work. Socket Mode looks like a long-lived consumer connection and is nothing of the kind: it is a delivery channel, and having three of them means Slack has three places it might put a payload, not three workers splitting a stream.
The non-determinism is what makes it so hard to pin down. The same message, sent twice, behaves differently. A pod restart changes the outcome. A quiet afternoon looks fine because there is enough headroom that nothing is ever slow enough to be retried. Then a busy morning produces four complaints in an hour and nobody can reproduce any of them. Every hypothesis fits some of the evidence, which is the signature of a routing problem rather than a logic problem.
Idempotency keys are the usual first attempt and they usually fail, for a reason worth understanding. A dedupe cache in process memory is per-pod. Pod A handles the original, pod B handles the retry, and neither one has ever seen the other's copy, so both are certain they are handling something new. Moving the cache to Redis fixes it — and at that point you have built the coordination layer that Socket Mode was supposed to save you from, in front of a transport that still drops payloads when a pod is restarting.
And the losses do not announce themselves. There is no dead-letter queue, no unacknowledged-message metric, no consumer lag. A payload routed to a pod that is thirty seconds into a rolling restart is simply never processed by anything, and the only record it existed is the human message in Slack that nobody replied to. That is why the check has to be built out of the workspace's own history rather than out of the app's telemetry: the app has no telemetry for events it never received.
Why it happens
Duplicates and misses together is the fingerprint, and no other note here produces it. Retries without replicas duplicate but do not lose. A leaked connection pool loses but does not duplicate, because each payload still goes to exactly one registration. A mishandled refresh loses in periodic bunches. Only several live listeners give you both at once, in the same channel, in the same hour. The script checks for the pair rather than for either half.
The determinism fraction is a better number than a duplicate count. “Fourteen duplicates this week” invites somebody to add a dedupe key. “Sixty-two per cent of triggers got exactly one answer” says the delivery path is non-deterministic, which is the actual finding, and it moves in the right direction as soon as the replica count comes down.
Spacing separates this from the retry note, and the script makes that call explicitly. Duplicates a few seconds apart, or one minute or five minutes apart, are Slack's retry schedule; if there are no misses beside them, one listener is mishandling retries and that is a different note. Duplicates less than a second apart with no misses are usually two subscriptions to the same conversation, which is a third note. The verdict names which of the three it found.
The replica count is an input, not an inference. The script cannot see your deployment and does not pretend to. You state how many replicas hold a connection, and the finding is reported against that number, because the repair is expressed in replicas and a diagnosis expressed in anything else is not actionable.
A per-process dedupe cache is not a fix and the script says so in the repair. The cache is correct; it is simply in the wrong place. If the app must run more than one instance, the deduplication has to live somewhere all of them can see, and even then the routing losses remain. That is why the repair is one receiver and a queue you control, not a smarter cache.
Nothing here opens a connection. Counting live connections would require minting one, which consumes part of the app's ten-connection budget and is a write. Everything above is read from the workspace and from a replica count you already know.
The fix, as a flow
The red arrow here is at the routing decision, and the two boxes after it are the outcomes of a coin toss rather than of a fault: every component in this picture is behaving exactly as designed, including the dedupe cache that misses. The fix branch sorts the outcome of one trigger, and it is the only branch in the batch whose rows name other notes, because duplicates and drops look identical apart from whether they appear together.
How to fix it
Count the replies to each mention rather than the duplicates overall
reply_counts walks conversations.history and returns, for every mention of the bot, how many app-authored messages followed it inside the window. That per-trigger count is what makes the rest possible: a total number of duplicates cannot tell you about the mentions that got nothing.
Sort the outcomes into none, once and many
outcome_mix reduces those counts to four numbers and a fraction. The fraction — triggers answered exactly once, over all triggers — is the finding. A healthy app sits at one. Anything materially below it has a delivery path that is not deterministic, whichever direction the errors go.
Look at the gaps between repeated answers
repeat_runs groups the app's own messages by identical text within a window and returns the intervals between them. spacing_kind then labels each run simultaneous, retry or spread. This is where the diagnosis stops being about counts and starts being about mechanism.
Let the combination pick the note
fingerprint takes the outcome mix and the spacings and returns one of five verdicts. Duplicates with misses is this note. Retry-spaced duplicates without misses is the retry note. Sub-second duplicates without misses is a double subscription. Misses alone is the connection cap or a refresh. Nothing at all is clean, and the script will say that too.
Read the verdict against the replica count you gave it
The output states the finding in replicas because that is what you can change. Three replicas each holding a connection is three places a payload might land and two of them are wrong at any moment; the number to get to is one.
One receiver, then scale behind it
The repair the script prints is architectural. Run Socket Mode as a singleton, restarted on failure, and have it do nothing but enqueue payloads into something you control; scale the workers that drain that queue as far as you like. If ingress genuinely has to be multi-node, that is what the HTTP Events API and a load balancer are for. A shared dedupe cache is a patch on the duplicates and does nothing for the drops.
How to check it worked
Scale back to one replica, wait for a busy period, and run it again. The fraction is the number to watch; the duplicate count will follow it.
python3 slack_socket_replica_fingerprint.py --channel C05SUP9QT --replicas 3
# identity U07BOT9QD (helpbot) in Northwind
# triggers 84 mention(s) read, answered inside 45s
# outcome none 9 once 63 many 12 of 84
# exactly-once 0.750
# spacing retry 7 run(s) repeated at Slack retry intervals
# spacing simultaneous 5 run(s) repeated less than a second apart
# verdict several-live-connections duplicates and misses in the same channel
# replicas 3 replica(s) hold a connection, so a payload has 3 places to land
# repair: run Socket Mode as a singleton and enqueue payloads into a queue you
# control; scale the workers, not the receiver
# repair: a dedupe cache in process memory cannot see the copy another pod
# handled; if instances must be plural, the cache has to be shared
# note: nothing here opened a connection; counting them would consume oneThe full code
The number this script exists to produce is exactly-once: the proportion of triggers that got one answer, no more and no less. Counting duplicates alone would hide the drops, and counting drops alone would hide the duplicates, and it is the pair that identifies the failure. spacing_kind is the second half of the argument, because duplicates at Slack's retry intervals with no drops beside them belong to a different note, and fingerprint is where those two observations are turned into a verdict that names which note you are in.
"""Tell several live Socket Mode connections apart from retries and drops.
Read only. Nothing here opens a connection: apps.connections.open mints one and
counts against the ten an app is allowed, so the number of live connections is
supplied by you as a replica count rather than measured. Everything else comes
from conversations.history, which is where the consequences are recorded.
The measurement is the fraction of triggers that got exactly one answer. Slack
routes each payload to one of the open connections without a distribution
guarantee, so more connections means more places a payload can land and more
places a retry can land separately from its original.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_socket_replica_fingerprint")
API = "https://slack.com/api/"
# Slack retries an unacknowledged event on a schedule. Duplicates spaced like
# this, with no drops beside them, are one listener mishandling retries - a
# different problem with a different repair.
RETRY_SPACINGS = (60.0, 300.0)
RETRY_TOLERANCE = 0.25
# Below this, two answers were produced at the same moment rather than in
# sequence, which points at two subscriptions rather than two connections.
SIMULTANEOUS_SECONDS = 1.0
def reply_counts(messages, bot_user_id, window=45.0):
"""How many answers did each mention get? Pure.
Per trigger rather than in total, because a total duplicate count cannot
say anything about the mentions that got nothing at all, and it is the two
together that identify this failure.
Returns [(mention_ts, replies), ...] oldest first.
"""
marker = "<@%s>" % bot_user_id if bot_user_id else None
own, mentions = [], []
for m in messages or []:
ts = str((m or {}).get("ts") or "")
if not ts:
continue
author = (m or {}).get("user") or ""
if (bot_user_id and author == bot_user_id) or (m or {}).get("bot_id"):
own.append(float(ts))
elif marker and marker in str((m or {}).get("text") or ""):
mentions.append(float(ts))
out = []
for ts in sorted(mentions):
out.append((ts, sum(1 for o in own if ts <= o <= ts + float(window))))
return out
def outcome_mix(counts):
"""Sort trigger outcomes into none, once and many. Pure.
Returns (mix, exactly_once). The fraction is the finding: a delivery path
that works answers every trigger once, and a fraction materially below one
is non-deterministic whichever direction the errors go in.
"""
rows = [int(n) for _ts, n in counts or []]
mix = {"none": sum(1 for n in rows if n == 0),
"once": sum(1 for n in rows if n == 1),
"many": sum(1 for n in rows if n > 1),
"total": len(rows)}
fraction = 0.0 if not rows else round(mix["once"] / float(len(rows)), 3)
return (mix, fraction)
def repeat_runs(messages, bot_user_id, gap=600.0):
"""Runs of identical app-authored messages, and the gaps between them. Pure.
Text is compared after trimming and lowercasing, which is crude and is
enough: the duplicates this note is about are the same answer produced
twice by two processes running the same code.
Returns [(text, [delta, ...]), ...] for runs of two or more.
"""
said = {}
for m in messages or []:
ts = str((m or {}).get("ts") or "")
author = (m or {}).get("user") or ""
is_app = (bot_user_id and author == bot_user_id) or (m or {}).get("bot_id")
text = str((m or {}).get("text") or "").strip().lower()
if not ts or not is_app or not text:
continue
said.setdefault(text, []).append(float(ts))
runs = []
for text, stamps in said.items():
stamps.sort()
current = [stamps[0]]
for ts in stamps[1:]:
if ts - current[-1] <= float(gap):
current.append(ts)
else:
if len(current) > 1:
runs.append((text, [b - a for a, b in zip(current, current[1:])]))
current = [ts]
if len(current) > 1:
runs.append((text, [b - a for a, b in zip(current, current[1:])]))
return sorted(runs)
def spacing_kind(deltas):
"""What produced this run of identical answers? Pure.
simultaneous under a second apart: two handlers acted on the same payload
at the same moment, which is usually two subscriptions.
retry at Slack's retry intervals: one listener that did not
acknowledge in time, which is a different note.
spread neither, so the run says nothing on its own.
"""
gaps = [float(d) for d in deltas or []]
if not gaps:
return "single"
if all(g < SIMULTANEOUS_SECONDS for g in gaps):
return "simultaneous"
for g in gaps:
for expected in RETRY_SPACINGS:
if abs(g - expected) <= expected * RETRY_TOLERANCE:
return "retry"
return "spread"
def fingerprint(mix, kinds):
"""Which failure is this? Pure. Returns (verdict, detail).
The combination is the whole diagnosis. Duplicates beside drops is the only
shape several live connections produce; retries duplicate without dropping,
and a leaked connection pool drops without duplicating.
"""
mix = mix or {}
many, none = int(mix.get("many") or 0), int(mix.get("none") or 0)
kinds = list(kinds or [])
if many and none:
return ("several-live-connections",
"duplicates and misses in the same channel. A retry landing on a "
"different instance from its original is handled twice, and a payload "
"routed to an instance that is restarting is handled by nobody")
if many and "retry" in kinds:
return ("retries-one-listener",
"duplicates at Slack's retry intervals with nothing dropped. One "
"listener that did not acknowledge in time, which is a different note")
if many and "simultaneous" in kinds:
return ("double-subscription",
"duplicates under a second apart with nothing dropped. Two "
"subscriptions delivering the same conversation, not two connections")
if many:
return ("duplicates-unclassified",
"repeats that match neither the retry schedule nor a simultaneous "
"pair; read the runs before concluding anything")
if none:
return ("losses-only",
"drops with no duplicates. Each payload still went to exactly one "
"place, so look at the connection cap or a mishandled refresh")
return ("clean", "every trigger in this window got exactly one answer")
def page_history(session, channel, limit, max_pages):
"""Page conversations.history. A read, and one of two calls made."""
out, cursor, pages = [], "", 0
while pages < max_pages:
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.error("history unavailable %s", body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = ((body.get("response_metadata") or {}).get("next_cursor") or "")
pages += 1
if not cursor:
break
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN")
ap.add_argument("--channel", required=True)
ap.add_argument("--replicas", type=int, default=1,
help="how many instances hold a Socket Mode connection")
ap.add_argument("--window", type=float, default=45.0)
ap.add_argument("--repeat-gap", type=float, default=600.0)
ap.add_argument("--limit", type=int, default=200)
ap.add_argument("--max-pages", type=int, default=5)
args = ap.parse_args()
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token with channels:history", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s (%s) in %s", who.get("user_id"), who.get("user"),
who.get("team"))
messages = page_history(s, args.channel, args.limit, args.max_pages)
counts = reply_counts(messages, who.get("user_id"), args.window)
mix, fraction = outcome_mix(counts)
log.info("triggers %d mention(s) read, answered inside %.0fs", mix["total"],
args.window)
if not mix["total"]:
log.info("verdict nothing to measure; no mentions in the window read")
return 0
log.info("outcome none %d once %d many %d of %d", mix["none"],
mix["once"], mix["many"], mix["total"])
log.info("exactly-once %.3f", fraction)
runs = repeat_runs(messages, who.get("user_id"), args.repeat_gap)
kinds = [spacing_kind(deltas) for _text, deltas in runs]
for kind in sorted(set(kinds)):
log.info("spacing %-12s %d run(s) of repeated answers", kind,
kinds.count(kind))
verdict, detail = fingerprint(mix, kinds)
if verdict == "clean":
log.info("verdict clean %s", detail)
return 0
if verdict != "several-live-connections":
log.info("verdict %-12s %s", verdict, detail)
return 0
log.warning("verdict %s %s", verdict, detail)
log.warning("replicas %d replica(s) hold a connection, so a payload has %d "
"places to land and %d of them are wrong at any moment",
args.replicas, args.replicas, max(0, args.replicas - 1))
log.warning(" repair: run Socket Mode as a singleton and enqueue payloads into a "
"queue you control; scale the workers, not the receiver")
log.warning(" repair: a dedupe cache in process memory cannot see the copy "
"another instance handled; if instances must be plural, share it")
log.warning(" repair: for genuinely multi-node ingress use the HTTP Events API "
"behind a load balancer, which does distribute")
log.warning(" note: nothing here opened a connection; counting them would "
"consume one of the ten this app is allowed")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Tell several live Socket Mode connections apart from retries and drops.
*
* Read only. Nothing here opens a connection: apps.connections.open mints one
* and counts against the ten an app is allowed, so the number of live
* connections is supplied as a replica count rather than measured. Everything
* else comes from conversations.history, where the consequences are recorded.
*
* The measurement is the fraction of triggers that got exactly one answer.
*/
const API = 'https://slack.com/api/';
// Slack retries an unacknowledged event on a schedule. Duplicates spaced like
// this, with no drops beside them, are one listener mishandling retries.
export const RETRY_SPACINGS = [60, 300];
export const RETRY_TOLERANCE = 0.25;
export const SIMULTANEOUS_SECONDS = 1;
/**
* How many answers did each mention get? Pure.
* Per trigger rather than in total, because a total duplicate count says
* nothing about the mentions that got nothing at all.
*/
export function replyCounts(messages, botUserId, window = 45) {
const marker = botUserId ? `<@${botUserId}>` : null;
const own = [];
const mentions = [];
for (const m of messages ?? []) {
const ts = String((m ?? {}).ts ?? '');
if (!ts) continue;
const author = (m ?? {}).user ?? '';
if ((botUserId && author === botUserId) || (m ?? {}).bot_id) own.push(Number(ts));
else if (marker && String((m ?? {}).text ?? '').includes(marker)) {
mentions.push(Number(ts));
}
}
return mentions.sort((a, b) => a - b)
.map((ts) => [ts, own.filter((o) => o >= ts && o <= ts + Number(window)).length]);
}
/** Sort trigger outcomes into none, once and many. Pure. [mix, exactlyOnce]. */
export function outcomeMix(counts) {
const rows = (counts ?? []).map(([, n]) => Number(n));
const mix = {
none: rows.filter((n) => n === 0).length,
once: rows.filter((n) => n === 1).length,
many: rows.filter((n) => n > 1).length,
total: rows.length,
};
const fraction = rows.length ? Math.round((mix.once / rows.length) * 1000) / 1000 : 0;
return [mix, fraction];
}
/**
* Runs of identical app-authored messages, and the gaps between them. Pure.
* Returns [[text, [delta, ...]], ...] for runs of two or more.
*/
export function repeatRuns(messages, botUserId, gap = 600) {
const said = new Map();
for (const m of messages ?? []) {
const ts = String((m ?? {}).ts ?? '');
const author = (m ?? {}).user ?? '';
const isApp = (botUserId && author === botUserId) || (m ?? {}).bot_id;
const text = String((m ?? {}).text ?? '').trim().toLowerCase();
if (!ts || !isApp || !text) continue;
if (!said.has(text)) said.set(text, []);
said.get(text).push(Number(ts));
}
const runs = [];
const deltas = (run) => run.slice(1).map((b, i) => b - run[i]);
for (const [text, stamps] of said) {
stamps.sort((a, b) => a - b);
let current = [stamps[0]];
for (const ts of stamps.slice(1)) {
if (ts - current[current.length - 1] <= Number(gap)) current.push(ts);
else {
if (current.length > 1) runs.push([text, deltas(current)]);
current = [ts];
}
}
if (current.length > 1) runs.push([text, deltas(current)]);
}
return runs.sort((a, b) => (a[0] < b[0] ? -1 : 1));
}
/** What produced this run of identical answers? Pure. */
export function spacingKind(deltas) {
const gaps = (deltas ?? []).map(Number);
if (!gaps.length) return 'single';
if (gaps.every((g) => g < SIMULTANEOUS_SECONDS)) return 'simultaneous';
for (const g of gaps) {
for (const expected of RETRY_SPACINGS) {
if (Math.abs(g - expected) <= expected * RETRY_TOLERANCE) return 'retry';
}
}
return 'spread';
}
/** Which failure is this? Pure. Returns [verdict, detail]. */
export function fingerprint(mix, kinds) {
const m = mix ?? {};
const many = Number(m.many) || 0;
const none = Number(m.none) || 0;
const seen = kinds ?? [];
if (many && none) {
return ['several-live-connections',
'duplicates and misses in the same channel. A retry landing on a different '
+ 'instance from its original is handled twice, and a payload routed to an '
+ 'instance that is restarting is handled by nobody'];
}
if (many && seen.includes('retry')) {
return ['retries-one-listener',
'duplicates at Slack retry intervals with nothing dropped. One listener that '
+ 'did not acknowledge in time, which is a different note'];
}
if (many && seen.includes('simultaneous')) {
return ['double-subscription',
'duplicates under a second apart with nothing dropped. Two subscriptions '
+ 'delivering the same conversation, not two connections'];
}
if (many) {
return ['duplicates-unclassified',
'repeats that match neither the retry schedule nor a simultaneous pair; read '
+ 'the runs before concluding anything'];
}
if (none) {
return ['losses-only',
'drops with no duplicates. Each payload still went to exactly one place, so '
+ 'look at the connection cap or a mishandled refresh'];
}
return ['clean', 'every trigger in this window got exactly one answer'];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function pageHistory(headers, channel, limit, maxPages) {
const out = [];
let cursor = '';
for (let page = 0; page < maxPages; page += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.error(`history unavailable ${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = (body.response_metadata ?? {}).next_cursor ?? '';
if (!cursor) break;
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
const channel = arg(args, '--channel', '');
if (!token || !channel) {
console.error(`set ${tokenEnv} to a bot token with channels:history, and pass `
+ '--channel');
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} (${who.user}) in ${who.team}`);
const window = Number(arg(args, '--window', '45'));
const replicas = Number(arg(args, '--replicas', '1'));
const messages = await pageHistory(headers, channel,
Number(arg(args, '--limit', '200')), Number(arg(args, '--max-pages', '5')));
const counts = replyCounts(messages, who.user_id, window);
const [mix, fraction] = outcomeMix(counts);
console.log(`triggers ${mix.total} mention(s) read, answered inside ${window}s`);
if (!mix.total) {
console.log('verdict nothing to measure; no mentions in the window read');
return;
}
console.log(`outcome none ${mix.none} once ${mix.once} many ${mix.many} of `
+ `${mix.total}`);
console.log(`exactly-once ${fraction}`);
const runs = repeatRuns(messages, who.user_id, Number(arg(args, '--repeat-gap',
'600')));
const kinds = runs.map(([, deltas]) => spacingKind(deltas));
for (const kind of [...new Set(kinds)].sort()) {
console.log(`spacing ${kind.padEnd(12)} `
+ `${kinds.filter((k) => k === kind).length} run(s) of repeated answers`);
}
const [verdict, detail] = fingerprint(mix, kinds);
if (verdict !== 'several-live-connections') {
console.log(`verdict ${verdict.padEnd(12)} ${detail}`);
return;
}
console.warn(`verdict ${verdict} ${detail}`);
console.warn(`replicas ${replicas} replica(s) hold a connection, so a payload has `
+ `${replicas} places to land and ${Math.max(0, replicas - 1)} of them are wrong `
+ 'at any moment');
console.warn(' repair: run Socket Mode as a singleton and enqueue payloads into a '
+ 'queue you control; scale the workers, not the receiver');
console.warn(' repair: a dedupe cache in process memory cannot see the copy another '
+ 'instance handled; if instances must be plural, share it');
console.warn(' repair: for genuinely multi-node ingress use the HTTP Events API '
+ 'behind a load balancer, which does distribute');
console.warn(' note: nothing here opened a connection; counting them would '
+ 'consume one of the ten this app is allowed');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Every test here is about a boundary between this note and one of its neighbours, because the evidence is nearly the same in all of them and only the combination differs. Duplicates with drops must come back as several connections; the same duplicates without drops must come back as retries or as a double subscription, depending on their spacing; drops without duplicates must be handed to the connection cap. outcome_mix is tested for the fraction on a clean run, which has to be exactly one, because a check that cannot say this is fine is not usable on a healthy app.
from slack_socket_replica_fingerprint import (
fingerprint, outcome_mix, repeat_runs, reply_counts, spacing_kind,
)
def test_a_mention_answered_once_counts_once():
msgs = [{"ts": "100", "user": "U1", "text": "<@UBOT> go"},
{"ts": "110", "user": "UBOT", "text": "on it"}]
assert reply_counts(msgs, "UBOT", 45) == [(100.0, 1)]
def test_two_answers_inside_the_window_count_as_two():
msgs = [{"ts": "100", "user": "U1", "text": "<@UBOT> go"},
{"ts": "110", "user": "UBOT", "text": "on it"},
{"ts": "111", "bot_id": "B1", "text": "on it"}]
assert reply_counts(msgs, "UBOT", 45) == [(100.0, 2)]
def test_a_mention_with_nothing_after_it_counts_as_zero():
msgs = [{"ts": "100", "user": "U1", "text": "<@UBOT> go"}]
assert reply_counts(msgs, "UBOT", 45) == [(100.0, 0)]
def test_a_clean_run_is_exactly_one():
mix, fraction = outcome_mix([(1, 1), (2, 1), (3, 1)])
assert mix == {"none": 0, "once": 3, "many": 0, "total": 3}
assert fraction == 1.0
def test_the_fraction_falls_for_drops_and_for_duplicates_alike():
_mix, fraction = outcome_mix([(1, 0), (2, 1), (3, 2), (4, 1)])
assert fraction == 0.5
def test_an_empty_window_divides_by_nothing():
assert outcome_mix([]) == ({"none": 0, "once": 0, "many": 0, "total": 0}, 0.0)
def test_identical_answers_close_together_are_one_run():
msgs = [{"ts": "100", "user": "UBOT", "text": "Done"},
{"ts": "100.4", "user": "UBOT", "text": "done"}]
runs = repeat_runs(msgs, "UBOT", 600)
assert len(runs) == 1
assert round(runs[0][1][0], 1) == 0.4
def test_a_single_answer_is_not_a_run():
assert repeat_runs([{"ts": "100", "user": "UBOT", "text": "done"}], "UBOT") == []
def test_the_same_answer_hours_apart_is_two_separate_runs_of_one():
msgs = [{"ts": "100", "user": "UBOT", "text": "done"},
{"ts": "99999", "user": "UBOT", "text": "done"}]
assert repeat_runs(msgs, "UBOT", 600) == []
def test_human_messages_are_never_counted_as_repeats():
msgs = [{"ts": "100", "user": "U1", "text": "same"},
{"ts": "101", "user": "U2", "text": "same"}]
assert repeat_runs(msgs, "UBOT") == []
def test_sub_second_repeats_are_simultaneous():
assert spacing_kind([0.2, 0.4]) == "simultaneous"
def test_a_one_minute_gap_is_slacks_retry_schedule():
assert spacing_kind([61.0]) == "retry"
assert spacing_kind([290.0]) == "retry"
def test_a_gap_that_is_neither_says_nothing_on_its_own():
assert spacing_kind([17.0]) == "spread"
assert spacing_kind([]) == "single"
def test_duplicates_beside_drops_is_this_note():
verdict, detail = fingerprint({"none": 4, "once": 60, "many": 6, "total": 70},
["retry", "simultaneous"])
assert verdict == "several-live-connections"
assert "different instance" in detail
def test_retry_spaced_duplicates_without_drops_belong_to_the_retry_note():
assert fingerprint({"none": 0, "once": 60, "many": 6, "total": 66},
["retry"])[0] == "retries-one-listener"
def test_simultaneous_duplicates_without_drops_are_a_double_subscription():
assert fingerprint({"none": 0, "once": 60, "many": 6, "total": 66},
["simultaneous"])[0] == "double-subscription"
def test_drops_without_duplicates_are_handed_to_the_cap_note():
verdict, detail = fingerprint({"none": 9, "once": 60, "many": 0, "total": 69}, [])
assert verdict == "losses-only"
assert "connection cap" in detail
def test_repeats_that_match_no_pattern_are_left_unclassified():
assert fingerprint({"none": 0, "once": 5, "many": 2, "total": 7},
["spread"])[0] == "duplicates-unclassified"
def test_a_healthy_channel_is_reported_as_clean():
assert fingerprint({"none": 0, "once": 70, "many": 0, "total": 70}, [])[0] == "clean"
assert fingerprint({}, [])[0] == "clean"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
fingerprint, outcomeMix, repeatRuns, replyCounts, spacingKind,
} from './slack-socket-replica-fingerprint.mjs';
test('a mention answered once counts once', () => {
const msgs = [{ ts: '100', user: 'U1', text: '<@UBOT> go' },
{ ts: '110', user: 'UBOT', text: 'on it' }];
assert.deepEqual(replyCounts(msgs, 'UBOT', 45), [[100, 1]]);
});
test('two answers inside the window count as two', () => {
const msgs = [{ ts: '100', user: 'U1', text: '<@UBOT> go' },
{ ts: '110', user: 'UBOT', text: 'on it' },
{ ts: '111', bot_id: 'B1', text: 'on it' }];
assert.deepEqual(replyCounts(msgs, 'UBOT', 45), [[100, 2]]);
});
test('a mention with nothing after it counts as zero', () => {
const msgs = [{ ts: '100', user: 'U1', text: '<@UBOT> go' }];
assert.deepEqual(replyCounts(msgs, 'UBOT', 45), [[100, 0]]);
});
test('a clean run is exactly one', () => {
const [mix, fraction] = outcomeMix([[1, 1], [2, 1], [3, 1]]);
assert.deepEqual(mix, {
none: 0, once: 3, many: 0, total: 3,
});
assert.equal(fraction, 1);
});
test('the fraction falls for drops and for duplicates alike', () => {
const [, fraction] = outcomeMix([[1, 0], [2, 1], [3, 2], [4, 1]]);
assert.equal(fraction, 0.5);
});
test('an empty window divides by nothing', () => {
assert.deepEqual(outcomeMix([]), [{
none: 0, once: 0, many: 0, total: 0,
}, 0]);
});
test('identical answers close together are one run', () => {
const msgs = [{ ts: '100', user: 'UBOT', text: 'Done' },
{ ts: '100.4', user: 'UBOT', text: 'done' }];
const runs = repeatRuns(msgs, 'UBOT', 600);
assert.equal(runs.length, 1);
assert.equal(Math.round(runs[0][1][0] * 10) / 10, 0.4);
});
test('a single answer is not a run', () => {
assert.deepEqual(repeatRuns([{ ts: '100', user: 'UBOT', text: 'done' }], 'UBOT'), []);
});
test('the same answer hours apart is two separate runs of one', () => {
const msgs = [{ ts: '100', user: 'UBOT', text: 'done' },
{ ts: '99999', user: 'UBOT', text: 'done' }];
assert.deepEqual(repeatRuns(msgs, 'UBOT', 600), []);
});
test('human messages are never counted as repeats', () => {
const msgs = [{ ts: '100', user: 'U1', text: 'same' },
{ ts: '101', user: 'U2', text: 'same' }];
assert.deepEqual(repeatRuns(msgs, 'UBOT'), []);
});
test('sub second repeats are simultaneous', () => {
assert.equal(spacingKind([0.2, 0.4]), 'simultaneous');
});
test('a one minute gap is Slack retry schedule', () => {
assert.equal(spacingKind([61]), 'retry');
assert.equal(spacingKind([290]), 'retry');
});
test('a gap that is neither says nothing on its own', () => {
assert.equal(spacingKind([17]), 'spread');
assert.equal(spacingKind([]), 'single');
});
test('duplicates beside drops is this note', () => {
const [verdict, detail] = fingerprint({
none: 4, once: 60, many: 6, total: 70,
}, ['retry', 'simultaneous']);
assert.equal(verdict, 'several-live-connections');
assert.match(detail, /different instance/);
});
test('retry spaced duplicates without drops belong to the retry note', () => {
assert.equal(fingerprint({
none: 0, once: 60, many: 6, total: 66,
}, ['retry'])[0], 'retries-one-listener');
});
test('simultaneous duplicates without drops are a double subscription', () => {
assert.equal(fingerprint({
none: 0, once: 60, many: 6, total: 66,
}, ['simultaneous'])[0], 'double-subscription');
});
test('drops without duplicates are handed to the cap note', () => {
const [verdict, detail] = fingerprint({
none: 9, once: 60, many: 0, total: 69,
}, []);
assert.equal(verdict, 'losses-only');
assert.match(detail, /connection cap/);
});
test('repeats that match no pattern are left unclassified', () => {
assert.equal(fingerprint({
none: 0, once: 5, many: 2, total: 7,
}, ['spread'])[0], 'duplicates-unclassified');
});
test('a healthy channel is reported as clean', () => {
assert.equal(fingerprint({
none: 0, once: 70, many: 0, total: 70,
}, [])[0], 'clean');
assert.equal(fingerprint({}, [])[0], 'clean');
});
FAQ
Does Slack really not distribute payloads across connections?
It routes each payload to one of the open connections, and the documentation is explicit that there is no predictable distribution pattern. That is a routing decision, not a work-sharing contract: there is no consumer group, no acknowledgement-based redelivery to a different consumer, and no ordering guarantee. Two connections do not mean half the load each; they mean two places a payload might arrive, one of which is chosen for you.
Why do duplicates happen at all if each payload goes to one connection?
Because retries are separate deliveries. When an event is not acknowledged in time Slack sends it again, and with several connections open the retry can be routed to a different instance from the original. The first instance may well have finished the work; the second has no idea it exists. That is also why a dedupe cache in process memory does not help: neither instance ever saw both copies, so both are correct in believing the payload is new.
Would a shared Redis dedupe cache fix it?
It fixes the duplicates and leaves the drops. A payload routed to an instance that is restarting, draining or wedged is not redelivered anywhere, and no cache can recover something that was never received. It is also worth noticing what a shared cache costs: at that point you have built a coordination layer in front of a transport chosen specifically to avoid needing infrastructure. One receiver and a queue you control is less code and loses nothing.
How is this different from the note about duplicate processing on retry?
By the drops. A single listener that mishandles retries produces duplicates and no misses, because every payload still reaches the one process that exists. Several live connections produce duplicates and misses together, in the same channel, in the same hour. The script checks for the combination and, when it finds duplicates without drops, sends you to the retry note instead of claiming this one.
We need more than one instance for availability. What then?
Keep one Socket Mode receiver and make it cheap to restart, so a failure is a short gap rather than a lost workload, and run your redundancy behind it: the receiver acknowledges and enqueues, and as many workers as you like drain the queue. If you need genuinely redundant ingress with no single receiver, that is the case for the HTTP Events API, where a load balancer distributes deliveries properly and the connection budget stops being a factor.
Related field notes
- the same duplicates without the drops
- two subscriptions rather than two connections
- the key that stops a repeat becoming a second message
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using Socket Mode - Slack Docs
- bolt-js #2487: multiple instances each receive some of the events
- bolt-python #445: scaling Socket Mode horizontally duplicates work
- conversations.history method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.