Diagnostic Slack
too_many_websockets: ten is the ceiling and ghosts fill it
Roughly three button clicks in four do nothing. Not the same button, not the same person, not the same time of day — three in four, forever, with no pattern anybody can find. Events mostly work. The handler is never invoked for the ones that vanish, so there is nothing to debug: no error, no timeout, no partial write, just a payload that was never delivered to this process.
The app runs on a single pod. It has been restarted eleven times this week by a rolling deploy, and each restart opened a new WebSocket without the old one ever being closed on Slack's side. There are now four registrations for one listener, and Slack sends each payload to exactly one of them.
An app gets ten concurrent Socket Mode connections. That is the whole budget, per app, and it is not per workspace, per replica or per token. Exceed it and the eleventh attempt is refused, the disconnect reason is too_many_websockets, and the connections already registered stay exactly where they are.
The part that produces the symptom is what happens below the cap. Slack routes each payload to one of the open connections, so a registration that no process is reading — a socket left standing by a reconnect that did not close its predecessor, a client behind NAT whose close never reached Slack — is a hole that swallows its share of the traffic. Three ghosts beside one live socket loses three payloads in four, and nothing anywhere reports a failure.
You cannot count the open connections with a read token, and you must not count them by opening one: apps.connections.open mints a connection, so the measurement would consume the resource being measured and could itself be the eleventh. Count them from the deployment instead — replicas times sockets per replica — and then run the observed loss fraction backwards. If three quarters of your triggers get no answer and exactly one socket is live, the arithmetic says four registrations exist and three of them are ghosts.
The problem in plain words
The loss is uniform, which is why every instinct about it is wrong. There is no bad channel, no bad user, no bad message type, no time of day. Each payload is an independent draw against the same set of registrations, so the failures scatter evenly across everything, and every hypothesis a team forms — it only fails for long messages, it only fails in the busy channel, it only fails after lunch — survives about ten minutes of testing before the counter-example arrives.
Interactions feel worse than events even though the loss rate is identical, and the reason is human. A button click has a person standing in front of it who expects something to happen within a second. A missed message event has nobody watching. So the bug is reported as “buttons are flaky” and diagnosed against the interactivity configuration, which is fine, because the interactivity configuration is not the problem.
The leak itself is unglamorous. A reconnect is supposed to close the old socket and open a new one; several client versions have got the ordering wrong, or have opened the replacement and left the original to time out on Slack's side, or have lost the close frame entirely because the process died before sending it. Every rolling deploy, every OOM kill, every network blip is an opportunity to leave a registration behind, and Slack has no way to know that the process at the other end of a quiet connection has been gone for a day. The registrations accumulate until either the cap refuses a new one or the loss rate becomes unbearable, whichever a human notices first.
Replicas make it arithmetic rather than accidental. Ten is not much: three pods with a client that opens two connections each is six before anything leaks, and a rolling deploy briefly doubles the count while the old pods drain. Teams that reach for horizontal scaling here are usually surprised to learn there is a small integer involved at all, because nothing else in their stack has a ceiling that low.
Why it happens
The cap is unobservable and the consequence is not. No read method returns the number of open Socket Mode connections. So the check is built out of two things you do have: the shape of the deployment, which you can state exactly, and the fraction of triggers that go unanswered, which the workspace records for you in conversations.history. Neither requires a connection to be opened.
Measuring by opening a socket would be self-defeating twice over. It consumes one of the ten, so the number it returns is one higher than the number it was asked about; and on an app already at the cap it would be refused, or would displace something live. This is the clearest case in the section for the rule the whole section runs on: the audit must not be capable of causing the incident.
The loss fraction is a measurement of the ghost count, not just a symptom. If payloads are distributed across N registrations of which L are live, the probability that any given payload is lost is (N - L) / N. Turn that around: given an observed loss fraction f and the number of sockets you believe are live, N = L / (1 - f). Three quarters lost with one live socket means four registrations; a quarter lost with three live sockets means four as well, and one ghost. That is a number you can hold against ten, and it is the only estimate of the leak available from outside.
Even scatter is the fingerprint, and the script checks for it rather than assuming it. Losses spread uniformly across the window are consistent with random routing across a leaky set. Losses bunched into a few tight clusters are not — those are a connection being taken away and replaced badly, which is a different note. The script buckets the window, measures how many buckets contain a miss, and refuses to blame the cap when the misses arrive in a huddle.
No duplicates. That matters as much as the misses. A leaked connection pool loses payloads; it never delivers one twice, because each payload goes to exactly one registration. If the same channel shows duplicated app responses as well as missing ones, several live listeners are involved and the diagnosis is different. The script says so instead of quietly fitting the evidence to this note.
The disconnect reason, if you have it, is the one piece of direct proof. too_many_websockets in your own logs is Slack saying the cap was hit. The script takes the reasons you have already collected and reports whether the cap is confirmed or merely implied, because an implied cap and a confirmed one deserve different confidence in the write-up.
The fix, as a flow
This is the only chain in the batch with a loop under it, and the loop is the note: every restart leaves one more registration standing, so the failure gets worse while nobody touches anything. The red arrow is late because each individual step is correct, including the reconnect. The fix branch sorts a budget of ten rather than a set of errors, and the fourth row is the one that keeps this check honest: losses that arrive in a huddle are not a leak, and attributing them here would produce a confident, wrong number.
How to fix it
State the connection budget from the deployment, not from Slack
sockets_declared multiplies replicas by sockets per replica and holds the result against the documented ten. It is arithmetic on facts you already know, and it is the only counting method available: there is no read that returns the live connection count, and the write that would is the one this section refuses to call.
Pair every mention with the reply that should have followed it
trigger_pairs walks conversations.history and marks each mention of the bot as answered if the app posted anything within --window seconds. loss_fraction reduces that to one number. A window is needed because the app is allowed to be slow; the default is generous on purpose, since counting a slow answer as a loss inflates the ghost estimate.
Check the losses are actually scattered
loss_scatter divides the observed window into buckets and reports how many of them contain a miss. Even occupancy is what random routing across a leaky set looks like. A low occupancy means the misses arrived in a huddle, which is a refresh handled badly rather than a cap, and the script hands you to that note instead of proceeding.
Run the loss fraction backwards into a ghost count
phantom_estimate takes the loss fraction and the number of sockets you believe are live and returns the implied number of registrations and how many of them nobody is reading. A total loss is refused as out of range: that is a connection that never opened, not a leaked one, and it is a different note.
Say whether the cap is confirmed or only implied
cap_evidence reads whatever disconnect reasons you already have and looks for too_many_websockets. Present, and the cap is confirmed by Slack itself. Absent, and the finding is an inference from the arithmetic, which is worth stating as such rather than blurring.
Close before you open, and keep the count small
The repair is one connection per instance and a small number of instances, with the client forced to close the old socket before opening the new one. Ten does not stretch, and Socket Mode does not scale horizontally: if the workload genuinely needs many processing nodes, one receiver that enqueues and many workers that drain is the shape that works, and a public Request URL behind a load balancer is the shape that scales.
How to check it worked
After the SDK upgrade and the replica reduction, re-run against the same channel and the same window. The number that should move is the implied registration count, and it should converge on the number of sockets you actually run.
python3 slack_socket_headroom.py --channel C05OPS9QT --replicas 3 --sockets-per-replica 1 \
--reasons too_many_websockets,too_many_websockets
# identity U07BOT9QD (opsbot) in Northwind
# budget fine 3 replica(s) x 1 socket = 3 of 10, 7 left before the cap
# triggers 64 mention(s) read, 48 answered within 45s
# loss 0.250 16 of 64 trigger(s) went unanswered
# scatter scattered misses touch 83% of the buckets in the window
# phantom leaked 4.0 registration(s) implied for 3 live socket(s): 1.0 ghost
# cap confirmed too_many_websockets appears 2 time(s) in the supplied reasons
# verdict 4.0 of the 10 permitted connection(s) are implied, and 25% of payloads
# land on one nobody reads
# repair: force-close the previous socket before opening the replacement
# repair: one connection per instance, and fewer instances; ten does not stretch
# note: nothing here opened a connection to count them; that would consume oneThe full code
The interesting function is phantom_estimate, and it is four lines of arithmetic. Slack picks one registration per payload, so the loss fraction is the ghost proportion, and inverting it turns an unexplained flakiness rate into a count you can compare with ten. loss_scatter is the guard around it: it refuses to blame the cap when the misses arrive in clusters rather than evenly, because that shape belongs to a different failure. Two GETs, no writes, and nothing that opens a socket.
"""Estimate how much of your ten-connection Socket Mode budget is wasted.
Read only. Slack does not expose the number of open Socket Mode connections to
any read method, and the method that would tell you - apps.connections.open -
mints a connection: calling it consumes one of the ten and could itself be the
one that trips the cap. So this script never calls it. The count comes from the
deployment you describe on the command line, and the leak comes from arithmetic
on the loss fraction the workspace already recorded.
Two GETs with a bot token: auth.test and conversations.history. No writes.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_socket_headroom")
API = "https://slack.com/api/"
# Documented, per app, and not negotiable. Ten is small enough that a rolling
# deploy over four replicas can breach it without anything having leaked.
CAP = 10
def sockets_declared(replicas, per_replica, cap=CAP):
"""How much of the budget does this deployment ask for? Pure.
Returns (in_use, cap, left, state). There is no read method that returns
the live connection count, so this arithmetic on facts you already know is
the only counting available. States: over, at-the-cap, tight, fine.
"""
count = max(0, int(replicas or 0)) * max(0, int(per_replica or 0))
left = cap - count
if count > cap:
state = "over"
elif count == cap:
state = "at-the-cap"
elif count >= cap - 2:
state = "tight"
else:
state = "fine"
return (count, cap, left, state)
def trigger_pairs(messages, bot_user_id, window=45.0):
"""Every mention of the bot, marked answered or not. Pure.
A mention counts as answered when the app posted anything within `window`
seconds of it. The window is generous on purpose: counting a slow answer as
a loss inflates the ghost estimate that the rest of this script builds on
top of it.
Returns [(ts, answered), ...] oldest first.
"""
marker = "<@%s>" % bot_user_id if bot_user_id else None
own, mentions = [], []
for m in messages or []:
ts = str((m or {}).get("ts") or "")
if not ts:
continue
author = (m or {}).get("user") or ""
if (bot_user_id and author == bot_user_id) or (m or {}).get("bot_id"):
own.append(float(ts))
elif marker and marker in str((m or {}).get("text") or ""):
mentions.append(float(ts))
own.sort()
out = []
for ts in sorted(mentions):
out.append((ts, any(ts <= o <= ts + float(window) for o in own)))
return out
def loss_fraction(pairs):
"""What proportion of triggers got no answer? Pure. Returns (missed, total, f)."""
total = len(pairs or [])
missed = sum(1 for _ts, ok in pairs or [] if not ok)
return (missed, total, 0.0 if not total else round(missed / float(total), 3))
def loss_scatter(miss_ts, buckets=12, min_misses=4):
"""Are the misses spread evenly, or bunched? Pure.
Random routing across a set of registrations loses payloads uniformly, so
even occupancy is the fingerprint of a leaked pool. Misses that arrive in a
huddle are a connection being taken away and replaced badly, which is a
different failure with a different repair, and this function exists so the
script cannot quietly attribute that shape to the cap.
Returns (shape, occupancy). Shapes: scattered, clustered, mixed, too-few.
"""
stamps = sorted(float(t) for t in miss_ts or [])
if len(stamps) < max(2, min_misses):
return ("too-few", 0.0)
span = stamps[-1] - stamps[0]
if span <= 0:
return ("clustered", round(1.0 / buckets, 3))
n = max(2, int(buckets))
hit = set()
for t in stamps:
idx = int((t - stamps[0]) / span * n)
hit.add(min(idx, n - 1))
occupancy = round(len(hit) / float(n), 3)
if occupancy >= 0.5:
return ("scattered", occupancy)
if occupancy <= 0.34:
return ("clustered", occupancy)
return ("mixed", occupancy)
def phantom_estimate(fraction, live, cap=CAP):
"""Turn an observed loss fraction into a registration count. Pure.
Slack routes each payload to one of the open connections, so with N
registrations of which L are read by a process, the chance a payload is
lost is (N - L) / N. Inverted: N = L / (1 - f). Three quarters of payloads
lost with one live socket means four registrations and three ghosts; a
quarter lost with three live sockets means four registrations and one.
Returns (ghosts, registrations, state). States: none, leaked, over-cap,
out-of-range.
"""
f = float(fraction or 0.0)
listeners = max(0, int(live or 0))
if listeners <= 0:
return (0.0, 0.0, "out-of-range")
if f <= 0:
return (0.0, float(listeners), "none")
if f >= 0.999:
return (0.0, 0.0, "out-of-range")
registrations = round(listeners / (1.0 - f), 1)
ghosts = round(registrations - listeners, 1)
if registrations > cap:
return (ghosts, registrations, "over-cap")
return (ghosts, registrations, "leaked" if ghosts >= 0.5 else "none")
def cap_evidence(reasons):
"""Did Slack itself say the cap was hit? Pure. Returns (state, hits).
too_many_websockets in your own disconnect logs is direct proof. Its
absence leaves the finding as an inference from arithmetic, and the two
deserve different confidence in whatever you write up afterwards.
"""
hits = sum(1 for r in reasons or [] if str(r or "").strip() == "too_many_websockets")
return (("confirmed" if hits else "implied"), hits)
def page_history(session, channel, limit, max_pages):
"""Page conversations.history. A read, and one of two calls made."""
out, cursor, pages = [], "", 0
while pages < max_pages:
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.error("history unavailable %s", body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = ((body.get("response_metadata") or {}).get("next_cursor") or "")
pages += 1
if not cursor:
break
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN")
ap.add_argument("--channel", required=True,
help="a channel the bot answers in")
ap.add_argument("--replicas", type=int, default=1)
ap.add_argument("--sockets-per-replica", type=int, default=1)
ap.add_argument("--window", type=float, default=45.0,
help="seconds an answer is allowed to take")
ap.add_argument("--reasons", default="",
help="comma separated disconnect reasons from your own logs")
ap.add_argument("--limit", type=int, default=200)
ap.add_argument("--max-pages", type=int, default=5)
args = ap.parse_args()
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token with channels:history", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s (%s) in %s", who.get("user_id"), who.get("user"),
who.get("team"))
live, cap, left, budget = sockets_declared(args.replicas, args.sockets_per_replica)
log.info("budget %-12s %d replica(s) x %d socket = %d of %d, %d left before "
"the cap", budget, args.replicas, args.sockets_per_replica, live, cap,
left)
messages = page_history(s, args.channel, args.limit, args.max_pages)
pairs = trigger_pairs(messages, who.get("user_id"), args.window)
missed, total, fraction = loss_fraction(pairs)
log.info("triggers %d mention(s) read, %d answered within %.0fs",
total, total - missed, args.window)
if not total:
log.info("verdict nothing to measure; this channel has no mentions in the "
"window read")
return 0
log.info("loss %-12.3f %d of %d trigger(s) went unanswered", fraction,
missed, total)
shape, occupancy = loss_scatter([ts for ts, ok in pairs if not ok])
log.info("scatter %-12s misses touch %.0f%% of the buckets in the window",
shape, occupancy * 100)
if shape == "clustered":
log.warning("verdict the misses are bunched rather than spread, which is "
"not what a leaked connection pool looks like")
log.warning(" next: read the note on refresh_requested disconnects; "
"periodic clusters are a swap that was not overlapped")
return 1
ghosts, registrations, state = phantom_estimate(fraction, live)
if state == "out-of-range":
log.warning("phantom out-of-range the loss is total or the live count is "
"zero; that is a connection that never opened, not a leaked one")
return 1
log.info("phantom %-12s %.1f registration(s) implied for %d live socket(s): "
"%.1f ghost", state, registrations, live, ghosts)
evidence, hits = cap_evidence([r.strip() for r in args.reasons.split(",")])
log.info("cap %-12s too_many_websockets appears %d time(s) in the supplied "
"reasons", evidence, hits)
if state == "none" and budget in ("fine", "tight"):
log.info("verdict clean the budget fits and the loss does not imply "
"a leak")
return 0
log.warning("verdict %.1f of the %d permitted connection(s) are implied, and "
"%.0f%% of payloads land on one nobody reads", registrations, cap,
fraction * 100)
log.warning(" repair: force-close the previous socket before opening the "
"replacement, and upgrade past the known close-path regressions")
log.warning(" repair: one connection per instance and fewer instances; for real "
"horizontal scale use the HTTP Events API behind a load balancer")
log.warning(" note: nothing here opened a connection to count them; doing so "
"would consume one of the ten it is trying to measure")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Estimate how much of your ten-connection Socket Mode budget is wasted.
*
* Read only. Slack exposes no read method for the number of open Socket Mode
* connections, and the method that would tell you - apps.connections.open -
* mints one: calling it consumes part of the budget it is measuring and could
* itself trip the cap. So this script never calls it. The count comes from the
* deployment you describe, the leak from arithmetic on the observed losses.
*/
const API = 'https://slack.com/api/';
// Documented, per app, and not negotiable.
export const CAP = 10;
/**
* How much of the budget does this deployment ask for? Pure.
* Returns [inUse, cap, left, state]; states over, at-the-cap, tight, fine.
*/
export function socketsDeclared(replicas, perReplica, cap = CAP) {
const count = Math.max(0, Number(replicas) || 0) * Math.max(0, Number(perReplica) || 0);
const left = cap - count;
let state = 'fine';
if (count > cap) state = 'over';
else if (count === cap) state = 'at-the-cap';
else if (count >= cap - 2) state = 'tight';
return [count, cap, left, state];
}
/**
* Every mention of the bot, marked answered or not. Pure.
* The window is generous on purpose: a slow answer counted as a loss inflates
* the ghost estimate built on top of it. Returns [[ts, answered], ...].
*/
export function triggerPairs(messages, botUserId, window = 45) {
const marker = botUserId ? `<@${botUserId}>` : null;
const own = [];
const mentions = [];
for (const m of messages ?? []) {
const ts = String((m ?? {}).ts ?? '');
if (!ts) continue;
const author = (m ?? {}).user ?? '';
if ((botUserId && author === botUserId) || (m ?? {}).bot_id) own.push(Number(ts));
else if (marker && String((m ?? {}).text ?? '').includes(marker)) {
mentions.push(Number(ts));
}
}
own.sort((a, b) => a - b);
return mentions.sort((a, b) => a - b)
.map((ts) => [ts, own.some((o) => o >= ts && o <= ts + Number(window))]);
}
/** What proportion of triggers got no answer? Pure. [missed, total, f]. */
export function lossFraction(pairs) {
const rows = pairs ?? [];
const total = rows.length;
const missed = rows.filter(([, ok]) => !ok).length;
const f = total ? Math.round((missed / total) * 1000) / 1000 : 0;
return [missed, total, f];
}
/**
* Are the misses spread evenly, or bunched? Pure.
* Random routing across registrations loses payloads uniformly, so even
* occupancy is the fingerprint of a leaked pool. A huddle is a connection
* replaced badly, which is a different note. Returns [shape, occupancy].
*/
export function lossScatter(missTs, buckets = 12, minMisses = 4) {
const stamps = (missTs ?? []).map(Number).sort((a, b) => a - b);
if (stamps.length < Math.max(2, minMisses)) return ['too-few', 0];
const span = stamps[stamps.length - 1] - stamps[0];
if (span <= 0) return ['clustered', Math.round((1000 / buckets)) / 1000];
const n = Math.max(2, Math.trunc(buckets));
const hit = new Set();
for (const t of stamps) {
hit.add(Math.min(Math.trunc(((t - stamps[0]) / span) * n), n - 1));
}
const occupancy = Math.round((hit.size / n) * 1000) / 1000;
if (occupancy >= 0.5) return ['scattered', occupancy];
if (occupancy <= 0.34) return ['clustered', occupancy];
return ['mixed', occupancy];
}
/**
* Turn an observed loss fraction into a registration count. Pure.
* With N registrations of which L are read, loss = (N - L) / N, so
* N = L / (1 - f). Returns [ghosts, registrations, state].
*/
export function phantomEstimate(fraction, live, cap = CAP) {
const f = Number(fraction) || 0;
const listeners = Math.max(0, Number(live) || 0);
if (listeners <= 0) return [0, 0, 'out-of-range'];
if (f <= 0) return [0, listeners, 'none'];
if (f >= 0.999) return [0, 0, 'out-of-range'];
const registrations = Math.round((listeners / (1 - f)) * 10) / 10;
const ghosts = Math.round((registrations - listeners) * 10) / 10;
if (registrations > cap) return [ghosts, registrations, 'over-cap'];
return [ghosts, registrations, ghosts >= 0.5 ? 'leaked' : 'none'];
}
/** Did Slack itself say the cap was hit? Pure. Returns [state, hits]. */
export function capEvidence(reasons) {
const hits = (reasons ?? [])
.filter((r) => String(r ?? '').trim() === 'too_many_websockets').length;
return [hits ? 'confirmed' : 'implied', hits];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function pageHistory(headers, channel, limit, maxPages) {
const out = [];
let cursor = '';
for (let page = 0; page < maxPages; page += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.error(`history unavailable ${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = (body.response_metadata ?? {}).next_cursor ?? '';
if (!cursor) break;
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
const channel = arg(args, '--channel', '');
if (!token || !channel) {
console.error(`set ${tokenEnv} to a bot token with channels:history, and pass `
+ '--channel');
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} (${who.user}) in ${who.team}`);
const replicas = Number(arg(args, '--replicas', '1'));
const perReplica = Number(arg(args, '--sockets-per-replica', '1'));
const window = Number(arg(args, '--window', '45'));
const [live, cap, left, budget] = socketsDeclared(replicas, perReplica);
console.log(`budget ${budget.padEnd(12)} ${replicas} replica(s) x ${perReplica} `
+ `socket = ${live} of ${cap}, ${left} left before the cap`);
const messages = await pageHistory(headers, channel,
Number(arg(args, '--limit', '200')), Number(arg(args, '--max-pages', '5')));
const pairs = triggerPairs(messages, who.user_id, window);
const [missed, total, fraction] = lossFraction(pairs);
console.log(`triggers ${total} mention(s) read, ${total - missed} answered within `
+ `${window}s`);
if (!total) {
console.log('verdict nothing to measure; this channel has no mentions in the '
+ 'window read');
return;
}
console.log(`loss ${String(fraction).padEnd(12)} ${missed} of ${total} `
+ 'trigger(s) went unanswered');
const [shape, occupancy] = lossScatter(pairs.filter(([, ok]) => !ok).map(([ts]) => ts));
console.log(`scatter ${shape.padEnd(12)} misses touch `
+ `${Math.round(occupancy * 100)}% of the buckets in the window`);
if (shape === 'clustered') {
console.warn('verdict the misses are bunched rather than spread, which is not '
+ 'what a leaked connection pool looks like');
console.warn(' next: read the note on refresh_requested disconnects; periodic '
+ 'clusters are a swap that was not overlapped');
process.exitCode = 1;
return;
}
const [ghosts, registrations, state] = phantomEstimate(fraction, live);
if (state === 'out-of-range') {
console.warn('phantom out-of-range the loss is total or the live count is zero; '
+ 'that is a connection that never opened, not a leaked one');
process.exitCode = 1;
return;
}
console.log(`phantom ${state.padEnd(12)} ${registrations} registration(s) implied `
+ `for ${live} live socket(s): ${ghosts} ghost`);
const [evidence, hits] = capEvidence(String(arg(args, '--reasons', '')).split(','));
console.log(`cap ${evidence.padEnd(12)} too_many_websockets appears ${hits} `
+ 'time(s) in the supplied reasons');
if (state === 'none' && (budget === 'fine' || budget === 'tight')) {
console.log('verdict clean the budget fits and the loss does not imply a '
+ 'leak');
return;
}
console.warn(`verdict ${registrations} of the ${cap} permitted connection(s) are `
+ `implied, and ${Math.round(fraction * 100)}% of payloads land on one nobody reads`);
console.warn(' repair: force-close the previous socket before opening the '
+ 'replacement, and upgrade past the known close-path regressions');
console.warn(' repair: one connection per instance and fewer instances; for real '
+ 'horizontal scale use the HTTP Events API behind a load balancer');
console.warn(' note: nothing here opened a connection to count them; doing so '
+ 'would consume one of the ten it is trying to measure');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The arithmetic gets the attention it deserves: a quarter lost with one live socket must come out as four registrations and three ghosts, and a loss of zero must come out as no leak rather than as a rounding artefact. Beyond that the tests are mostly about refusals. phantom_estimate refuses a total loss, because that is a connection that never opened and belongs to another note. loss_scatter refuses to call a huddle of misses scattered. And cap_evidence keeps Slack said so and we worked it out in separate words, because those are not the same claim.
from slack_socket_headroom import (
CAP, cap_evidence, loss_fraction, loss_scatter, phantom_estimate,
sockets_declared, trigger_pairs,
)
def test_the_documented_ceiling_is_ten():
assert CAP == 10
def test_the_budget_is_replicas_times_sockets():
assert sockets_declared(3, 1)[:3] == (3, 10, 7)
assert sockets_declared(3, 2)[0] == 6
def test_more_than_ten_is_over_and_exactly_ten_is_at_the_cap():
assert sockets_declared(11, 1)[3] == "over"
assert sockets_declared(5, 2)[3] == "at-the-cap"
def test_eight_of_ten_is_called_tight_rather_than_fine():
assert sockets_declared(8, 1)[3] == "tight"
assert sockets_declared(2, 1)[3] == "fine"
def test_a_mention_answered_inside_the_window_is_not_a_loss():
msgs = [{"ts": "100", "user": "U1", "text": "<@UBOT> go"},
{"ts": "110", "user": "UBOT", "text": "on it"}]
assert trigger_pairs(msgs, "UBOT", 45) == [(100.0, True)]
def test_an_answer_after_the_window_is_a_loss():
msgs = [{"ts": "100", "user": "U1", "text": "<@UBOT> go"},
{"ts": "500", "user": "UBOT", "text": "on it"}]
assert trigger_pairs(msgs, "UBOT", 45) == [(100.0, False)]
def test_an_earlier_bot_message_does_not_answer_a_later_mention():
msgs = [{"ts": "90", "user": "UBOT", "text": "morning"},
{"ts": "100", "user": "U1", "text": "<@UBOT> go"}]
assert trigger_pairs(msgs, "UBOT", 45) == [(100.0, False)]
def test_the_loss_fraction_is_missed_over_total():
assert loss_fraction([(1, True), (2, False), (3, True), (4, False)]) == (2, 4, 0.5)
assert loss_fraction([]) == (0, 0, 0.0)
def test_three_quarters_lost_with_one_live_socket_implies_four_registrations():
ghosts, registrations, state = phantom_estimate(0.75, 1)
assert (ghosts, registrations) == (3.0, 4.0)
assert state == "leaked"
def test_a_quarter_lost_with_three_live_sockets_implies_one_ghost():
ghosts, registrations, _state = phantom_estimate(0.25, 3)
assert (ghosts, registrations) == (1.0, 4.0)
def test_an_implied_count_above_ten_is_flagged_against_the_cap():
assert phantom_estimate(0.5, 6)[2] == "over-cap"
def test_no_loss_means_no_ghosts():
assert phantom_estimate(0.0, 3) == (0.0, 3.0, "none")
def test_a_total_loss_is_out_of_range_because_it_is_a_different_note():
assert phantom_estimate(1.0, 3)[2] == "out-of-range"
assert phantom_estimate(0.25, 0)[2] == "out-of-range"
def test_misses_spread_across_the_window_are_scattered():
shape, occupancy = loss_scatter([0, 100, 200, 300, 400, 500, 600, 700])
assert shape == "scattered"
assert occupancy >= 0.5
def test_misses_in_one_huddle_are_clustered_and_belong_elsewhere():
assert loss_scatter([0, 1, 2, 3, 4, 5, 1000])[0] == "clustered"
def test_too_few_misses_to_judge_says_so_rather_than_guessing():
assert loss_scatter([1, 2])[0] == "too-few"
assert loss_scatter([])[0] == "too-few"
def test_identical_timestamps_are_a_huddle_and_not_a_spread():
assert loss_scatter([5, 5, 5, 5])[0] == "clustered"
def test_slack_saying_too_many_websockets_is_confirmation():
assert cap_evidence(["too_many_websockets", "refresh_requested"]) == ("confirmed", 1)
def test_without_that_reason_the_finding_is_only_implied():
assert cap_evidence(["refresh_requested"]) == ("implied", 0)
assert cap_evidence([]) == ("implied", 0)
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
CAP, capEvidence, lossFraction, lossScatter, phantomEstimate, socketsDeclared,
triggerPairs,
} from './slack-socket-headroom.mjs';
test('the documented ceiling is ten', () => {
assert.equal(CAP, 10);
});
test('the budget is replicas times sockets', () => {
assert.deepEqual(socketsDeclared(3, 1).slice(0, 3), [3, 10, 7]);
assert.equal(socketsDeclared(3, 2)[0], 6);
});
test('more than ten is over and exactly ten is at the cap', () => {
assert.equal(socketsDeclared(11, 1)[3], 'over');
assert.equal(socketsDeclared(5, 2)[3], 'at-the-cap');
});
test('eight of ten is called tight rather than fine', () => {
assert.equal(socketsDeclared(8, 1)[3], 'tight');
assert.equal(socketsDeclared(2, 1)[3], 'fine');
});
test('a mention answered inside the window is not a loss', () => {
const msgs = [{ ts: '100', user: 'U1', text: '<@UBOT> go' },
{ ts: '110', user: 'UBOT', text: 'on it' }];
assert.deepEqual(triggerPairs(msgs, 'UBOT', 45), [[100, true]]);
});
test('an answer after the window is a loss', () => {
const msgs = [{ ts: '100', user: 'U1', text: '<@UBOT> go' },
{ ts: '500', user: 'UBOT', text: 'on it' }];
assert.deepEqual(triggerPairs(msgs, 'UBOT', 45), [[100, false]]);
});
test('an earlier bot message does not answer a later mention', () => {
const msgs = [{ ts: '90', user: 'UBOT', text: 'morning' },
{ ts: '100', user: 'U1', text: '<@UBOT> go' }];
assert.deepEqual(triggerPairs(msgs, 'UBOT', 45), [[100, false]]);
});
test('the loss fraction is missed over total', () => {
assert.deepEqual(lossFraction([[1, true], [2, false], [3, true], [4, false]]),
[2, 4, 0.5]);
assert.deepEqual(lossFraction([]), [0, 0, 0]);
});
test('three quarters lost with one live socket implies four registrations', () => {
const [ghosts, registrations, state] = phantomEstimate(0.75, 1);
assert.deepEqual([ghosts, registrations], [3, 4]);
assert.equal(state, 'leaked');
});
test('a quarter lost with three live sockets implies one ghost', () => {
const [ghosts, registrations] = phantomEstimate(0.25, 3);
assert.deepEqual([ghosts, registrations], [1, 4]);
});
test('an implied count above ten is flagged against the cap', () => {
assert.equal(phantomEstimate(0.5, 6)[2], 'over-cap');
});
test('no loss means no ghosts', () => {
assert.deepEqual(phantomEstimate(0, 3), [0, 3, 'none']);
});
test('a total loss is out of range because it is a different note', () => {
assert.equal(phantomEstimate(1, 3)[2], 'out-of-range');
assert.equal(phantomEstimate(0.25, 0)[2], 'out-of-range');
});
test('misses spread across the window are scattered', () => {
const [shape, occupancy] = lossScatter([0, 100, 200, 300, 400, 500, 600, 700]);
assert.equal(shape, 'scattered');
assert.ok(occupancy >= 0.5);
});
test('misses in one huddle are clustered and belong elsewhere', () => {
assert.equal(lossScatter([0, 1, 2, 3, 4, 5, 1000])[0], 'clustered');
});
test('too few misses to judge says so rather than guessing', () => {
assert.equal(lossScatter([1, 2])[0], 'too-few');
assert.equal(lossScatter([])[0], 'too-few');
});
test('identical timestamps are a huddle and not a spread', () => {
assert.equal(lossScatter([5, 5, 5, 5])[0], 'clustered');
});
test('Slack saying too_many_websockets is confirmation', () => {
assert.deepEqual(capEvidence(['too_many_websockets', 'refresh_requested']),
['confirmed', 1]);
});
test('without that reason the finding is only implied', () => {
assert.deepEqual(capEvidence(['refresh_requested']), ['implied', 0]);
assert.deepEqual(capEvidence([]), ['implied', 0]);
});
FAQ
Can I just count the open connections?
Not from outside. No read method returns the live Socket Mode connection count, and the only method that touches connections at all is apps.connections.open, which mints one. Calling it to count would add to the number it was asked about, and on an app already at ten it would either be refused or displace something live. The two available substitutes are the deployment's own arithmetic and the loss fraction the workspace recorded, which is what this script uses.
Where does the ghost estimate come from?
From the routing rule. Slack sends each payload to one of the open connections, so if N registrations exist and only L of them are attached to a running process, the chance any payload is lost is (N minus L) over N. Observe the loss fraction f and you can invert it: N equals L over (1 minus f). Three quarters lost with one live socket means four registrations, three of them ghosts; a quarter lost with three live sockets means four registrations and one ghost. It is an estimate, not a measurement, and it is the only number of its kind available without opening a connection.
We run one pod. How can we be near a limit of ten?
Because registrations outlive processes. A reconnect that opens the replacement before closing the original, a pod killed before its close frame is sent, a NAT timeout that leaves Slack believing a socket is still there: each of those leaves a registration standing that nothing will ever read. Eleven restarts in a week on a client with a broken close path is enough to fill most of the budget on its own, and Slack has no way to know the far end is gone.
Why does the script refuse to conclude anything when the misses are clustered?
Because clustered misses are somebody else's failure. A leaked connection pool loses payloads at random, so the misses spread evenly across whatever window you look at. Losses that arrive in a few tight bunches, especially at a regular interval, are a connection being taken away and replaced without overlap, which is the refresh_requested note. Fitting that evidence to the cap would produce a confident, wrong ghost count and send somebody to upgrade an SDK that was fine.
Would moving to the HTTP Events API fix this?
It removes the ceiling, yes, because HTTP delivery scales with your load balancer rather than with a per-app connection budget. It also costs you the reason most teams chose Socket Mode: no public endpoint, no URL verification, nothing to expose from behind a firewall. The cheaper repair is usually to keep Socket Mode with one connection per instance and a small instance count, and to put concurrency behind the receiver in a queue you control.
Related field notes
- the transport Socket Mode replaced, and its own limits
- the other deadline that silently drops work
- what happens when the misses turn into repeats
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using Socket Mode - Slack Docs
- python-slack-sdk #1940: too_many_websockets after leaked reconnects
- node-slack-sdk #1654: old socket not closed before the new one opens
- conversations.history method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.