Diagnostic Slack
One quota bucket, eight workers that think they are alone
The backfill was slow, so it was given eight workers. It got slower. Each worker has its own client, its own connection pool and its own carefully written backoff, and each of them spends most of its life asleep.
Nothing in that list is wrong. The mistake is upstream of all of it: the quota is not stored per process. Eight workers is not eight budgets. It is one budget, divided eight ways, by eight things that do not know the other seven exist.
Slack's rate limit is keyed on method, workspace and app. Not on process, host, container, connection or client instance. Ten replicas of the same deployment holding the same token draw from exactly one bucket, so adding concurrency to a rate-limited API is negative work: total throughput does not move, and per-request latency gets worse because every worker now spends time in backoff it did not spend before.
Then the second-order failure. Each replica gets the same Retry-After, because it is the same window. Each sleeps exactly that long. Each wakes at the same instant and retries together, so the bucket is emptied in the first few milliseconds of every window and the herd re-forms on the next refusal. Backoff without jitter does not spread load; it synchronises it.
The script below establishes the shared identity from auth.test bodies collected from each replica, does the fan-out arithmetic against the tier, and infers unseen callers from a paced sample: being throttled at a rate well below the documented floor means somebody else is spending your window.
The problem in plain words
Every intuition you have about scaling a worker pool is correct except for the one that matters here. More workers usually means more throughput because the bottleneck is local: CPU, a connection, a disk. When the bottleneck is a quota held by somebody else's server and keyed on your app rather than on your process, the pool size is a divisor and not a multiplier. Eight workers each achieving one-eighth of the rate is the good case; the realistic case is worse, because the coordination overhead is paid in wall-clock sleeps.
The synchronisation is what makes it pathological rather than merely futile. Slack hands every refused caller the same number of seconds, because the window is a property of the window and not of the caller. Eight replicas that each implement the documented behaviour perfectly will therefore sleep for identical durations and wake within milliseconds of each other. The first two calls of the new window succeed and the other six are refused, all at once, and receive an identical Retry-After again. The pool converges on a lockstep that nobody designed and nobody can see from inside any single process.
And the diagnosis is genuinely hard from one process, because every log you have is a log of one replica behaving reasonably. It made twelve calls a minute against a method whose floor is twenty and it was throttled, which reads as a Slack problem. It is not: it is the other four replicas, or the cron on the analytics box that nobody remembers, or the staging deployment still pointed at the production token.
Why it happens
The key does not contain your process. This is the whole note. Method, workspace, app. Two tokens of the same app in the same workspace share. Two replicas of the same container share. A laptop running the script during the incident shares. Only a genuinely different app, or a different workspace, gets its own bucket.
Being throttled below the floor is evidence about somebody else. If you make twelve calls a minute against a method with a floor of twenty and are refused, at least eight calls a minute came from somewhere that is not you. That subtraction is the most useful thing a single process can compute, and it is the only way to find the caller nobody remembers deploying.
Identical backoff is worse than random backoff. Correct, documented, uniform backoff produces a thundering herd, because uniformity is exactly what you do not want when the wake-up time is shared. Jitter is not a refinement here; without it the pool cannot de-synchronise, ever.
Per-replica limiting cannot work in principle. A token bucket inside each process can only enforce a rate it can observe, and it cannot observe the other seven. The limiter has to live somewhere all callers meet: a shared store, or a single sender service that everything else queues into.
Parallelising across channels is a different question and a real answer. chat.postMessage is limited per channel, so N channels genuinely means N envelopes. That is the one axis on which concurrency buys throughput, and it is worth separating from the axis on which it does not.
The fix, as a flow
This is the only chain in the batch that loops, and the loop is the note: every replica received the same Retry-After, so every replica wakes at the same instant and empties the new window together. The fix branch is about the key rather than the rate, because two processes that land in one group are one caller however many hosts they run on.
How to fix it
Collect an auth.test from every process that holds a token
Every replica, every cron, every notebook, staging included. Save the response bodies with a label. Two of them carrying the same team_id and the same bot_id are the same app in the same workspace, which is the quota key, which means they are one caller as far as Slack is concerned however many hosts they run on.
Add up the rates you believe each one makes
Calls per minute per replica, times the replica count, summed. Against the documented floor for that method. This arithmetic usually ends the investigation on its own, because nobody has ever written the total down and the total is often several times the budget.
Take a paced sample and see whether it is refused
The script makes a small number of read calls, deliberately spaced well under the tier, and reports whether any came back throttled. It refuses to go below a spacing floor. If a rate that low is refused, the bucket was already empty when the script arrived, and that is a measurement of everybody else.
Subtract to find the callers you did not know about
The floor minus your observed rate is the traffic that has to be coming from somewhere else. Expressed as a multiple of your own rate it is a headcount: roughly how many more processes like this one are on the same method. That number is what you take to the team to ask whose job it is.
Measure the herd, not just the rate
Replica count, the Retry-After they all received, and the jitter fraction in your backoff. With no jitter the retry window has zero width and every replica lands in the same millisecond. The script computes how many arrive in the first hundred milliseconds, which is the number that decides whether the next window is also wasted.
Move the limiter out of the process
One shared token bucket, in Redis or in a single sender service, so the app respects the tier regardless of how many copies of it are running. Add jitter to every backoff. Then, if throughput still matters, parallelise along the axis that actually has more than one bucket: across channels for posting, or across workspaces, never across workers on one method.
How to check it worked
Re-run after the shared limiter is in front of the sender. The sample should come back clean and the fan-out arithmetic should sit under the floor.
python3 slack_shared_quota_audit.py --identities replicas.json --tier-floor 20
# identity U0APPBOT11 bot B0APPBOT99 in acme
# bucket separate-buckets 2 group(s), no two callers share a workspace and app
# fanout headroom 8.0/min across 4 caller(s) against a floor of 20/min
# sample 5 call(s) at 5.0s spacing, 0 throttled
# competitors nothing was throttled, so nothing is implied about other callers
# herd de-synchronised retries spread over 15.0s, about 1 in the first 100ms
# 0 finding(s)The full code
Five pure functions and one paced GET loop. same_bucket is the one that ends arguments: it groups auth.test bodies by the key Slack actually uses, and two replicas that land in one group are one caller no matter what the deployment diagram says. implied_competitors is the subtraction that finds the process nobody remembers. herd_window is about time rather than rate, because a pool can be under budget and still waste every window by arriving in it together.
"""Find out how many callers are sharing one Slack rate-limit bucket.
Read only. Slack keys its quota on method, workspace and app, so replicas of one
deployment share a single budget and adding workers to a throttled job makes it
slower. This groups auth.test bodies by that key, adds up the fan-out against the
documented tier floor, takes a deliberately slow sample to see whether the bucket
is already empty, and measures how tightly the pool's retries synchronise.
The sample is paced well below the tier and refuses to go faster. Nothing here
tries to exhaust a window: that would be a denial of service against the
workspace under audit, and against everybody else's app in it.
"""
import argparse
import json
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_shared_quota_audit")
API = "https://slack.com/api/"
# The sample will not go faster than this, whatever the flags say. An audit that
# can be pointed at a production workspace has to be incapable of becoming the
# incident it was sent to diagnose.
MIN_SPACING = 3.0
# Retries landing inside this window of each other are one wave rather than a
# spread. A hundred milliseconds is generous to the client and still small
# against any Retry-After Slack returns.
BURST = 0.1
def same_bucket(identities):
"""Group callers by the key Slack's quota actually uses. Pure.
identities: [(label, auth_test_body), ...].
The documented key is (method, workspace, app). team_id is the workspace and
bot_id is the closest thing a read-only caller has to the app's identity in
it: two different tokens of the same app in the same workspace return the
same bot_id. A user token has no bot_id, so it falls back to user_id, which
is a weaker key and is labelled as such in the output.
"""
groups = {}
for label, body in identities or []:
body = body if isinstance(body, dict) else {}
team = str(body.get("team_id") or "").strip() or "unknown-team"
actor = str(body.get("bot_id") or "").strip()
if not actor:
actor = str(body.get("user_id") or "").strip() or "unknown-app"
groups.setdefault("%s/%s" % (team, actor), []).append(str(label))
return groups
def bucket_verdict(groups):
"""Say whether anything is sharing. Pure. Returns (verdict, detail)."""
if not groups:
return ("nothing-supplied",
"no identities were given, so the sharing question is unanswered. "
"One auth.test per process, saved with a label, is the whole "
"input and it is the evidence that ends the argument.")
shared = {key: labels for key, labels in groups.items() if len(labels) > 1}
if not shared:
return ("separate-buckets",
"%d group(s), no two callers share a workspace and app"
% len(groups))
biggest = max(shared.items(), key=lambda pair: len(pair[1]))
return ("one-bucket",
"%d caller(s) share the key %s: %s. Slack sees one app in one "
"workspace, so they hold one budget between them however many hosts "
"they run on." % (len(biggest[1]), biggest[0], ", ".join(biggest[1])))
def aggregate_rate(workers):
"""Total calls per minute against one method. Pure.
workers: [{"label", "calls_per_minute", "replicas"?}]. Nobody has usually
written this total down, and the total is the finding.
"""
total = 0.0
for worker in workers or []:
try:
rate = float((worker or {}).get("calls_per_minute") or 0)
except (TypeError, ValueError):
rate = 0.0
try:
replicas = max(1, int((worker or {}).get("replicas") or 1))
except (TypeError, ValueError):
replicas = 1
total += max(0.0, rate) * replicas
return total
def fanout_verdict(total_rate, tier_floor):
"""The pool's demand against the method's floor. Pure.
Returns (verdict, share, detail).
"""
floor = max(1.0, float(tier_floor or 1))
share = float(total_rate) / floor
if share > 1.0:
return ("saturated", share,
"%.1f/min across the pool against a floor of %.0f/min. The pool is "
"asking for %.1f times the budget, so its real throughput is the "
"floor and the extra workers buy queueing rather than work."
% (total_rate, floor, share))
if share > 0.8:
return ("at-the-edge", share,
"%.1f/min across the pool against a floor of %.0f/min. One more "
"replica, or one retry storm, puts it over."
% (total_rate, floor))
return ("headroom", share,
"%.1f/min against a floor of %.0f/min" % (total_rate, floor))
def implied_competitors(observed_rate, tier_floor, throttled):
"""Callers you cannot see, inferred by subtraction. Pure.
Returns (others, detail). others is a multiple of your own rate, or None
when nothing can be inferred. This is the most useful thing one process can
compute about the rest of them.
"""
floor = max(0.0, float(tier_floor or 0))
observed = max(0.0, float(observed_rate or 0))
if not throttled:
return (None,
"nothing was throttled, so nothing is implied about other callers. "
"A clean sample is not proof the bucket is yours alone; it is "
"proof it was not empty in the seconds you looked.")
if observed <= 0:
return (None, "throttled at an observed rate of zero, which means the "
"sample never got a call away. Nothing to subtract from.")
if observed >= floor:
return (None,
"throttled at %.1f/min against a floor of %.0f/min, which you "
"reached on your own. This sample says nothing about anybody else."
% (observed, floor))
hidden = floor - observed
others = hidden / observed
return (others,
"throttled at %.1f/min against a floor of %.0f/min, so at least "
"%.1f call(s) a minute on this method came from somewhere other than "
"this process. That is roughly %.1f more caller(s) at your rate, and "
"finding them is the job." % (observed, floor, hidden, others))
def herd_window(replicas, retry_after, jitter_fraction=0.0):
"""How tightly the pool's retries land together. Pure.
Returns (width, first_wave, verdict, detail). Slack hands every refused
caller the same Retry-After because the window belongs to the window and not
to the caller, so uniform backoff is exactly the wrong shape: it guarantees
the pool wakes in lockstep.
"""
try:
replicas = max(1, int(replicas))
except (TypeError, ValueError):
replicas = 1
try:
retry_after = max(0.0, float(retry_after))
except (TypeError, ValueError):
retry_after = 0.0
try:
jitter = min(1.0, max(0.0, float(jitter_fraction)))
except (TypeError, ValueError):
jitter = 0.0
width = retry_after * jitter
if width <= 0:
return (0.0, replicas, "thundering-herd",
"no jitter, so the retry window has zero width and all %d "
"replica(s) wake in the same millisecond. The first calls of the "
"new window are spent instantly and the rest are refused "
"together, which re-forms the herd on the next Retry-After."
% replicas)
first = max(1, int(round(replicas * min(1.0, BURST / width))))
verdict = "thundering-herd" if first >= 2 else "de-synchronised"
return (width, first, verdict,
"retries spread over %.1fs, about %d in the first %.0fms"
% (width, first, BURST * 1000))
def paced_sample(session, method, params, samples, spacing):
"""A deliberately slow read loop. GET only.
Spacing is floored: this audit will not add meaningful load to the bucket it
is measuring, and it will not chase a 429 to prove one exists.
"""
spacing = max(MIN_SPACING, float(spacing))
throttled = 0
started = time.monotonic()
made = 0
for i in range(max(1, int(samples))):
if i:
time.sleep(spacing)
res = session.get(API + method, params=params, timeout=30)
made += 1
try:
body = res.json()
except ValueError:
body = {}
if res.status_code == 429 or body.get("error") == "ratelimited":
throttled += 1
elapsed = max(1e-6, time.monotonic() - started)
return (made, throttled, made / elapsed * 60.0, spacing)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--identities", default="",
help="JSON list of [label, auth.test body] from each process")
ap.add_argument("--workers", default="",
help="JSON list of {label, calls_per_minute, replicas}")
ap.add_argument("--tier-floor", type=float, default=20.0,
help="documented requests per minute for the method under audit")
ap.add_argument("--method", default="conversations.list",
help="read method to sample")
ap.add_argument("--samples", type=int, default=5, help="calls in the sample")
ap.add_argument("--spacing", type=float, default=5.0,
help="seconds between sample calls, floored at 3")
ap.add_argument("--replicas", type=int, default=1,
help="how many copies of this process run")
ap.add_argument("--retry-after", type=float, default=30.0,
help="the Retry-After they all received")
ap.add_argument("--jitter", type=float, default=0.0,
help="jitter as a fraction of the wait, 0 for none")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding the bot token")
ap.add_argument("--no-sample", action="store_true",
help="skip the live half and do the arithmetic only")
args = ap.parse_args()
findings = 0
identities = []
if args.identities:
identities = json.loads(open(args.identities, encoding="utf-8").read())
groups = same_bucket(identities)
verdict, detail = bucket_verdict(groups)
if verdict == "one-bucket":
findings += 1
log.warning("bucket %-17s %s", verdict, detail)
log.warning(" repair: one shared token bucket in front of Slack, in Redis "
"or in a single sender service. A limiter inside each process "
"can only enforce a rate it can see")
else:
log.info("bucket %-17s %s", verdict, detail)
if args.workers:
workers = json.loads(open(args.workers, encoding="utf-8").read())
total = aggregate_rate(workers)
fan, share, fan_detail = fanout_verdict(total, args.tier_floor)
if fan == "headroom":
log.info("fanout %-17s %.1f/min across %d caller(s) against a floor "
"of %.0f/min", fan, total, len(workers), args.tier_floor)
else:
findings += 1
log.warning("fanout %-17s %s", fan, fan_detail)
log.warning(" repair: concurrency is a divisor here, not a multiplier. "
"Parallelise across channels or workspaces, which have more "
"than one bucket, rather than across workers on one method")
if not args.no_sample:
token = os.environ.get(args.token_env)
if not token:
log.error("set %s, or pass --no-sample", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
identity = s.get(API + "auth.test", timeout=30).json()
if identity.get("ok") is not True:
log.error("auth.test answered ok: false, error=%s", identity.get("error"))
return 2
log.info("identity %s bot %s in %s", identity.get("user_id"),
identity.get("bot_id"), identity.get("team"))
made, throttled, rate, spacing = paced_sample(
s, args.method, {"limit": 1, "exclude_archived": "true"},
args.samples, args.spacing)
log.info("sample %d call(s) at %.1fs spacing, %d throttled",
made, spacing, throttled)
others, why = implied_competitors(rate, args.tier_floor, throttled > 0)
if others is None:
log.info("competitors %s", why)
else:
findings += 1
log.warning("competitors %s", why)
width, first, herd, herd_detail = herd_window(args.replicas, args.retry_after,
args.jitter)
if herd == "thundering-herd" and args.replicas > 1:
findings += 1
log.warning("herd %-17s %s", herd, herd_detail)
log.warning(" repair: add jitter to every backoff. Uniform, correct, "
"documented backoff synchronises the pool, because Slack gives "
"every caller the same number")
else:
log.info("herd %-17s %s", herd, herd_detail)
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find out how many callers are sharing one Slack rate-limit bucket.
*
* Read only. Slack keys its quota on method, workspace and app, so replicas of
* one deployment share a single budget and adding workers to a throttled job
* makes it slower. This groups auth.test bodies by that key, adds up the fan-out
* against the documented tier floor, takes a deliberately slow sample to see
* whether the bucket is already empty, and measures how tightly the pool's
* retries synchronise.
*
* The sample is paced well below the tier and refuses to go faster.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://slack.com/api/';
// The sample will not go faster than this, whatever the flags say.
const MIN_SPACING = 3.0;
// Retries landing inside this window of each other are one wave, not a spread.
const BURST = 0.1;
/**
* Group callers by the key Slack's quota actually uses. Pure.
* identities: [[label, authTestBody], ...].
*/
export function sameBucket(identities) {
const groups = new Map();
for (const [label, body] of identities ?? []) {
const doc = (body && typeof body === 'object') ? body : {};
const team = String(doc.team_id ?? '').trim() || 'unknown-team';
let actor = String(doc.bot_id ?? '').trim();
if (!actor) actor = String(doc.user_id ?? '').trim() || 'unknown-app';
const key = `${team}/${actor}`;
if (!groups.has(key)) groups.set(key, []);
groups.get(key).push(String(label));
}
return groups;
}
/** Say whether anything is sharing. Pure. Returns [verdict, detail]. */
export function bucketVerdict(groups) {
if (!groups || groups.size === 0) {
return ['nothing-supplied',
'no identities were given, so the sharing question is unanswered. One ' +
'auth.test per process, saved with a label, is the whole input and it is ' +
'the evidence that ends the argument.'];
}
const shared = [...groups.entries()].filter(([, labels]) => labels.length > 1);
if (shared.length === 0) {
return ['separate-buckets',
`${groups.size} group(s), no two callers share a workspace and app`];
}
const biggest = shared.reduce((a, b) => (b[1].length > a[1].length ? b : a));
return ['one-bucket',
`${biggest[1].length} caller(s) share the key ${biggest[0]}: ` +
`${biggest[1].join(', ')}. Slack sees one app in one workspace, so they ` +
'hold one budget between them however many hosts they run on.'];
}
/** Total calls per minute against one method. Pure. */
export function aggregateRate(workers) {
let total = 0;
for (const worker of workers ?? []) {
const rate = Number(worker?.calls_per_minute ?? 0);
const replicas = Math.max(1, Math.trunc(Number(worker?.replicas ?? 1) || 1));
total += Math.max(0, Number.isFinite(rate) ? rate : 0) * replicas;
}
return total;
}
/** The pool's demand against the method's floor. Pure. Returns [verdict, share, detail]. */
export function fanoutVerdict(totalRate, tierFloor) {
const floor = Math.max(1, Number(tierFloor) || 1);
const share = Number(totalRate) / floor;
if (share > 1.0) {
return ['saturated', share,
`${totalRate.toFixed(1)}/min across the pool against a floor of ` +
`${floor.toFixed(0)}/min. The pool is asking for ${share.toFixed(1)} times ` +
'the budget, so its real throughput is the floor and the extra workers buy ' +
'queueing rather than work.'];
}
if (share > 0.8) {
return ['at-the-edge', share,
`${totalRate.toFixed(1)}/min across the pool against a floor of ` +
`${floor.toFixed(0)}/min. One more replica, or one retry storm, puts it over.`];
}
return ['headroom', share,
`${totalRate.toFixed(1)}/min against a floor of ${floor.toFixed(0)}/min`];
}
/**
* Callers you cannot see, inferred by subtraction. Pure.
* Returns [others, detail]; others is a multiple of your own rate, or null.
*/
export function impliedCompetitors(observedRate, tierFloor, throttled) {
const floor = Math.max(0, Number(tierFloor) || 0);
const observed = Math.max(0, Number(observedRate) || 0);
if (!throttled) {
return [null,
'nothing was throttled, so nothing is implied about other callers. A clean ' +
'sample is not proof the bucket is yours alone; it is proof it was not ' +
'empty in the seconds you looked.'];
}
if (observed <= 0) {
return [null, 'throttled at an observed rate of zero, which means the sample ' +
'never got a call away. Nothing to subtract from.'];
}
if (observed >= floor) {
return [null,
`throttled at ${observed.toFixed(1)}/min against a floor of ` +
`${floor.toFixed(0)}/min, which you reached on your own. This sample says ` +
'nothing about anybody else.'];
}
const hidden = floor - observed;
const others = hidden / observed;
return [others,
`throttled at ${observed.toFixed(1)}/min against a floor of ` +
`${floor.toFixed(0)}/min, so at least ${hidden.toFixed(1)} call(s) a minute ` +
'on this method came from somewhere other than this process. That is roughly ' +
`${others.toFixed(1)} more caller(s) at your rate, and finding them is the job.`];
}
/**
* How tightly the pool's retries land together. Pure.
* Returns [width, firstWave, verdict, detail].
*/
export function herdWindow(replicas, retryAfter, jitterFraction = 0) {
const count = Math.max(1, Math.trunc(Number(replicas) || 1));
const wait = Math.max(0, Number(retryAfter) || 0);
const jitter = Math.min(1, Math.max(0, Number(jitterFraction) || 0));
const width = wait * jitter;
if (width <= 0) {
return [0, count, 'thundering-herd',
`no jitter, so the retry window has zero width and all ${count} replica(s) ` +
'wake in the same millisecond. The first calls of the new window are spent ' +
'instantly and the rest are refused together, which re-forms the herd on ' +
'the next Retry-After.'];
}
const first = Math.max(1, Math.round(count * Math.min(1, BURST / width)));
const verdict = first >= 2 ? 'thundering-herd' : 'de-synchronised';
return [width, first, verdict,
`retries spread over ${width.toFixed(1)}s, about ${first} in the first ` +
`${(BURST * 1000).toFixed(0)}ms`];
}
const sleep = (ms) => new Promise((resolve) => { setTimeout(resolve, ms); });
/** A deliberately slow read loop. GET only, and spacing is floored. */
async function pacedSample(token, method, params, samples, spacing) {
const gap = Math.max(MIN_SPACING, Number(spacing) || MIN_SPACING);
const qs = new URLSearchParams(params);
let throttled = 0;
let made = 0;
const started = Date.now();
for (let i = 0; i < Math.max(1, Math.trunc(samples)); i += 1) {
if (i) await sleep(gap * 1000);
const res = await fetch(`${API}${method}?${qs}`, {
headers: { Authorization: `Bearer ${token}` },
});
made += 1;
let body = {};
try { body = await res.json(); } catch { body = {}; }
if (res.status === 429 || body.error === 'ratelimited') throttled += 1;
}
const elapsed = Math.max(1e-6, (Date.now() - started) / 1000);
return [made, throttled, (made / elapsed) * 60, gap];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function main() {
const args = process.argv.slice(2);
const tierFloor = Number(arg(args, '--tier-floor', 20));
let findings = 0;
const identitiesFile = arg(args, '--identities', '');
const identities = identitiesFile
? JSON.parse(await readFile(identitiesFile, 'utf8')) : [];
const groups = sameBucket(identities);
const [verdict, detail] = bucketVerdict(groups);
if (verdict === 'one-bucket') {
findings += 1;
console.warn(`bucket ${verdict.padEnd(17)} ${detail}`);
console.warn(' repair: one shared token bucket in front of Slack, in Redis or ' +
'in a single sender service. A limiter inside each process can only enforce ' +
'a rate it can see');
} else {
console.log(`bucket ${verdict.padEnd(17)} ${detail}`);
}
const workersFile = arg(args, '--workers', '');
if (workersFile) {
const workers = JSON.parse(await readFile(workersFile, 'utf8'));
const total = aggregateRate(workers);
const [fan, , fanDetail] = fanoutVerdict(total, tierFloor);
if (fan === 'headroom') {
console.log(`fanout ${fan.padEnd(17)} ${total.toFixed(1)}/min across ` +
`${workers.length} caller(s) against a floor of ${tierFloor}/min`);
} else {
findings += 1;
console.warn(`fanout ${fan.padEnd(17)} ${fanDetail}`);
console.warn(' repair: concurrency is a divisor here, not a multiplier. ' +
'Parallelise across channels or workspaces, which have more than one ' +
'bucket, rather than across workers on one method');
}
}
if (!args.includes('--no-sample')) {
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv}, or pass --no-sample`);
process.exitCode = 2;
return;
}
const res = await fetch(`${API}auth.test`, {
headers: { Authorization: `Bearer ${token}` },
});
const identity = await res.json();
if (identity.ok !== true) {
console.error(`auth.test answered ok: false, error=${identity.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${identity.user_id} bot ${identity.bot_id} in ` +
`${identity.team}`);
const [made, throttled, rate, gap] = await pacedSample(
token, arg(args, '--method', 'conversations.list'),
{ limit: '1', exclude_archived: 'true' },
Number(arg(args, '--samples', 5)), Number(arg(args, '--spacing', 5)));
console.log(`sample ${made} call(s) at ${gap.toFixed(1)}s spacing, ` +
`${throttled} throttled`);
const [others, why] = impliedCompetitors(rate, tierFloor, throttled > 0);
if (others === null) {
console.log(`competitors ${why}`);
} else {
findings += 1;
console.warn(`competitors ${why}`);
}
}
const replicas = Number(arg(args, '--replicas', 1));
const [, , herd, herdDetail] = herdWindow(replicas,
Number(arg(args, '--retry-after', 30)), Number(arg(args, '--jitter', 0)));
if (herd === 'thundering-herd' && replicas > 1) {
findings += 1;
console.warn(`herd ${herd.padEnd(17)} ${herdDetail}`);
console.warn(' repair: add jitter to every backoff. Uniform, correct, ' +
'documented backoff synchronises the pool, because Slack gives every ' +
'caller the same number');
} else {
console.log(`herd ${herd.padEnd(17)} ${herdDetail}`);
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
// Only run when invoked directly, so importing this module in the tests does not
// execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The fixture that matters is four auth.test bodies: two replicas of the same deployment, a cron box holding a second token of the same app, and a staging install in another workspace. The tests pin that the first three land in one group and the fourth does not, which is the whole claim. The rest pin the arithmetic: subtraction only produces a headcount when the sample was actually throttled below the floor, and uniform backoff is reported as a herd rather than as correct behaviour.
from slack_shared_quota_audit import (aggregate_rate, bucket_verdict,
fanout_verdict, herd_window,
implied_competitors, same_bucket)
# One app, one workspace, three processes. The third holds a different token of
# the same app, which is the case people are most confident is separate.
WORKER_A = {"ok": True, "team_id": "T0ACME1234", "bot_id": "B0APPBOT99",
"user_id": "U0APPBOT11"}
WORKER_B = {"ok": True, "team_id": "T0ACME1234", "bot_id": "B0APPBOT99",
"user_id": "U0APPBOT11"}
CRON_BOX = {"ok": True, "team_id": "T0ACME1234", "bot_id": "B0APPBOT99",
"user_id": "U0APPBOT11"}
STAGING = {"ok": True, "team_id": "T0STAGE999", "bot_id": "B0APPBOT99",
"user_id": "U0APPBOT11"}
def test_replicas_of_one_deployment_land_in_one_group():
groups = same_bucket([("worker-a", WORKER_A), ("worker-b", WORKER_B),
("cron", CRON_BOX)])
assert list(groups) == ["T0ACME1234/B0APPBOT99"]
assert groups["T0ACME1234/B0APPBOT99"] == ["worker-a", "worker-b", "cron"]
def test_a_different_workspace_is_a_different_bucket():
groups = same_bucket([("prod", WORKER_A), ("staging", STAGING)])
assert len(groups) == 2
def test_sharing_is_the_finding_and_is_named_with_its_members():
groups = same_bucket([("worker-a", WORKER_A), ("worker-b", WORKER_B),
("staging", STAGING)])
verdict, detail = bucket_verdict(groups)
assert verdict == "one-bucket"
assert "worker-a, worker-b" in detail
assert "staging" not in detail
def test_no_sharing_and_no_input_are_different_answers():
assert bucket_verdict(same_bucket([("prod", WORKER_A)]))[0] == "separate-buckets"
assert bucket_verdict({})[0] == "nothing-supplied"
def test_a_user_token_without_a_bot_id_still_gets_a_key():
groups = same_bucket([("script", {"team_id": "T0ACME1234", "user_id": "U0HUMAN1"})])
assert list(groups) == ["T0ACME1234/U0HUMAN1"]
assert list(same_bucket([("mystery", {})])) == ["unknown-team/unknown-app"]
def test_replicas_multiply_the_rate_they_were_given():
workers = [{"label": "worker", "calls_per_minute": 5, "replicas": 8},
{"label": "cron", "calls_per_minute": 2}]
assert aggregate_rate(workers) == 42.0
assert aggregate_rate([]) == 0.0
assert aggregate_rate([{"calls_per_minute": "many"}]) == 0.0
def test_the_pool_is_saturated_when_the_total_passes_the_floor():
verdict, share, detail = fanout_verdict(42.0, 20)
assert verdict == "saturated"
assert round(share, 1) == 2.1
assert "divisor" not in detail
assert "queueing rather than work" in detail
def test_the_edge_and_the_headroom_are_separate_verdicts():
assert fanout_verdict(18.0, 20)[0] == "at-the-edge"
assert fanout_verdict(8.0, 20)[0] == "headroom"
def test_throttling_below_the_floor_implies_callers_you_cannot_see():
others, detail = implied_competitors(12.0, 20, True)
assert round(others, 2) == 0.67
assert "8.0 call(s) a minute" in detail
def test_a_clean_sample_implies_nothing_at_all():
others, detail = implied_competitors(12.0, 20, False)
assert others is None
assert "not proof the bucket is yours alone" in detail
def test_reaching_the_floor_yourself_is_not_evidence_of_anybody_else():
others, _ = implied_competitors(25.0, 20, True)
assert others is None
assert implied_competitors(0.0, 20, True)[0] is None
def test_uniform_backoff_is_reported_as_a_herd():
width, first, verdict, detail = herd_window(8, 30, 0.0)
assert width == 0.0
assert first == 8
assert verdict == "thundering-herd"
assert "same millisecond" in detail
def test_jitter_spreads_the_pool_across_the_window():
width, first, verdict, _ = herd_window(8, 30, 0.5)
assert width == 15.0
assert first == 1
assert verdict == "de-synchronised"
def test_a_single_replica_cannot_form_a_herd_worth_reporting():
assert herd_window(1, 30, 0.0)[1] == 1
assert herd_window("eight", "soon", "none")[2] == "thundering-herd"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { aggregateRate, bucketVerdict, fanoutVerdict, herdWindow,
impliedCompetitors, sameBucket } from './slack-shared-quota-audit.mjs';
// One app, one workspace, three processes. The third holds a different token of
// the same app, which is the case people are most confident is separate.
const WORKER_A = { ok: true, team_id: 'T0ACME1234', bot_id: 'B0APPBOT99',
user_id: 'U0APPBOT11' };
const WORKER_B = { ...WORKER_A };
const CRON_BOX = { ...WORKER_A };
const STAGING = { ok: true, team_id: 'T0STAGE999', bot_id: 'B0APPBOT99',
user_id: 'U0APPBOT11' };
test('replicas of one deployment land in one group', () => {
const groups = sameBucket([['worker-a', WORKER_A], ['worker-b', WORKER_B],
['cron', CRON_BOX]]);
assert.deepEqual([...groups.keys()], ['T0ACME1234/B0APPBOT99']);
assert.deepEqual(groups.get('T0ACME1234/B0APPBOT99'),
['worker-a', 'worker-b', 'cron']);
});
test('a different workspace is a different bucket', () => {
assert.equal(sameBucket([['prod', WORKER_A], ['staging', STAGING]]).size, 2);
});
test('sharing is the finding and is named with its members', () => {
const groups = sameBucket([['worker-a', WORKER_A], ['worker-b', WORKER_B],
['staging', STAGING]]);
const [verdict, detail] = bucketVerdict(groups);
assert.equal(verdict, 'one-bucket');
assert.match(detail, /worker-a, worker-b/);
assert.ok(!detail.includes('staging'));
});
test('no sharing and no input are different answers', () => {
assert.equal(bucketVerdict(sameBucket([['prod', WORKER_A]]))[0], 'separate-buckets');
assert.equal(bucketVerdict(new Map())[0], 'nothing-supplied');
});
test('a user token without a bot id still gets a key', () => {
const groups = sameBucket([['script', { team_id: 'T0ACME1234', user_id: 'U0HUMAN1' }]]);
assert.deepEqual([...groups.keys()], ['T0ACME1234/U0HUMAN1']);
assert.deepEqual([...sameBucket([['mystery', {}]]).keys()],
['unknown-team/unknown-app']);
});
test('replicas multiply the rate they were given', () => {
const workers = [{ label: 'worker', calls_per_minute: 5, replicas: 8 },
{ label: 'cron', calls_per_minute: 2 }];
assert.equal(aggregateRate(workers), 42);
assert.equal(aggregateRate([]), 0);
assert.equal(aggregateRate([{ calls_per_minute: 'many' }]), 0);
});
test('the pool is saturated when the total passes the floor', () => {
const [verdict, share, detail] = fanoutVerdict(42, 20);
assert.equal(verdict, 'saturated');
assert.equal(Number(share.toFixed(1)), 2.1);
assert.match(detail, /queueing rather than work/);
});
test('the edge and the headroom are separate verdicts', () => {
assert.equal(fanoutVerdict(18, 20)[0], 'at-the-edge');
assert.equal(fanoutVerdict(8, 20)[0], 'headroom');
});
test('throttling below the floor implies callers you cannot see', () => {
const [others, detail] = impliedCompetitors(12, 20, true);
assert.equal(Number(others.toFixed(2)), 0.67);
assert.match(detail, /8\.0 call\(s\) a minute/);
});
test('a clean sample implies nothing at all', () => {
const [others, detail] = impliedCompetitors(12, 20, false);
assert.equal(others, null);
assert.match(detail, /not proof the bucket is yours alone/);
});
test('reaching the floor yourself is not evidence of anybody else', () => {
assert.equal(impliedCompetitors(25, 20, true)[0], null);
assert.equal(impliedCompetitors(0, 20, true)[0], null);
});
test('uniform backoff is reported as a herd', () => {
const [width, first, verdict, detail] = herdWindow(8, 30, 0);
assert.equal(width, 0);
assert.equal(first, 8);
assert.equal(verdict, 'thundering-herd');
assert.match(detail, /same millisecond/);
});
test('jitter spreads the pool across the window', () => {
const [width, first, verdict] = herdWindow(8, 30, 0.5);
assert.equal(width, 15);
assert.equal(first, 1);
assert.equal(verdict, 'de-synchronised');
});
test('a single replica cannot form a herd worth reporting', () => {
assert.equal(herdWindow(1, 30, 0)[1], 1);
assert.equal(herdWindow('eight', 'soon', 'none')[2], 'thundering-herd');
});
FAQ
Does giving each worker its own token split the quota?
No. Slack keys the limit on the app and the workspace, not on the token, so two tokens belonging to the same app in the same workspace draw on one bucket. That is the case people are most confident is separate, and it is the reason the audit groups on team_id and bot_id rather than on the credential.
How do I find the caller nobody remembers deploying?
By subtraction. Take a paced sample well under the documented floor; if it is refused anyway, the difference between the floor and your own rate is traffic from somewhere else. Expressed as a multiple of your rate it is roughly a headcount, and it is usually a staging deployment still pointed at production, an analytics cron, or a laptop.
Is exponential backoff enough if every worker implements it correctly?
Correctly is the problem. Slack hands every refused caller the same Retry-After, so identical, correct backoff makes the pool wake in lockstep and empty the new window in its first milliseconds. Jitter is not a refinement here; without it the workers cannot de-synchronise, and the herd re-forms on every refusal.
Where should the shared limiter live?
Somewhere all callers meet: a token bucket in Redis, or a single sender service everything else queues into. The rule is that the limiter has to see every call the app makes, which no in-process limiter can. A sender service is usually the smaller change, because it also gives you one place to add batching and one place to log.
Is there any axis where more concurrency does help?
Yes, where there is genuinely more than one bucket. chat.postMessage is limited per channel, so posting into N channels means N envelopes and parallelism is real. Separate workspaces are separate buckets too. Across workers hitting the same method in the same workspace there is exactly one, and adding workers divides it.
Related field notes
- the budget one process is spending
- the header every replica reads the same way
- installs keyed on team_id alone collide
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits — Slack Docs
- auth.test method reference — Slack Docs
- conversations.list method reference — Slack Docs
- Using the Web API — Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.