Diagnostic Slack
One missed ack becomes four deliveries and a 429
It is fine at ten events a minute and it is fine at forty, and then one afternoon it is not fine at anything. The handler slows down a little, some deliveries miss the deadline, Slack sends them again, the extra deliveries make the same API calls, the calls start coming back ratelimited, the handler waits on them, and now it is missing far more deadlines than it was a minute ago.
Nothing in that sequence is a bug. Every component is behaving exactly as designed, and together they converge on total failure and stay there until the traffic stops.
Slack retries a delivery it believes failed up to three times, at roughly zero, sixty and three hundred seconds. That is a ceiling of four deliveries per event, so the amplification is bounded and small — on its own it would be a nuisance. What makes it a storm is the second multiplier: each delivery runs your handler, and each run of your handler makes some number of Web API calls. Four deliveries times three calls is twelve calls where you budgeted three, against a per-method quota that did not grow.
So this note is arithmetic rather than a state to detect. Events per minute, times deliveries per event, times calls per handler run, held against the method's budget. And then the part people miss: the miss rate is not an input, it is an output. Being throttled makes handlers slower, slower handlers miss more deadlines, more misses mean more deliveries, and the script iterates that loop until it settles — which it always does, because the retry ceiling bounds it. Where it settles is the finding. The same forty events a minute settle at a five percent miss rate with one API call per handler run and at ninety-three percent with three, and ninety-three percent is not a slow app: it is a hair under the failure rate at which Slack switches your event subscriptions off.
The problem in plain words
The reason this arrives as a surprise is that every number involved looks fine in isolation. Forty events a minute is nothing. Three API calls per event is modest. A five percent timeout rate is a good day. A Tier 3 method allows fifty requests a minute. Every one of those is defensible in a design review and the combination is over the line by a factor of two.
What makes it a storm rather than a shortfall is that one of the four numbers depends on the other three. If the demand exceeds the budget, calls start being refused, and a handler that waits on a refused call takes longer, and a handler that takes longer misses the three second acknowledgement, and a missed acknowledgement produces another delivery, which makes another set of calls. The miss rate that started at five percent is now thirty, then sixty. There is no component in that loop that is misbehaving and no log line that says what happened.
It also recovers strangely, which is the tell. The system does not degrade gracefully and then improve as load falls; it collapses, stays collapsed while the sixty-second and three-hundred-second retries keep arriving from events that are already minutes old, and then comes back all at once when the backlog is finally exhausted. People describe it as the service "unsticking itself", which is exactly what a feedback loop looks like from outside.
Why it happens
The amplification is bounded at four and the consequence is not. Three retries is the ceiling and it never goes higher, so the tempting conclusion is that the worst case is a four times spike, which any system with headroom absorbs. That is true of the deliveries and false of the effect, because the four times lands on a quota that four times exceeded then feeds back into the thing producing the retries.
The loop has an equilibrium, and the equilibrium is the problem. Because the amplification is capped, the feedback cannot run away to infinity; it settles. What the arithmetic shows, and what makes this worth computing rather than intuiting, is that the settling point for an ordinary over-budget configuration is a miss rate in the nineties. That is not a degraded service. It is within a percentage point or two of the sustained failure rate at which Slack disables event delivery outright — which is why an app that survives one storm sometimes does not survive the next.
The retry spacing is why the pain outlasts the incident. Deliveries at sixty and three hundred seconds mean that five minutes after the load spike ends you are still receiving retries of events from before it started, all of them still making calls. The tail is longer than the incident, and a graph of API calls will show a second and third hump that no traffic in your product explains.
Per-method, per-workspace, per-app is what the budget is scoped to. Not per handler, and not per instance. Adding replicas does not add quota, so scaling out to serve the extra deliveries makes the throttling worse rather than better, which is the single most common wrong move at this point in the incident.
The multiplicand you control is the calls per handler run. You cannot change the retry schedule and you should not want to change the deadline. What you can change is how many API calls one delivery makes: a handler that posts a message, then updates it, then looks up two users, is four calls where one would do, and that four is multiplied by everything else in the chain.
An audit must not demonstrate this. Provoking the loop to prove it exists means exhausting a real workspace's quota for every app installed in it, yours and everybody else's. The script below computes and iterates a model; the only thing it measures is how many messages the workspace already produced, which it reads out of history.
The fix, as a flow
The one chain here with an arrow that goes backwards, and the arrow is the mechanism rather than a flourish: the box that fails is the input to the box that caused it. Every step in it is a component behaving as designed. The fix branch is arithmetic with a ladder at the end, and the top rung is not a performance verdict at all.
How to fix it
Measure how many events a minute the app is really on the hook for
One page of conversations.history in a sample of the channels the bot belongs to, and the span between the oldest and newest ts, gives messages per minute. For an app subscribed to message.channels that is the delivery rate directly. It is a floor rather than a census, and the script says so.
Count the API calls one handler run makes
Read the handler and count them: every chat.postMessage, every lookup, every update. This is the one number in the model that is nobody's estimate, and it is also the one you can change this afternoon.
Expand the deliveries for the miss rate you have today
One event is one delivery plus a retry for each miss, capped at three. At a five percent miss rate that is 1.05 deliveries per event, which is nothing. At forty percent it is 1.7, and at eighty percent it is 3.4. The function is not linear in the miss rate and that is exactly why it feels like a cliff.
Hold the product against the method's budget
Events times deliveries times calls, against requests per minute for the method your handler leans on hardest. Under half the budget is fine. Over the budget is not a warning, it is the point at which some of your calls are already being refused.
Iterate it, because the miss rate is an output
Feed the throttled fraction back into the miss rate and run it again, and again. It will settle; the ceiling on retries guarantees that. The question is where, and the four answers are not variations on each other: unchanged, elevated, collapsed at a fixed point that happens to be failure, or settled above the rate at which delivery is switched off. The script prints the path so the walk is visible rather than asserted.
Cut a multiplicand rather than adding capacity
Fewer calls per handler run, or fewer events arriving, or a narrower subscription. More replicas add nothing because the quota is per app, not per process. And the deliveries factor closes itself once the deadline is met, which is a different note and the one to read next.
How to check it worked
Re-run with the reduced call count. The event rate will be identical, and the loop should settle instead of walking to the ceiling.
python3 slack_retry_amplification.py --calls-per-event 1 --miss-rate 0.05 --tier-per-minute 50
# rate observed 38.4 message(s)/minute across 5 channel(s), a floor
# demand tight 40.4 call(s)/minute against a budget of 50 (80.8%)
# loop stable settles at a miss rate of 0.05 after 1 round(s)
# loop path 0.05 -> 0.05
# verdict ok no feedback: the throttled fraction stays at zeroThe full code
Three pure functions and one page of history per sampled channel. deliveries_per_event is the retry schedule as a geometric sum capped at three. demand multiplies the three factors and holds the product against the budget. converge is the note's actual claim: it feeds the throttled fraction back into the miss rate and iterates until the loop settles, and settle_verdict reads the ladder off the fixed point, whose top rung is not slowness but the failure rate that gets an app switched off.
"""Model the retry amplification of a Slack event handler against its quota.
Read only, and it does not demonstrate the problem it describes: provoking a
retry storm means exhausting a real workspace's per-method quota for every app
installed in it. The only measurement taken is how many messages the workspace
already produced, read from one page of history per sampled channel. Everything
else is arithmetic over numbers the caller supplies.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_retry_amplification")
API = "https://slack.com/api/"
# Slack retries a delivery it believes failed three times: immediately, after
# about a minute, and after about five minutes. Four deliveries is the ceiling
# and the spacing is why the tail outlasts the incident.
MAX_RETRIES = 3
RETRY_SCHEDULE_SECONDS = (0, 60, 300)
# The miss rate is capped just short of one so the arithmetic stays meaningful.
# The number that matters more is the other one: Slack disables an app's event
# subscriptions when it fails more than 95% of deliveries in an hour, so a loop
# whose equilibrium sits above that does not end in a slow app.
MISS_CAP = 0.99
DISABLE_THRESHOLD = 0.95
def deliveries_per_event(miss_rate, max_retries=MAX_RETRIES):
"""Expected deliveries for one event at a given miss rate. Pure.
A geometric sum truncated at the retry ceiling. It is deliberately not
linear: five percent gives 1.05 and eighty percent gives 3.4, and the
distance between those is why this feels like a cliff rather than a slope.
"""
p = min(max(float(miss_rate or 0.0), 0.0), 1.0)
total = 1.0
term = 1.0
for _ in range(max(int(max_retries or 0), 0)):
term *= p
total += term
return round(total, 4)
def demand(events_per_minute, calls_per_event, miss_rate, tier_per_minute):
"""Multiply the three factors and hold the product against the budget. Pure.
Returns the demanded call rate, the share of the budget it takes, and the
fraction of calls that must be refused. That last number is the input to
the feedback loop, which is what makes this more than a multiplication.
"""
events = max(float(events_per_minute or 0.0), 0.0)
calls = max(float(calls_per_event or 0.0), 0.0)
budget = max(float(tier_per_minute or 0.0), 1.0)
per_event = deliveries_per_event(miss_rate)
demanded = events * per_event * calls
share = round(demanded * 100.0 / budget, 1)
throttled = 0.0 if demanded <= budget else round(1.0 - (budget / demanded), 4)
if demanded > budget:
verdict = "over-budget"
elif share >= 80.0:
verdict = "saturated"
elif share >= 50.0:
verdict = "tight"
else:
verdict = "stable"
return {"deliveries_per_event": per_event, "calls_per_minute": round(demanded, 1),
"budget_per_minute": round(budget, 1), "share_percent": share,
"throttled_fraction": throttled, "verdict": verdict}
def settle_verdict(final_miss_rate, base_miss_rate):
"""Name the equilibrium the loop arrived at. Pure.
The loop always has a fixed point, because the retry ceiling bounds the
amplification. Whether that fixed point is somewhere you can live is the
entire question, and the top rung is not a performance problem: an app
failing more than 95% of deliveries for an hour has its event
subscriptions switched off, and they do not come back on their own.
"""
final = float(final_miss_rate or 0.0)
base = float(base_miss_rate or 0.0)
if final <= base + 0.01:
return ("stable", "no feedback: the throttled fraction never leaves zero")
if final >= DISABLE_THRESHOLD:
return ("disabling",
"the equilibrium is above the %d%% failure rate that gets event "
"subscriptions switched off. This loop does not end in a slow "
"app, it ends in a disabled one." % int(DISABLE_THRESHOLD * 100))
if final >= 0.5:
return ("collapsed",
"the loop has a fixed point and the fixed point is failure: "
"most deliveries miss, most events are retried, and it stays "
"there until the traffic stops")
return ("elevated",
"the feedback is real and the equilibrium is survivable; there is "
"no headroom left for the next increase in traffic")
def converge(events_per_minute, calls_per_event, base_miss_rate, tier_per_minute,
rounds=12):
"""Feed the throttled fraction back into the miss rate and iterate. Pure.
The claim of this note is that the miss rate is an output rather than an
input: throttled calls make handlers slower, slower handlers miss the three
second deadline, and missed deadlines produce the deliveries that caused
the throttling. The retry ceiling means this always settles somewhere. The
path is returned so the walk can be read rather than asserted.
"""
p = min(max(float(base_miss_rate or 0.0), 0.0), MISS_CAP)
base = p
path = [round(p, 4)]
for step in range(max(int(rounds or 0), 1)):
state = demand(events_per_minute, calls_per_event, p, tier_per_minute)
nxt = min(base + state["throttled_fraction"], MISS_CAP)
path.append(round(nxt, 4))
if abs(nxt - p) < 0.001:
verdict, detail = settle_verdict(nxt, base)
return {"verdict": verdict, "detail": detail,
"miss_rate": round(nxt, 4), "rounds": step + 1, "path": path,
"deliveries_per_event": deliveries_per_event(nxt)}
p = nxt
return {"verdict": "unsettled",
"detail": "still climbing after %d round(s), which is the same "
"finding as collapse with less arithmetic" % int(rounds),
"miss_rate": round(p, 4), "rounds": int(rounds), "path": path,
"deliveries_per_event": deliveries_per_event(p)}
def observed_rate(timestamps):
"""Messages per minute across a sample of history. Pure.
A floor, not a census: one page per channel, and a workspace whose history
calls are clamped returns fewer objects than were asked for. Both of those
bias the number down, never up, which is the safe direction for a model
whose output is a warning.
"""
values = sorted(float(t) for t in timestamps or [] if t)
if len(values) < 2:
return {"count": len(values), "span_seconds": 0.0, "per_minute": 0.0,
"note": "not enough messages in the sample to derive a rate"}
span = values[-1] - values[0]
if span <= 0:
return {"count": len(values), "span_seconds": 0.0, "per_minute": 0.0,
"note": "every sampled message shares a timestamp"}
per_minute = round(len(values) * 60.0 / span, 1)
return {"count": len(values), "span_seconds": round(span, 1),
"per_minute": per_minute,
"note": "a floor: one page per channel, so the real rate is higher"}
def get(session, method, **params):
r = session.get(API + method, params=params, timeout=30)
try:
return r.json()
except ValueError:
return {"ok": False, "error": "unparseable_body"}
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN")
ap.add_argument("--calls-per-event", type=float, default=3.0,
help="Web API calls one handler run makes; count them")
ap.add_argument("--miss-rate", type=float, default=0.05,
help="fraction of deliveries that miss the three second ack")
ap.add_argument("--tier-per-minute", type=float, default=50.0,
help="requests a minute the busiest method in the handler allows")
ap.add_argument("--events-per-minute", type=float, default=0.0,
help="override the measured rate instead of sampling history")
ap.add_argument("--channels", type=int, default=5,
help="how many of the bot's channels to sample")
args = ap.parse_args()
events = args.events_per_minute
token = os.environ.get(args.token_env)
if events <= 0 and not token:
log.error("set %s or pass --events-per-minute", args.token_env)
return 2
if events <= 0:
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = get(s, "auth.test")
if who.get("ok") is not True:
log.error("auth.test answered 200 with ok: false, error=%s", who.get("error"))
return 2
mine = get(s, "users.conversations", user=who.get("user_id", ""),
types="public_channel,private_channel", limit=200)
ids = [c.get("id") for c in (mine.get("channels") or [])][:max(args.channels, 1)]
stamps, sampled = [], 0
for cid in ids:
page = get(s, "conversations.history", channel=cid, limit=200)
if page.get("ok") is not True:
log.warning("rate skipped %s on %s", page.get("error"), cid)
continue
sampled += 1
stamps.extend(m.get("ts") for m in (page.get("messages") or []))
rate = observed_rate(stamps)
events = rate["per_minute"]
log.info("rate observed %s message(s)/minute across %d "
"channel(s), %s", rate["per_minute"], sampled, rate["note"])
state = demand(events, args.calls_per_event, args.miss_rate, args.tier_per_minute)
(log.info if state["verdict"] == "stable" else log.warning)(
"demand %-14s %.1f call(s)/minute against a budget of %.0f (%.1f%%)",
state["verdict"], state["calls_per_minute"], state["budget_per_minute"],
state["share_percent"])
log.info("deliveries %-14s %.2f per event at a miss rate of %.2f, retried at "
"%s seconds", "expanded", state["deliveries_per_event"], args.miss_rate,
", ".join(str(x) for x in RETRY_SCHEDULE_SECONDS))
loop = converge(events, args.calls_per_event, args.miss_rate, args.tier_per_minute)
(log.info if loop["verdict"] == "stable" else log.warning)(
"loop %-14s settles at a miss rate of %.2f after %d round(s), "
"%.2f deliveries/event", loop["verdict"], loop["miss_rate"],
loop["rounds"], loop["deliveries_per_event"])
log.info("loop path %s", " -> ".join("%.2f" % p for p in loop["path"]))
(log.info if loop["verdict"] == "stable" else log.warning)(
"loop %-14s %s", "meaning", loop["detail"])
if loop["verdict"] == "stable" and state["verdict"] in ("stable", "tight"):
log.info("verdict %-14s no feedback: the throttled fraction stays at "
"zero", "ok")
return 0
log.warning(" repair: cut the calls one handler run makes. That factor "
"multiplies everything else in the chain and it is the only one "
"of the four you can change today")
log.warning(" repair: narrow the subscription so fewer events arrive at all; "
"an app that receives every message in every channel is paying "
"the multiplier on traffic it will never act on")
log.warning(" repair: do not add replicas. The quota is per method, per "
"workspace, per app, so another process shares the same bucket "
"and arrives at the same refusals sooner")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Model the retry amplification of a Slack event handler against its quota.
*
* Read only, and it does not demonstrate the problem it describes. The only
* measurement taken is how many messages the workspace already produced, read
* from one page of history per sampled channel. Everything else is arithmetic
* over numbers the caller supplies.
*/
const API = 'https://slack.com/api/';
// Three retries: immediately, after about a minute, and after about five.
const MAX_RETRIES = 3;
const RETRY_SCHEDULE_SECONDS = [0, 60, 300];
// The miss rate is capped just short of one so the arithmetic stays meaningful.
// The number that matters more is the other one: Slack disables an app's event
// subscriptions when it fails more than 95% of deliveries in an hour.
const MISS_CAP = 0.99;
const DISABLE_THRESHOLD = 0.95;
/**
* Expected deliveries for one event at a given miss rate. Pure.
* A geometric sum truncated at the retry ceiling, and deliberately not linear.
*/
export function deliveriesPerEvent(missRate, maxRetries = MAX_RETRIES) {
const p = Math.min(Math.max(Number(missRate) || 0, 0), 1);
let total = 1;
let term = 1;
for (let i = 0; i < Math.max(Number(maxRetries) || 0, 0); i += 1) {
term *= p;
total += term;
}
return Math.round(total * 10000) / 10000;
}
/**
* Multiply the three factors and hold the product against the budget. Pure.
* The throttled fraction is the input to the feedback loop, which is what
* makes this more than a multiplication.
*/
export function demand(eventsPerMinute, callsPerEvent, missRate, tierPerMinute) {
const events = Math.max(Number(eventsPerMinute) || 0, 0);
const calls = Math.max(Number(callsPerEvent) || 0, 0);
const budget = Math.max(Number(tierPerMinute) || 0, 1);
const perEvent = deliveriesPerEvent(missRate);
const demanded = events * perEvent * calls;
const share = Math.round((demanded * 1000.0) / budget) / 10;
const throttled = demanded <= budget
? 0 : Math.round((1 - budget / demanded) * 10000) / 10000;
let verdict = 'stable';
if (demanded > budget) verdict = 'over-budget';
else if (share >= 80.0) verdict = 'saturated';
else if (share >= 50.0) verdict = 'tight';
return {
deliveriesPerEvent: perEvent,
callsPerMinute: Math.round(demanded * 10) / 10,
budgetPerMinute: Math.round(budget * 10) / 10,
sharePercent: share,
throttledFraction: throttled,
verdict,
};
}
/**
* Name the equilibrium the loop arrived at. Pure.
* The loop always has a fixed point, because the retry ceiling bounds the
* amplification. Whether that fixed point is somewhere you can live is the
* whole question, and the top rung is not a performance problem.
*/
export function settleVerdict(finalMissRate, baseMissRate) {
const final = Number(finalMissRate) || 0;
const base = Number(baseMissRate) || 0;
if (final <= base + 0.01) {
return ['stable', 'no feedback: the throttled fraction never leaves zero'];
}
if (final >= DISABLE_THRESHOLD) {
return ['disabling',
`the equilibrium is above the ${Math.round(DISABLE_THRESHOLD * 100)}% ` +
'failure rate that gets event subscriptions switched off. This loop does ' +
'not end in a slow app, it ends in a disabled one.'];
}
if (final >= 0.5) {
return ['collapsed',
'the loop has a fixed point and the fixed point is failure: most ' +
'deliveries miss, most events are retried, and it stays there until the ' +
'traffic stops'];
}
return ['elevated',
'the feedback is real and the equilibrium is survivable; there is no ' +
'headroom left for the next increase in traffic'];
}
/**
* Feed the throttled fraction back into the miss rate and iterate. Pure.
* The miss rate is an output rather than an input. The retry ceiling means
* this always settles somewhere; where it settles is the finding.
*/
export function converge(eventsPerMinute, callsPerEvent, baseMissRate,
tierPerMinute, rounds = 12) {
let p = Math.min(Math.max(Number(baseMissRate) || 0, 0), MISS_CAP);
const base = p;
const path = [Math.round(p * 10000) / 10000];
const limit = Math.max(Number(rounds) || 0, 1);
for (let step = 0; step < limit; step += 1) {
const state = demand(eventsPerMinute, callsPerEvent, p, tierPerMinute);
const next = Math.min(base + state.throttledFraction, MISS_CAP);
path.push(Math.round(next * 10000) / 10000);
if (Math.abs(next - p) < 0.001) {
const [verdict, detail] = settleVerdict(next, base);
return {
verdict,
detail,
missRate: Math.round(next * 10000) / 10000,
rounds: step + 1,
path,
deliveriesPerEvent: deliveriesPerEvent(next),
};
}
p = next;
}
return {
verdict: 'unsettled',
detail: `still climbing after ${limit} round(s), which is the same finding ` +
'as collapse with less arithmetic',
missRate: Math.round(p * 10000) / 10000,
rounds: limit,
path,
deliveriesPerEvent: deliveriesPerEvent(p),
};
}
/**
* Messages per minute across a sample of history. Pure.
* A floor, not a census: one page per channel, and a clamped history returns
* fewer objects than were asked for. Both bias the number down.
*/
export function observedRate(timestamps) {
const values = (timestamps ?? []).filter(Boolean).map(Number).sort((a, b) => a - b);
if (values.length < 2) {
return {
count: values.length,
spanSeconds: 0,
perMinute: 0,
note: 'not enough messages in the sample to derive a rate',
};
}
const span = values[values.length - 1] - values[0];
if (span <= 0) {
return {
count: values.length,
spanSeconds: 0,
perMinute: 0,
note: 'every sampled message shares a timestamp',
};
}
return {
count: values.length,
spanSeconds: Math.round(span * 10) / 10,
perMinute: Math.round((values.length * 600.0) / span) / 10,
note: 'a floor: one page per channel, so the real rate is higher',
};
}
async function get(token, method, params) {
const qs = new URLSearchParams(params);
const res = await fetch(`${API}${method}?${qs}`, {
headers: { Authorization: `Bearer ${token}` },
});
try { return await res.json(); } catch { return { ok: false, error: 'unparseable_body' }; }
}
function arg(args, name, fallback = '') {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function main() {
const args = process.argv.slice(2);
const callsPerEvent = Number(arg(args, '--calls-per-event', 3));
const missRate = Number(arg(args, '--miss-rate', 0.05));
const tierPerMinute = Number(arg(args, '--tier-per-minute', 50));
const channels = Number(arg(args, '--channels', 5));
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
let events = Number(arg(args, '--events-per-minute', 0));
if (events <= 0 && !token) {
console.error(`set ${tokenEnv} or pass --events-per-minute`);
process.exitCode = 2;
return;
}
if (events <= 0) {
const who = await get(token, 'auth.test', {});
if (who.ok !== true) {
console.error(`auth.test answered 200 with ok: false, error=${who.error}`);
process.exitCode = 2;
return;
}
const mine = await get(token, 'users.conversations', {
user: who.user_id ?? '', types: 'public_channel,private_channel', limit: '200',
});
const ids = (mine.channels ?? []).map((c) => c.id).slice(0, Math.max(channels, 1));
const stamps = [];
let sampled = 0;
for (const cid of ids) {
// eslint-disable-next-line no-await-in-loop
const page = await get(token, 'conversations.history', { channel: cid, limit: '200' });
if (page.ok !== true) {
console.warn(`rate skipped ${page.error} on ${cid}`);
continue;
}
sampled += 1;
for (const m of page.messages ?? []) stamps.push(m.ts);
}
const rate = observedRate(stamps);
events = rate.perMinute;
console.log(`rate observed ${rate.perMinute} message(s)/minute ` +
`across ${sampled} channel(s), ${rate.note}`);
}
const state = demand(events, callsPerEvent, missRate, tierPerMinute);
(state.verdict === 'stable' ? console.log : console.warn)(
`demand ${state.verdict.padEnd(14)} ${state.callsPerMinute} call(s)/minute ` +
`against a budget of ${state.budgetPerMinute} (${state.sharePercent}%)`);
console.log(`deliveries expanded ${state.deliveriesPerEvent} per event at a ` +
`miss rate of ${missRate}, retried at ${RETRY_SCHEDULE_SECONDS.join(', ')} seconds`);
const loop = converge(events, callsPerEvent, missRate, tierPerMinute);
const calm = loop.verdict === 'stable';
(calm ? console.log : console.warn)(
`loop ${loop.verdict.padEnd(14)} settles at a miss rate of ` +
`${loop.missRate} after ${loop.rounds} round(s), ${loop.deliveriesPerEvent} ` +
'deliveries/event');
console.log(`loop path ${loop.path.map((p) => p.toFixed(2)).join(' -> ')}`);
(calm ? console.log : console.warn)(`loop meaning ${loop.detail}`);
if (calm && (state.verdict === 'stable' || state.verdict === 'tight')) {
console.log('verdict ok no feedback: the throttled fraction ' +
'stays at zero');
return;
}
console.warn(' repair: cut the calls one handler run makes. That factor ' +
'multiplies everything else in the chain and it is the only one of the four ' +
'you can change today');
console.warn(' repair: narrow the subscription so fewer events arrive at all; ' +
'an app that receives every message in every channel is paying the multiplier ' +
'on traffic it will never act on');
console.warn(' repair: do not add replicas. The quota is per method, per ' +
'workspace, per app, so another process shares the same bucket and arrives ' +
'at the same refusals sooner');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The tests pin the cliff and the fixed point, because those are the two claims. Deliveries per event is asserted to be almost nothing at a five percent miss rate and nearly three at eighty percent, from the same bounded schedule, which is the non-linearity people budget as if it were linear. Then the same forty events a minute are asserted to settle at five percent with one call per handler run and in the nineties with three, and a heavier configuration is asserted to settle above the failure rate that gets event delivery switched off — the rung where this stops being a performance finding.
from slack_retry_amplification import (converge, deliveries_per_event, demand,
observed_rate, settle_verdict)
def test_the_amplification_is_bounded_at_four_deliveries():
assert deliveries_per_event(1.0) == 4.0
assert deliveries_per_event(0.0) == 1.0
def test_the_expansion_is_a_cliff_rather_than_a_slope():
assert deliveries_per_event(0.05) < 1.06
assert deliveries_per_event(0.4) > 1.6
assert deliveries_per_event(0.8) > 2.9
def test_a_saturated_handler_has_nothing_refused_yet():
state = demand(40, 1, 0.05, 50)
assert state["verdict"] == "saturated"
assert state["share_percent"] > 80
assert state["throttled_fraction"] == 0.0
def test_three_calls_on_the_same_traffic_is_over_the_same_budget():
state = demand(40, 3, 0.05, 50)
assert state["verdict"] == "over-budget"
assert state["calls_per_minute"] > 50
assert state["throttled_fraction"] > 0.5
def test_the_throttled_fraction_is_what_feeds_the_loop():
assert demand(40, 3, 0.05, 50)["throttled_fraction"] > demand(
40, 2, 0.05, 50)["throttled_fraction"]
def test_the_same_traffic_is_stable_at_one_call_and_collapses_at_three():
assert converge(40, 1, 0.05, 50)["verdict"] == "stable"
assert converge(40, 3, 0.05, 50)["verdict"] == "collapsed"
def test_a_stable_loop_keeps_the_miss_rate_it_started_with():
loop = converge(40, 1, 0.05, 50)
assert loop["miss_rate"] == 0.05
assert loop["rounds"] == 1
def test_the_collapsed_loop_still_has_a_fixed_point():
loop = converge(40, 3, 0.05, 50)
assert loop["path"][0] == 0.05
assert loop["path"] == sorted(loop["path"])
assert 0.9 < loop["miss_rate"] < 0.95
assert loop["deliveries_per_event"] > 3.5
def test_enough_load_settles_above_the_rate_that_gets_you_switched_off():
loop = converge(200, 5, 0.05, 50)
assert loop["verdict"] == "disabling"
assert loop["miss_rate"] >= 0.95
assert "switched off" in loop["detail"]
def test_the_ladder_is_read_off_the_fixed_point_not_the_journey():
assert settle_verdict(0.05, 0.05)[0] == "stable"
assert settle_verdict(0.3, 0.05)[0] == "elevated"
assert settle_verdict(0.7, 0.05)[0] == "collapsed"
assert settle_verdict(0.96, 0.05)[0] == "disabling"
def test_zero_traffic_cannot_feed_back():
assert converge(0, 10, 0.05, 50)["verdict"] == "stable"
def test_a_rate_needs_two_messages_and_a_span():
assert observed_rate([])["per_minute"] == 0.0
assert observed_rate(["1735689600.000100"])["count"] == 1
same = observed_rate(["1735689600.000100", "1735689600.000100"])
assert same["per_minute"] == 0.0
assert "shares a timestamp" in same["note"]
def test_the_measured_rate_is_reported_as_a_floor():
rate = observed_rate(["1735689600.000100", "1735689630.000100",
"1735689660.000100"])
assert rate["span_seconds"] == 60.0
assert rate["per_minute"] == 3.0
assert "floor" in rate["note"]
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
converge, deliveriesPerEvent, demand, observedRate, settleVerdict,
} from './slack-retry-amplification.mjs';
test('the amplification is bounded at four deliveries', () => {
assert.equal(deliveriesPerEvent(1.0), 4);
assert.equal(deliveriesPerEvent(0), 1);
});
test('the expansion is a cliff rather than a slope', () => {
assert.ok(deliveriesPerEvent(0.05) < 1.06);
assert.ok(deliveriesPerEvent(0.4) > 1.6);
assert.ok(deliveriesPerEvent(0.8) > 2.9);
});
test('a saturated handler has nothing refused yet', () => {
const state = demand(40, 1, 0.05, 50);
assert.equal(state.verdict, 'saturated');
assert.ok(state.sharePercent > 80);
assert.equal(state.throttledFraction, 0);
});
test('three calls on the same traffic is over the same budget', () => {
const state = demand(40, 3, 0.05, 50);
assert.equal(state.verdict, 'over-budget');
assert.ok(state.callsPerMinute > 50);
assert.ok(state.throttledFraction > 0.5);
});
test('the throttled fraction is what feeds the loop', () => {
assert.ok(demand(40, 3, 0.05, 50).throttledFraction
> demand(40, 2, 0.05, 50).throttledFraction);
});
test('the same traffic is stable at one call and collapses at three', () => {
assert.equal(converge(40, 1, 0.05, 50).verdict, 'stable');
assert.equal(converge(40, 3, 0.05, 50).verdict, 'collapsed');
});
test('a stable loop keeps the miss rate it started with', () => {
const loop = converge(40, 1, 0.05, 50);
assert.equal(loop.missRate, 0.05);
assert.equal(loop.rounds, 1);
});
test('the collapsed loop still has a fixed point', () => {
const loop = converge(40, 3, 0.05, 50);
assert.equal(loop.path[0], 0.05);
assert.deepEqual(loop.path, [...loop.path].sort((a, b) => a - b));
assert.ok(loop.missRate > 0.9 && loop.missRate < 0.95);
assert.ok(loop.deliveriesPerEvent > 3.5);
});
test('enough load settles above the rate that gets you switched off', () => {
const loop = converge(200, 5, 0.05, 50);
assert.equal(loop.verdict, 'disabling');
assert.ok(loop.missRate >= 0.95);
assert.match(loop.detail, /switched off/);
});
test('the ladder is read off the fixed point not the journey', () => {
assert.equal(settleVerdict(0.05, 0.05)[0], 'stable');
assert.equal(settleVerdict(0.3, 0.05)[0], 'elevated');
assert.equal(settleVerdict(0.7, 0.05)[0], 'collapsed');
assert.equal(settleVerdict(0.96, 0.05)[0], 'disabling');
});
test('zero traffic cannot feed back', () => {
assert.equal(converge(0, 10, 0.05, 50).verdict, 'stable');
});
test('a rate needs two messages and a span', () => {
assert.equal(observedRate([]).perMinute, 0);
assert.equal(observedRate(['1735689600.000100']).count, 1);
const same = observedRate(['1735689600.000100', '1735689600.000100']);
assert.equal(same.perMinute, 0);
assert.match(same.note, /shares a timestamp/);
});
test('the measured rate is reported as a floor', () => {
const rate = observedRate(['1735689600.000100', '1735689630.000100',
'1735689660.000100']);
assert.equal(rate.spanSeconds, 60);
assert.equal(rate.perMinute, 3);
assert.match(rate.note, /floor/);
});
FAQ
Is four deliveries per event really the worst case?
For a single event, yes: the original plus three retries, and Slack does not go beyond that. The reason it still becomes a storm is that the ceiling applies per event while the quota applies per method per minute across every event at once. A four times spike on a system running at forty percent of its budget is fine; the same spike on one running at sixty percent is not, and the difference between those two workspaces is invisible until it happens.
Why not just add more workers to absorb the retries?
Because the rate limit is per method, per workspace, per app, and not per process. Two replicas share one bucket, so doubling the workers doubles the rate at which the bucket is emptied and brings the refusals forward. It is the most natural response to a queue that is not draining and it is the one that makes this particular failure worse.
Where do the sixty and three hundred second retries show up?
As humps in your API call graph that no product traffic explains, minutes after the load that caused them has gone. That spacing is also the fingerprint in the workspace itself: duplicate app messages roughly a minute and roughly five minutes apart are Slack's retry schedule, and reading those gaps is how a duplicate is attributed to a retry rather than to a double subscription.
Does the model need my real miss rate to be useful?
It needs a starting one, and any honest number will do, because the interesting output is where the loop settles rather than where it starts. Take whatever fraction of deliveries your logs show taking over three seconds. If a five percent start settles in the nineties, then no plausible starting value saves this configuration; if a twenty percent start settles back near twenty, the feedback is not engaged and you have real headroom.
Could the script measure the storm instead of modelling it?
It could measure the aftermath, which is duplicate messages in a channel, and that is a different note with a different script. What it will not do is produce the storm to prove the model, because the quota being exhausted belongs to the whole workspace: every other app installed there would be refused alongside yours, and none of their owners agreed to the experiment.
Related field notes
- the missed deadline that starts the loop
- what to do with the refusal when it arrives
- who else is spending the same bucket
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Events API - Slack Docs
- Rate limits - Slack Docs
- conversations.history method reference - Slack Docs
- users.conversations method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.