Diagnostic Slack
message_limit_exceeded: the workspace posting cap
Your app posts eleven messages a day. This morning every one of them failed with message_limit_exceeded, and the human-readable half of the error says that members on this team are sending too many messages. Nobody on your team is sending anything. Your retry logic, which is well written and honours Retry-After, does not help, because this is not your quota.
Somebody else's migration script is running, and it has taken the whole workspace with it.
message_limit_exceeded is a workspace-wide ceiling on message volume, sitting above and apart from the per-method rate limits your app has. When it is exhausted, every app in the workspace starts failing at once, which means the app that reports the error is frequently not the app that caused it. Conflating it with ratelimited sends the investigation to your own backoff code, which is the one place the answer definitely is not.
A read-only token cannot query the workspace's remaining budget, and there is no method that would. What it can do is count. The script below reads recent history from the channels you name, buckets every app-authored message into whole minutes, and ranks the sending apps by their peak minute. Volume is attributed by app_id and bot_id, so the output distinguishes your app from the three others in the workspace, and the loudest sender is usually one line in somebody's loop.
The problem in plain words
The error looks like a rate limit and is filed as one. It is not: ratelimited is per method, per workspace, per app, and it is a statement about your traffic. message_limit_exceeded is a statement about the workspace's traffic, all of it, from every integration and every user, and your app can be at the bottom of that list and still be refused. Backoff will not clear it, retrying will not clear it, and a second app token will not clear it either, because the ceiling is not attached to the token.
What makes it genuinely difficult is that the evidence is not in your logs. Your logs contain your sends, which are not the problem. The volume that exhausted the ceiling is in the workspace, posted by something you may not own, and the only place a read-only token can see it is in the channels themselves. Nobody thinks to look there, because the instinct when an error mentions limits is to look at the code that hit the limit.
And the cause is nearly always mundane. A migration script that posts one message per row instead of one message per batch. A monitoring integration that lost its deduplication and is re-announcing the same alert every four seconds. An automation that got wired into a loop with another automation. In every case the shape in the history is unmistakable once somebody counts it: one sender, one channel, several hundred messages in a minute, in a workspace where the normal figure is single digits.
Why it happens
The ceiling belongs to the workspace, so the fix might not belong to you. This is the rare Slack failure where the correct outcome of an investigation can be a message to another team. Establishing that early saves the day that would otherwise go into tuning your own sender, and the only way to establish it is to attribute the volume by app rather than counting your own.
ratelimited and message_limit_exceeded must never be caught by the same branch. One is answered by honouring Retry-After and slowing your own sends. The other is answered by finding a different app. A handler that treats them alike will retry politely into a wall for as long as the incident lasts.
Whole minutes are the right unit, and seconds are not. The workspace ceiling is an aggregate over time, so the number that describes it is messages per minute summed across channels and senders. How closely spaced two consecutive messages from one app are in one channel is a different measurement answering a different question, and mixing the two produces a report where a chatty deploy bot outranks the migration script that caused the incident.
Attribution is readable and free. Messages posted by apps carry app_id, bot_id and often a bot_profile with a name. That is enough to build a leaderboard of senders from history alone, without any admin scope, and to say plainly whether the app at the top of it is yours.
The sample can be too small to divide into minutes, and then the script says nothing. One page of history from a quiet channel spans hours and produces buckets of one. A ranking built on that is noise dressed as evidence, so the run reports the window it actually got and declines to rank when there is not enough of it.
The fix, as a flow
The box that goes red here is not the failure. The failure is three steps earlier and belongs to somebody else; the red box is the investigation arriving at the wrong sender, which is the outcome this note exists to prevent. The fix branch is a leaderboard rather than a diagnosis, and the healthy row is your own app being cleared in writing.
How to fix it
Separate the two errors in the handler before anything else
attribute_ceiling is offline and takes ten seconds to wire in. Until message_limit_exceeded and ratelimited land in different branches with different log lines, every future incident of this kind starts by being misfiled.
Pick the channels where the volume would be
Alert channels, deploy channels, whatever the migration is writing to. The script takes them as arguments because a read-only token cannot ask the workspace where its traffic is, and guessing across every channel would cost more requests than the answer is worth.
Read history and keep only what an app posted
Human messages are not what exhausts this ceiling and counting them buries the signal. sender_key keeps messages carrying app_id or bot_id and drops the rest, returning a stable key so the same app is one row across every channel.
Bucket into whole minutes, deliberately
Peak messages per minute, per app, summed across the channels you named. Not gaps between consecutive messages: that measures the cadence of one sender in one channel, which is a real problem and a different note, and it will rank a steady bot above a burst that took the workspace down.
Rank the senders and find yourself in the list
auth.test gives your own bot_id, and the ranking marks your rows. Being third with four messages a minute behind something posting six hundred is the finding, and it is the one that ends the argument about whose sender needs work.
Take the number to whoever owns the loop
The printed repair is batching: one message with many blocks rather than N messages, a per-workspace send budget in each app so no single integration can exhaust a shared ceiling, and, where the volume is genuinely legitimate, a conversation with Slack, because this ceiling is not self-service.
How to check it worked
Re-run after the offending sender is batched. The peak minute should fall by an order of magnitude and no sender should be labelled a likely cause.
python3 slack_workspace_send_volume.py C0ALERTS99 C0DEPLOY11
# identity B0OURAPP11 in acme
# window 412 message(s) across 2 channel(s), 143 from apps, spanning 96 minute(s)
# sender background B0OURAPP11 (this app) peak 4/min, 31 total
# sender background A0PAGERDUTY peak 6/min, 44 total
# sender background A0MIGRATION peak 9/min, 68 total
# 0 sender(s) at a rate that would exhaust a workspace ceilingThe full code
Four pure functions and one paginated read per channel. sender_key decides what counts as an app-authored message and gives the same app one identity across channels. per_minute_by_app does the arithmetic in whole minutes, which is the unit this ceiling is measured in. rank_senders turns that into a leaderboard that marks your own app. attribute_ceiling is the ten-second fix: it keeps this error away from your retry code.
"""Rank the apps sending into a workspace, in messages per minute.
Read only. Reads recent history from the channels you name, keeps the messages
an app posted, buckets them into whole minutes and ranks the senders. Nothing
is sent: the workspace message ceiling is an aggregate over everybody's
traffic, and the point of this script is to find out whose.
Whole minutes on purpose. How closely spaced one app's consecutive messages are
in one channel is a different measurement answering a different question.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_workspace_send_volume")
API = "https://slack.com/api/"
# Errors that get caught by the same branch and should not be. The first
# belongs to the workspace, the second to your app, and they have opposite
# repairs: find another team, or slow your own sender down.
CEILINGS = {
"message_limit_exceeded": (
"workspace-ceiling",
"a workspace-wide cap on message volume, not your app's quota. Every app "
"in the workspace fails at once when it is exhausted, so the app that "
"reports it is often not the app that caused it. Backoff does not clear "
"it and a second token does not either."),
"ratelimited": (
"app-quota",
"your app's own per-method window, answered by honouring Retry-After. It "
"is a statement about your traffic, which the workspace ceiling is not."),
"rate_limited": (
"app-quota",
"the same limit under its older spelling. Still your traffic, still "
"answered by backoff."),
"accesslimited": (
"network-policy",
"the calling IP is outside an allowed range. Nothing to do with volume."),
"msg_too_long": (
"message-shape",
"one message exceeded the size limit. A single message problem rather "
"than a rate of them."),
}
# Below this many app messages, or this many distinct minutes, the window is
# too thin to divide into a rate. A ranking built on three messages spread over
# four hours is noise wearing the clothes of evidence.
MIN_APP_MESSAGES = 20
MIN_MINUTES = 3
def sender_key(message):
"""The app that posted this message, or None if a person did. Pure.
app_id is preferred because it is stable for the app across workspaces and
across reinstalls; bot_id is the fallback for messages that carry only
that. Human messages are dropped: they are not what exhausts this ceiling
and counting them buries the sender that is.
"""
m = message or {}
app_id = str(m.get("app_id") or "").strip()
if app_id:
return app_id
bot_id = str(m.get("bot_id") or "").strip()
if bot_id:
return bot_id
if m.get("subtype") == "bot_message":
return "bot:unattributed"
return None
def per_minute_by_app(messages):
"""Bucket app-authored messages into whole minutes, per sender. Pure.
Returns {key: {"total", "minutes", "peak_per_minute", "mean_per_minute"}}.
Whole minutes because the workspace ceiling is an aggregate over time; the
gap between two consecutive sends is a cadence question and belongs to the
per-channel posting rate rather than here.
"""
buckets = {}
for message in messages or []:
key = sender_key(message)
if key is None:
continue
try:
minute = int(float(message.get("ts"))) // 60
except (TypeError, ValueError):
continue
buckets.setdefault(key, {})
buckets[key][minute] = buckets[key].get(minute, 0) + 1
out = {}
for key, minutes in buckets.items():
counts = list(minutes.values())
out[key] = {
"total": sum(counts),
"minutes": len(counts),
"peak_per_minute": max(counts),
"mean_per_minute": round(sum(counts) / float(len(counts)), 1),
}
return out
def rank_senders(volume, own_ids=(), threshold=60):
"""Order senders by their peak minute and label them. Pure.
Returns [(key, verdict, detail), ...], loudest first. own_ids are the
identities auth.test gave for this app, so the reader can see at a glance
whether the app at the top of the list is theirs.
"""
mine = {str(i) for i in own_ids if i}
rows = []
for key, stats in volume.items():
peak = stats["peak_per_minute"]
if peak >= threshold:
verdict = "likely-cause"
elif peak >= max(threshold // 4, 1):
verdict = "contributor"
else:
verdict = "background"
label = "%s (this app)" % key if key in mine else key
rows.append((peak, label, verdict,
"peak %d/min, %d total over %d minute(s), mean %s/min"
% (peak, stats["total"], stats["minutes"],
stats["mean_per_minute"])))
rows.sort(key=lambda row: (-row[0], row[1]))
return [(label, verdict, detail) for _, label, verdict, detail in rows]
def attribute_ceiling(error):
"""Keep the workspace ceiling away from your retry code. Pure and offline.
The ten second fix in this note: two errors that look alike, land in the
same handler, and want opposite repairs.
"""
text = str(error or "").strip()
known = CEILINGS.get(text)
if known is not None:
return known
if not text:
return ("none-recorded",
"no error was supplied, so the ranking below stands on its own as "
"a volume measurement.")
return ("unattributed",
"%s is not a volume limit. Whatever refused the message, the workspace "
"ceiling is not it." % text)
def history(session, channel, limit, pages):
"""Recent messages from one channel. GET only, bounded."""
out, cursor = [], ""
for _ in range(max(int(pages), 1)):
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.warning("%-10s %-14s %s", channel, "unreadable",
"conversations.history answered ok: false, error=%s"
% body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = (body.get("response_metadata") or {}).get("next_cursor") or ""
if not cursor:
break
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("channels", nargs="+",
help="channel IDs where the volume would be")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding the bot token")
ap.add_argument("--limit", type=int, default=200,
help="page size for conversations.history")
ap.add_argument("--pages", type=int, default=2,
help="pages per channel, so the audit stays cheap")
ap.add_argument("--threshold", type=int, default=60,
help="peak messages per minute at which a sender is a likely cause")
ap.add_argument("--observed-error", default="",
help="an error your app recorded, attributed offline")
args = ap.parse_args()
token = os.environ.get(args.token_env)
if not token:
log.error("set %s (channels:history on the named channels is enough)",
args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test answered 200 with ok: false, error=%s", who.get("error"))
return 2
own = [who.get("bot_id"), who.get("user_id")]
log.info("identity %s in %s", who.get("bot_id") or who.get("user_id"),
who.get("team"))
messages = []
for channel in args.channels:
messages.extend(history(s, channel, args.limit, args.pages))
volume = per_minute_by_app(messages)
app_messages = sum(v["total"] for v in volume.values())
minutes = len({int(float(m["ts"])) // 60 for m in messages if m.get("ts")})
log.info("window %d message(s) across %d channel(s), %d from apps, "
"spanning %d minute(s)", len(messages), len(args.channels),
app_messages, minutes)
if args.observed_error:
source, why = attribute_ceiling(args.observed_error)
(log.warning if source == "workspace-ceiling" else log.info)(
"recorded %-14s %s", source, why)
if app_messages < MIN_APP_MESSAGES or minutes < MIN_MINUTES:
log.info("ranking %-14s %d app message(s) over %d minute(s) is not "
"enough to divide into a rate. Widen --pages or name a busier "
"channel.", "sample-thin", app_messages, minutes)
return 0
loud = 0
for key, verdict, detail in rank_senders(volume, own, args.threshold):
(log.warning if verdict != "background" else log.info)(
"sender %-14s %-24s %s", verdict, key, detail)
if verdict == "likely-cause":
loud += 1
if loud:
log.warning(" repair: batch. One message with many blocks instead of N "
"messages is the single largest reduction available")
log.warning(" repair: give every app a per-workspace send budget so no "
"one integration can exhaust a ceiling all of them share")
log.warning(" repair: if the volume is legitimate, contact Slack; the "
"workspace ceiling is not self-service")
log.info("%d sender(s) at a rate that would exhaust a workspace ceiling", loud)
return 1 if loud else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Rank the apps sending into a workspace, in messages per minute.
*
* Read only. Reads recent history from the channels you name, keeps the
* messages an app posted, buckets them into whole minutes and ranks the
* senders. Nothing is sent: the workspace message ceiling is an aggregate over
* everybody's traffic, and the point of this script is to find out whose.
*
* Whole minutes on purpose. How closely spaced one app's consecutive messages
* are in one channel is a different measurement answering a different question.
*/
const API = 'https://slack.com/api/';
// Errors that get caught by the same branch and should not be. The first
// belongs to the workspace, the second to your app, and they have opposite
// repairs.
const CEILINGS = new Map([
['message_limit_exceeded', ['workspace-ceiling',
"a workspace-wide cap on message volume, not your app's quota. Every app in " +
'the workspace fails at once when it is exhausted, so the app that reports ' +
'it is often not the app that caused it. Backoff does not clear it and a ' +
'second token does not either.']],
['ratelimited', ['app-quota',
'your app\'s own per-method window, answered by honouring Retry-After. It is ' +
'a statement about your traffic, which the workspace ceiling is not.']],
['rate_limited', ['app-quota',
'the same limit under its older spelling. Still your traffic, still answered ' +
'by backoff.']],
['accesslimited', ['network-policy',
'the calling IP is outside an allowed range. Nothing to do with volume.']],
['msg_too_long', ['message-shape',
'one message exceeded the size limit. A single message problem rather than a ' +
'rate of them.']],
]);
// Below this many app messages, or this many distinct minutes, the window is
// too thin to divide into a rate.
const MIN_APP_MESSAGES = 20;
const MIN_MINUTES = 3;
/**
* The app that posted this message, or null if a person did. Pure.
* app_id first because it is stable across reinstalls; bot_id is the fallback.
*/
export function senderKey(message) {
const m = message ?? {};
const appId = String(m.app_id ?? '').trim();
if (appId) return appId;
const botId = String(m.bot_id ?? '').trim();
if (botId) return botId;
if (m.subtype === 'bot_message') return 'bot:unattributed';
return null;
}
/**
* Bucket app-authored messages into whole minutes, per sender. Pure.
* Whole minutes because the workspace ceiling is an aggregate over time; the
* gap between two consecutive sends is a cadence question and belongs
* elsewhere.
*/
export function perMinuteByApp(messages) {
const buckets = new Map();
for (const message of messages ?? []) {
const key = senderKey(message);
if (key === null) continue;
const seconds = Number(message.ts);
if (!Number.isFinite(seconds)) continue;
const minute = Math.floor(seconds / 60);
if (!buckets.has(key)) buckets.set(key, new Map());
const per = buckets.get(key);
per.set(minute, (per.get(minute) ?? 0) + 1);
}
const out = {};
for (const [key, minutes] of buckets) {
const counts = [...minutes.values()];
const total = counts.reduce((a, b) => a + b, 0);
out[key] = {
total,
minutes: counts.length,
peak_per_minute: Math.max(...counts),
mean_per_minute: Math.round((total * 10) / counts.length) / 10,
};
}
return out;
}
/**
* Order senders by their peak minute and label them. Pure.
* ownIds are the identities auth.test gave for this app, so the reader can see
* at a glance whether the app at the top of the list is theirs.
*/
export function rankSenders(volume, ownIds = [], threshold = 60) {
const mine = new Set([...ownIds].filter(Boolean).map(String));
const rows = [];
for (const [key, stats] of Object.entries(volume)) {
const peak = stats.peak_per_minute;
let verdict = 'background';
if (peak >= threshold) verdict = 'likely-cause';
else if (peak >= Math.max(Math.floor(threshold / 4), 1)) verdict = 'contributor';
const label = mine.has(key) ? `${key} (this app)` : key;
rows.push([peak, label, verdict,
`peak ${peak}/min, ${stats.total} total over ${stats.minutes} minute(s), ` +
`mean ${stats.mean_per_minute}/min`]);
}
rows.sort((a, b) => (b[0] - a[0]) || a[1].localeCompare(b[1]));
return rows.map(([, label, verdict, detail]) => [label, verdict, detail]);
}
/**
* Keep the workspace ceiling away from your retry code. Pure and offline.
* Two errors that look alike, land in the same handler, and want opposite
* repairs.
*/
export function attributeCeiling(error) {
const text = String(error ?? '').trim();
const known = CEILINGS.get(text);
if (known !== undefined) return known;
if (!text) {
return ['none-recorded',
'no error was supplied, so the ranking below stands on its own as a volume ' +
'measurement.'];
}
return ['unattributed',
`${text} is not a volume limit. Whatever refused the message, the workspace ` +
'ceiling is not it.'];
}
async function history(token, channel, limit, pages) {
const out = [];
let cursor = '';
for (let i = 0; i < Math.max(pages, 1); i += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const res = await fetch(`${API}conversations.history?${params}`, {
headers: { Authorization: `Bearer ${token}` },
});
const body = await res.json();
if (body.ok !== true) {
console.warn(`${channel.padEnd(10)} ${'unreadable'.padEnd(14)} ` +
`conversations.history answered ok: false, error=${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = body.response_metadata?.next_cursor ?? '';
if (!cursor) break;
}
return out;
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
function positionals(args) {
const out = [];
for (let i = 0; i < args.length; i += 1) {
if (args[i].startsWith('--')) { i += 1; continue; }
out.push(args[i]);
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const channels = positionals(args);
if (channels.length === 0) {
console.error('usage: <channel id>... [--pages 2] [--threshold 60] ' +
'[--observed-error message_limit_exceeded]');
process.exitCode = 2;
return;
}
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} (channels:history on the named channels is enough)`);
process.exitCode = 2;
return;
}
const limit = Number(arg(args, '--limit', 200));
const pages = Number(arg(args, '--pages', 2));
const threshold = Number(arg(args, '--threshold', 60));
const whoRes = await fetch(`${API}auth.test`, {
headers: { Authorization: `Bearer ${token}` },
});
const who = await whoRes.json();
if (who.ok !== true) {
console.error(`auth.test answered 200 with ok: false, error=${who.error}`);
process.exitCode = 2;
return;
}
const own = [who.bot_id, who.user_id];
console.log(`identity ${who.bot_id ?? who.user_id} in ${who.team}`);
const messages = [];
for (const channel of channels) {
messages.push(...await history(token, channel, limit, pages));
}
const volume = perMinuteByApp(messages);
const appMessages = Object.values(volume).reduce((a, v) => a + v.total, 0);
const minutes = new Set(messages.filter((m) => m.ts)
.map((m) => Math.floor(Number(m.ts) / 60))).size;
console.log(`window ${messages.length} message(s) across ${channels.length} ` +
`channel(s), ${appMessages} from apps, spanning ${minutes} minute(s)`);
const observed = arg(args, '--observed-error', '');
if (observed) {
const [source, why] = attributeCeiling(observed);
(source === 'workspace-ceiling' ? console.warn : console.log)(
`recorded ${source.padEnd(14)} ${why}`);
}
if (appMessages < MIN_APP_MESSAGES || minutes < MIN_MINUTES) {
console.log(`ranking ${'sample-thin'.padEnd(14)} ${appMessages} app ` +
`message(s) over ${minutes} minute(s) is not enough to divide into a rate. ` +
'Widen --pages or name a busier channel.');
return;
}
let loud = 0;
for (const [key, verdict, detail] of rankSenders(volume, own, threshold)) {
(verdict === 'background' ? console.log : console.warn)(
`sender ${verdict.padEnd(14)} ${key.padEnd(24)} ${detail}`);
if (verdict === 'likely-cause') loud += 1;
}
if (loud) {
console.warn(' repair: batch. One message with many blocks instead of N ' +
'messages is the single largest reduction available');
console.warn(' repair: give every app a per-workspace send budget so no one ' +
'integration can exhaust a ceiling all of them share');
console.warn(' repair: if the volume is legitimate, contact Slack; the ' +
'workspace ceiling is not self-service');
}
console.log(`${loud} sender(s) at a rate that would exhaust a workspace ceiling`);
process.exitCode = loud ? 1 : 0;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The fixture is one minute of an incident: a migration app posting sixty times inside a single minute while the app running the audit posts twice, in the same channel, over the same window. The tests assert that the migration is ranked first and labelled a likely cause, that this app is found in the list and labelled as its own, and that human messages are excluded entirely. The last two pin the discipline: message_limit_exceeded and ratelimited never come back with the same source.
from slack_workspace_send_volume import (attribute_ceiling, per_minute_by_app,
rank_senders, sender_key)
MINUTE = 1756400000
# One minute of an incident, plus a quiet minute either side. The migration app
# posts sixty times inside a single minute; the app running the audit posts
# twice; a person says something, which is not what exhausts this ceiling.
MESSAGES = (
[{"ts": "%d.0000%02d" % (MINUTE, i), "app_id": "A0MIGRATION",
"bot_id": "B0MIGRATION"} for i in range(60)]
+ [{"ts": "%d.000000" % (MINUTE - 60), "app_id": "A0MIGRATION"}]
+ [{"ts": "%d.000000" % (MINUTE + 60), "app_id": "A0MIGRATION"}]
+ [{"ts": "%d.000100" % MINUTE, "bot_id": "B0OURAPP11"},
{"ts": "%d.000200" % (MINUTE + 60), "bot_id": "B0OURAPP11"}]
+ [{"ts": "%d.000300" % MINUTE, "user": "U0HUMAN111", "text": "what is going on"}]
)
def test_a_human_message_has_no_sender_key():
assert sender_key({"ts": "1.0", "user": "U0HUMAN111"}) is None
assert sender_key({}) is None
assert sender_key(None) is None
def test_app_id_is_preferred_over_bot_id_so_one_app_is_one_row():
assert sender_key({"app_id": "A0MIGRATION", "bot_id": "B0MIGRATION"}) == "A0MIGRATION"
assert sender_key({"bot_id": "B0OURAPP11"}) == "B0OURAPP11"
assert sender_key({"subtype": "bot_message"}) == "bot:unattributed"
def test_the_peak_minute_is_the_number_this_ceiling_is_measured_in():
volume = per_minute_by_app(MESSAGES)
assert volume["A0MIGRATION"]["peak_per_minute"] == 60
assert volume["A0MIGRATION"]["total"] == 62
assert volume["A0MIGRATION"]["minutes"] == 3
def test_a_steady_sender_never_reaches_a_peak():
volume = per_minute_by_app(MESSAGES)
assert volume["B0OURAPP11"]["peak_per_minute"] == 1
assert volume["B0OURAPP11"]["total"] == 2
def test_humans_are_not_counted_at_all():
assert set(per_minute_by_app(MESSAGES)) == {"A0MIGRATION", "B0OURAPP11"}
def test_a_message_without_a_usable_timestamp_is_skipped_not_guessed():
volume = per_minute_by_app([{"app_id": "A0X"}, {"ts": "later", "app_id": "A0X"}])
assert volume == {}
def test_the_loudest_sender_is_first_and_named_as_the_likely_cause():
rows = rank_senders(per_minute_by_app(MESSAGES), ["B0OURAPP11"])
assert rows[0][0] == "A0MIGRATION"
assert rows[0][1] == "likely-cause"
assert "peak 60/min" in rows[0][2]
def test_your_own_app_is_marked_so_the_answer_is_unmissable():
rows = rank_senders(per_minute_by_app(MESSAGES), ["B0OURAPP11"])
assert rows[1][0] == "B0OURAPP11 (this app)"
assert rows[1][1] == "background"
def test_the_threshold_is_a_dial_rather_than_a_law():
volume = per_minute_by_app(MESSAGES)
assert rank_senders(volume, [], threshold=1000)[0][1] == "background"
assert rank_senders(volume, [], threshold=200)[0][1] == "contributor"
assert rank_senders(volume, [], threshold=60)[0][1] == "likely-cause"
def test_the_workspace_ceiling_is_never_confused_with_your_own_quota():
source, why = attribute_ceiling("message_limit_exceeded")
assert source == "workspace-ceiling"
assert "not your app" in why
assert attribute_ceiling("ratelimited")[0] == "app-quota"
assert attribute_ceiling("rate_limited")[0] == "app-quota"
def test_an_unrelated_error_is_not_read_as_volume():
assert attribute_ceiling("msg_too_long")[0] == "message-shape"
assert attribute_ceiling("channel_not_found")[0] == "unattributed"
assert attribute_ceiling("")[0] == "none-recorded"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { attributeCeiling, perMinuteByApp, rankSenders, senderKey }
from './slack-workspace-send-volume.mjs';
const MINUTE = 1756400000;
// One minute of an incident, plus a quiet minute either side.
const MESSAGES = [
...Array.from({ length: 60 }, (_, i) => ({
ts: `${MINUTE}.0000${String(i).padStart(2, '0')}`,
app_id: 'A0MIGRATION', bot_id: 'B0MIGRATION',
})),
{ ts: `${MINUTE - 60}.000000`, app_id: 'A0MIGRATION' },
{ ts: `${MINUTE + 60}.000000`, app_id: 'A0MIGRATION' },
{ ts: `${MINUTE}.000100`, bot_id: 'B0OURAPP11' },
{ ts: `${MINUTE + 60}.000200`, bot_id: 'B0OURAPP11' },
{ ts: `${MINUTE}.000300`, user: 'U0HUMAN111', text: 'what is going on' },
];
test('a human message has no sender key', () => {
assert.equal(senderKey({ ts: '1.0', user: 'U0HUMAN111' }), null);
assert.equal(senderKey({}), null);
assert.equal(senderKey(null), null);
});
test('app_id is preferred over bot_id so one app is one row', () => {
assert.equal(senderKey({ app_id: 'A0MIGRATION', bot_id: 'B0MIGRATION' }),
'A0MIGRATION');
assert.equal(senderKey({ bot_id: 'B0OURAPP11' }), 'B0OURAPP11');
assert.equal(senderKey({ subtype: 'bot_message' }), 'bot:unattributed');
});
test('the peak minute is the number this ceiling is measured in', () => {
const volume = perMinuteByApp(MESSAGES);
assert.equal(volume.A0MIGRATION.peak_per_minute, 60);
assert.equal(volume.A0MIGRATION.total, 62);
assert.equal(volume.A0MIGRATION.minutes, 3);
});
test('a steady sender never reaches a peak', () => {
const volume = perMinuteByApp(MESSAGES);
assert.equal(volume.B0OURAPP11.peak_per_minute, 1);
assert.equal(volume.B0OURAPP11.total, 2);
});
test('humans are not counted at all', () => {
assert.deepEqual(Object.keys(perMinuteByApp(MESSAGES)).sort(),
['A0MIGRATION', 'B0OURAPP11']);
});
test('a message without a usable timestamp is skipped not guessed', () => {
assert.deepEqual(perMinuteByApp([{ app_id: 'A0X' }, { ts: 'later', app_id: 'A0X' }]),
{});
});
test('the loudest sender is first and named as the likely cause', () => {
const rows = rankSenders(perMinuteByApp(MESSAGES), ['B0OURAPP11']);
assert.equal(rows[0][0], 'A0MIGRATION');
assert.equal(rows[0][1], 'likely-cause');
assert.match(rows[0][2], /peak 60\/min/);
});
test('your own app is marked so the answer is unmissable', () => {
const rows = rankSenders(perMinuteByApp(MESSAGES), ['B0OURAPP11']);
assert.equal(rows[1][0], 'B0OURAPP11 (this app)');
assert.equal(rows[1][1], 'background');
});
test('the threshold is a dial rather than a law', () => {
const volume = perMinuteByApp(MESSAGES);
assert.equal(rankSenders(volume, [], 1000)[0][1], 'background');
assert.equal(rankSenders(volume, [], 200)[0][1], 'contributor');
assert.equal(rankSenders(volume, [], 60)[0][1], 'likely-cause');
});
test('the workspace ceiling is never confused with your own quota', () => {
const [source, why] = attributeCeiling('message_limit_exceeded');
assert.equal(source, 'workspace-ceiling');
assert.match(why, /not your app/);
assert.equal(attributeCeiling('ratelimited')[0], 'app-quota');
assert.equal(attributeCeiling('rate_limited')[0], 'app-quota');
});
test('an unrelated error is not read as volume', () => {
assert.equal(attributeCeiling('msg_too_long')[0], 'message-shape');
assert.equal(attributeCeiling('channel_not_found')[0], 'unattributed');
assert.equal(attributeCeiling('')[0], 'none-recorded');
});
FAQ
Is message_limit_exceeded the same as ratelimited?
No, and treating them as the same is the mistake this note exists to prevent. ratelimited is per method, per workspace, per app, and it describes your traffic; message_limit_exceeded is a workspace-wide ceiling on message volume that every app in the workspace shares. One is answered by honouring Retry-After, the other by finding whichever integration is flooding the place.
Can a read-only token see how much of the ceiling is left?
No. There is no method that reports remaining budget for anything in Slack, and rate limit posture is always inferred from live calls rather than queried. What a read token can do is count what has already been posted, which is why this script measures history rather than asking for a quota that no endpoint returns.
Why whole minutes rather than the gap between messages?
Because they answer different questions. The workspace ceiling is an aggregate over time, so peak messages per minute summed across senders is the figure that describes it. Sub-second spacing between one app's consecutive posts in one channel describes that app's cadence against the per-channel send rate, and ranking by it would put a chatty deploy bot above the migration that caused the outage.
The top sender is not our app. What do we actually do?
Send the number to whoever owns it. That is the useful outcome: a named app, a channel, and a peak minute is a message somebody can act on, where a screenshot of your own failures is not. In the meantime the only thing your app can do for itself is post less, so batching is worth doing regardless of who caused this one.
Our volume is legitimate. Is there a way to raise the ceiling?
Not by yourself. The workspace ceiling is not a setting in the admin console and it is not a plan upgrade, so the route is a conversation with Slack about the workload. Before starting it, batch what can be batched: replacing N messages with one message carrying N blocks usually removes most of the volume and makes the remaining case much easier to argue.
Related field notes
- two apps generating volume out of each other
- the other limit that is not your fault
- why the refusal arrived as a success
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- chat.postMessage method reference - Slack Docs
- Rate limits - Slack Docs
- conversations.history method reference - Slack Docs
- auth.test method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.