Diagnostic Slack
The install store keeps rows for uninstalled workspaces
The nightly digest job iterates four hundred installations and logs a hundred and twenty errors. It has done that every night for eleven months. Nobody looks at the log any more, because the log is a hundred and twenty lines of token_revoked followed by the two lines that actually matter, and after the fourth week of that you stop scrolling.
Slack is not doing anything wrong here. Every one of those hundred and twenty workspaces removed your app, and Slack told you so at the time: it emitted app_uninstalled, it emitted tokens_revoked, and it invalidated the token immediately. What it did not do — what nothing does — is delete the row in your database. That was always your job, and the failure is sitting in your own table.
Deleting the installation record is the app's responsibility, and the SDK installation stores do not do it for you. Handle app_uninstalled and tokens_revoked by removing the row and everything derived from it; without that, an install that ended two years ago is still iterated on every sweep, still consuming a request, and still writing an error line.
The measurement is a population, not an error. Walk the store, call auth.test once per token, and sort the answers into live, revoked, deactivated, expired and unreadable. What you want out of it is one number: the fraction of the store that is dead. That fraction is the finding, and it is the thing no single error message can tell you, because every individual error looks like ordinary bad luck.
The second number is the one that turns a fact into a diagnosis. Run the sweep twice, a fortnight apart, and compare. A dead fraction that shrinks means your cleanup works. A dead fraction that holds steady or grows means nothing is deleting anything, and every uninstall from here on is permanent sediment. Keep token_expired out of that count: rotation lapsing is recoverable by refreshing, and tombstoning it throws away a live customer.
The problem in plain words
The reason this survives for years is that it is not an outage. Nothing is down. The app works perfectly for every workspace that still has it installed, and the errors are all attached to workspaces that do not exist as customers any more. There is no user to complain, no dashboard to go red, and no page to receive. It is a slow accumulation of things that are true and useless, and slow accumulations do not get tickets.
What it does instead is ruin your instrumentation. When a hundred and twenty of every four hundred iterations fail by design, the error rate for the job is thirty percent and always has been, so nobody can use the error rate for anything. A real failure — a genuine token problem in a paying tenant — arrives as one more line in a wall of identical lines, and the alerting threshold that would have caught it was raised months ago precisely because of the noise. The dead rows do not merely waste effort; they hide the failures that matter.
The events that should have prevented this are real, and apps that handle them still end up here. app_uninstalled and tokens_revoked arrive in an order that is not guaranteed, so a handler written to expect one before the other drops the second. tokens_revoked does not fire for every case — a single user revoking their own authorization is not the same as the app being removed — so a store keyed on user tokens keeps rows the event never mentioned. And an app that was deployed before the handler was written has a backlog that no future event will ever clear, because the events for those installs were emitted and discarded years ago.
Then there is the residue, which is the part people forget entirely. The installation row is not the only thing that referenced that workspace. There are scheduled messages queued against channels in it, cached channel and user ids, incoming webhook URLs, per-tenant cron entries, a row in a billing table. Deleting the token and leaving the rest produces a second generation of orphans that fail in less obvious ways: a scheduled message that will never send, a cron entry that wakes every hour to do nothing, a webhook URL that answers 404 forever.
Why it happens
The unit of the finding is the store, not the token. A single token_revoked is a fact about one workspace and has its own note; it means the app was removed there, and there is nothing to diagnose. What this check produces is a proportion: how much of your table is sediment. That number cannot be reached by reading an error, only by walking every row and counting, which is why the script takes a store export rather than a log.
Two audits beat one, and the second one is the whole argument. A dead fraction of thirty percent is a fact. A dead fraction that was twenty-nine percent a fortnight ago is a diagnosis: nothing is being cleaned up, the number only goes one way, and every future uninstall is permanent. So the script accepts a previous run's counts and reports the trend, because “stable” here means “broken” and that is not obvious.
Recoverable and dead are different, and merging them deletes customers. token_revoked and account_inactive mean the grant is gone. token_expired means rotation is switched on and the refresh has lapsed, which is a live installation with a stale credential and is fixed by refreshing rather than by deletion. A sweep that tombstones on any error at all will eventually remove a paying tenant during a rotation outage, which is a considerably worse bug than the one it was written to fix.
Rate limiting is not evidence of death, and neither is a network blip. A sweep of four hundred tokens will hit ratelimited somewhere in the middle, and a row that could not be read is unknown rather than dead. The classifier says so explicitly and the counts keep unknown rows in their own column, because rolling them into either side turns a transient into a permanent decision.
Nothing prints a token, and the script does not need one to. Every row is identified by its store id and its team or enterprise id. Where the shape of a credential is genuinely useful — a store that has been quietly accumulating user tokens where bot tokens were intended — the script reports the prefix class and nothing else: xoxb, xoxp, xapp, or unrecognised. No value, no fragment, no length.
The derived records are named because deleting the row alone creates a second generation. Scheduled messages, cached ids, webhook URLs and per-tenant schedules all outlive the install and all keep costing something. The script reports which of them a dead row still carries, so the cleanup you write covers them the first time rather than after a second incident.
The fix, as a flow
The red arrow is the first one, and nothing to the left of it went wrong. The workspace removed the app, which customers are entitled to do, and Slack did everything correctly: it invalidated the token immediately and announced the end twice, with app_uninstalled and with tokens_revoked. The failure is the box the arrow points at, which is a row in your own database that nothing deleted, and every box after it is the same day repeating for eleven months. That is also why the fix branch sorts rows of a table rather than responses from an API. The row that keeps the whole thing honest is the third: a lapsed rotation looks like every other failure and is a live customer, so a cleanup that deletes on any error will eventually remove paying tenants in bulk.
How to fix it
Export the store, tokens included, and keep the export out of your logs
The script reads a JSON list of rows carrying at least an id, a team id and a token. It calls auth.test once per row and prints nothing but ids and states. Treat the export as a credential file for as long as it exists, because that is what it is.
Classify each answer into an action, not just a state
classify_install returns both: revoked and inactive map to tombstone, expired maps to refresh, rate-limited and unreadable map to retry, and anything unrecognised maps to inspect rather than to deletion. The action column is what a cleanup job would act on, and its default is deliberately to do nothing.
Size the dead fraction
store_health reports clean, residue, dominated or empty, with the counts behind each. Dominated means the majority of your sweep is spent on workspaces that removed the app, which is the state most multi-tenant Slack apps discover themselves to be in the first time anybody counts.
Compare against the last audit
dead_trend takes the previous run's counts and answers growing, stable, shrinking or no-baseline. Stable is a failure result: it means the number of dead rows is not going down, which is only possible if nothing deletes them.
Name what the dead row still points at
orphan_residue reads the row itself and reports the derived records that also need removing: scheduled message ids, cached channel ids, an incoming webhook URL, a per-tenant schedule. Each of those fails on its own timetable once the token is gone.
Handle both events, then reconcile anyway
Subscribe to app_uninstalled and tokens_revoked and delete on either, idempotently, because the order is not guaranteed. Then keep this sweep as a scheduled reconciliation, because events get dropped during deploys and no event will ever arrive for the installs that ended before the handler existed.
How to check it worked
Write the cleanup, let it run, and re-run the sweep a fortnight later with --previous pointing at the earlier counts. The line to read is trend: it should say shrinking.
python3 slack_install_store_sweep.py --store installs.json --previous last-audit.json
# store 412 row(s) read from installs.json
# row inst-0031 revoked tombstone T04AB1 xoxb
# row inst-0044 inactive tombstone T04CD2 xoxp
# row inst-0102 expired refresh T04EF3 xoxb
# row inst-0188 rate-limited retry T04GH4 xoxb
# health dominated 412 row(s): 118 live, 279 dead, 12 recoverable, 3 unknown
# trend stable the dead share was 67.2% and is 67.7%; nothing is
# deleting anything
# residue inst-0031 4 scheduled message(s), 1 webhook url, 1 channel cache
# cost wasted 279 call(s) per sweep, 8,370 per day at 30 sweeps
# verdict 2 finding(s)
# repair: handle app_uninstalled and tokens_revoked, delete the row idempotently on
# either, and delete every derived record with it
# repair: keep this sweep as a scheduled reconciliation; no event will ever arrive
# for the installs that ended before the handler existed
# note: token_expired rows were left alone; that is a refresh, not a deletionThe full code
Everything that decides anything is pure and takes a parsed response or a row, so the whole classifier can be tested without a token. classify_install returns a state and an action, and its default action is to do nothing, because a sweep that guesses wrong deletes a customer. store_health turns the states into the one number worth having. dead_trend is the function that makes the number mean something, and it treats a flat line as a failure. orphan_residue reads what else the row points at, and token_shape reports a prefix class so that nothing anywhere prints a credential.
"""Find the installations in your own store that no longer exist in Slack.
Read only. One GET is made per stored row, to auth.test, which is the cheapest
method that answers "is this token still a token". Nothing is written, nothing
is deleted, and no cleanup is performed: the script counts, classifies and
prints the repair for you to run.
No token value is printed by any path in this file. Rows are identified by
their store id and their team or enterprise id, and where the shape of a
credential matters the prefix class is reported and nothing else.
The finding here is not an error from Slack. It is a proportion of your own
table, which is why the script takes a store export rather than a log, and why
it accepts the previous audit's counts: a dead fraction that holds steady
between two audits is the proof that nothing is cleaning up.
"""
import argparse
import json
import logging
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_install_store_sweep")
API = "https://slack.com/api/"
# The grant is gone and the row is sediment.
DEAD = ("revoked", "inactive")
# The installation is alive and the credential is stale. Deleting one of these
# during a rotation outage removes a paying customer, which is a considerably
# worse bug than the one this script exists to find.
RECOVERABLE = ("expired",)
# Nothing was learned about this row. Not dead, not alive, counted separately.
UNKNOWN = ("rate-limited", "unreadable", "unrecognised")
# Errors that mean the grant itself has ended.
REVOKED_ERRORS = ("token_revoked", "invalid_auth", "not_authed")
INACTIVE_ERRORS = ("account_inactive", "team_disabled", "user_removed_from_team")
EXPIRED_ERRORS = ("token_expired",)
TRANSIENT_ERRORS = ("ratelimited", "service_unavailable", "fatal_error",
"request_timeout", "unparseable_body")
# Fields on a stored row that point at something outside it. Each one outlives
# the token and fails on its own timetable once the install has ended.
RESIDUE_FIELDS = (
("scheduled_message_ids", "scheduled message(s) queued against a workspace "
"that removed the app"),
("webhook_url", "an incoming webhook URL that will answer 404 forever"),
("cached_channels", "cached channel id(s) that can never be resolved again"),
("schedule_id", "a per-tenant schedule that wakes up and does nothing"),
)
# Where a dead fraction stops being untidy and starts being the whole sweep.
DOMINATED_SHARE = 0.5
RESIDUE_SHARE = 0.05
# Two audits will never land on the same number exactly; this is the band
# inside which "it did not change" is the honest reading.
TREND_TOLERANCE = 0.02
def token_shape(token):
"""The prefix class of a credential, and nothing else. Pure.
Returns "xoxb", "xoxp", "xoxe", "xapp" or "unrecognised". Never returns any
part of the value, and no caller in this file has any reason to want one.
A store quietly accumulating user tokens where bot tokens were intended is
worth seeing; the tokens themselves are not.
"""
text = str(token or "")
for prefix in ("xoxb", "xoxp", "xoxe", "xapp", "xoxa"):
if text.startswith(prefix + "-"):
return prefix
return "unrecognised"
def classify_install(body):
"""Read one auth.test response as a state and an action. Pure.
Returns (state, action). The action is the half a cleanup job would act on,
and its default is to do nothing:
live ok true -> keep
revoked the grant has ended -> tombstone
inactive the user or workspace is gone -> tombstone
expired rotation lapsed, still a tenant -> refresh
rate-limited transient, nothing learned -> retry
unreadable no ok field at all -> retry
unrecognised an error this script does not know -> inspect
"""
doc = body or {}
if doc.get("ok") is True:
return ("live", "keep")
error = str(doc.get("error") or "")
if not error and "ok" not in doc:
return ("unreadable", "retry")
if error in EXPIRED_ERRORS:
return ("expired", "refresh")
if error in REVOKED_ERRORS:
return ("revoked", "tombstone")
if error in INACTIVE_ERRORS:
return ("inactive", "tombstone")
if error in TRANSIENT_ERRORS:
return ("rate-limited", "retry")
return ("unrecognised", "inspect")
def store_health(states):
"""What fraction of the store is dead? Pure.
states: the state string from classify_install, one per row.
Returns (state, counts). Unknown rows are kept in their own column rather
than pushed to either side, because a rate limit in the middle of a sweep
is not evidence of anything.
"""
seen = [str(s) for s in (states or [])]
counts = {"rows": len(seen), "live": 0, "dead": 0, "recoverable": 0, "unknown": 0}
for state in seen:
if state == "live":
counts["live"] += 1
elif state in DEAD:
counts["dead"] += 1
elif state in RECOVERABLE:
counts["recoverable"] += 1
else:
counts["unknown"] += 1
if not seen:
return ("empty", counts)
share = counts["dead"] / float(len(seen))
counts["dead_share"] = round(share, 4)
if share >= DOMINATED_SHARE:
return ("dominated", counts)
if share > RESIDUE_SHARE:
return ("residue", counts)
if counts["dead"]:
return ("residue", counts)
return ("clean", counts)
def dead_trend(previous, current):
"""Compare this audit's dead share against the last one. Pure.
Returns (state, detail). Stable is a failure result and the detail says so:
a dead fraction that does not fall is only possible if nothing is deleting
anything, and every uninstall from here on is permanent.
"""
def share(counts):
rows = float((counts or {}).get("rows") or 0)
if not rows:
return None
return float((counts or {}).get("dead") or 0) / rows
was, now = share(previous), share(current)
if now is None:
return ("no-data", "this audit read no rows, so there is nothing to compare")
if was is None:
return ("no-baseline", "no previous audit was given; run this again in a "
"fortnight and compare, because the trend is the "
"finding rather than the number")
delta = now - was
detail = "the dead share was %.1f%% and is %.1f%%" % (was * 100, now * 100)
if delta > TREND_TOLERANCE:
return ("growing", detail + "; the store is accumulating dead rows faster "
"than anything removes them")
if delta < -TREND_TOLERANCE:
return ("shrinking", detail + "; something is cleaning up")
return ("stable", detail + "; nothing is deleting anything")
def orphan_residue(row):
"""What else does this row still point at? Pure.
Returns a list of (code, detail). Deleting the installation and leaving
these behind produces a second generation of orphans that fail on their own
timetables: a message that will never send, a webhook that answers 404, a
schedule that wakes every hour to do nothing.
"""
doc = row or {}
out = []
for field, why in RESIDUE_FIELDS:
value = doc.get(field)
if isinstance(value, (list, tuple, set)):
if value:
out.append((field, "%d %s" % (len(value), why)))
elif value:
out.append((field, why))
return out
def sweep_cost(counts, sweeps_per_day=1):
"""How much of every sweep is spent on workspaces that left? Pure.
Returns (state, numbers). The point of the number is not the money, which
is negligible; it is that a job whose error rate is thirty percent by
design has no usable error rate at all.
"""
dead = int((counts or {}).get("dead") or 0)
rows = int((counts or {}).get("rows") or 0)
per_day = dead * max(int(sweeps_per_day or 0), 0)
numbers = {"calls_per_sweep": dead, "calls_per_day": per_day,
"share": round(dead / float(rows), 4) if rows else 0.0}
return (("wasted", numbers) if dead else ("none", numbers))
def auth_test(session, token):
"""One GET, with one stored token. Returns the parsed body."""
r = session.get(API + "auth.test",
headers={"Authorization": "Bearer " + token}, timeout=30)
try:
return r.json()
except ValueError:
return {"ok": False, "error": "unparseable_body"}
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--store", required=True,
help="your installation store as JSON: a list of rows carrying "
"at least id, team_id and token")
ap.add_argument("--previous", default="",
help="the counts written by a previous run, to read the trend")
ap.add_argument("--sweeps-per-day", type=int, default=1)
args = ap.parse_args()
with open(args.store, encoding="utf-8") as handle:
rows = json.load(handle) or []
log.info("store %d row(s) read from %s", len(rows), args.store)
session = requests.Session()
states = []
dead_rows = []
for row in rows:
row = row or {}
token = row.get("token") or ""
state, action = classify_install(auth_test(session, token) if token
else {"ok": False, "error": "not_authed"})
states.append(state)
level = log.info if state == "live" else log.warning
level("row %-16s %-14s %-14s %-7s %s", row.get("id"), state, action,
row.get("team_id") or row.get("enterprise_id") or "-",
token_shape(token))
if state in DEAD:
dead_rows.append(row)
health, counts = store_health(states)
(log.info if health == "clean" else log.warning)(
"health %-14s %s", health, counts)
previous = {}
if args.previous:
with open(args.previous, encoding="utf-8") as handle:
previous = json.load(handle) or {}
trend, detail = dead_trend(previous, counts)
(log.info if trend == "shrinking" else log.warning)(
"trend %-14s %s", trend, detail)
for row in dead_rows:
residue = orphan_residue(row)
if residue:
log.warning("residue %-16s %s", row.get("id"),
"; ".join(d for _c, d in residue))
cost, numbers = sweep_cost(counts, args.sweeps_per_day)
(log.info if cost == "none" else log.warning)("cost %-14s %s", cost, numbers)
findings = (1 if health != "clean" else 0) + (1 if trend in ("stable", "growing")
else 0)
if not findings:
log.info("verdict clean the store holds no dead rows")
return 0
log.warning("verdict %d finding(s)", findings)
log.warning(" repair: handle app_uninstalled and tokens_revoked, delete the row "
"idempotently on either, and delete every derived record with it")
log.warning(" repair: keep this sweep as a scheduled reconciliation; no event "
"will ever arrive for the installs that ended before the handler "
"existed")
log.warning(" note: token_expired rows were left alone; that is a refresh, "
"not a deletion")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Find the installations in your own store that no longer exist in Slack.
*
* Read only. One GET is made per stored row, to auth.test. Nothing is written,
* nothing is deleted, and no cleanup is performed: the script counts,
* classifies and prints the repair for you to run.
*
* No token value is printed by any path in this file. Rows are identified by
* their store id and their team or enterprise id, and where the shape of a
* credential matters the prefix class is reported and nothing else.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://slack.com/api/';
// The grant is gone and the row is sediment.
export const DEAD = ['revoked', 'inactive'];
// Alive, with a stale credential. Deleting one of these removes a customer.
export const RECOVERABLE = ['expired'];
export const REVOKED_ERRORS = ['token_revoked', 'invalid_auth', 'not_authed'];
export const INACTIVE_ERRORS = ['account_inactive', 'team_disabled',
'user_removed_from_team'];
export const EXPIRED_ERRORS = ['token_expired'];
export const TRANSIENT_ERRORS = ['ratelimited', 'service_unavailable', 'fatal_error',
'request_timeout', 'unparseable_body'];
// Fields on a stored row that point at something outside it.
export const RESIDUE_FIELDS = [
['scheduled_message_ids', 'scheduled message(s) queued against a workspace that '
+ 'removed the app'],
['webhook_url', 'an incoming webhook URL that will answer 404 forever'],
['cached_channels', 'cached channel id(s) that can never be resolved again'],
['schedule_id', 'a per-tenant schedule that wakes up and does nothing'],
];
const DOMINATED_SHARE = 0.5;
const RESIDUE_SHARE = 0.05;
const TREND_TOLERANCE = 0.02;
/** The prefix class of a credential, and nothing else. Pure. */
export function tokenShape(token) {
const text = String(token ?? '');
for (const prefix of ['xoxb', 'xoxp', 'xoxe', 'xapp', 'xoxa']) {
if (text.startsWith(`${prefix}-`)) return prefix;
}
return 'unrecognised';
}
/**
* Read one auth.test response as a state and an action. Pure.
* Returns [state, action]; the default action is to do nothing.
*/
export function classifyInstall(body) {
const doc = body ?? {};
if (doc.ok === true) return ['live', 'keep'];
const error = String(doc.error ?? '');
if (!error && !('ok' in doc)) return ['unreadable', 'retry'];
if (EXPIRED_ERRORS.includes(error)) return ['expired', 'refresh'];
if (REVOKED_ERRORS.includes(error)) return ['revoked', 'tombstone'];
if (INACTIVE_ERRORS.includes(error)) return ['inactive', 'tombstone'];
if (TRANSIENT_ERRORS.includes(error)) return ['rate-limited', 'retry'];
return ['unrecognised', 'inspect'];
}
/**
* What fraction of the store is dead? Pure.
* Returns [state, counts]; empty, clean, residue, dominated.
*/
export function storeHealth(states) {
const seen = (states ?? []).map((s) => String(s));
const counts = { rows: seen.length, live: 0, dead: 0, recoverable: 0, unknown: 0 };
for (const state of seen) {
if (state === 'live') counts.live += 1;
else if (DEAD.includes(state)) counts.dead += 1;
else if (RECOVERABLE.includes(state)) counts.recoverable += 1;
else counts.unknown += 1;
}
if (!seen.length) return ['empty', counts];
const share = counts.dead / seen.length;
counts.dead_share = Math.round(share * 10000) / 10000;
if (share >= DOMINATED_SHARE) return ['dominated', counts];
if (share > RESIDUE_SHARE) return ['residue', counts];
if (counts.dead) return ['residue', counts];
return ['clean', counts];
}
/**
* Compare this audit's dead share against the last one. Pure.
* Stable is a failure result and the detail says so.
*/
export function deadTrend(previous, current) {
const share = (counts) => {
const rows = Number((counts ?? {}).rows ?? 0);
if (!rows) return null;
return Number((counts ?? {}).dead ?? 0) / rows;
};
const was = share(previous);
const now = share(current);
if (now === null) {
return ['no-data', 'this audit read no rows, so there is nothing to compare'];
}
if (was === null) {
return ['no-baseline', 'no previous audit was given; run this again in a '
+ 'fortnight and compare, because the trend is the finding rather than the '
+ 'number'];
}
const delta = now - was;
const detail = `the dead share was ${(was * 100).toFixed(1)}% and is `
+ `${(now * 100).toFixed(1)}%`;
if (delta > TREND_TOLERANCE) {
return ['growing', `${detail}; the store is accumulating dead rows faster than `
+ 'anything removes them'];
}
if (delta < -TREND_TOLERANCE) return ['shrinking', `${detail}; something is cleaning up`];
return ['stable', `${detail}; nothing is deleting anything`];
}
/** What else does this row still point at? Pure. */
export function orphanResidue(row) {
const doc = row ?? {};
const out = [];
for (const [field, why] of RESIDUE_FIELDS) {
const value = doc[field];
if (Array.isArray(value)) {
if (value.length) out.push([field, `${value.length} ${why}`]);
} else if (value) {
out.push([field, why]);
}
}
return out;
}
/** How much of every sweep is spent on workspaces that left? Pure. */
export function sweepCost(counts, sweepsPerDay = 1) {
const dead = Number((counts ?? {}).dead ?? 0);
const rows = Number((counts ?? {}).rows ?? 0);
const perDay = dead * Math.max(Number(sweepsPerDay ?? 0), 0);
const numbers = {
calls_per_sweep: dead,
calls_per_day: perDay,
share: rows ? Math.round((dead / rows) * 10000) / 10000 : 0,
};
return [dead ? 'wasted' : 'none', numbers];
}
function arg(args, name, fallback = '') {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function authTest(token) {
const r = await fetch(`${API}auth.test`,
{ headers: { Authorization: `Bearer ${token}` } });
try {
return await r.json();
} catch {
return { ok: false, error: 'unparseable_body' };
}
}
async function main() {
const args = process.argv.slice(2);
const storePath = arg(args, '--store');
if (!storePath) {
console.error('pass --store with your installation store as JSON');
process.exitCode = 2;
return;
}
const rows = JSON.parse(await readFile(storePath, 'utf8')) ?? [];
console.log(`store ${rows.length} row(s) read from ${storePath}`);
const states = [];
const deadRows = [];
for (const raw of rows) {
const row = raw ?? {};
const token = row.token ?? '';
// eslint-disable-next-line no-await-in-loop
const body = token ? await authTest(token) : { ok: false, error: 'not_authed' };
const [state, action] = classifyInstall(body);
states.push(state);
const id = String(row.id ?? '').padEnd(16);
const team = row.team_id ?? row.enterprise_id ?? '-';
const line = `row ${id} ${state.padEnd(14)} ${action.padEnd(14)} `
+ `${String(team).padEnd(7)} ${tokenShape(token)}`;
if (state === 'live') console.log(line);
else console.warn(line);
if (DEAD.includes(state)) deadRows.push(row);
}
const [health, counts] = storeHealth(states);
const healthLine = `health ${health.padEnd(14)} ${JSON.stringify(counts)}`;
if (health === 'clean') console.log(healthLine);
else console.warn(healthLine);
let previous = {};
const previousPath = arg(args, '--previous');
if (previousPath) previous = JSON.parse(await readFile(previousPath, 'utf8')) ?? {};
const [trend, detail] = deadTrend(previous, counts);
const trendLine = `trend ${trend.padEnd(14)} ${detail}`;
if (trend === 'shrinking') console.log(trendLine);
else console.warn(trendLine);
for (const row of deadRows) {
const residue = orphanResidue(row);
if (residue.length) {
console.warn(`residue ${String(row.id ?? '').padEnd(16)} `
+ `${residue.map(([, d]) => d).join('; ')}`);
}
}
const [cost, numbers] = sweepCost(counts, Number(arg(args, '--sweeps-per-day', '1')));
const costLine = `cost ${cost.padEnd(14)} ${JSON.stringify(numbers)}`;
if (cost === 'none') console.log(costLine);
else console.warn(costLine);
const findings = (health === 'clean' ? 0 : 1)
+ (trend === 'stable' || trend === 'growing' ? 1 : 0);
if (!findings) {
console.log('verdict clean the store holds no dead rows');
return;
}
console.warn(`verdict ${findings} finding(s)`);
console.warn(' repair: handle app_uninstalled and tokens_revoked, delete the row '
+ 'idempotently on either, and delete every derived record with it');
console.warn(' repair: keep this sweep as a scheduled reconciliation; no event '
+ 'will ever arrive for the installs that ended before the handler existed');
console.warn(' note: token_expired rows were left alone; that is a refresh, not '
+ 'a deletion');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main().
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The fixture tokens are obviously fake and deliberately short, because nothing in these tests needs a credential and a realistic-looking one in a repository is a liability with no upside. The assertions that carry the argument are the two conservative ones: token_expired must map to refresh rather than to tombstone, and an error the classifier has never seen must map to inspect, so that a cleanup written against this never removes a live tenant it did not understand. After that it is the trend, where a flat line has to read as a failure rather than as stability.
from slack_install_store_sweep import (
classify_install, dead_trend, orphan_residue, store_health, sweep_cost,
token_shape,
)
BOT = "xoxb-fake-1"
USER = "xoxp-fake-2"
def test_a_healthy_row_is_kept_and_says_so():
assert classify_install({"ok": True, "team_id": "T1"}) == ("live", "keep")
def test_a_revoked_grant_is_the_one_state_that_earns_a_tombstone():
assert classify_install({"ok": False, "error": "token_revoked"}) \
== ("revoked", "tombstone")
def test_a_deactivated_account_is_dead_but_reported_separately():
state, action = classify_install({"ok": False, "error": "account_inactive"})
assert state == "inactive"
assert action == "tombstone"
def test_an_expired_token_is_refreshed_and_never_deleted():
assert classify_install({"ok": False, "error": "token_expired"}) \
== ("expired", "refresh")
def test_a_rate_limit_in_the_middle_of_a_sweep_is_not_evidence_of_death():
assert classify_install({"ok": False, "error": "ratelimited"}) \
== ("rate-limited", "retry")
def test_an_unreadable_body_is_retried_rather_than_judged():
assert classify_install({}) == ("unreadable", "retry")
def test_an_error_the_classifier_has_never_seen_is_inspected_not_deleted():
state, action = classify_install({"ok": False, "error": "something_new"})
assert state == "unrecognised"
assert action == "inspect"
def test_the_token_shape_is_a_prefix_class_and_nothing_else():
assert token_shape(BOT) == "xoxb"
assert token_shape(USER) == "xoxp"
assert token_shape("xapp-fake-3") == "xapp"
assert token_shape("nope") == "unrecognised"
assert token_shape("") == "unrecognised"
def test_no_part_of_a_token_survives_the_shape_function():
shape = token_shape(BOT)
assert "fake" not in shape
assert shape not in BOT[5:]
def test_a_store_where_most_rows_are_dead_is_dominated():
states = ["live"] + ["revoked"] * 6 + ["inactive"] * 3
state, counts = store_health(states)
assert state == "dominated"
assert counts["dead"] == 9
assert counts["dead_share"] == 0.9
def test_a_single_dead_row_is_residue_rather_than_clean():
state, counts = store_health(["live"] * 99 + ["revoked"])
assert state == "residue"
assert counts["dead"] == 1
def test_recoverable_and_unknown_rows_are_kept_out_of_the_dead_count():
_state, counts = store_health(["live", "expired", "rate-limited", "unrecognised"])
assert counts["dead"] == 0
assert counts["recoverable"] == 1
assert counts["unknown"] == 2
def test_a_store_of_only_live_rows_is_clean():
assert store_health(["live", "live"])[0] == "clean"
def test_an_empty_store_is_empty_rather_than_clean():
assert store_health([])[0] == "empty"
def test_a_flat_dead_share_between_audits_is_a_failure_not_stability():
state, detail = dead_trend({"rows": 400, "dead": 269},
{"rows": 412, "dead": 279})
assert state == "stable"
assert "nothing is deleting anything" in detail
def test_a_rising_dead_share_says_the_store_is_accumulating():
state, detail = dead_trend({"rows": 100, "dead": 10},
{"rows": 100, "dead": 40})
assert state == "growing"
assert "accumulating" in detail
def test_a_falling_dead_share_is_the_only_good_answer():
assert dead_trend({"rows": 100, "dead": 40},
{"rows": 100, "dead": 5})[0] == "shrinking"
def test_without_a_previous_audit_the_script_asks_for_a_second_one():
state, detail = dead_trend({}, {"rows": 10, "dead": 4})
assert state == "no-baseline"
assert "fortnight" in detail
def test_an_empty_audit_cannot_be_compared_at_all():
assert dead_trend({"rows": 10, "dead": 4}, {"rows": 0, "dead": 0})[0] == "no-data"
def test_the_residue_on_a_dead_row_is_named_field_by_field():
row = {"id": "inst-1", "token": BOT,
"scheduled_message_ids": ["Q1", "Q2"],
"webhook_url": "https://hooks.example/x",
"cached_channels": [], "schedule_id": "sched-9"}
codes = [c for c, _d in orphan_residue(row)]
assert codes == ["scheduled_message_ids", "webhook_url", "schedule_id"]
def test_an_empty_collection_is_not_reported_as_residue():
assert orphan_residue({"scheduled_message_ids": [], "cached_channels": []}) == []
def test_a_row_with_nothing_derived_produces_nothing():
assert orphan_residue({"id": "inst-2", "token": BOT}) == []
def test_the_cost_is_reported_per_sweep_and_per_day():
state, numbers = sweep_cost({"rows": 400, "dead": 120}, sweeps_per_day=30)
assert state == "wasted"
assert numbers["calls_per_sweep"] == 120
assert numbers["calls_per_day"] == 3600
assert numbers["share"] == 0.3
def test_a_clean_store_costs_nothing():
assert sweep_cost({"rows": 400, "dead": 0}, sweeps_per_day=30)[0] == "none"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
classifyInstall, deadTrend, orphanResidue, storeHealth, sweepCost, tokenShape,
} from './slack-install-store-sweep.mjs';
const BOT = 'xoxb-fake-1';
const USER = 'xoxp-fake-2';
test('a healthy row is kept and says so', () => {
assert.deepEqual(classifyInstall({ ok: true, team_id: 'T1' }), ['live', 'keep']);
});
test('a revoked grant is the one state that earns a tombstone', () => {
assert.deepEqual(classifyInstall({ ok: false, error: 'token_revoked' }),
['revoked', 'tombstone']);
});
test('a deactivated account is dead but reported separately', () => {
const [state, action] = classifyInstall({ ok: false, error: 'account_inactive' });
assert.equal(state, 'inactive');
assert.equal(action, 'tombstone');
});
test('an expired token is refreshed and never deleted', () => {
assert.deepEqual(classifyInstall({ ok: false, error: 'token_expired' }),
['expired', 'refresh']);
});
test('a rate limit in the middle of a sweep is not evidence of death', () => {
assert.deepEqual(classifyInstall({ ok: false, error: 'ratelimited' }),
['rate-limited', 'retry']);
});
test('an unreadable body is retried rather than judged', () => {
assert.deepEqual(classifyInstall({}), ['unreadable', 'retry']);
});
test('an error the classifier has never seen is inspected not deleted', () => {
const [state, action] = classifyInstall({ ok: false, error: 'something_new' });
assert.equal(state, 'unrecognised');
assert.equal(action, 'inspect');
});
test('the token shape is a prefix class and nothing else', () => {
assert.equal(tokenShape(BOT), 'xoxb');
assert.equal(tokenShape(USER), 'xoxp');
assert.equal(tokenShape('xapp-fake-3'), 'xapp');
assert.equal(tokenShape('nope'), 'unrecognised');
assert.equal(tokenShape(''), 'unrecognised');
});
test('no part of a token survives the shape function', () => {
const shape = tokenShape(BOT);
assert.equal(shape.includes('fake'), false);
assert.equal(BOT.slice(5).includes(shape), false);
});
test('a store where most rows are dead is dominated', () => {
const states = ['live', ...Array(6).fill('revoked'), ...Array(3).fill('inactive')];
const [state, counts] = storeHealth(states);
assert.equal(state, 'dominated');
assert.equal(counts.dead, 9);
assert.equal(counts.dead_share, 0.9);
});
test('a single dead row is residue rather than clean', () => {
const [state, counts] = storeHealth([...Array(99).fill('live'), 'revoked']);
assert.equal(state, 'residue');
assert.equal(counts.dead, 1);
});
test('recoverable and unknown rows are kept out of the dead count', () => {
const [, counts] = storeHealth(['live', 'expired', 'rate-limited', 'unrecognised']);
assert.equal(counts.dead, 0);
assert.equal(counts.recoverable, 1);
assert.equal(counts.unknown, 2);
});
test('a store of only live rows is clean', () => {
assert.equal(storeHealth(['live', 'live'])[0], 'clean');
});
test('an empty store is empty rather than clean', () => {
assert.equal(storeHealth([])[0], 'empty');
});
test('a flat dead share between audits is a failure not stability', () => {
const [state, detail] = deadTrend({ rows: 400, dead: 269 },
{ rows: 412, dead: 279 });
assert.equal(state, 'stable');
assert.match(detail, /nothing is deleting anything/);
});
test('a rising dead share says the store is accumulating', () => {
const [state, detail] = deadTrend({ rows: 100, dead: 10 }, { rows: 100, dead: 40 });
assert.equal(state, 'growing');
assert.match(detail, /accumulating/);
});
test('a falling dead share is the only good answer', () => {
assert.equal(deadTrend({ rows: 100, dead: 40 }, { rows: 100, dead: 5 })[0],
'shrinking');
});
test('without a previous audit the script asks for a second one', () => {
const [state, detail] = deadTrend({}, { rows: 10, dead: 4 });
assert.equal(state, 'no-baseline');
assert.match(detail, /fortnight/);
});
test('an empty audit cannot be compared at all', () => {
assert.equal(deadTrend({ rows: 10, dead: 4 }, { rows: 0, dead: 0 })[0], 'no-data');
});
test('the residue on a dead row is named field by field', () => {
const row = {
id: 'inst-1',
token: BOT,
scheduled_message_ids: ['Q1', 'Q2'],
webhook_url: 'https://hooks.example/x',
cached_channels: [],
schedule_id: 'sched-9',
};
assert.deepEqual(orphanResidue(row).map(([c]) => c),
['scheduled_message_ids', 'webhook_url', 'schedule_id']);
});
test('an empty collection is not reported as residue', () => {
assert.deepEqual(orphanResidue({ scheduled_message_ids: [], cached_channels: [] }),
[]);
});
test('a row with nothing derived produces nothing', () => {
assert.deepEqual(orphanResidue({ id: 'inst-2', token: BOT }), []);
});
test('the cost is reported per sweep and per day', () => {
const [state, numbers] = sweepCost({ rows: 400, dead: 120 }, 30);
assert.equal(state, 'wasted');
assert.equal(numbers.calls_per_sweep, 120);
assert.equal(numbers.calls_per_day, 3600);
assert.equal(numbers.share, 0.3);
});
test('a clean store costs nothing', () => {
assert.equal(sweepCost({ rows: 400, dead: 0 }, 30)[0], 'none');
});
FAQ
Slack already sends app_uninstalled and tokens_revoked. Why do I need a sweep as well?
Because events are a best effort and your store is the record. The two events arrive in an order that is not guaranteed, so a handler written to expect one before the other drops the second. tokens_revoked does not fire for every case, so a store holding user tokens keeps rows the event never mentioned. Events are missed during deploys and restarts. And most decisively, no event will ever arrive for the installs that ended before you wrote the handler, so an app that adds one today still has years of sediment that only a reconciliation pass will find.
Is this not just token_revoked with extra steps?
No, and the difference is the unit. token_revoked is a fact about one workspace: the app was removed there, and there is nothing further to work out. This note is about the population. What fraction of your table is dead, is that fraction going up or down between audits, and what else does each dead row still point at. None of that can be read from an error message, because every individual error looks exactly like ordinary bad luck; it only appears when you count.
Why does the sweep refuse to treat token_expired as dead?
Because it is not. token_expired means token rotation is switched on and the refresh has lapsed, which is a live installation with a stale credential, and the repair is to redeem the refresh token rather than to delete the customer. A cleanup that tombstones on any error at all will eventually run during a rotation outage and remove paying tenants in bulk, which is a far worse bug than the one it was written to fix. The classifier keeps expired in its own column with the action refresh, and the same caution applies to rate limits and unreadable responses.
What counts as a derived record, and why does deleting the token not cover it?
Anything your app created that only makes sense while that installation exists. Scheduled messages queued against channels in that workspace, cached channel and user ids, an incoming webhook URL, a per-tenant cron entry, a row in a billing table. The token is only the credential; these are the work. Delete the token alone and you get a second generation of orphans that fail on their own timetables, most of them silently, and each one eventually produces its own confusing incident.
The sweep says my store is dominated. Is there anything to do besides delete?
Tombstone rather than hard delete, at least at first. Keep the row with a state and a timestamp so that a reinstall from the same workspace can be recognised and so that you can prove what happened when somebody asks why a tenant vanished from a report. Then exclude tombstoned rows from every sweep, which is where the actual saving is, and keep the reconciliation on a schedule so the number keeps falling. The measurement to watch afterwards is the trend, not the total.
Related field notes
- the single dead token, and what it means on its own
- the other way a stored credential stops working
- one kind of derived record the row leaves behind
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- tokens_revoked event reference - Slack Docs
- auth.test method reference - Slack Docs
- bolt-js #1203: the installation store never deletes a record when the app is uninstalled
- bolt-js #673: handling app_uninstalled and tokens_revoked
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.