Diagnostic Slack
Scheduled messages outlive the app that queued them
The reminder bot is redeployed on Tuesday with a rewritten scheduler. On Wednesday morning people start getting two nudges for every task: one from the new code, and one from the old code, for a ticket that was closed nine days ago.
Nobody deployed the old code. Nobody can find it running. The old code is not running — it stopped existing on Tuesday. What is still running is Slack, working through a queue of sends the old scheduler handed it weeks ago and which nothing has ever told it to forget.
chat.scheduleMessage is not a timer inside your app. It is a handover. Slack takes the message, keeps it for up to 120 days, and delivers it at post_at regardless of what happens to you in the meantime. Deleting the database row does not cancel it. Redeploying does not cancel it. Uninstalling and reinstalling the app does not cancel it. Only chat.deleteScheduledMessage, with the scheduled_message_id that the original call returned, cancels it.
So the queue is a resource with a lifecycle, and almost nobody gives it one. Every scheduling change that ships without draining the queue leaves a tail behind it, and the tail is invisible until it fires.
chat.scheduledMessages.list reads the queue, and reading it is the whole diagnosis. Hold each entry's id against your own records: an entry you cannot account for is an orphan, and an entry you can account for but whose reason has closed is worse, because you will find it and decide it is fine. Then check where each one is aimed, because a channel archived last month is a send that is already doomed. Run it twice a week apart and the count itself is the finding: a queue that grows run over run is a queue nobody drains.
The problem in plain words
The reason this is expensive is that the symptom arrives with an alibi. Users report messages from a version of the app that no longer exists, and every instinct says look at what is deployed. What is deployed is fine. The sender is Slack, on your behalf, from an instruction you gave it in the past and can no longer see from inside your own process.
There are four ways a queued message becomes wrong, and they want different repairs. The reason closed. A reminder was scheduled for a task, the task was completed, the row was deleted, and the send was never cancelled because deleting a row is not an API call. The scheduler was replaced. A deploy changed the cadence, the text, or the channel, and the previous version's queue was left to fire alongside the new one, which is the duplicate-reminder shape. The target moved. The channel was archived, made private, or the bot was removed from it in a spring clean, and the send will fail on arrival with is_archived or not_in_channel weeks after anybody could connect the two. The install was replaced. The app was uninstalled and reinstalled, the token is new, and the queue from the old install is still there under the same app.
What all four share is that nothing in your application is in a position to notice. There is no callback when a scheduled message fires, no event, and no error surface for a send that fails at delivery time. The only place the state lives is in Slack's queue, and the only way to see it is to ask.
The queue is also silently capped. A workspace holds a limited number of pending scheduled messages per channel and per app, and an app that schedules without cancelling eventually starts getting rejections on new sends while the old ones sit there. By that point the failure is a scheduling failure, several layers away from the leak that caused it, and it gets fixed by raising a retry count.
And the horizon is long enough to hide the problem across an entire project. 120 days is a third of a year. A message scheduled in March fires in July, into a channel that has been renamed twice, from an app that has been rewritten once, for a person who left in May.
Why it happens
The queue is state you own and cannot see from your own process. Every other piece of scheduling state is in your database, where a migration or a delete reaches it. This one is not. It sits in Slack, it is only reachable by an API call, and the only reason it is ever wrong is that nobody made that call.
An id you cannot account for is the finding, not a count. "There are 340 pending messages" changes nothing. "There are 340 pending messages and 291 of them have ids that appear nowhere in your records" is a decision. The script exists to do that join, because it is the only comparison that separates a healthy backlog from a leak.
A tracked entry whose reason has closed is worse than an untracked one. The orphan is at least obviously wrong. The entry you can still look up is the one you will glance at, confirm exists in your table, and move on from — without noticing the row says the task was completed on the fourth. The script reads the state on your own record rather than only its presence.
Where it is aimed matters as much as whether you meant it. A send you still want, into a channel that was archived, is a failure already scheduled. It is worth knowing now rather than in six weeks, and conversations.info answers it with a read.
Two runs beat one run. A single snapshot cannot tell a queue that is being drained properly from one that is filling up, because both look like a number. The difference between two snapshots does: the ids that fired, the ids that appeared, and the ids that have been sitting there through both.
Cancelling is a write, so this script prints it. The repair for an orphan is chat.deleteScheduledMessage per id, and a script that ran that itself would be one bad ledger away from silently cancelling every reminder in the workspace. It prints the lines. You decide.
The fix, as a flow
The only chain in this batch with a loop under it, and the loop is the whole failure rather than an illustration of it. Each deploy hands Slack a fresh set of sends and takes nothing back, so the queue is topped up faster than it drains and the tail grows quietly for months. The red box is the fourth one, where nothing happens, because nothing happening is the bug: no call was made, so there is no error to find. The fix branch sorts the queue by who still stands behind each entry, and the top two rows are the two that most often get counted together and should not be.
How to fix it
Read the queue before you read anything else
chat.scheduledMessages.list with limit=100 and cursor pagination returns every pending send: id, channel_id, post_at and date_created. This is a read. Run it first and print the raw count, because the count alone is often the moment somebody says it should not be anywhere near that.
Join each entry against your own records by scheduled_message_id
ledger_verdict takes one queue entry and your records and returns tracked, superseded, or unknown-to-you. If you are not storing the id that chat.scheduleMessage returned, every entry comes back unknown-to-you and that is itself the finding: you have no way to cancel any of them.
Separate superseded from unknown rather than counting both as orphans
superseded means your own record says this send should not happen — the task is done, the reminder was cancelled, the row is tombstoned — and the send is still queued anyway. That is a missing cancel call in a specific code path, which is a bug with an address. unknown-to-you is a whole scheduler that never stored its ids, which is a design change.
Check where every pending send is aimed
target_verdict reads conversations.info for each distinct channel in the queue and returns target-archived, target-left, target-unreadable, or deliverable. These are sends that will fail on arrival with nobody watching, and they are worth cancelling even when you still want the message, because the message is not going to happen.
Look at the distribution, not just the total
bucket sorts each entry into within-the-hour, today, this-week, months-out and beyond-horizon. A healthy reminder queue is front-loaded. A queue with a long flat tail months out is a queue that has been accumulating, and the tail is where the orphans concentrate because nobody has been around long enough to remember them.
Run it again a week later and diff the ids
Save the id list, pass it back as --previous, and queue_drift reports what fired, what was added and what has been held through both runs. growing means more is being scheduled than is being cancelled or delivered, which is the leak stated as a rate rather than as a number. The held set is your orphan list, confirmed twice.
How to check it worked
Run it against the workspace and your own scheduling table. The finding is a set of ids and a reason, and the repair is a list of lines you run yourself.
python3 slack_scheduled_orphans.py --ledger reminders.json --previous last-week.json
# identity U024BE7LH in Acme
# queue 312 pending scheduled message(s) across 9 channel(s)
# buckets within-the-hour 2 today 14 this-week 51 months-out 245
# ledger unknown-to-you Q1298ABCD C024BE91L post_at=2026-11-04T09:00Z
# no record of this id; nothing in your app can cancel it
# ledger superseded Q1298XYZW C024BE91L post_at=2026-09-02T09:00Z
# your record says state=completed on 2026-08-20
# target target-archived C07J4K2QT 14 pending send(s) aimed at a
# channel archived on 2026-06-11; every one will fail
# drift growing fired 26 added 61 held 251
# verdict 291 of 312 pending send(s) have no owner
# repair: chat.deleteScheduledMessage channel=C024BE91L scheduled_message_id=Q1298ABCD
# repair: store the scheduled_message_id chat.scheduleMessage returns, beside the
# row that justifies the send, and cancel it when that row closesThe full code
Five pure functions and two reads. ledger_verdict is the join that makes the whole note, and it separates you never knew about this from you knew and changed your mind because those are different bugs. target_verdict asks whether the destination still accepts messages. bucket turns the queue into a shape. queue_drift is the only function here that needs two runs, and it is the one that answers whether this is a backlog or a leak. cancel_line writes the repair out as text and never runs it.
"""Find the scheduled messages Slack still intends to send on your behalf.
Read only. chat.scheduledMessages.list is a read and is the only thing this
touches in the queue; chat.deleteScheduledMessage is a write, so cancellations
are printed as lines for you to run and are never executed here.
The question is not whether post_at was computed correctly. It is whether the
send still has a reason to exist. Your app was redeployed, the task closed, the
channel was archived, and Slack knows about none of that: it took the message
and it will deliver it up to 120 days later exactly as instructed.
"""
import argparse
import json
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_scheduled_orphans")
API = "https://slack.com/api/"
HOUR = 3600
DAY = 86400
HORIZON_DAYS = 120
# States on your own record that mean the send should not happen. Anything else
# is treated as live, because guessing that an unknown state means cancelled is
# the one error that would make this script recommend cancelling a real send.
CLOSED_STATES = ("cancelled", "canceled", "completed", "done", "superseded",
"deleted", "tombstoned", "obsolete")
def ledger_verdict(entry, ledger):
"""Does your application still stand behind this queued send? Pure.
Returns (verdict, detail). tracked is the only one that needs no action.
superseded and unknown-to-you are both orphans, and they are kept apart
because they are different bugs: superseded is one missing cancel call in a
code path you can name, unknown-to-you is a scheduler that never stored the
ids it was given and therefore cannot cancel anything at all.
"""
entry = entry or {}
sid = str(entry.get("id") or "")
if not sid:
return ("unidentified", "the queue entry has no id, so there is nothing to "
"join against and nothing to cancel it with")
row = (ledger or {}).get(sid)
if row is None:
return ("unknown-to-you", "no record of %s. Nothing in your application can "
"cancel this send, because cancelling needs exactly "
"this id and you did not keep it" % sid)
if not isinstance(row, dict):
return ("tracked", "%s appears in your records" % sid)
state = str(row.get("state") or row.get("status") or "live").strip().lower()
if state in CLOSED_STATES:
return ("superseded", "your record says state=%s%s, and the send is still "
"queued. The row was closed and the cancel call was "
"never made" % (state, (" on " + str(row["closed_at"]))
if row.get("closed_at") else ""))
return ("tracked", "%s appears in your records as state=%s" % (sid, state))
def target_verdict(entry, channels):
"""Will the channel this is aimed at still take a message? Pure.
channels maps a channel id to the conversations.info payload, or to None
when the read failed. deliverable does not promise the send is wanted, only
that it is possible; everything else is a send that is already going to
fail, weeks from now, with nobody watching.
"""
entry = entry or {}
cid = str(entry.get("channel_id") or entry.get("channel") or "")
if not cid:
return ("target-missing", "the entry names no channel")
if cid not in (channels or {}):
return ("target-unchecked", "%s was not read; pass channels:read to check "
"where the queue is aimed" % cid)
info = (channels or {}).get(cid)
if not info:
return ("target-unreadable", "%s could not be read. A private channel the "
"bot has been removed from looks exactly like a "
"channel that does not exist" % cid)
if info.get("is_archived") is True:
return ("target-archived", "%s was archived. Archiving is permanent until "
"somebody undoes it by hand, and an archived "
"channel refuses every message" % cid)
if info.get("is_member") is False:
return ("target-left", "the bot is no longer in %s, so this send will fail "
"with not_in_channel when it fires" % cid)
return ("deliverable", "%s still accepts messages from this bot" % cid)
def bucket(entry, now=None):
"""How far into the future is this send, in words? Pure.
A healthy reminder queue is front loaded. A long flat tail months out is
where orphans collect, because by the time those fire nobody remembers
scheduling them.
"""
now = time.time() if now is None else now
try:
post_at = float((entry or {}).get("post_at") or 0)
except (TypeError, ValueError):
return ("unparseable", "post_at is not a number")
if not post_at:
return ("unparseable", "post_at is missing")
delta = post_at - now
if delta < 0:
return ("overdue", "post_at is %.0f second(s) in the past and it is still "
"listed as pending" % -delta)
if delta < HOUR:
return ("within-the-hour", "%.0f minute(s) away" % (delta / 60))
if delta < DAY:
return ("today", "%.1f hour(s) away" % (delta / HOUR))
if delta < 7 * DAY:
return ("this-week", "%.1f day(s) away" % (delta / DAY))
if delta <= HORIZON_DAYS * DAY:
return ("months-out", "%.0f day(s) away" % (delta / DAY))
return ("beyond-horizon", "%.0f day(s) away, past the %d day limit the API "
"accepts" % (delta / DAY, HORIZON_DAYS))
def queue_drift(before, after):
"""What happened to the queue between two runs? Pure.
Returns (verdict, detail, counts). One snapshot is a number and a number
cannot tell a backlog from a leak. Two snapshots can: if more ids appear
than leave, the queue is filling faster than it drains, and the ids present
in both runs are the orphan list confirmed twice.
"""
a = {str(x) for x in (before or [])}
b = {str(x) for x in (after or [])}
counts = {"fired": len(a - b), "added": len(b - a), "held": len(a & b)}
if not a:
return ("first-run", "no previous snapshot, so there is nothing to compare. "
"Keep this id list and pass it back next time",
counts)
if counts["added"] > counts["fired"]:
return ("growing", "%d id(s) appeared and %d left. The queue is being filled "
"faster than it is drained, which is the leak stated as a "
"rate" % (counts["added"], counts["fired"]), counts)
if counts["fired"] > counts["added"]:
return ("draining", "%d id(s) left and %d appeared"
% (counts["fired"], counts["added"]), counts)
return ("flat", "%d id(s) in and %d out; %d have been held through both runs"
% (counts["added"], counts["fired"], counts["held"]), counts)
def cancel_line(entry):
"""The repair for one orphan, as text. Printed, never executed. Pure.
Cancelling is a write and this script does not make writes. It is also the
one operation here that a wrong ledger would turn into deleting every
reminder in the workspace, which is a decision for a person.
"""
entry = entry or {}
return ("chat.deleteScheduledMessage channel=%s scheduled_message_id=%s"
% (entry.get("channel_id") or entry.get("channel") or "?",
entry.get("id") or "?"))
def load_ledger(path):
"""Read your own scheduling records. Deliberately forgiving of shape."""
if not path:
return {}
raw = json.loads(open(path, encoding="utf-8").read())
if isinstance(raw, dict) and not any(isinstance(v, dict) for v in raw.values()):
return {str(k): {"state": str(v)} for k, v in raw.items()}
if isinstance(raw, dict):
rows = raw.get("scheduled") or raw.get("records") or raw
if isinstance(rows, dict):
return {str(k): v for k, v in rows.items()}
raw = rows
out = {}
for row in raw or []:
if not isinstance(row, dict):
out[str(row)] = {}
continue
sid = (row.get("scheduled_message_id") or row.get("id")
or row.get("slack_id") or "")
if sid:
out[str(sid)] = row
return out
def read_queue(session, channel=""):
"""Follow the cursor to the end. Listing is a read; cancelling is not."""
entries, cursor = [], ""
while True:
params = {"limit": "100"}
if channel:
params["channel"] = channel
if cursor:
params["cursor"] = cursor
body = session.get(API + "chat.scheduledMessages.list", timeout=30,
params=params).json()
if body.get("ok") is not True:
log.error("queue unavailable %s", body.get("error"))
return entries, False
entries.extend(body.get("scheduled_messages") or [])
cursor = ((body.get("response_metadata") or {}).get("next_cursor") or "")
if not cursor:
return entries, True
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ledger", default="",
help="a JSON file of your own scheduling records, keyed by or "
"carrying the scheduled_message_id Slack returned")
ap.add_argument("--previous", default="",
help="a JSON array of ids emitted by an earlier run")
ap.add_argument("--channel", default="", help="limit the queue read to one channel")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding the bot token")
ap.add_argument("--emit", action="store_true",
help="print the current id list as JSON on stdout, to save and "
"pass back as --previous next time")
ap.add_argument("--skip-targets", action="store_true",
help="do not read conversations.info for the queued channels")
args = ap.parse_args()
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token that can list its own scheduled messages",
args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s in %s", who.get("user_id"), who.get("team"))
entries, complete = read_queue(s, args.channel)
if not complete:
return 2
channels = sorted({str(e.get("channel_id") or "") for e in entries} - {""})
log.info("queue %d pending scheduled message(s) across %d channel(s)",
len(entries), len(channels))
shape = {}
for entry in entries:
verdict, _ = bucket(entry)
shape[verdict] = shape.get(verdict, 0) + 1
log.info("buckets %s", " ".join("%s %d" % kv for kv in sorted(shape.items())))
info = {}
if not args.skip_targets:
for cid in channels:
body = s.get(API + "conversations.info", timeout=30,
params={"channel": cid}).json()
info[cid] = body.get("channel") if body.get("ok") is True else None
ledger = load_ledger(args.ledger)
if args.ledger:
log.info("ledger %d record(s) read from %s", len(ledger), args.ledger)
orphans, doomed = [], {}
for entry in entries:
verdict, detail = ledger_verdict(entry, ledger)
if verdict != "tracked":
orphans.append(entry)
log.warning("ledger %-14s %s %s post_at=%s", verdict,
entry.get("id"), entry.get("channel_id"), entry.get("post_at"))
log.warning(" %s", detail)
tverdict, tdetail = target_verdict(entry, info)
if tverdict not in ("deliverable", "target-unchecked"):
doomed.setdefault((entry.get("channel_id"), tverdict, tdetail), 0)
doomed[(entry.get("channel_id"), tverdict, tdetail)] += 1
for (cid, tverdict, tdetail), count in sorted(doomed.items(), key=str):
log.warning("target %-16s %s %d pending send(s); %s",
tverdict, cid, count, tdetail)
ids = [str(e.get("id") or "") for e in entries if e.get("id")]
if args.previous:
before = json.loads(open(args.previous, encoding="utf-8").read())
dverdict, ddetail, counts = queue_drift(before, ids)
line = "drift %-14s fired %d added %d held %d" % (
dverdict, counts["fired"], counts["added"], counts["held"])
(log.warning if dverdict == "growing" else log.info)(line)
log.info(" %s", ddetail)
if args.emit:
print(json.dumps(ids))
if orphans or doomed:
log.warning("verdict %d of %d pending send(s) have no owner",
len(orphans), len(entries))
for entry in orphans[:10]:
log.warning(" repair: %s", cancel_line(entry))
if len(orphans) > 10:
log.warning(" repair: and %d more, listed the same way",
len(orphans) - 10)
log.warning(" repair: store the scheduled_message_id chat.scheduleMessage "
"returns beside the row that justifies the send, and cancel it "
"when that row closes")
log.warning(" repair: drain the queue as a migration step on any deploy that "
"changes scheduling semantics, rather than leaving it to fire")
return 1
log.info("verdict clean every pending send is accounted for")
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find the scheduled messages Slack still intends to send on your behalf.
*
* Read only. chat.scheduledMessages.list is a read and is the only thing this
* touches in the queue; chat.deleteScheduledMessage is a write, so
* cancellations are printed as lines for you to run and are never executed.
*
* The question is not whether post_at was computed correctly. It is whether
* the send still has a reason to exist. Your app was redeployed, the task
* closed, the channel was archived, and Slack knows about none of that.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://slack.com/api/';
const HOUR = 3600;
const DAY = 86400;
const HORIZON_DAYS = 120;
// States on your own record that mean the send should not happen. Anything
// else is treated as live: guessing that an unknown state means cancelled is
// the one error that would make this recommend cancelling a real send.
const CLOSED_STATES = ['cancelled', 'canceled', 'completed', 'done', 'superseded',
'deleted', 'tombstoned', 'obsolete'];
/**
* Does your application still stand behind this queued send? Pure.
* Returns [verdict, detail]; tracked is the only one that needs no action.
*/
export function ledgerVerdict(entry, ledger) {
const e = entry ?? {};
const sid = String(e.id ?? '');
if (!sid) {
return ['unidentified', 'the queue entry has no id, so there is nothing to join '
+ 'against and nothing to cancel it with'];
}
const row = (ledger ?? {})[sid];
if (row === undefined || row === null) {
return ['unknown-to-you', `no record of ${sid}. Nothing in your application can `
+ 'cancel this send, because cancelling needs exactly this id and you did not '
+ 'keep it'];
}
if (typeof row !== 'object') return ['tracked', `${sid} appears in your records`];
const state = String(row.state ?? row.status ?? 'live').trim().toLowerCase();
if (CLOSED_STATES.includes(state)) {
return ['superseded', `your record says state=${state}`
+ `${row.closed_at ? ` on ${row.closed_at}` : ''}, and the send is still queued. `
+ 'The row was closed and the cancel call was never made'];
}
return ['tracked', `${sid} appears in your records as state=${state}`];
}
/**
* Will the channel this is aimed at still take a message? Pure.
* channels maps a channel id to the conversations.info payload, or to null.
*/
export function targetVerdict(entry, channels) {
const e = entry ?? {};
const cid = String(e.channel_id ?? e.channel ?? '');
if (!cid) return ['target-missing', 'the entry names no channel'];
if (!Object.prototype.hasOwnProperty.call(channels ?? {}, cid)) {
return ['target-unchecked', `${cid} was not read; pass channels:read to check `
+ 'where the queue is aimed'];
}
const info = (channels ?? {})[cid];
if (!info) {
return ['target-unreadable', `${cid} could not be read. A private channel the bot `
+ 'has been removed from looks exactly like a channel that does not exist'];
}
if (info.is_archived === true) {
return ['target-archived', `${cid} was archived. Archiving is permanent until `
+ 'somebody undoes it by hand, and an archived channel refuses every message'];
}
if (info.is_member === false) {
return ['target-left', `the bot is no longer in ${cid}, so this send will fail `
+ 'with not_in_channel when it fires'];
}
return ['deliverable', `${cid} still accepts messages from this bot`];
}
/** How far into the future is this send, in words? Pure. */
export function bucket(entry, now = Date.now() / 1000) {
const raw = (entry ?? {}).post_at;
const postAt = Number(raw);
if (raw === undefined || raw === null || raw === '' || !Number.isFinite(postAt)) {
return ['unparseable', 'post_at is missing or is not a number'];
}
if (!postAt) return ['unparseable', 'post_at is missing'];
const delta = postAt - now;
if (delta < 0) {
return ['overdue', `post_at is ${(-delta).toFixed(0)} second(s) in the past and it `
+ 'is still listed as pending'];
}
if (delta < HOUR) return ['within-the-hour', `${(delta / 60).toFixed(0)} minute(s) away`];
if (delta < DAY) return ['today', `${(delta / HOUR).toFixed(1)} hour(s) away`];
if (delta < 7 * DAY) return ['this-week', `${(delta / DAY).toFixed(1)} day(s) away`];
if (delta <= HORIZON_DAYS * DAY) {
return ['months-out', `${(delta / DAY).toFixed(0)} day(s) away`];
}
return ['beyond-horizon', `${(delta / DAY).toFixed(0)} day(s) away, past the `
+ `${HORIZON_DAYS} day limit the API accepts`];
}
/**
* What happened to the queue between two runs? Pure.
* Returns [verdict, detail, counts].
*/
export function queueDrift(before, after) {
const a = new Set((before ?? []).map(String));
const b = new Set((after ?? []).map(String));
const counts = {
fired: [...a].filter((x) => !b.has(x)).length,
added: [...b].filter((x) => !a.has(x)).length,
held: [...a].filter((x) => b.has(x)).length,
};
if (!a.size) {
return ['first-run', 'no previous snapshot, so there is nothing to compare. Keep '
+ 'this id list and pass it back next time', counts];
}
if (counts.added > counts.fired) {
return ['growing', `${counts.added} id(s) appeared and ${counts.fired} left. The `
+ 'queue is being filled faster than it is drained, which is the leak stated as '
+ 'a rate', counts];
}
if (counts.fired > counts.added) {
return ['draining', `${counts.fired} id(s) left and ${counts.added} appeared`,
counts];
}
return ['flat', `${counts.added} id(s) in and ${counts.fired} out; ${counts.held} `
+ 'have been held through both runs', counts];
}
/** The repair for one orphan, as text. Printed, never executed. Pure. */
export function cancelLine(entry) {
const e = entry ?? {};
return `chat.deleteScheduledMessage channel=${e.channel_id ?? e.channel ?? '?'} `
+ `scheduled_message_id=${e.id ?? '?'}`;
}
function loadLedger(raw) {
if (!raw) return {};
if (!Array.isArray(raw) && typeof raw === 'object') {
const values = Object.values(raw);
if (values.length && !values.some((v) => v && typeof v === 'object')) {
return Object.fromEntries(Object.entries(raw).map(([k, v]) => [k, { state: String(v) }]));
}
const rows = raw.scheduled ?? raw.records ?? raw;
if (!Array.isArray(rows)) return rows;
return loadLedger(rows);
}
const out = {};
for (const row of raw ?? []) {
if (!row || typeof row !== 'object') { out[String(row)] = {}; continue; }
const sid = row.scheduled_message_id ?? row.id ?? row.slack_id ?? '';
if (sid) out[String(sid)] = row;
}
return out;
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function main() {
const args = process.argv.slice(2);
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} to a bot token that can list its own scheduled `
+ 'messages');
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const read = async (method, params = {}) => {
const qs = new URLSearchParams(params).toString();
return (await fetch(`${API}${method}${qs ? `?${qs}` : ''}`, { headers })).json();
};
const who = await read('auth.test');
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} in ${who.team}`);
const channel = arg(args, '--channel', '');
const entries = [];
let cursor = '';
for (;;) {
const params = { limit: '100' };
if (channel) params.channel = channel;
if (cursor) params.cursor = cursor;
// eslint-disable-next-line no-await-in-loop
const body = await read('chat.scheduledMessages.list', params);
if (body.ok !== true) {
console.error(`queue unavailable ${body.error}`);
process.exitCode = 2;
return;
}
entries.push(...(body.scheduled_messages ?? []));
cursor = body.response_metadata?.next_cursor ?? '';
if (!cursor) break;
}
const channels = [...new Set(entries.map((e) => String(e.channel_id ?? '')))]
.filter(Boolean).sort();
console.log(`queue ${entries.length} pending scheduled message(s) across `
+ `${channels.length} channel(s)`);
const shape = {};
for (const entry of entries) {
const [verdict] = bucket(entry);
shape[verdict] = (shape[verdict] ?? 0) + 1;
}
console.log(`buckets ${Object.entries(shape).sort()
.map(([k, v]) => `${k} ${v}`).join(' ')}`);
const info = {};
if (!args.includes('--skip-targets')) {
for (const cid of channels) {
// eslint-disable-next-line no-await-in-loop
const body = await read('conversations.info', { channel: cid });
info[cid] = body.ok === true ? body.channel : null;
}
}
const ledgerFile = arg(args, '--ledger', '');
const ledger = ledgerFile
? loadLedger(JSON.parse(await readFile(ledgerFile, 'utf8')))
: {};
if (ledgerFile) {
console.log(`ledger ${Object.keys(ledger).length} record(s) read from ${ledgerFile}`);
}
const orphans = [];
const doomed = new Map();
for (const entry of entries) {
const [verdict, detail] = ledgerVerdict(entry, ledger);
if (verdict !== 'tracked') {
orphans.push(entry);
console.warn(`ledger ${verdict.padEnd(14)} ${entry.id} ${entry.channel_id} `
+ `post_at=${entry.post_at}`);
console.warn(` ${detail}`);
}
const [tv, td] = targetVerdict(entry, info);
if (tv !== 'deliverable' && tv !== 'target-unchecked') {
const key = `${entry.channel_id}\u0000${tv}\u0000${td}`;
doomed.set(key, (doomed.get(key) ?? 0) + 1);
}
}
for (const [key, count] of [...doomed.entries()].sort()) {
const [cid, tv, td] = key.split('\u0000');
console.warn(`target ${tv.padEnd(16)} ${cid} ${count} pending send(s); ${td}`);
}
const ids = entries.map((e) => String(e.id ?? '')).filter(Boolean);
const previous = arg(args, '--previous', '');
if (previous) {
const before = JSON.parse(await readFile(previous, 'utf8'));
const [dv, dd, counts] = queueDrift(before, ids);
const line = `drift ${dv.padEnd(14)} fired ${counts.fired} `
+ `added ${counts.added} held ${counts.held}`;
if (dv === 'growing') console.warn(line); else console.log(line);
console.log(` ${dd}`);
}
if (args.includes('--emit')) console.log(JSON.stringify(ids));
if (orphans.length || doomed.size) {
console.warn(`verdict ${orphans.length} of ${entries.length} pending send(s) `
+ 'have no owner');
for (const entry of orphans.slice(0, 10)) console.warn(` repair: ${cancelLine(entry)}`);
if (orphans.length > 10) {
console.warn(` repair: and ${orphans.length - 10} more, listed the same way`);
}
console.warn(' repair: store the scheduled_message_id chat.scheduleMessage returns '
+ 'beside the row that justifies the send, and cancel it when that row closes');
console.warn(' repair: drain the queue as a migration step on any deploy that '
+ 'changes scheduling semantics, rather than leaving it to fire');
process.exitCode = 1;
} else {
console.log('verdict clean every pending send is accounted for');
}
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The assertions worth having are the ones that keep two orphans apart and keep one non-orphan out. ledger_verdict has to call a closed record superseded rather than folding it in with the ids you never stored, because those two findings send you to different code. It also has to treat an unrecognised state as live, since the cost of a false superseded is a cancelled reminder somebody needed. And queue_drift has to distinguish a first run from a flat one, because zero movement and no comparison are not the same answer.
from slack_scheduled_orphans import (bucket, cancel_line, ledger_verdict, queue_drift,
target_verdict)
NOW = 1_800_000_000.0
def entry(**kw):
row = {"id": "Q1298ABCD", "channel_id": "C024BE91L", "post_at": NOW + 3600,
"date_created": NOW}
row.update(kw)
return row
def test_an_id_you_stored_is_tracked():
verdict, detail = ledger_verdict(entry(), {"Q1298ABCD": {"state": "live"}})
assert verdict == "tracked"
assert "Q1298ABCD" in detail
def test_an_id_you_never_stored_is_the_orphan_you_cannot_cancel():
verdict, detail = ledger_verdict(entry(), {})
assert verdict == "unknown-to-you"
assert "can cancel this send" in detail
def test_a_closed_record_with_a_live_send_is_its_own_verdict():
verdict, detail = ledger_verdict(
entry(), {"Q1298ABCD": {"state": "completed", "closed_at": "2026-08-20"}})
assert verdict == "superseded"
assert "2026-08-20" in detail
def test_superseded_and_unknown_are_never_collapsed():
assert ledger_verdict(entry(), {})[0] != ledger_verdict(
entry(), {"Q1298ABCD": {"state": "cancelled"}})[0]
def test_an_unrecognised_state_is_treated_as_live():
for state in ("live", "pending", "queued", "wat", ""):
assert ledger_verdict(entry(), {"Q1298ABCD": {"state": state}})[0] == "tracked"
def test_an_entry_with_no_id_cannot_be_joined_or_cancelled():
verdict, detail = ledger_verdict(entry(id=""), {"Q1298ABCD": {}})
assert verdict == "unidentified"
assert "nothing to cancel" in detail
def test_a_plain_string_record_still_counts_as_tracked():
assert ledger_verdict(entry(), {"Q1298ABCD": "live"})[0] == "tracked"
def test_a_channel_that_still_takes_messages_is_deliverable():
verdict, _ = target_verdict(entry(), {"C024BE91L": {"is_archived": False,
"is_member": True}})
assert verdict == "deliverable"
def test_an_archived_target_is_a_send_that_is_already_doomed():
verdict, detail = target_verdict(entry(), {"C024BE91L": {"is_archived": True}})
assert verdict == "target-archived"
assert "refuses every message" in detail
def test_a_channel_the_bot_left_names_the_error_it_will_fail_with():
verdict, detail = target_verdict(entry(), {"C024BE91L": {"is_archived": False,
"is_member": False}})
assert verdict == "target-left"
assert "not_in_channel" in detail
def test_an_unreadable_channel_is_not_reported_as_gone():
verdict, detail = target_verdict(entry(), {"C024BE91L": None})
assert verdict == "target-unreadable"
assert "looks exactly like" in detail
def test_a_channel_that_was_never_read_is_unchecked_rather_than_fine():
assert target_verdict(entry(), {})[0] == "target-unchecked"
def test_the_buckets_sort_by_distance_into_the_future():
assert bucket(entry(post_at=NOW + 600), NOW)[0] == "within-the-hour"
assert bucket(entry(post_at=NOW + 5 * 3600), NOW)[0] == "today"
assert bucket(entry(post_at=NOW + 3 * 86400), NOW)[0] == "this-week"
assert bucket(entry(post_at=NOW + 60 * 86400), NOW)[0] == "months-out"
def test_a_send_past_the_horizon_is_named_as_past_it():
verdict, detail = bucket(entry(post_at=NOW + 200 * 86400), NOW)
assert verdict == "beyond-horizon"
assert "120 day limit" in detail
def test_a_pending_entry_behind_the_clock_is_overdue():
assert bucket(entry(post_at=NOW - 90), NOW)[0] == "overdue"
def test_a_missing_post_at_is_unparseable_rather_than_zero():
assert bucket(entry(post_at=None), NOW)[0] == "unparseable"
assert bucket(entry(post_at="soon"), NOW)[0] == "unparseable"
def test_more_arriving_than_leaving_is_a_leak_stated_as_a_rate():
verdict, detail, counts = queue_drift(["a", "b"], ["b", "c", "d"])
assert verdict == "growing"
assert counts == {"fired": 1, "added": 2, "held": 1}
assert "faster than it is drained" in detail
def test_more_leaving_than_arriving_is_draining():
verdict, _, counts = queue_drift(["a", "b", "c"], ["c"])
assert verdict == "draining"
assert counts["fired"] == 2
def test_no_previous_snapshot_is_not_the_same_answer_as_no_movement():
assert queue_drift([], ["a"])[0] == "first-run"
assert queue_drift(["a"], ["a"])[0] == "flat"
def test_the_held_set_is_the_orphan_list_confirmed_twice():
_, _, counts = queue_drift(["a", "b", "c"], ["b", "c", "d"])
assert counts["held"] == 2
def test_the_cancel_line_is_text_and_carries_both_arguments():
line = cancel_line(entry())
assert "chat.deleteScheduledMessage" in line
assert "channel=C024BE91L" in line
assert "scheduled_message_id=Q1298ABCD" in line
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
bucket, cancelLine, ledgerVerdict, queueDrift, targetVerdict,
} from './slack-scheduled-orphans.mjs';
const NOW = 1800000000;
const entry = (over = {}) => ({
id: 'Q1298ABCD', channel_id: 'C024BE91L', post_at: NOW + 3600, date_created: NOW,
...over,
});
test('an id you stored is tracked', () => {
const [verdict, detail] = ledgerVerdict(entry(), { Q1298ABCD: { state: 'live' } });
assert.equal(verdict, 'tracked');
assert.match(detail, /Q1298ABCD/);
});
test('an id you never stored is the orphan you cannot cancel', () => {
const [verdict, detail] = ledgerVerdict(entry(), {});
assert.equal(verdict, 'unknown-to-you');
assert.match(detail, /can cancel this send/);
});
test('a closed record with a live send is its own verdict', () => {
const [verdict, detail] = ledgerVerdict(entry(),
{ Q1298ABCD: { state: 'completed', closed_at: '2026-08-20' } });
assert.equal(verdict, 'superseded');
assert.match(detail, /2026-08-20/);
});
test('superseded and unknown are never collapsed', () => {
assert.notEqual(ledgerVerdict(entry(), {})[0],
ledgerVerdict(entry(), { Q1298ABCD: { state: 'cancelled' } })[0]);
});
test('an unrecognised state is treated as live', () => {
for (const state of ['live', 'pending', 'queued', 'wat', '']) {
assert.equal(ledgerVerdict(entry(), { Q1298ABCD: { state } })[0], 'tracked');
}
});
test('an entry with no id cannot be joined or cancelled', () => {
const [verdict, detail] = ledgerVerdict(entry({ id: '' }), { Q1298ABCD: {} });
assert.equal(verdict, 'unidentified');
assert.match(detail, /nothing to cancel/);
});
test('a plain string record still counts as tracked', () => {
assert.equal(ledgerVerdict(entry(), { Q1298ABCD: 'live' })[0], 'tracked');
});
test('a channel that still takes messages is deliverable', () => {
assert.equal(targetVerdict(entry(),
{ C024BE91L: { is_archived: false, is_member: true } })[0], 'deliverable');
});
test('an archived target is a send that is already doomed', () => {
const [verdict, detail] = targetVerdict(entry(), { C024BE91L: { is_archived: true } });
assert.equal(verdict, 'target-archived');
assert.match(detail, /refuses every message/);
});
test('a channel the bot left names the error it will fail with', () => {
const [verdict, detail] = targetVerdict(entry(),
{ C024BE91L: { is_archived: false, is_member: false } });
assert.equal(verdict, 'target-left');
assert.match(detail, /not_in_channel/);
});
test('an unreadable channel is not reported as gone', () => {
const [verdict, detail] = targetVerdict(entry(), { C024BE91L: null });
assert.equal(verdict, 'target-unreadable');
assert.match(detail, /looks exactly like/);
});
test('a channel that was never read is unchecked rather than fine', () => {
assert.equal(targetVerdict(entry(), {})[0], 'target-unchecked');
});
test('the buckets sort by distance into the future', () => {
assert.equal(bucket(entry({ post_at: NOW + 600 }), NOW)[0], 'within-the-hour');
assert.equal(bucket(entry({ post_at: NOW + 5 * 3600 }), NOW)[0], 'today');
assert.equal(bucket(entry({ post_at: NOW + 3 * 86400 }), NOW)[0], 'this-week');
assert.equal(bucket(entry({ post_at: NOW + 60 * 86400 }), NOW)[0], 'months-out');
});
test('a send past the horizon is named as past it', () => {
const [verdict, detail] = bucket(entry({ post_at: NOW + 200 * 86400 }), NOW);
assert.equal(verdict, 'beyond-horizon');
assert.match(detail, /120 day limit/);
});
test('a pending entry behind the clock is overdue', () => {
assert.equal(bucket(entry({ post_at: NOW - 90 }), NOW)[0], 'overdue');
});
test('a missing post_at is unparseable rather than zero', () => {
assert.equal(bucket(entry({ post_at: null }), NOW)[0], 'unparseable');
assert.equal(bucket(entry({ post_at: 'soon' }), NOW)[0], 'unparseable');
});
test('more arriving than leaving is a leak stated as a rate', () => {
const [verdict, detail, counts] = queueDrift(['a', 'b'], ['b', 'c', 'd']);
assert.equal(verdict, 'growing');
assert.deepEqual(counts, { fired: 1, added: 2, held: 1 });
assert.match(detail, /faster than it is drained/);
});
test('more leaving than arriving is draining', () => {
const [verdict, , counts] = queueDrift(['a', 'b', 'c'], ['c']);
assert.equal(verdict, 'draining');
assert.equal(counts.fired, 2);
});
test('no previous snapshot is not the same answer as no movement', () => {
assert.equal(queueDrift([], ['a'])[0], 'first-run');
assert.equal(queueDrift(['a'], ['a'])[0], 'flat');
});
test('the held set is the orphan list confirmed twice', () => {
const [, , counts] = queueDrift(['a', 'b', 'c'], ['b', 'c', 'd']);
assert.equal(counts.held, 2);
});
test('the cancel line is text and carries both arguments', () => {
const line = cancelLine(entry());
assert.match(line, /chat\.deleteScheduledMessage/);
assert.match(line, /channel=C024BE91L/);
assert.match(line, /scheduled_message_id=Q1298ABCD/);
});
FAQ
Does uninstalling the app cancel its scheduled messages?
No. Neither does reinstalling it, rotating the token, deleting the database, or deleting the code. The scheduled message lives in Slack, keyed by an id Slack gave you, and the only thing that removes it is chat.deleteScheduledMessage with that id. This surprises people because every other kind of pending work in an integration dies with the process that created it.
How is this different from a scheduled message that fails with time_in_past?
That failure happens at scheduling time and is arithmetic: post_at was computed in milliseconds, or in local time, or too close to now. This one happens at delivery time and is bookkeeping: post_at was perfect, Slack accepted it, and then the world changed underneath it. The first never enters the queue. The second is already in it.
We do not store the scheduled_message_id at all. Where does that leave us?
With a queue you can read and cannot cancel, which is exactly what the script will tell you: every entry comes back unknown-to-you. Reading gives you back the ids, so a one-off drain is still possible by hand, matching entries to channels and post_at values. The lasting fix is to store the id the scheduling call returns beside the row that justifies the send.
Is there an event when a scheduled message fires or fails?
No. The delivery is invisible from your side. If the channel was archived in the meantime the send simply fails, at a moment when nothing of yours is running and nothing is watching, which is why an audit that checks where the queue is aimed is worth more here than logging would be.
Why does the script print the cancel commands instead of running them?
Because everything in this section reads, and because this is the one repair where a mistake is expensive in the other direction. If your ledger is incomplete for a boring reason, a script with a write path would quietly cancel real reminders that people are relying on. Printed lines let you look at the list first.
Related field notes
- the other scheduling failure, at the other end of the call
- where a queued send lands when the channel closed
- why a reinstall does not clear anything the old install left
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- chat.scheduledMessages.list method reference - Slack Docs
- chat.deleteScheduledMessage method reference - Slack Docs
- chat.scheduleMessage method reference - Slack Docs
- conversations.info method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.