Diagnostic Slack
file_deleted: the message survives and the file does not
The knowledge base was built two years ago by walking every channel, pulling every attachment, and storing the file ids beside the extracted text. It worked. It still works, for most of it. Roughly one search result in nine now opens to {"ok": false, "error": "file_deleted"}, and last quarter it was one in twelve.
Nobody broke anything. People delete files, admins tidy up, and retention policies run on a schedule. What nobody did was tell the index — and the messages that carried those files are still sitting in the channels, still rendering a link, still looking exactly like they did when the file was there.
Deleting a file removes the file. It does not touch the message that linked to it, the id you stored in a database, or the search index you built from either. The reference is not invalidated, it is orphaned, and a F id stays a plausible-looking string forever, so nothing fails until somebody follows it.
The symptom shows up on the message side and the cause is on the file side. In channel history a deleted attachment leaves a stub: the message still exists, its files array still has an entry, and that entry has lost the fields that made it a file — often carrying mode: "tombstone", always missing url_private and mimetype. Links pasted as text are worse: the URL is still there in full and there is nothing about it that looks wrong.
Measure it rather than fixing rows. Harvest every file reference you can find — from conversations.history, from your own stored ids — batch them through files.info, and count. The number worth reporting is the dangling fraction of the references you could decide: file_deleted against ok. A not_visible in the same batch is a membership problem and must be excluded, or the number measures two things at once and means neither.
The problem in plain words
The reason this decays instead of breaking is that every individual piece behaves correctly. Slack deletes the file because someone asked it to. The message stays because deleting a file is not deleting a message. Your database keeps the row because nothing told it otherwise. Each component is right and the system as a whole has quietly stopped being true, one file at a time, at whatever rate your workspace deletes things.
Slack does emit a file_deleted event, and subscribing to it is the actual fix. Almost nobody does, because the event exists for a case that has not happened yet when the integration is written: the day you ingest a file, its deletion is hypothetical. It stays hypothetical for months. By the time the first support ticket arrives about a broken link, there are thousands of stale rows and no record of which ones went when, because the event you were not subscribed to is the only thing that would have told you.
The message-side view is the part that misleads people during the investigation. Someone reports a broken link, a developer opens the channel, and the message is right there with an attachment on it. It looks completely normal in the client - the deletion is rendered as a small absence rather than as an error. The developer concludes the app has the wrong id. In the API the same message carries a files entry stripped down to almost nothing, and it is a stub rather than a file, but only if you know to look at which fields have gone.
And there is a second, quieter source of the same symptom that must not be confused with it. A message may reference a file the bot simply cannot read, which answers not_visible, and that has nothing to do with deletion. If those land in the same bucket, the dangling fraction rises whenever the bot is removed from a channel and falls whenever it is added to one, which makes the number useless for the thing it is meant to measure: whether your references are rotting and how fast.
Why it happens
The symptom is on the message side and the cause is on the file side. That split is why this gets misdiagnosed. What people see is an old post with a link that opens nothing; what happened is a file object being removed somewhere else entirely, possibly by a retention policy with no human involved. The script deliberately reads both ends - references out of history, verdicts out of files.info - because either alone tells half the story.
A file id never stops looking valid. There is no checksum, no expiry in the string, and no shape difference between an id that resolves and one that does not. This is the property that makes the decay silent, and it is the reason the only honest check is to ask the API about every reference rather than to validate them locally.
The fraction is the finding; the count is noise. Forty dangling references means nothing on its own. Forty out of four hundred means one in ten of your index is dead, and forty out of forty thousand means somebody deleted a folder once. The script divides, and it divides only by the references it could actually decide.
not_visible is excluded from the denominator on purpose. Unreadable is not deleted. Mixing them makes the number move when the bot's channel membership changes, which is a different note and a different repair, and a metric that responds to two unrelated causes cannot be used to detect either.
Links pasted as text rot invisibly, and blocks hide them. A file permalink can sit in a message's text, inside a rich_text block several levels down, or in an attachment. Harvesting only the files array finds the attachments and misses every link somebody pasted, which is usually the older and more valuable half of the corpus.
The repair is an event subscription, and the fallback is a tombstone. Subscribe to file_deleted and remove the reference when it fires. For everything already stored, re-validate on a schedule and mark the dead ones as dead rather than serving a link that fails - a result that says this file was deleted is a better answer than a link that goes nowhere, and it costs one column.
The fix, as a flow
The only chain in this batch with a loop under it, and the loop is the whole point: this is the one failure of the four that gets worse while nobody touches anything. Every ingestion adds references and every deletion quietly kills some of them, so the corpus decays at whatever rate the workspace deletes. The red box is late because the reference was fine on the day it was stored. The fix branch sorts the fate of a stored id, and the fourth row is the one deliberately kept out of the number.
How to fix it
Harvest references from both places they hide
file_ids_in takes one message and returns every file id it mentions: the entries in files, and the ids inside any Slack file permalink in text, in nested blocks, or in attachments. The pasted links are the half that a naive harvest misses, and they are usually the older half.
Read the message-side stub before you ask the API anything
attachment_state classifies a files entry as tombstone, stub or present. A tombstone is Slack telling you outright; a stub is an entry that has lost mimetype and url_private, which happens both for deletion and for a file this token cannot read. The stub verdict is a suspicion, not a verdict, and the script says so.
Batch the unique ids through files.info and classify each answer
classify_ref maps a response to live, deleted, unknown, unreadable or undecided. Deduplicate first: a popular file referenced in forty messages is one API call, and the rate limit on files.info is not generous.
Compute the fraction over decidable references only
rot divides deleted by deleted + live. Everything else - unreadable, undecided - is reported beside the fraction and kept out of it. A rot rate that moves when the bot joins a channel is measuring channel membership, not rot.
Run it twice, a month apart, and compare
A single run gives you a number with no meaning attached. Two runs give you a direction, and a direction is what tells you whether this is a historical clean-up that already happened or a policy that is still eating your corpus. The script prints the fraction in a form that is trivial to store.
Subscribe to file_deleted, then tombstone what is already dead
The repair is two changes. Subscribe to the file_deleted event with files:read so future deletions remove the reference at the time they happen. Then mark the existing dead references as dead rather than deleting the rows - a search result that says the file was removed is a better answer than a link that fails, and it also preserves the evidence for the next run.
How to check it worked
The output to keep is the last line. Store it, run it again next month, and compare the two numbers rather than reading either one on its own.
python3 slack_file_link_rot.py --channel C024BE91L --refs index-file-ids.json
# identity U07BOT9QD in Northwind
# refs 412 reference(s): 316 from history, 96 from index-file-ids.json
# message tombstone F06AAA1QQ Slack has already replaced this attachment
# message stub F06BBB2QQ an id and nothing else; files.info decides
# check deleted F06AAA1QQ the file is gone and the message that carried
# it is still in the channel
# check unreadable F07CCC3QQ not_visible, which is membership and not rot
# rot 47 of 391 decidable reference(s) no longer resolve (12.0%)
# rot 21 reference(s) left out: 18 unreadable, 3 undecided
# repair: subscribe to file_deleted and drop the reference when it fires
# repair: tombstone the dead references rather than serving a link that failsThe full code
One harvest, one batch of reads, one division. file_ids_in walks a message including its nested blocks, because a permalink pasted as text is a reference and looks nothing like an attachment. attachment_state reads the message-side symptom and is careful to call it a suspicion. classify_ref turns each files.info answer into one word, and rot divides by the references it could decide and no others - which is the one design decision in this script that keeps the number meaning something.
"""Measure how much of what you stored about Slack files no longer exists.
Read only. conversations.history and files.info are reads; nothing here deletes
a file, removes a message or repairs a reference. The repairs are printed.
The shape of the bug is that deleting a file does not touch anything that points
at it, so references rot silently and at whatever rate your workspace deletes
things. The number worth having is a fraction, and the one design decision here
is what goes in the denominator: only references this token could actually
decide. A file it cannot read is not a file that was deleted.
"""
import argparse
import json
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_file_link_rot")
API = "https://slack.com/api/"
# A Slack file permalink carries the file id as a path segment, between the
# uploader and the filename. Matching inside the URL rather than matching bare
# F-ids anywhere in the text, because "F" plus capitals is a common enough
# string in ordinary prose to poison the harvest.
FILE_URL = re.compile(r"https?://[^\s|<>]*slack\.com/files/[^\s|<>]*?"
r"/(F[A-Z0-9]{6,})", re.IGNORECASE)
def _strings_in(node, out, depth=0):
"""Every string anywhere in a blocks tree. Recursive, depth-capped."""
if depth > 12:
return out
if isinstance(node, str):
out.append(node)
elif isinstance(node, dict):
for value in node.values():
_strings_in(value, out, depth + 1)
elif isinstance(node, (list, tuple)):
for value in node:
_strings_in(value, out, depth + 1)
return out
def file_ids_in(message):
"""Every file this message refers to, from both places they hide. Pure.
Two kinds of reference, and a harvest that finds only the first is the
version of this check that reports a healthy corpus:
attachments entries in the message's `files` array, which is what most
people look at because it is the obvious one.
links a file permalink pasted into `text`, buried in a rich_text
block, or sitting in an attachment. Older, usually more
valuable, and completely invisible to a harvest that reads
the files array alone.
Returns ids in first-seen order, deduplicated and upper-cased.
"""
m = message or {}
found = []
for entry in m.get("files") or []:
fid = (entry or {}).get("id")
if isinstance(fid, str) and fid:
found.append(fid)
haystack = _strings_in(m.get("text"), [])
_strings_in(m.get("blocks"), haystack)
_strings_in(m.get("attachments"), haystack)
for chunk in haystack:
for hit in FILE_URL.finditer(chunk):
found.append(hit.group(1))
seen, out = set(), []
for fid in found:
upper = fid.upper()
if upper not in seen:
seen.add(upper)
out.append(upper)
return out
def attachment_state(entry):
"""What does the message itself say about this attachment? Pure.
Returns (state, detail). This is the message-side symptom and it is
deliberately weaker than the API-side verdict:
tombstone Slack has replaced the file with a marker. Conclusive.
stub the entry has an id and has lost the fields that made it a
file. Consistent with deletion and equally consistent with a
file this token cannot read, so it is a suspicion.
present the entry still describes a file.
"""
e = entry or {}
if str(e.get("mode") or "").lower() == "tombstone":
return ("tombstone", "Slack has already replaced this attachment with a "
"tombstone, so the deletion is not in doubt")
if not e.get("mimetype") and not e.get("url_private"):
return ("stub", "an id and nothing else. That is what history returns for a "
"file that was deleted and also for one this token cannot "
"read, so files.info decides which")
return ("present", "the entry still describes a file")
def classify_ref(payload):
"""One files.info answer, in one word. Pure.
live resolves, and counts in the denominator.
deleted file_deleted, and counts in both.
unknown file_not_found: no such file for this token. Counts as dangling
because a reference you cannot resolve is dead to you either
way, and is reported separately so a workspace mix-up is
visible.
unreadable not_visible or access_denied. NOT rot, and excluded from the
fraction: mixing it in makes the number move when the bot's
channel membership changes.
undecided anything else, including a rate limit or a scope problem.
"""
p = payload or {}
if p.get("ok") is True:
return "live"
error = str(p.get("error") or "")
if error == "file_deleted":
return "deleted"
if error == "file_not_found":
return "unknown"
if error in ("not_visible", "access_denied",
"slack_connect_file_link_sharing_blocked"):
return "unreadable"
return "undecided"
def rot(verdicts):
"""The dangling fraction, over the references that could be decided. Pure.
Returns (dangling, decidable, percent, excluded). A reference this token
could not resolve one way or the other is left out of both halves rather
than counted as healthy, because counting it as healthy is how a rot rate
quietly reports success while the corpus dies.
"""
counts = {}
for v in verdicts or []:
counts[v] = counts.get(v, 0) + 1
dangling = counts.get("deleted", 0) + counts.get("unknown", 0)
decidable = dangling + counts.get("live", 0)
excluded = counts.get("unreadable", 0) + counts.get("undecided", 0)
percent = 0.0 if not decidable else round(100.0 * dangling / decidable, 1)
return (dangling, decidable, percent, excluded)
def read_history(session, channel, pages, limit):
"""Page conversations.history. A read."""
out, cursor = [], ""
for _ in range(pages):
params = {"channel": channel, "limit": str(limit)}
if cursor:
params["cursor"] = cursor
body = session.get(API + "conversations.history", params=params,
timeout=30).json()
if body.get("ok") is not True:
log.warning("history unavailable %s: %s", channel, body.get("error"))
return out
out.extend(body.get("messages") or [])
cursor = (body.get("response_metadata") or {}).get("next_cursor") or ""
if not cursor:
break
return out
def load_refs(path):
"""The ids your own store believes in."""
raw = json.loads(open(path, encoding="utf-8").read())
if isinstance(raw, dict):
raw = raw.get("files") or raw.get("ids") or raw.get("refs") or []
out = []
for row in raw or []:
if isinstance(row, str):
out.append(row.upper())
elif isinstance(row, dict) and isinstance(row.get("id"), str):
out.append(row["id"].upper())
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--channel", action="append", default=[],
help="a channel id to harvest references from; repeatable")
ap.add_argument("--refs", default="",
help="a JSON file of file ids your own store holds")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN")
ap.add_argument("--pages", type=int, default=10)
ap.add_argument("--limit", type=int, default=200)
ap.add_argument("--max-checks", type=int, default=500,
help="cap on files.info calls; it is a tier 4 method, not free")
args = ap.parse_args()
if not args.channel and not args.refs:
log.error("pass --channel C..., or --refs FILE, or both")
return 2
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token with files:read", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s in %s", who.get("user_id"), who.get("team"))
from_history, order = [], []
for channel in args.channel:
for message in read_history(s, channel, args.pages, args.limit):
for entry in message.get("files") or []:
state, detail = attachment_state(entry)
if state != "present":
log.info("message %-14s %s %s", state,
(entry or {}).get("id") or "?", detail)
from_history.extend(file_ids_in(message))
from_store = load_refs(args.refs) if args.refs else []
seen = set()
for fid in from_history + from_store:
if fid not in seen:
seen.add(fid)
order.append(fid)
log.info("refs %d reference(s): %d from history, %d from %s",
len(order), len(set(from_history)), len(set(from_store)),
args.refs or "no store")
verdicts = []
for fid in order[:args.max_checks]:
body = s.get(API + "files.info", params={"file": fid}, timeout=30).json()
verdict = classify_ref(body)
verdicts.append(verdict)
if verdict == "deleted":
log.warning("check deleted %s the file is gone and anything "
"that points at it is not", fid)
elif verdict == "unknown":
log.warning("check unknown %s file_not_found: no such file for "
"this token, which may be a different workspace", fid)
elif verdict == "unreadable":
log.info("check unreadable %s not_visible, which is membership "
"and not rot", fid)
elif verdict == "undecided":
log.info("check undecided %s neither resolved nor refused "
"cleanly", fid)
dangling, decidable, percent, excluded = rot(verdicts)
if not decidable:
log.warning("rot nothing decidable in this run; check files:read and "
"the bot's channel membership")
return 1
log.info("rot %d of %d decidable reference(s) no longer resolve (%.1f%%)",
dangling, decidable, percent)
if excluded:
log.info("rot %d reference(s) left out of that fraction on purpose",
excluded)
if not dangling:
return 0
log.warning(" repair: subscribe to the file_deleted event and drop the reference "
"when it fires; that is the only version of this that stays fixed")
log.warning(" repair: tombstone the dead references rather than serving a link "
"that fails, and keep the row so the next run can compare")
log.warning(" repair: if the content has to outlive Slack, copy the bytes at "
"ingestion time; Slack is a transport, not an archive")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Measure how much of what you stored about Slack files no longer exists.
*
* Read only. conversations.history and files.info are reads; nothing here
* deletes a file, removes a message or repairs a reference. Repairs are printed.
*
* Deleting a file does not touch anything that points at it, so references rot
* silently. The number worth having is a fraction, and the one design decision
* here is what goes in the denominator: only references this token could
* actually decide. A file it cannot read is not a file that was deleted.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://slack.com/api/';
// A Slack file permalink carries the id as a path segment. Matching inside the
// URL rather than matching bare F-ids anywhere in the text, because "F" plus
// capitals is a common enough string in prose to poison the harvest.
const FILE_URL = /https?:\/\/[^\s|<>]*slack\.com\/files\/[^\s|<>]*?\/(F[A-Z0-9]{6,})/gi;
function stringsIn(node, out, depth = 0) {
if (depth > 12) return out;
if (typeof node === 'string') out.push(node);
else if (Array.isArray(node)) for (const v of node) stringsIn(v, out, depth + 1);
else if (node && typeof node === 'object') {
for (const v of Object.values(node)) stringsIn(v, out, depth + 1);
}
return out;
}
/**
* Every file this message refers to, from both places they hide. Pure.
*
* Attachments live in the `files` array and are the obvious half. Permalinks
* pasted into `text`, buried in a rich_text block or sitting in an attachment
* are the older half, and a harvest that reads only `files` reports a healthy
* corpus while missing most of it.
*/
export function fileIdsIn(message) {
const m = message ?? {};
const found = [];
for (const entry of m.files ?? []) {
const fid = (entry ?? {}).id;
if (typeof fid === 'string' && fid) found.push(fid);
}
const haystack = stringsIn(m.text, []);
stringsIn(m.blocks, haystack);
stringsIn(m.attachments, haystack);
for (const chunk of haystack) {
FILE_URL.lastIndex = 0;
let hit = FILE_URL.exec(chunk);
while (hit) {
found.push(hit[1]);
hit = FILE_URL.exec(chunk);
}
}
const seen = new Set();
const out = [];
for (const fid of found) {
const upper = fid.toUpperCase();
if (!seen.has(upper)) {
seen.add(upper);
out.push(upper);
}
}
return out;
}
/**
* What does the message itself say about this attachment? Pure.
* Returns [state, detail]; stub is a suspicion rather than a verdict.
*/
export function attachmentState(entry) {
const e = entry ?? {};
if (String(e.mode ?? '').toLowerCase() === 'tombstone') {
return ['tombstone', 'Slack has already replaced this attachment with a tombstone, '
+ 'so the deletion is not in doubt'];
}
if (!e.mimetype && !e.url_private) {
return ['stub', 'an id and nothing else. That is what history returns for a file '
+ 'that was deleted and also for one this token cannot read, so files.info '
+ 'decides which'];
}
return ['present', 'the entry still describes a file'];
}
/**
* One files.info answer, in one word. Pure.
* live, deleted, unknown, unreadable or undecided; unreadable is not rot.
*/
export function classifyRef(payload) {
const p = payload ?? {};
if (p.ok === true) return 'live';
const error = String(p.error ?? '');
if (error === 'file_deleted') return 'deleted';
if (error === 'file_not_found') return 'unknown';
if (['not_visible', 'access_denied',
'slack_connect_file_link_sharing_blocked'].includes(error)) return 'unreadable';
return 'undecided';
}
/**
* The dangling fraction, over the references that could be decided. Pure.
* Returns [dangling, decidable, percent, excluded].
*/
export function rot(verdicts) {
const counts = {};
for (const v of verdicts ?? []) counts[v] = (counts[v] ?? 0) + 1;
const dangling = (counts.deleted ?? 0) + (counts.unknown ?? 0);
const decidable = dangling + (counts.live ?? 0);
const excluded = (counts.unreadable ?? 0) + (counts.undecided ?? 0);
const percent = decidable ? Math.round((1000 * dangling) / decidable) / 10 : 0;
return [dangling, decidable, percent, excluded];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
function argAll(args, name) {
const out = [];
for (let i = 0; i < args.length; i += 1) {
if (args[i] === name && args[i + 1] !== undefined) out.push(args[i + 1]);
}
return out;
}
async function readHistory(headers, channel, pages, limit) {
const out = [];
let cursor = '';
for (let i = 0; i < pages; i += 1) {
const params = new URLSearchParams({ channel, limit: String(limit) });
if (cursor) params.set('cursor', cursor);
const body = await (await fetch(`${API}conversations.history?${params}`,
{ headers })).json();
if (body.ok !== true) {
console.warn(`history unavailable ${channel}: ${body.error}`);
return out;
}
out.push(...(body.messages ?? []));
cursor = (body.response_metadata ?? {}).next_cursor ?? '';
if (!cursor) break;
}
return out;
}
function loadRefs(raw) {
let rows = raw;
if (rows && !Array.isArray(rows) && typeof rows === 'object') {
rows = rows.files ?? rows.ids ?? rows.refs ?? [];
}
const out = [];
for (const row of rows ?? []) {
if (typeof row === 'string') out.push(row.toUpperCase());
else if (row && typeof row.id === 'string') out.push(row.id.toUpperCase());
}
return out;
}
async function main() {
const args = process.argv.slice(2);
const channels = argAll(args, '--channel');
const refsFile = arg(args, '--refs', '');
if (!channels.length && !refsFile) {
console.error('pass --channel C..., or --refs FILE, or both');
process.exitCode = 2;
return;
}
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} to a bot token with files:read`);
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} in ${who.team}`);
const pages = Number(arg(args, '--pages', '10'));
const limit = Number(arg(args, '--limit', '200'));
const fromHistory = [];
for (const channel of channels) {
for (const message of await readHistory(headers, channel, pages, limit)) {
for (const entry of message.files ?? []) {
const [state, detail] = attachmentState(entry);
if (state !== 'present') {
console.log(`message ${state.padEnd(14)} ${(entry ?? {}).id ?? '?'} `
+ `${detail}`);
}
}
fromHistory.push(...fileIdsIn(message));
}
}
const fromStore = refsFile
? loadRefs(JSON.parse(await readFile(refsFile, 'utf8'))) : [];
const order = [...new Set([...fromHistory, ...fromStore])];
console.log(`refs ${order.length} reference(s): ${new Set(fromHistory).size} `
+ `from history, ${new Set(fromStore).size} from ${refsFile || 'no store'}`);
const maxChecks = Number(arg(args, '--max-checks', '500'));
const verdicts = [];
for (const fid of order.slice(0, maxChecks)) {
const params = new URLSearchParams({ file: fid });
const body = await (await fetch(`${API}files.info?${params}`, { headers })).json();
const verdict = classifyRef(body);
verdicts.push(verdict);
if (verdict === 'deleted') {
console.warn(`check deleted ${fid} the file is gone and anything `
+ 'that points at it is not');
} else if (verdict === 'unknown') {
console.warn(`check unknown ${fid} file_not_found: no such file for `
+ 'this token, which may be a different workspace');
} else if (verdict === 'unreadable') {
console.log(`check unreadable ${fid} not_visible, which is membership `
+ 'and not rot');
} else if (verdict === 'undecided') {
console.log(`check undecided ${fid} neither resolved nor refused `
+ 'cleanly');
}
}
const [dangling, decidable, percent, excluded] = rot(verdicts);
if (!decidable) {
console.warn('rot nothing decidable in this run; check files:read and the '
+ "bot's channel membership");
process.exitCode = 1;
return;
}
console.log(`rot ${dangling} of ${decidable} decidable reference(s) no longer `
+ `resolve (${percent}%)`);
if (excluded) {
console.log(`rot ${excluded} reference(s) left out of that fraction on `
+ 'purpose');
}
if (!dangling) return;
console.warn(' repair: subscribe to the file_deleted event and drop the reference '
+ 'when it fires; that is the only version of this that stays fixed');
console.warn(' repair: tombstone the dead references rather than serving a link that '
+ 'fails, and keep the row so the next run can compare');
console.warn(' repair: if the content has to outlive Slack, copy the bytes at '
+ 'ingestion time; Slack is a transport, not an archive');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The harvest is tested against a permalink buried three levels down in a rich_text block, because that is the shape a real pasted link has and the shape a flat scan misses. rot is tested for its denominator more than for its numerator: an unreadable reference must not count as healthy and must not count as dangling, and there is an explicit test that a run consisting entirely of unreadable references reports nothing decidable rather than a comfortable zero per cent.
from slack_file_link_rot import attachment_state, classify_ref, file_ids_in, rot
PERMALINK = "https://northwind.slack.com/files/U07BOT9QD/F06AAA1QQ/report.pdf"
def test_an_attachment_is_the_obvious_half_of_the_harvest():
assert file_ids_in({"files": [{"id": "F06AAA1QQ"}]}) == ["F06AAA1QQ"]
def test_a_permalink_pasted_as_text_is_the_half_that_gets_missed():
assert file_ids_in({"text": "see %s for the numbers" % PERMALINK}) == ["F06AAA1QQ"]
def test_a_permalink_buried_in_a_rich_text_block_is_still_found():
message = {"blocks": [{"type": "rich_text", "elements": [
{"type": "rich_text_section", "elements": [
{"type": "link", "url": PERMALINK, "text": "the report"}]}]}]}
assert file_ids_in(message) == ["F06AAA1QQ"]
def test_a_permalink_in_an_attachment_counts_as_a_reference():
assert file_ids_in({"attachments": [{"text": PERMALINK}]}) == ["F06AAA1QQ"]
def test_the_same_file_referenced_twice_is_one_call():
message = {"files": [{"id": "F06AAA1QQ"}], "text": PERMALINK}
assert file_ids_in(message) == ["F06AAA1QQ"]
def test_an_id_shaped_word_in_prose_is_not_harvested():
assert file_ids_in({"text": "FRIDAY and F06AAA1QQ are not links"}) == []
def test_a_message_with_no_files_at_all_yields_nothing():
assert file_ids_in({"text": "no attachments here"}) == []
assert file_ids_in(None) == []
def test_a_tombstone_is_conclusive():
state, detail = attachment_state({"id": "F1", "mode": "tombstone"})
assert state == "tombstone"
assert "not in doubt" in detail
def test_a_stripped_entry_is_a_suspicion_rather_than_a_verdict():
state, detail = attachment_state({"id": "F1"})
assert state == "stub"
assert "files.info decides which" in detail
def test_a_real_attachment_is_present():
assert attachment_state({"id": "F1", "mimetype": "application/pdf",
"url_private": "https://files.slack.com/x"})[0] == "present"
def test_the_four_files_info_answers_are_four_different_words():
assert classify_ref({"ok": True}) == "live"
assert classify_ref({"ok": False, "error": "file_deleted"}) == "deleted"
assert classify_ref({"ok": False, "error": "file_not_found"}) == "unknown"
assert classify_ref({"ok": False, "error": "not_visible"}) == "unreadable"
def test_anything_unexpected_is_undecided_rather_than_assumed_healthy():
assert classify_ref({"ok": False, "error": "ratelimited"}) == "undecided"
assert classify_ref({}) == "undecided"
assert classify_ref(None) == "undecided"
def test_the_fraction_is_over_decidable_references_only():
dangling, decidable, percent, excluded = rot(
["deleted", "deleted", "live", "live", "unreadable"])
assert (dangling, decidable, excluded) == (2, 4, 1)
assert percent == 50.0
def test_an_unreadable_reference_is_neither_healthy_nor_rotten():
with_it = rot(["deleted", "live", "unreadable"])
without_it = rot(["deleted", "live"])
assert with_it[2] == without_it[2]
def test_a_run_of_nothing_but_unreadable_reports_nothing_decidable():
dangling, decidable, percent, excluded = rot(["unreadable", "unreadable"])
assert decidable == 0
assert percent == 0.0
assert excluded == 2
def test_file_not_found_counts_as_dangling_and_is_still_named_separately():
assert rot(["unknown", "live"])[0] == 1
assert classify_ref({"ok": False, "error": "file_not_found"}) != "deleted"
def test_an_empty_run_does_not_divide_by_zero():
assert rot([]) == (0, 0, 0.0, 0)
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
attachmentState, classifyRef, fileIdsIn, rot,
} from './slack-file-link-rot.mjs';
const PERMALINK = 'https://northwind.slack.com/files/U07BOT9QD/F06AAA1QQ/report.pdf';
test('an attachment is the obvious half of the harvest', () => {
assert.deepEqual(fileIdsIn({ files: [{ id: 'F06AAA1QQ' }] }), ['F06AAA1QQ']);
});
test('a permalink pasted as text is the half that gets missed', () => {
assert.deepEqual(fileIdsIn({ text: `see ${PERMALINK} for the numbers` }),
['F06AAA1QQ']);
});
test('a permalink buried in a rich_text block is still found', () => {
const message = {
blocks: [{
type: 'rich_text',
elements: [{
type: 'rich_text_section',
elements: [{ type: 'link', url: PERMALINK, text: 'the report' }],
}],
}],
};
assert.deepEqual(fileIdsIn(message), ['F06AAA1QQ']);
});
test('a permalink in an attachment counts as a reference', () => {
assert.deepEqual(fileIdsIn({ attachments: [{ text: PERMALINK }] }), ['F06AAA1QQ']);
});
test('the same file referenced twice is one call', () => {
assert.deepEqual(fileIdsIn({ files: [{ id: 'F06AAA1QQ' }], text: PERMALINK }),
['F06AAA1QQ']);
});
test('the global regex does not skip a second message', () => {
assert.deepEqual(fileIdsIn({ text: PERMALINK }), ['F06AAA1QQ']);
assert.deepEqual(fileIdsIn({ text: PERMALINK }), ['F06AAA1QQ']);
});
test('an id shaped word in prose is not harvested', () => {
assert.deepEqual(fileIdsIn({ text: 'FRIDAY and F06AAA1QQ are not links' }), []);
});
test('a message with no files at all yields nothing', () => {
assert.deepEqual(fileIdsIn({ text: 'no attachments here' }), []);
assert.deepEqual(fileIdsIn(null), []);
});
test('a tombstone is conclusive', () => {
const [state, detail] = attachmentState({ id: 'F1', mode: 'tombstone' });
assert.equal(state, 'tombstone');
assert.match(detail, /not in doubt/);
});
test('a stripped entry is a suspicion rather than a verdict', () => {
const [state, detail] = attachmentState({ id: 'F1' });
assert.equal(state, 'stub');
assert.match(detail, /files\.info decides which/);
});
test('a real attachment is present', () => {
assert.equal(attachmentState({
id: 'F1', mimetype: 'application/pdf', url_private: 'https://files.slack.com/x',
})[0], 'present');
});
test('the four files.info answers are four different words', () => {
assert.equal(classifyRef({ ok: true }), 'live');
assert.equal(classifyRef({ ok: false, error: 'file_deleted' }), 'deleted');
assert.equal(classifyRef({ ok: false, error: 'file_not_found' }), 'unknown');
assert.equal(classifyRef({ ok: false, error: 'not_visible' }), 'unreadable');
});
test('anything unexpected is undecided rather than assumed healthy', () => {
assert.equal(classifyRef({ ok: false, error: 'ratelimited' }), 'undecided');
assert.equal(classifyRef({}), 'undecided');
assert.equal(classifyRef(null), 'undecided');
});
test('the fraction is over decidable references only', () => {
const [dangling, decidable, percent, excluded] = rot(
['deleted', 'deleted', 'live', 'live', 'unreadable']);
assert.deepEqual([dangling, decidable, excluded], [2, 4, 1]);
assert.equal(percent, 50);
});
test('an unreadable reference is neither healthy nor rotten', () => {
assert.equal(rot(['deleted', 'live', 'unreadable'])[2],
rot(['deleted', 'live'])[2]);
});
test('a run of nothing but unreadable reports nothing decidable', () => {
const [, decidable, percent, excluded] = rot(['unreadable', 'unreadable']);
assert.equal(decidable, 0);
assert.equal(percent, 0);
assert.equal(excluded, 2);
});
test('file_not_found counts as dangling and is still named separately', () => {
assert.equal(rot(['unknown', 'live'])[0], 1);
assert.notEqual(classifyRef({ ok: false, error: 'file_not_found' }), 'deleted');
});
test('an empty run does not divide by zero', () => {
assert.deepEqual(rot([]), [0, 0, 0, 0]);
});
FAQ
Why does deleting a file not remove the message that shared it?
Because they are different objects with different owners. A message belongs to the person who posted it and lives in a conversation; a file belongs to whoever uploaded it and lives in the workspace's file store. Deleting one does not authorise deleting the other, and Slack does not. What you get instead is a message whose attachment has been reduced to a marker, which reads as normal in the client and reads as a stub in the API.
Is there a way to be told when a file is deleted?
Yes, and it is the actual fix. Slack emits a file_deleted event to apps subscribed to it with files:read. Handle it by removing or tombstoning the reference at the moment it happens, and the decay stops. The reason so few integrations do this is timing: on the day you build the ingestion, deletion is hypothetical, and by the time it is not, there are thousands of stale rows and no record of when any of them died.
Should a not_visible reference count towards the rot rate?
No, and this is the one number in the script that would be easy to get wrong. Unreadable is not deleted. If they share a bucket, the rot rate rises whenever the bot is removed from a channel and falls whenever somebody adds it to one, so the metric responds to two unrelated causes and cannot be used to detect either. The script divides by the references it could decide and reports the excluded ones beside the fraction.
Can I get a deleted file back?
Not through the API. Deletion is not reversible for an app, and files removed by a workspace retention policy are gone in the same way. That is the argument for copying anything that has to outlive Slack into your own store at ingestion time, using the authenticated download URL - Slack is a transport for files, not an archive of them, and treating it as the second is how a retention policy becomes a data-loss policy nobody chose.
The rot rate is 12% and I do not know whether that is bad. What should I compare it to?
The same number from last month. A single reading has no meaning attached; a direction has. If it is flat, some files were deleted once and the corpus is stable. If it is climbing, something is deleting continuously - most often a retention policy - and the ingestion needs to copy rather than reference. The script prints the fraction in a form you can store precisely so that the second run is a comparison rather than a fresh guess.
Related field notes
- the unreadable references this one refuses to count
- why file_deleted arrived inside a successful response
- harvesting one page of history and calling it the corpus
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- files.info method reference - Slack Docs
- file_deleted event reference - Slack Docs
- conversations.history method reference - Slack Docs
- files.list method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.