Diagnostic Slack
Half finished uploads leave a file id that names nothing
The report generator says it uploaded the PDF. The log line is there, no exception was raised, and the message announcing it went out on time. The message links to a file that does not exist.
Uploading to Slack is three network operations and only the first and last are Slack API calls. The middle one goes to a URL Slack handed you, so it is invisible to every Slack-shaped piece of monitoring you have. A break anywhere along the way leaves a file_id that was issued before any bytes existed and never became anything, and the SDK helper that wraps the sequence reports the whole thing as one failure with no indication of which third of it broke.
The modern upload is: files.getUploadURLExternal returns an upload_url and a file_id; you send the bytes to that URL; files.completeUploadExternal registers the file and shares it. The file_id exists from the first call, before a single byte has been transferred. That is what makes the failure so quiet: you have an id, you write it down, you put it in a message, and nothing about it says whether it ever became a file.
Break the sequence at step two and the id names nothing. Break it at step three and the bytes are in Slack's storage attached to a file that was never registered. Either way files.info on your recorded id comes back {"ok": false, "error": "file_not_found"} — and that answer, against an id your own logs say you own, is the proof.
The most common cause is a scope. files:write is needed for the completion call and not for the byte transfer, so an app missing it gets all the way through the expensive part and fails at the end. The SDK reports it as an upload error, which sends people to look at the file, the size, the mime type: everything except the grant.
Fix the retry direction too. A sequence that failed at step three should repeat step three. Restarting from step one instead mints a second file_id, uploads the bytes again, and leaves the first orphan exactly where it was.
The problem in plain words
The thing that makes this expensive is that the sequence is not atomic and nothing in it says so. Three operations, two systems, no transaction, and an id issued at the start that looks equally valid whether the story finished or not. Every other resource in the Slack API is either there or not there. This one has a middle state that your code holds a handle to.
The middle operation is invisible in an unusual way. It does not go to slack.com/api, it is not a Web API method, and it does not answer with ok. If your instrumentation wraps the Slack client, it does not see this call at all: no metric, no trace span, no retry policy, no timeout you set on purpose. A proxy that rewrites large request bodies, a network that drops the connection halfway through fifty megabytes, a hop that truncates the stream — all of that happens outside everything you have instrumented, and the next thing your code does is confidently call step three.
Then there is the length. files.getUploadURLExternal takes a length, and the value you declare has to be the number of bytes that actually arrive. Compute it from a string length rather than the encoded byte count, or from the file size before compression, and you have declared one thing and sent another. The failure surfaces later and elsewhere.
The scope trap is the one that catches whole teams at once. files:write is required for the completion step. An app installed with files:read alone will happily mint upload URLs and transfer bytes, and then die on the last call. Everything before the failure worked, which is exactly why nobody looks at the token: the natural reading of a failure at the end of an upload is that something is wrong with the file.
And the SDK helpers hide the seam. filesUploadV2 and files_upload_v2 exist because hand-rolling three steps is worse, but they turn three failure modes into one exception, and there have been versions that dropped a per-call token override between steps and produced not_authed in the middle of a sequence whose first call had authenticated perfectly.
Why it happens
An id issued before the resource exists is the whole problem, so the audit is keyed on the id. Your ledger holds ids your uploader was given. files.info says whether each one became a file. That one join separates a real upload from a receipt for an upload that never happened, and nothing else in the API will tell you.
file_not_found on your own id is not ambiguous, and that is rare here. Most Slack errors could mean several things. This one, against an id your logs say you were handed, means the sequence did not finish. It is one of the few places in this section where a single error string is conclusive.
Which step broke decides which step to repeat, and getting it backwards doubles the mess. A failure at completion should be retried at completion, using the id you already have. Restarting from the beginning mints a new id, sends the bytes again, and leaves the first orphan behind. A retry policy that always starts over converts one stalled upload into a growing pile.
The scope has to be checked before the sequence, not after it fails. files:write is a precondition of a step that runs last. Checking it up front costs one header read; discovering it at the end costs the bytes, the time, and a half-created file.
Zero bytes is a distinct verdict from no file. A file registered with a size of zero means the completion call succeeded and the transfer did not, which is the opposite diagnosis from an id that names nothing, and it points at the middle operation nobody is watching.
The reverse direction finds what a ledger cannot. Listing the files this bot owns turns up ids your records have never heard of — uploads from a code path that does not write to the ledger at all. A one-directional audit will always report those as clean.
The fix, as a flow
The red box is the third one, in the middle, and that position is the argument. An upload is three operations across two systems with no transaction around them, and the id you write down is handed out by the first one before a single byte exists. Break the sequence anywhere and you keep a perfectly valid looking handle to nothing. The fix branch is the only one in this batch whose rows end in a retry direction rather than a repair, because going the wrong way here does not fail, it duplicates.
How to fix it
Check files:write before you look at a single file
precondition reads the X-OAuth-Scopes header and reports no-files-write first, because that scope is a requirement of the last step in the sequence. An app without it can mint URLs and transfer bytes all day and will fail every completion, after the expensive part is done.
Say how far each recorded upload got
step_reached reads one ledger row and returns nothing, url-minted, bytes-sent or completed. This is your side of the story, and it is worth stating before Slack's side, because the two disagreeing is the finding.
Ask files.info about every id you recorded
reconcile takes your step and Slack's answer and returns never-registered, empty-file, registered-unshared, deleted, unseeable or complete. file_not_found against an id from your own logs is the conclusive one: the sequence stopped before the file was created.
Read zero bytes as a different failure from no file
empty-file means completion succeeded and the transfer did not. That is the middle operation failing, the one that never goes through your Slack client and is therefore not in your traces. It is also where a wrong length shows up, which is what length_fault is for.
Take the retry the finding implies, not the one the wrapper implies
retry_plan names the single step to repeat. A completion that failed wants completion retried with the id you already hold. Starting the sequence over mints a second id, sends the bytes again, and leaves the first orphan untouched, which is how a stalled upload becomes a pile of them.
Look in the other direction as well
files.list filtered to the bot's own user id returns what Slack thinks this app owns. Ids in that list which are absent from your ledger came from a code path that does not record anything, and no amount of auditing your records will ever surface them.
How to check it worked
Run it against the ids your uploader wrote down. The finding is a specific id, a specific step, and one line saying which step to repeat.
python3 slack_upload_gaps.py --ledger uploads.json
# identity U024BE7LH in Acme
# scopes files:read,chat:write,channels:history
# scope no-files-write the completion step needs files:write. Without it the
# sequence always fails after the bytes are already up
# ledger 146 recorded upload(s)
# gap never-registered F07ABCD1234 step reached: bytes-sent
# the bytes went up and files.completeUploadExternal was
# never answered, so the id names nothing
# retry: repeat the completion for F07ABCD1234; do not mint a second upload URL
# gap empty-file F07ABCD5678 registered with 0 bytes; completion
# succeeded and the transfer did not
# gap length F07ABCD9999 declared 41821 bytes and sent 41 bytes;
# a character count was used where a byte count belongs
# inverse 17 file(s) owned by this bot are absent from your ledger
# verdict 31 of 146 recorded upload(s) never became a file
# repair: grant files:write before the first call, so the sequence cannot fail
# at the one step that needs it
# repair: record the file_id at step one and the outcome at step three, and
# retry the step that failed rather than the whole sequenceThe full code
Five pure functions and two reads. step_reached states your side, reconcile states Slack's, and the interesting rows are where they disagree. length_fault catches the declared byte count that was really a character count, which is the quiet way the middle operation goes wrong. precondition checks the scope that a step at the end of the sequence needs at the beginning. retry_plan exists because getting the retry direction wrong is what turns one stalled upload into a hundred.
"""Find the uploads that stopped between the three calls that make one.
Read only. This never mints an upload URL, never sends bytes anywhere, and
never completes an upload: those are writes. It reads the ids your uploader
already recorded and asks files.info whether each one ever became a file.
The modern upload is three network operations and only the first and last are
Slack API calls. The file_id is handed out by the first one, before any bytes
exist, so an id on its own proves nothing at all.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_upload_gaps")
API = "https://slack.com/api/"
def precondition(scopes):
"""Can this app finish an upload, and can this script audit one? Pure.
files:write is a requirement of the last of the three steps, which is why
it is checked first here. An app without it mints URLs and sends bytes
perfectly well and then fails at the end, once the expensive part is done.
"""
if scopes is None:
return ("scopes-unread", "the granted scopes were not read, so the one "
"precondition that matters is unverified")
held = {str(s).strip() for s in scopes}
if "files:write" not in held:
return ("no-files-write", "the completion step needs files:write. Without it "
"the sequence always fails after the bytes are "
"already up, which is the state that leaves orphans")
if "files:read" not in held:
return ("no-files-read", "files:write is granted so uploads can complete, but "
"this audit needs files:read to ask about any of them")
return ("ready", "files:write and files:read are both granted")
def step_reached(record):
"""How far did your own uploader get with this file? Pure.
This is your side of the story and it is stated before Slack's, because the
two disagreeing is the entire finding.
"""
row = record or {}
if row.get("completed_at") or row.get("completed") is True:
return ("completed", "your record says the completion call was answered")
if row.get("bytes_sent") or row.get("uploaded_at"):
return ("bytes-sent", "the bytes went up and the completion call is not "
"recorded as having been answered")
if row.get("file_id") or row.get("upload_url"):
return ("url-minted", "an upload URL and a file id were issued and nothing "
"after that is recorded")
return ("nothing", "the row records no file id at all, so there is nothing to ask "
"Slack about")
def length_fault(declared, actual):
"""Did you send the number of bytes you said you would? Pure.
The declared length is the one number in the sequence that has to agree
with a fact about the file, and the usual mistake is a character count
where a byte count belongs, which only diverges once the text is not
plain ASCII.
"""
try:
want = int(declared)
got = int(actual)
except (TypeError, ValueError):
return ("unknown", "the row does not carry both a declared length and a sent "
"byte count")
if want <= 0:
return ("zero", "a declared length of %d. The call takes the byte count of the "
"file, and zero is never right" % want)
if got == want:
return ("match", "%d byte(s) declared and sent" % want)
if got < want:
return ("short", "declared %d byte(s) and sent %d. A truncated transfer, or a "
"length taken before compression" % (want, got))
return ("over", "declared %d byte(s) and sent %d. A character count was very "
"likely used where a byte count belongs" % (want, got))
def reconcile(step, info):
"""What does Slack say about the id your ledger recorded? Pure.
file_not_found against an id your own logs say you were handed is the
conclusive answer in this note: the sequence stopped before the file was
ever created. Most Slack errors are ambiguous. This one is not.
"""
body = info or {}
if body.get("ok") is False:
error = str(body.get("error") or "")
if error == "file_not_found":
return ("never-registered", "Slack has no file under this id. It was "
"issued by the first call before any bytes "
"existed, and nothing ever completed it")
if error == "not_visible":
return ("unseeable", "the file exists and this token cannot see it, which "
"is a membership question rather than a broken "
"sequence")
if error == "file_deleted":
return ("deleted", "the file was created and later deleted, so the "
"sequence did finish")
return ("unreadable", "files.info answered %s, which decides nothing either "
"way" % (error or "with no error string"))
row = body.get("file") if isinstance(body.get("file"), dict) else body
if not row:
return ("unreadable", "no file object came back")
try:
size = int(row.get("size") or 0)
except (TypeError, ValueError):
size = 0
if size <= 0:
return ("empty-file", "registered with %d byte(s). The completion call "
"succeeded and the transfer did not, which is the "
"middle operation failing" % size)
shares = row.get("shares") or {}
targets = (list(row.get("channels") or []) + list(row.get("groups") or [])
+ list(row.get("ims") or []) + list(shares.get("public") or [])
+ list(shares.get("private") or []))
if not targets:
return ("registered-unshared", "the file exists with %d byte(s) and is shared "
"nowhere. The sequence finished; the channel "
"argument is a separate question" % size)
if step == "completed":
return ("complete", "%d byte(s), shared in %d place(s)" % (size, len(targets)))
return ("complete-unrecorded", "the file exists and your ledger stopped at %s, so "
"your records are behind Slack rather than ahead of "
"it" % step)
def retry_plan(step, verdict):
"""Which single step should be repeated? Pure. Printed, never run.
Getting this backwards is what turns one stalled upload into a pile of
them: restarting from the first call mints a second id, sends the bytes a
second time, and leaves the first orphan exactly where it was.
"""
if verdict == "never-registered" and step in ("bytes-sent", "completed"):
return ("repeat the completion for this file id and nothing else. The bytes "
"are already up; minting a second upload URL would create a second "
"orphan beside this one")
if verdict == "never-registered":
return ("start this upload again and discard the recorded id. Nothing was "
"ever transferred under it")
if verdict == "empty-file":
return ("start this upload again. The upload URL is single use and spent, and "
"the step that failed was the byte transfer")
if verdict == "registered-unshared":
return ("nothing in the sequence needs repeating. Share the file that exists "
"rather than uploading it again")
if verdict in ("unseeable", "deleted"):
return "nothing to repeat; the sequence finished"
if verdict == "complete-unrecorded":
return ("nothing to repeat. Record the outcome so the next audit does not "
"raise this file again")
return "nothing to repeat"
def load_ledger(path):
raw = json.loads(open(path, encoding="utf-8").read())
if isinstance(raw, dict):
raw = raw.get("uploads") or raw.get("files") or []
return [r for r in raw or [] if isinstance(r, dict)]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--ledger", required=True,
help="a JSON file of upload records, each with file_id and "
"whatever of upload_url, bytes_sent and completed_at you keep")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding the bot token")
ap.add_argument("--inverse", action="store_true",
help="also list files this bot owns that your ledger has never "
"heard of")
ap.add_argument("--count", type=int, default=100,
help="files to read in the inverse pass")
args = ap.parse_args()
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token holding files:read", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
r = s.get(API + "auth.test", timeout=30)
who = r.json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
header = r.headers.get("X-OAuth-Scopes")
scopes = [p.strip() for p in header.split(",") if p.strip()] if header else None
log.info("identity %s in %s", who.get("user_id"), who.get("team"))
log.info("scopes %s", ",".join(scopes) if scopes else "not reported")
sverdict, sdetail = precondition(scopes)
(log.info if sverdict == "ready" else log.warning)("scope %-16s %s",
sverdict, sdetail)
ledger = load_ledger(args.ledger)
log.info("ledger %d recorded upload(s)", len(ledger))
broken = 0
seen = set()
for row in ledger:
step, _ = step_reached(row)
file_id = str(row.get("file_id") or "")
if not file_id:
log.warning("gap no-file-id a row with no id; step reached: %s",
step)
broken += 1
continue
seen.add(file_id)
lverdict, ldetail = length_fault(row.get("length"), row.get("bytes_sent"))
if lverdict in ("short", "over", "zero"):
log.warning("gap length %s %s", file_id, ldetail)
body = s.get(API + "files.info", timeout=30,
params={"file": file_id}).json()
verdict, detail = reconcile(step, body)
if verdict in ("complete", "complete-unrecorded", "deleted"):
continue
broken += 1
log.warning("gap %-16s %s step reached: %s", verdict, file_id, step)
log.warning(" %s", detail)
log.warning(" retry: %s", retry_plan(step, verdict))
if args.inverse:
body = s.get(API + "files.list", timeout=30,
params={"user": who.get("user_id") or "",
"count": str(args.count),
"types": "all"}).json()
if body.get("ok") is True:
owned = {str(f.get("id") or "") for f in body.get("files") or []} - {""}
unknown = owned - seen
if unknown:
log.warning("inverse %d file(s) owned by this bot are absent from "
"your ledger", len(unknown))
log.warning(" %s",
", ".join(sorted(unknown)[:8]))
else:
log.warning("files.list unavailable %s", body.get("error"))
if broken or sverdict not in ("ready", "no-files-read"):
log.warning("verdict %d of %d recorded upload(s) never became a file",
broken, len(ledger))
log.warning(" repair: grant files:write before the first call, so the "
"sequence cannot fail at the one step that needs it")
log.warning(" repair: record the file_id at step one and the outcome at step "
"three, and retry the step that failed rather than the whole "
"sequence")
log.warning(" repair: pass the channel as an id in channel_id on the "
"completion call; a channel name is rejected there")
return 1
log.info("verdict clean every recorded upload became a real file")
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find the uploads that stopped between the three calls that make one.
*
* Read only. This never mints an upload URL, never sends bytes anywhere, and
* never completes an upload: those are writes. It reads the ids your uploader
* already recorded and asks files.info whether each one became a file.
*
* The modern upload is three network operations and only the first and last
* are Slack API calls. The file_id comes from the first one, before any bytes
* exist, so an id on its own proves nothing at all.
*/
import { readFile } from 'node:fs/promises';
const API = 'https://slack.com/api/';
/** Can this app finish an upload, and can this script audit one? Pure. */
export function precondition(scopes) {
if (scopes === null || scopes === undefined) {
return ['scopes-unread', 'the granted scopes were not read, so the one '
+ 'precondition that matters is unverified'];
}
const held = new Set(scopes.map((s) => String(s).trim()));
if (!held.has('files:write')) {
return ['no-files-write', 'the completion step needs files:write. Without it the '
+ 'sequence always fails after the bytes are already up, which is the state that '
+ 'leaves orphans'];
}
if (!held.has('files:read')) {
return ['no-files-read', 'files:write is granted so uploads can complete, but this '
+ 'audit needs files:read to ask about any of them'];
}
return ['ready', 'files:write and files:read are both granted'];
}
/** How far did your own uploader get with this file? Pure. */
export function stepReached(record) {
const row = record ?? {};
if (row.completed_at || row.completed === true) {
return ['completed', 'your record says the completion call was answered'];
}
if (row.bytes_sent || row.uploaded_at) {
return ['bytes-sent', 'the bytes went up and the completion call is not recorded '
+ 'as having been answered'];
}
if (row.file_id || row.upload_url) {
return ['url-minted', 'an upload URL and a file id were issued and nothing after '
+ 'that is recorded'];
}
return ['nothing', 'the row records no file id at all, so there is nothing to ask '
+ 'Slack about'];
}
/** Did you send the number of bytes you said you would? Pure. */
export function lengthFault(declared, actual) {
const want = Number(declared);
const got = Number(actual);
if (declared === null || declared === undefined || actual === null
|| actual === undefined || !Number.isFinite(want) || !Number.isFinite(got)) {
return ['unknown', 'the row does not carry both a declared length and a sent byte '
+ 'count'];
}
if (want <= 0) {
return ['zero', `a declared length of ${want}. The call takes the byte count of `
+ 'the file, and zero is never right'];
}
if (got === want) return ['match', `${want} byte(s) declared and sent`];
if (got < want) {
return ['short', `declared ${want} byte(s) and sent ${got}. A truncated transfer, `
+ 'or a length taken before compression'];
}
return ['over', `declared ${want} byte(s) and sent ${got}. A character count was `
+ 'very likely used where a byte count belongs'];
}
/** What does Slack say about the id your ledger recorded? Pure. */
export function reconcile(step, info) {
const body = info ?? {};
if (body.ok === false) {
const error = String(body.error ?? '');
if (error === 'file_not_found') {
return ['never-registered', 'Slack has no file under this id. It was issued by '
+ 'the first call before any bytes existed, and nothing ever completed it'];
}
if (error === 'not_visible') {
return ['unseeable', 'the file exists and this token cannot see it, which is a '
+ 'membership question rather than a broken sequence'];
}
if (error === 'file_deleted') {
return ['deleted', 'the file was created and later deleted, so the sequence did '
+ 'finish'];
}
return ['unreadable', `files.info answered ${error || 'with no error string'}, `
+ 'which decides nothing either way'];
}
const row = (body.file && typeof body.file === 'object') ? body.file : body;
if (!row || !Object.keys(row).length) {
return ['unreadable', 'no file object came back'];
}
const size = Number(row.size ?? 0) || 0;
if (size <= 0) {
return ['empty-file', `registered with ${size} byte(s). The completion call `
+ 'succeeded and the transfer did not, which is the middle operation failing'];
}
const shares = row.shares ?? {};
const targets = [...(row.channels ?? []), ...(row.groups ?? []), ...(row.ims ?? []),
...Object.keys(shares.public ?? {}), ...Object.keys(shares.private ?? {})];
if (!targets.length) {
return ['registered-unshared', `the file exists with ${size} byte(s) and is shared `
+ 'nowhere. The sequence finished; the channel argument is a separate question'];
}
if (step === 'completed') {
return ['complete', `${size} byte(s), shared in ${targets.length} place(s)`];
}
return ['complete-unrecorded', `the file exists and your ledger stopped at ${step}, `
+ 'so your records are behind Slack rather than ahead of it'];
}
/** Which single step should be repeated? Pure. Printed, never run. */
export function retryPlan(step, verdict) {
if (verdict === 'never-registered' && ['bytes-sent', 'completed'].includes(step)) {
return 'repeat the completion for this file id and nothing else. The bytes are '
+ 'already up; minting a second upload URL would create a second orphan beside '
+ 'this one';
}
if (verdict === 'never-registered') {
return 'start this upload again and discard the recorded id. Nothing was ever '
+ 'transferred under it';
}
if (verdict === 'empty-file') {
return 'start this upload again. The upload URL is single use and spent, and the '
+ 'step that failed was the byte transfer';
}
if (verdict === 'registered-unshared') {
return 'nothing in the sequence needs repeating. Share the file that exists rather '
+ 'than uploading it again';
}
if (['unseeable', 'deleted'].includes(verdict)) {
return 'nothing to repeat; the sequence finished';
}
if (verdict === 'complete-unrecorded') {
return 'nothing to repeat. Record the outcome so the next audit does not raise '
+ 'this file again';
}
return 'nothing to repeat';
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function main() {
const args = process.argv.slice(2);
const ledgerFile = arg(args, '--ledger', '');
if (!ledgerFile) {
console.error('pass --ledger FILE with the file ids your uploader recorded');
process.exitCode = 2;
return;
}
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} to a bot token holding files:read`);
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const read = async (method, params = {}) => {
const qs = new URLSearchParams(params).toString();
return fetch(`${API}${method}${qs ? `?${qs}` : ''}`, { headers });
};
const r = await read('auth.test');
const who = await r.json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
const header = r.headers.get('x-oauth-scopes');
const scopes = header ? header.split(',').map((p) => p.trim()).filter(Boolean) : null;
console.log(`identity ${who.user_id} in ${who.team}`);
console.log(`scopes ${scopes ? scopes.join(',') : 'not reported'}`);
const [sv, sd] = precondition(scopes);
const scopeLine = `scope ${sv.padEnd(16)} ${sd}`;
if (sv === 'ready') console.log(scopeLine); else console.warn(scopeLine);
let ledger = JSON.parse(await readFile(ledgerFile, 'utf8'));
if (!Array.isArray(ledger)) ledger = ledger.uploads ?? ledger.files ?? [];
ledger = ledger.filter((row) => row && typeof row === 'object');
console.log(`ledger ${ledger.length} recorded upload(s)`);
let broken = 0;
const seen = new Set();
for (const row of ledger) {
const [step] = stepReached(row);
const fileId = String(row.file_id ?? '');
if (!fileId) {
console.warn(`gap no-file-id a row with no id; step reached: ${step}`);
broken += 1;
continue;
}
seen.add(fileId);
const [lv, ld] = lengthFault(row.length, row.bytes_sent);
if (['short', 'over', 'zero'].includes(lv)) {
console.warn(`gap length ${fileId} ${ld}`);
}
// eslint-disable-next-line no-await-in-loop
const body = await (await read('files.info', { file: fileId })).json();
const [verdict, detail] = reconcile(step, body);
if (['complete', 'complete-unrecorded', 'deleted'].includes(verdict)) continue;
broken += 1;
console.warn(`gap ${verdict.padEnd(16)} ${fileId} step reached: ${step}`);
console.warn(` ${detail}`);
console.warn(` retry: ${retryPlan(step, verdict)}`);
}
if (args.includes('--inverse')) {
const body = await (await read('files.list', {
user: who.user_id ?? '', count: String(arg(args, '--count', '100')), types: 'all',
})).json();
if (body.ok === true) {
const owned = new Set((body.files ?? []).map((f) => String(f.id ?? ''))
.filter(Boolean));
const unknown = [...owned].filter((id) => !seen.has(id));
if (unknown.length) {
console.warn(`inverse ${unknown.length} file(s) owned by this bot are `
+ 'absent from your ledger');
console.warn(` ${unknown.sort().slice(0, 8).join(', ')}`);
}
} else {
console.warn(`files.list unavailable ${body.error}`);
}
}
if (broken || (sv !== 'ready' && sv !== 'no-files-read')) {
console.warn(`verdict ${broken} of ${ledger.length} recorded upload(s) never `
+ 'became a file');
console.warn(' repair: grant files:write before the first call, so the sequence '
+ 'cannot fail at the one step that needs it');
console.warn(' repair: record the file_id at step one and the outcome at step '
+ 'three, and retry the step that failed rather than the whole sequence');
console.warn(' repair: pass the channel as an id in channel_id on the completion '
+ 'call; a channel name is rejected there');
process.exitCode = 1;
} else {
console.log('verdict clean every recorded upload became a real file');
}
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing ledger.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The assertions that matter are the ones that stop a retry going the wrong way. retry_plan has to say repeat the completion when the bytes are already up and start again when they are not, because those two look identical from inside an SDK wrapper and one of them doubles the orphans. reconcile has to keep never-registered and empty-file apart, since the first means nothing was created and the second means the middle operation failed. And precondition has to name files:write before files:read, because one of them breaks uploads and the other only breaks this script.
from slack_upload_gaps import (length_fault, precondition, reconcile, retry_plan,
step_reached)
def ok_file(**kw):
row = {"id": "F07ABCD1234", "size": 41821, "channels": ["C024BE91L"],
"shares": {}}
row.update(kw)
return {"ok": True, "file": row}
def test_files_write_is_the_precondition_that_is_reported_first():
verdict, detail = precondition(["files:read", "chat:write"])
assert verdict == "no-files-write"
assert "after the bytes are already up" in detail
def test_a_missing_read_scope_only_breaks_the_audit():
verdict, detail = precondition(["files:write"])
assert verdict == "no-files-read"
assert "this audit" in detail
def test_both_scopes_present_is_ready():
assert precondition(["files:write", "files:read"])[0] == "ready"
def test_unread_scopes_are_not_reported_as_granted():
assert precondition(None)[0] == "scopes-unread"
def test_each_recorded_stage_is_named():
assert step_reached({"completed_at": "2026-08-20"})[0] == "completed"
assert step_reached({"file_id": "F1", "bytes_sent": 12})[0] == "bytes-sent"
assert step_reached({"file_id": "F1"})[0] == "url-minted"
assert step_reached({})[0] == "nothing"
def test_a_row_with_only_an_upload_url_got_no_further_than_step_one():
verdict, detail = step_reached({"upload_url": "https://files.slack.com/upload/v1/x"})
assert verdict == "url-minted"
assert "nothing after that is recorded" in detail
def test_matching_byte_counts_are_a_match():
assert length_fault(41821, 41821)[0] == "match"
def test_sending_fewer_bytes_than_declared_is_short():
verdict, detail = length_fault(41821, 41)
assert verdict == "short"
assert "truncated" in detail
def test_sending_more_than_declared_names_the_character_count_mistake():
verdict, detail = length_fault(41, 41821)
assert verdict == "over"
assert "character count" in detail
def test_a_declared_length_of_zero_is_never_right():
assert length_fault(0, 0)[0] == "zero"
def test_a_row_without_both_numbers_is_unknown_rather_than_a_match():
assert length_fault(None, 100)[0] == "unknown"
assert length_fault(100, None)[0] == "unknown"
def test_file_not_found_on_your_own_id_is_the_conclusive_answer():
verdict, detail = reconcile("bytes-sent",
{"ok": False, "error": "file_not_found"})
assert verdict == "never-registered"
assert "before any bytes existed" in detail
def test_a_zero_byte_file_is_the_middle_operation_failing():
verdict, detail = reconcile("completed", ok_file(size=0))
assert verdict == "empty-file"
assert "middle operation" in detail
def test_never_registered_and_empty_file_are_never_collapsed():
assert reconcile("bytes-sent", {"ok": False, "error": "file_not_found"})[0] \
!= reconcile("completed", ok_file(size=0))[0]
def test_a_file_shared_nowhere_is_reported_as_a_separate_question():
verdict, detail = reconcile("completed", ok_file(channels=[]))
assert verdict == "registered-unshared"
assert "separate question" in detail
def test_a_finished_upload_is_complete():
assert reconcile("completed", ok_file())[0] == "complete"
def test_a_file_slack_has_and_your_ledger_stopped_short_of_is_its_own_verdict():
verdict, detail = reconcile("bytes-sent", ok_file())
assert verdict == "complete-unrecorded"
assert "behind Slack" in detail
def test_not_visible_is_a_membership_answer_not_a_broken_sequence():
verdict, detail = reconcile("completed", {"ok": False, "error": "not_visible"})
assert verdict == "unseeable"
assert "membership question" in detail
def test_a_deleted_file_did_finish_the_sequence():
assert reconcile("completed", {"ok": False, "error": "file_deleted"})[0] == "deleted"
def test_an_unrecognised_error_decides_nothing():
assert reconcile("completed", {"ok": False, "error": "ratelimited"})[0] == "unreadable"
def test_bytes_already_up_means_repeat_the_completion_only():
plan = retry_plan("bytes-sent", "never-registered")
assert "repeat the completion" in plan
assert "second orphan" in plan
def test_nothing_transferred_means_start_again():
plan = retry_plan("url-minted", "never-registered")
assert "start this upload again" in plan
assert "discard the recorded id" in plan
def test_the_two_retry_directions_are_never_the_same_line():
assert retry_plan("bytes-sent", "never-registered") \
!= retry_plan("url-minted", "never-registered")
def test_an_empty_file_is_restarted_because_the_url_is_spent():
assert "single use and spent" in retry_plan("completed", "empty-file")
def test_an_unshared_file_is_not_uploaded_again():
assert "rather than uploading it again" in retry_plan("completed",
"registered-unshared")
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
lengthFault, precondition, reconcile, retryPlan, stepReached,
} from './slack-upload-gaps.mjs';
const okFile = (over = {}) => ({
ok: true,
file: {
id: 'F07ABCD1234', size: 41821, channels: ['C024BE91L'], shares: {}, ...over,
},
});
test('files:write is the precondition that is reported first', () => {
const [verdict, detail] = precondition(['files:read', 'chat:write']);
assert.equal(verdict, 'no-files-write');
assert.match(detail, /after the bytes are already up/);
});
test('a missing read scope only breaks the audit', () => {
const [verdict, detail] = precondition(['files:write']);
assert.equal(verdict, 'no-files-read');
assert.match(detail, /this audit/);
});
test('both scopes present is ready', () => {
assert.equal(precondition(['files:write', 'files:read'])[0], 'ready');
});
test('unread scopes are not reported as granted', () => {
assert.equal(precondition(null)[0], 'scopes-unread');
});
test('each recorded stage is named', () => {
assert.equal(stepReached({ completed_at: '2026-08-20' })[0], 'completed');
assert.equal(stepReached({ file_id: 'F1', bytes_sent: 12 })[0], 'bytes-sent');
assert.equal(stepReached({ file_id: 'F1' })[0], 'url-minted');
assert.equal(stepReached({})[0], 'nothing');
});
test('a row with only an upload url got no further than step one', () => {
const [verdict, detail] = stepReached(
{ upload_url: 'https://files.slack.com/upload/v1/x' });
assert.equal(verdict, 'url-minted');
assert.match(detail, /nothing after that is recorded/);
});
test('matching byte counts are a match', () => {
assert.equal(lengthFault(41821, 41821)[0], 'match');
});
test('sending fewer bytes than declared is short', () => {
const [verdict, detail] = lengthFault(41821, 41);
assert.equal(verdict, 'short');
assert.match(detail, /truncated/);
});
test('sending more than declared names the character count mistake', () => {
const [verdict, detail] = lengthFault(41, 41821);
assert.equal(verdict, 'over');
assert.match(detail, /character count/);
});
test('a declared length of zero is never right', () => {
assert.equal(lengthFault(0, 0)[0], 'zero');
});
test('a row without both numbers is unknown rather than a match', () => {
assert.equal(lengthFault(null, 100)[0], 'unknown');
assert.equal(lengthFault(100, undefined)[0], 'unknown');
});
test('file_not_found on your own id is the conclusive answer', () => {
const [verdict, detail] = reconcile('bytes-sent',
{ ok: false, error: 'file_not_found' });
assert.equal(verdict, 'never-registered');
assert.match(detail, /before any bytes existed/);
});
test('a zero byte file is the middle operation failing', () => {
const [verdict, detail] = reconcile('completed', okFile({ size: 0 }));
assert.equal(verdict, 'empty-file');
assert.match(detail, /middle operation/);
});
test('never-registered and empty-file are never collapsed', () => {
assert.notEqual(reconcile('bytes-sent', { ok: false, error: 'file_not_found' })[0],
reconcile('completed', okFile({ size: 0 }))[0]);
});
test('a file shared nowhere is reported as a separate question', () => {
const [verdict, detail] = reconcile('completed', okFile({ channels: [] }));
assert.equal(verdict, 'registered-unshared');
assert.match(detail, /separate question/);
});
test('a finished upload is complete', () => {
assert.equal(reconcile('completed', okFile())[0], 'complete');
});
test('a file Slack has and your ledger stopped short of is its own verdict', () => {
const [verdict, detail] = reconcile('bytes-sent', okFile());
assert.equal(verdict, 'complete-unrecorded');
assert.match(detail, /behind Slack/);
});
test('not_visible is a membership answer not a broken sequence', () => {
const [verdict, detail] = reconcile('completed', { ok: false, error: 'not_visible' });
assert.equal(verdict, 'unseeable');
assert.match(detail, /membership question/);
});
test('a deleted file did finish the sequence', () => {
assert.equal(reconcile('completed', { ok: false, error: 'file_deleted' })[0],
'deleted');
});
test('an unrecognised error decides nothing', () => {
assert.equal(reconcile('completed', { ok: false, error: 'ratelimited' })[0],
'unreadable');
});
test('bytes already up means repeat the completion only', () => {
const plan = retryPlan('bytes-sent', 'never-registered');
assert.match(plan, /repeat the completion/);
assert.match(plan, /second orphan/);
});
test('nothing transferred means start again', () => {
const plan = retryPlan('url-minted', 'never-registered');
assert.match(plan, /start this upload again/);
assert.match(plan, /discard the recorded id/);
});
test('the two retry directions are never the same line', () => {
assert.notEqual(retryPlan('bytes-sent', 'never-registered'),
retryPlan('url-minted', 'never-registered'));
});
test('an empty file is restarted because the url is spent', () => {
assert.match(retryPlan('completed', 'empty-file'), /single use and spent/);
});
test('an unshared file is not uploaded again', () => {
assert.match(retryPlan('completed', 'registered-unshared'),
/rather than uploading it again/);
});
FAQ
The upload raised no exception. How can the file not exist?
Because the id you were given came from the first call, and the first call only reserves a slot. It hands back an upload URL and a file id before any bytes have moved. If the transfer or the completion fails, the id you already wrote down stays exactly as valid-looking as it was. files.info on that id is the only thing that distinguishes a real file from a receipt for one that never happened.
Why does the failure so often turn out to be a scope?
Because files:write is needed by the last step and by nothing before it. An app without it mints the URL fine, transfers the bytes fine, and fails at completion after all the work is done. The SDK reports that as an upload error, which naturally sends people to look at the file rather than at the token. Checking the scope up front costs one header read.
Should I retry the whole upload when it fails?
Only if it failed before the bytes went up. If the transfer succeeded and completion did not, retry completion with the id you already hold. Starting over mints a second id and sends the bytes again, leaving the first orphan in place, which is how a single stalled upload turns into a steadily growing pile of them.
The file exists but nobody can find it. Is that this problem?
No, that one is about sharing rather than about the sequence. If files.info returns a real file with real bytes, all three steps finished and what is missing is a channel to share it into. This script reports that case and hands it on, because the fix is a different argument on a call that already worked rather than a broken sequence.
Can I detect this without keeping a ledger of file ids?
Only partly. Without recorded ids there is nothing to ask files.info about, so the forward audit is unavailable. Listing the files this bot owns still shows what Slack has, which is worth running on its own, but it cannot show you the uploads that produced no file at all. Those exist only as ids in your logs, which is the argument for recording them.
Related field notes
- why the upload became three calls in the first place
- the other way a file ends up somewhere you did not intend
- reading what the token was actually granted
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- files.getUploadURLExternal method reference - Slack Docs
- files.completeUploadExternal method reference - Slack Docs
- files.info method reference - Slack Docs
- files.list method reference - Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.