Diagnostic Slack
url_private without a bearer header saves the login page
The archiver has been running for a year and has never logged an error. Every fetch returned 200, every write succeeded, and the folder has fourteen thousand files in it. Then somebody tries to open one and their PDF reader says the file is damaged.
All fourteen thousand are three kilobytes. All fourteen thousand start with <!DOCTYPE html>. Fetching url_private without an Authorization header does not return 401 — it returns 200 and Slack's sign-in page, and a downloader that follows redirects and writes the body writes the login screen to disk, once per file, silently, for a year.
url_private and url_private_download are authenticated URLs on files.slack.com. They serve the file only to a request carrying Authorization: Bearer <token>. Without that header the response is HTTP 200 with an HTML sign-in page, not a 401, so every check that trusts the status code passes and every downloader that saves the body saves the wrong bytes.
This is the same trap as the Web API's ok: false inside a 200, moved to a different host. It has two extra ways to bite. permalink is a page in the Slack client rather than a file and must never be used as a download URL. And several HTTP clients strip Authorization when a request is redirected to another host, which reproduces the bug exactly on a client that is sending the header.
You can find every corrupted file without downloading anything. files.info reports the file's mimetype and size; the bytes are already on your disk. Compare them. An HTML header where the API says application/pdf, or three kilobytes where the API says four megabytes, is the whole finding — and doing it this way means the audit never needs the credential it is auditing and never sends the file anywhere.
The problem in plain words
Every layer here behaves plausibly and the result is still garbage. Slack answers 200 because it is serving a real page: you asked an anonymous browser-shaped question and got the anonymous browser-shaped answer, which is please sign in. The HTTP client returns success because the request succeeded. The code writes the body because the body is what it asked for. Nothing in that chain is a bug, and nothing in that chain is checking whether the thing on disk is the thing that was wanted.
The corruption is uniform, which is what makes it survive so long. Every file in the folder is the same three kilobytes of the same markup, so spot-checking two of them looks consistent, file counts look right, disk usage looks plausible for a lot of small screenshots, and the archive appears healthy by every measure except opening a file. Teams usually find this when somebody needs one specific document a year later, which is the worst possible moment.
The near-miss version is more interesting, because the code looks correct. A client that sets the header on the initial request but drops it across a cross-host redirect gets exactly the same login page. Python's requests strips Authorization when a redirect changes host; several other stacks do the same, on purpose, as a credential-leak defence. So we do send the bearer token is true and the file on disk is still HTML, and the fix is about redirect handling rather than about headers.
Then there is the URL that was never going to work. permalink is the human-facing page for a file, and it is the most obvious-looking field in the response, so it gets stored as the URL of the file constantly. Fetching it authenticated returns a Slack web page; fetching it unauthenticated returns a sign-in page. permalink_public, when it exists at all, is a wrapper page rather than raw bytes, and it only exists for files that were deliberately made public - which is a separate and worse problem than this one.
The last thing worth saying is what this script deliberately does not do. It does not fetch anything from files.slack.com, and it does not print a url_private. Fetching to prove the point would either need the exact credential under audit or would pull the content of every file through the auditing process, and a url_private in a log line is one paste away from being a capability somebody else holds. The API describes the file, the bytes are already on the disk, and the comparison is offline.
Why it happens
The status code is not the answer, again. Slack's Web API puts failures in a 200 body and its file host puts them in a 200 page. Once you know that, the rule generalises: on anything Slack-shaped, the check is what came back, not what number came with it.
The API already told you what the file should be. mimetype and size come back from files.info for free, and together they are a complete contract for what should be on disk. Almost nobody compares against them, which is why this corruption is invisible - not because it is hard to detect, but because there is no line of code anywhere asking the question.
The audit must not need the credential it is auditing. A check that downloads the files to test them requires the working token, which is exactly the thing you suspect is missing, and it moves the content of every file through one more process. Reading the first few hundred bytes off your own disk answers the same question with no token and no transfer.
A url_private is a capability once you attach a token, so it does not belong in a log. The script classifies URLs and prints the class and a redacted host-and-path-kind, never the URL itself and never a query string. pub_secret values in particular are the whole secret.
Redirect handling is a separate failure with an identical symptom. Because it looks like the header is being sent - it is, on the first hop - it survives code review and gets diagnosed as a Slack problem. The script cannot see your HTTP client, so the note names it explicitly as the thing to check when the header is provably present and the file on disk is still HTML.
Size and type are different findings and want different fixes. HTML in place of a PDF is authentication. A file that is the right type and the wrong length is a truncated write, a stream closed early, or a mis-declared length at upload time. Reporting both as download broken merges an auth bug with an I/O bug.
A validation step at write time makes the whole class of bug impossible. Compare Content-Type against the API's mimetype before saving, and refuse to write a file whose length does not match size. That is four lines, and it converts a year of silent corruption into an error on the first file.
The fix, as a flow
This chain is red at the very first arrow, at the request, and every box after it is a consequence wearing the costume of a success: a 200, a body, a write, a folder that looks right by every measure except opening a file. The fix branch is the only one in this batch that sorts bytes rather than API answers, and the order of its rows is the design. HTML is decided before length, because a sign-in page is also the wrong size and the wrong size sends you to the wrong part of your own code.
How to fix it
Classify the URL your app stored, with no token at all
--urls stored-urls.json runs url_class over the download URLs your app kept and needs no credential. web-page means somebody stored permalink, which was never going to return a file. public-link means the file was made public, which is a different note and a worse problem. private-bytes is the only class that is correct here, and it is the one that needs the header.
Ask files.info what the file is supposed to be
mimetype, size, filetype and name, for each id your downloader wrote. This is the contract. It is one read per file and it needs files:read and nothing else.
Read the first bytes of what you already saved
sniff takes the leading bytes off your local copy and names the format from its magic number - %PDF-, the PNG signature, PK\x03\x04 for anything zip-shaped including Office documents - and, crucially, recognises an HTML document. No network, and no need for the token.
Hold the two against each other
saved_verdict returns login-page, empty, truncated, size-mismatch, type-mismatch or ok. The HTML check runs before the size checks on purpose: a login page is also the wrong size, and reporting it as size-mismatch would send somebody to look at their write buffer instead of their headers.
Separate the auth finding from the I/O finding
login-page is authentication and the repair is a header. truncated is a write that stopped early or a stream that was not drained. They arrive in the same run and they have nothing to do with each other, and a single corrupt bucket costs a day.
If the header is provably present, look at redirects next
A client that sets Authorization and then follows a redirect to another host may drop it - Python's requests does exactly this, deliberately. Same symptom, different cause, and the script cannot see it from here, so it prints the check rather than performing it: log the final URL and whether the header survived the last hop.
Validate at write time so this can never be silent again
The printed repair is a code change: compare the response Content-Type against the API's mimetype and the byte count against size, and refuse the write when either disagrees. Four lines, and the failure becomes loud on the first file instead of quiet for a year.
How to check it worked
Point it at the folder your downloader filled. Nothing is fetched from Slack's file host and no URL is printed in full.
python3 slack_download_audit.py --saved archive-index.json
# identity U07BOT9QD in Northwind
# url private-bytes F08K2M4QX https://files.slack.com/files-pri/... needs a
# bearer header on every hop
# saved login-page F08K2M4QX the bytes on disk begin an HTML document and
# files.info says application/pdf. That is the sign-in page
# saved truncated F08K2M55Z 3 KB on disk against 4.1 MB reported; a write
# that stopped early, which is not the same bug
# url web-page F08K2M77A https://northwind.slack.com/files/... a
# permalink is a page in the client, not the file
# saved ok F08K2M91B runbook.pdf 812 KB, application/pdf
# verdict 2 of 4 saved file(s) are not what the API says they are
# repair: send Authorization: Bearer on every fetch of url_private_download
# repair: compare Content-Type against mimetype before writing, and refuse on a
# mismatch rather than saving the bodyThe full code
Nothing here fetches a file and nothing here prints a URL. url_class decides which of the four Slack URL kinds you stored and whether it needs a header; redact exists so the answer can be logged without logging a capability. sniff reads the leading bytes of your own copy, and saved_verdict holds them against the mimetype and size the API reports. The order of the checks inside that last function is the design: HTML is tested before length, because a login page is also the wrong size and the wrong size sends people to the wrong place.
"""Check whether your downloader saved Slack files or Slack's sign-in page.
Read only, and narrower than that: the only network call is files.info, which
describes a file without transferring it. This script never fetches
url_private, never follows a Slack file URL, and never prints one.
Fetching to prove the point would be wrong twice over. It would need the exact
credential under audit - the missing header is the thing being investigated -
and it would pull the content of every file through one more process. The API
already reports the mimetype and the size, and the bytes are already on your
disk, so the comparison is offline and needs no token for the interesting half.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_download_audit")
API = "https://slack.com/api/"
HEAD_BYTES = 512
# Leading bytes, as latin-1 text so the table reads as a table. The zip
# signature covers docx, xlsx, pptx and every other Office format, which is why
# compatibility below is more forgiving than equality.
MAGIC = (
("%PDF-", "application/pdf"),
("\x89PNG\r\n\x1a\n", "image/png"),
("\xff\xd8\xff", "image/jpeg"),
("GIF87a", "image/gif"),
("GIF89a", "image/gif"),
("PK\x03\x04", "application/zip"),
("\x1f\x8b", "application/gzip"),
("OggS", "audio/ogg"),
("ID3", "audio/mpeg"),
("RIFF", "audio/wav"),
("\x00\x00\x00", "video/mp4"),
)
HTML_STARTS = ("<!doctype html", "<html", "<head", "<!-- ")
def _text(head):
"""The leading bytes as latin-1 text. Never decodes lossily enough to lie."""
if head is None:
return ""
if isinstance(head, str):
return head[:HEAD_BYTES]
return bytes(head)[:HEAD_BYTES].decode("latin-1", "replace")
def sniff(head):
"""What are these bytes, judged only by how they start? Pure.
Returns a mimetype or "". The case this function exists for is the first
one checked: an HTML document where a file was expected is Slack's sign-in
page, served with HTTP 200 to a request that carried no Authorization
header.
"""
text = _text(head)
stripped = text.lstrip()
lowered = stripped.lower()
if any(lowered.startswith(start) for start in HTML_STARTS):
return "text/html"
if "<title>slack" in lowered or "signin_form" in lowered:
return "text/html"
for prefix, mimetype in MAGIC:
if text.startswith(prefix):
return mimetype
return ""
def compatible(sniffed, declared):
"""Do a sniffed type and an API mimetype describe the same file? Pure.
Zip is the whole reason this is not equality: every Office format is a zip
archive, so PK\x03\x04 is the honest answer for a .docx and reporting it as
a mismatch would bury the real findings under paperwork.
"""
if not sniffed or not declared:
return True
left, right = sniffed.lower(), declared.lower()
if left == right:
return True
if left == "application/zip":
return any(token in right for token in
("zip", "officedocument", "opendocument", "epub", "jar"))
if left.split("/")[0] == right.split("/")[0] == "image":
return True
return False
def url_class(url):
"""Which kind of Slack URL is this, and does it need a header? Pure.
Returns (kind, needs_header, detail), and never the URL itself.
private-bytes files-pri or files-tmb: the file, served only to a request
carrying Authorization: Bearer. This is the correct one.
web-page a permalink. A page in the Slack client, not the file, and
the most common thing to have stored by mistake because it
is the friendliest looking field in the response.
public-link a pub_secret link, which serves the bytes to anyone at all.
A different and worse problem.
elsewhere not a Slack file URL; an external file lives on its own host.
"""
text = str(url or "").strip()
if not text:
return ("missing", False, "no download URL was stored for this file")
low = text.lower()
if not low.startswith(("http://", "https://")):
return ("not-a-url", False, "not a URL at all, so nothing was ever going to "
"come back from it")
if "pub_secret=" in low or "slack-files.com" in low:
return ("public-link", False, "a public file link. It serves the bytes to "
"anyone holding it, with no Slack login, which "
"is a data exposure rather than a download bug")
if "/files-pri/" in low or "/files-tmb/" in low:
return ("private-bytes", True, "the file itself, served only to a request "
"carrying an Authorization bearer header on "
"every hop including redirects")
if "/files/" in low or "/archives/" in low:
return ("web-page", False, "a permalink, which is a page in the Slack client "
"rather than the file. Authenticated or not, this "
"returns markup")
return ("elsewhere", False, "not a Slack file host, so this is an external file "
"and its access rules are somebody else's")
def redact(url):
"""Host and path kind, with the identifiers and the query removed. Pure.
A url_private is a capability the moment a token is attached to it, and a
pub_secret query string is a capability on its own. Neither belongs in a log
line, a ticket or a screenshot, so this is what the script prints instead.
"""
text = str(url or "").strip()
if not text:
return ""
text = text.split("?", 1)[0].split("#", 1)[0]
parts = text.split("/")
if len(parts) < 4:
return text
return "/".join(parts[:4]) + "/..."
def saved_verdict(info, saved_size, head=b""):
"""Is the file on disk the file the API describes? Pure.
Returns (verdict, detail). The order of the checks is the design of this
function: the HTML test runs before the length tests, because a sign-in page
is also the wrong size, and reporting it as a size problem sends somebody to
look at their write buffer when the fault is in their headers.
"""
f = info or {}
declared = str(f.get("mimetype") or "")
size = f.get("size")
size = size if isinstance(size, int) and not isinstance(size, bool) else None
sniffed = sniff(head)
if not saved_size:
return ("empty", "nothing was written. A zero byte file is a fetch that "
"returned no body, or a write that never happened")
if sniffed == "text/html" and "html" not in declared.lower():
return ("login-page", "the bytes on disk begin an HTML document and files.info "
"says %s. That is Slack's sign-in page, served with HTTP "
"200 to a fetch that carried no Authorization header"
% (declared or "something else"))
if size is not None and size > 0 and saved_size != size:
if saved_size * 4 < size:
return ("truncated", "%d byte(s) on disk against %d reported. A write that "
"stopped early or a stream that was not drained, which "
"is not the same bug as a missing header"
% (saved_size, size))
return ("size-mismatch", "%d byte(s) on disk against %d reported" % (saved_size,
size))
if not compatible(sniffed, declared):
return ("type-mismatch", "the bytes look like %s and files.info says %s"
% (sniffed, declared))
return ("ok", "%s, %d byte(s), matching what files.info reports"
% (declared or "unknown type", saved_size))
def load_saved(path):
"""Whatever your downloader wrote down: {id: path}, or a list of records."""
raw = json.loads(open(path, encoding="utf-8").read())
if isinstance(raw, dict) and not raw.get("files"):
return [(str(k), str(v)) for k, v in raw.items() if isinstance(v, str)]
rows = raw.get("files") if isinstance(raw, dict) else raw
out = []
for row in rows or []:
if isinstance(row, dict):
fid = row.get("id") or row.get("file_id") or ""
where = row.get("path") or row.get("saved") or ""
if fid and where:
out.append((str(fid), str(where)))
return out
def head_and_size(path):
"""The leading bytes and the length of a local file. No network."""
try:
with open(path, "rb") as fh:
head = fh.read(HEAD_BYTES)
return head, os.path.getsize(path)
except OSError as err:
log.warning("saved unreadable %s %s", path, err)
return b"", None
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--saved", default="",
help="a JSON map of file id to the local path your downloader wrote")
ap.add_argument("--urls", default="",
help="a JSON list of stored download URLs; classified with no token")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN")
args = ap.parse_args()
if not args.saved and not args.urls:
log.error("pass --saved FILE, or --urls FILE to classify stored URLs offline")
return 2
bad_urls = 0
if args.urls:
for url in json.loads(open(args.urls, encoding="utf-8").read()) or []:
kind, needs, detail = url_class(url)
line = ("url %-14s %s %s", kind, redact(url), detail)
if kind == "private-bytes":
log.info(*line)
else:
bad_urls += 1
log.warning(*line)
if needs:
log.info(" this one is correct and it is the one that needs "
"the header")
if not args.saved:
if bad_urls:
log.warning("verdict %d stored URL(s) are not the file", bad_urls)
return 1
log.info("verdict clean every stored URL is a private file URL")
return 0
token = os.environ.get(args.token_env)
if not token:
log.error("set %s to a bot token with files:read", args.token_env)
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
who = s.get(API + "auth.test", timeout=30).json()
if who.get("ok") is not True:
log.error("auth.test unavailable %s", who.get("error"))
return 2
log.info("identity %s in %s", who.get("user_id"), who.get("team"))
rows = load_saved(args.saved)
if not rows:
log.error("no {id: path} records found in %s", args.saved)
return 2
broken = 0
for fid, where in rows:
body = s.get(API + "files.info", params={"file": fid}, timeout=30).json()
if body.get("ok") is not True:
log.warning("info unavailable %s %s", fid, body.get("error"))
continue
info = body.get("file") or {}
kind, _needs, detail = url_class(info.get("url_private_download")
or info.get("url_private"))
log.info("url %-14s %s %s %s", kind, fid,
redact(info.get("url_private_download") or info.get("url_private")),
detail)
head, saved_size = head_and_size(where)
if saved_size is None:
continue
verdict, why = saved_verdict(info, saved_size, head)
if verdict == "ok":
log.info("saved ok %s %s", fid, why)
continue
broken += 1
log.warning("saved %-14s %s %s", verdict, fid, why)
if not broken:
log.info("verdict clean every saved file matches what files.info "
"reports")
return 0
log.warning("verdict %d of %d saved file(s) are not what the API says they are",
broken, len(rows))
log.warning(" repair: send Authorization: Bearer on every fetch of "
"url_private_download, and never use permalink as a download URL")
log.warning(" repair: compare the response Content-Type against the API mimetype "
"before writing, and refuse the write on a mismatch")
log.warning(" repair: if the header is provably sent, check redirects; several "
"HTTP clients drop Authorization when a redirect changes host")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Check whether your downloader saved Slack files or Slack's sign-in page.
*
* Read only, and narrower than that: the only network call is files.info, which
* describes a file without transferring it. This script never fetches
* url_private, never follows a Slack file URL, and never prints one.
*
* Fetching to prove the point would need the exact credential under audit - the
* missing header is the thing being investigated - and would pull the content
* of every file through one more process. The API reports the mimetype and the
* size, the bytes are already on disk, so the comparison is offline.
*/
import { readFile, stat } from 'node:fs/promises';
const API = 'https://slack.com/api/';
const HEAD_BYTES = 512;
// Leading bytes, as latin-1 text so the table reads as a table. The zip
// signature covers every Office format, which is why compatibility below is
// more forgiving than equality.
const MAGIC = [
['%PDF-', 'application/pdf'],
['\x89PNG\r\n\x1a\n', 'image/png'],
['\xff\xd8\xff', 'image/jpeg'],
['GIF87a', 'image/gif'],
['GIF89a', 'image/gif'],
['PK\x03\x04', 'application/zip'],
['\x1f\x8b', 'application/gzip'],
['OggS', 'audio/ogg'],
['ID3', 'audio/mpeg'],
['RIFF', 'audio/wav'],
['\x00\x00\x00', 'video/mp4'],
];
const HTML_STARTS = ['<!doctype html', '<html', '<head', '<!-- '];
function asText(head) {
if (head === null || head === undefined) return '';
if (typeof head === 'string') return head.slice(0, HEAD_BYTES);
let out = '';
for (let i = 0; i < Math.min(head.length, HEAD_BYTES); i += 1) {
out += String.fromCharCode(head[i]);
}
return out;
}
/**
* What are these bytes, judged only by how they start? Pure.
* Returns a mimetype or ''. HTML is checked first because HTML is the note.
*/
export function sniff(head) {
const text = asText(head);
const lowered = text.trimStart().toLowerCase();
if (HTML_STARTS.some((start) => lowered.startsWith(start))) return 'text/html';
if (lowered.includes('<title>slack') || lowered.includes('signin_form')) {
return 'text/html';
}
for (const [prefix, mimetype] of MAGIC) {
if (text.startsWith(prefix)) return mimetype;
}
return '';
}
/**
* Do a sniffed type and an API mimetype describe the same file? Pure.
* Zip is why this is not equality: every Office format is a zip archive.
*/
export function compatible(sniffed, declared) {
if (!sniffed || !declared) return true;
const left = sniffed.toLowerCase();
const right = declared.toLowerCase();
if (left === right) return true;
if (left === 'application/zip') {
return ['zip', 'officedocument', 'opendocument', 'epub', 'jar']
.some((token) => right.includes(token));
}
if (left.split('/')[0] === 'image' && right.split('/')[0] === 'image') return true;
return false;
}
/**
* Which kind of Slack URL is this, and does it need a header? Pure.
* Returns [kind, needsHeader, detail], and never the URL itself.
*/
export function urlClass(url) {
const text = String(url ?? '').trim();
if (!text) return ['missing', false, 'no download URL was stored for this file'];
const low = text.toLowerCase();
if (!low.startsWith('http://') && !low.startsWith('https://')) {
return ['not-a-url', false, 'not a URL at all, so nothing was ever going to come '
+ 'back from it'];
}
if (low.includes('pub_secret=') || low.includes('slack-files.com')) {
return ['public-link', false, 'a public file link. It serves the bytes to anyone '
+ 'holding it, with no Slack login, which is a data exposure rather than a '
+ 'download bug'];
}
if (low.includes('/files-pri/') || low.includes('/files-tmb/')) {
return ['private-bytes', true, 'the file itself, served only to a request carrying '
+ 'an Authorization bearer header on every hop including redirects'];
}
if (low.includes('/files/') || low.includes('/archives/')) {
return ['web-page', false, 'a permalink, which is a page in the Slack client rather '
+ 'than the file. Authenticated or not, this returns markup'];
}
return ['elsewhere', false, 'not a Slack file host, so this is an external file and '
+ "its access rules are somebody else's"];
}
/**
* Host and path kind, with the identifiers and the query removed. Pure.
* A url_private is a capability once a token is attached; a pub_secret query
* string is a capability on its own. Neither belongs in a log line.
*/
export function redact(url) {
let text = String(url ?? '').trim();
if (!text) return '';
text = text.split('?')[0];
text = text.split('#')[0];
const parts = text.split('/');
if (parts.length < 4) return text;
return `${parts.slice(0, 4).join('/')}/...`;
}
/**
* Is the file on disk the file the API describes? Pure.
* Returns [verdict, detail]. The HTML test runs before the length tests, because
* a sign-in page is also the wrong size and the wrong size sends people to the
* wrong place.
*/
export function savedVerdict(info, savedSize, head = '') {
const f = info ?? {};
const declared = String(f.mimetype ?? '');
const size = (typeof f.size === 'number' && Number.isFinite(f.size)) ? f.size : null;
const sniffed = sniff(head);
if (!savedSize) {
return ['empty', 'nothing was written. A zero byte file is a fetch that returned '
+ 'no body, or a write that never happened'];
}
if (sniffed === 'text/html' && !declared.toLowerCase().includes('html')) {
return ['login-page', 'the bytes on disk begin an HTML document and files.info says '
+ `${declared || 'something else'}. That is Slack's sign-in page, served with `
+ 'HTTP 200 to a fetch that carried no Authorization header'];
}
if (size !== null && size > 0 && savedSize !== size) {
if (savedSize * 4 < size) {
return ['truncated', `${savedSize} byte(s) on disk against ${size} reported. A `
+ 'write that stopped early or a stream that was not drained, which is not the '
+ 'same bug as a missing header'];
}
return ['size-mismatch', `${savedSize} byte(s) on disk against ${size} reported`];
}
if (!compatible(sniffed, declared)) {
return ['type-mismatch', `the bytes look like ${sniffed} and files.info says `
+ `${declared}`];
}
return ['ok', `${declared || 'unknown type'}, ${savedSize} byte(s), matching what `
+ 'files.info reports'];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
function loadSaved(raw) {
if (raw && !Array.isArray(raw) && typeof raw === 'object' && !raw.files) {
return Object.entries(raw).filter(([, v]) => typeof v === 'string')
.map(([k, v]) => [String(k), String(v)]);
}
const rows = (raw && !Array.isArray(raw)) ? raw.files : raw;
const out = [];
for (const row of rows ?? []) {
if (row && typeof row === 'object') {
const fid = row.id ?? row.file_id ?? '';
const where = row.path ?? row.saved ?? '';
if (fid && where) out.push([String(fid), String(where)]);
}
}
return out;
}
async function headAndSize(path) {
try {
const bytes = await readFile(path);
const info = await stat(path);
return [bytes.subarray(0, HEAD_BYTES), info.size];
} catch (err) {
console.warn(`saved unreadable ${path} ${err.message}`);
return ['', null];
}
}
async function main() {
const args = process.argv.slice(2);
const savedFile = arg(args, '--saved', '');
const urlsFile = arg(args, '--urls', '');
if (!savedFile && !urlsFile) {
console.error('pass --saved FILE, or --urls FILE to classify stored URLs offline');
process.exitCode = 2;
return;
}
let badUrls = 0;
if (urlsFile) {
for (const url of JSON.parse(await readFile(urlsFile, 'utf8')) ?? []) {
const [kind, needs, detail] = urlClass(url);
const line = `url ${kind.padEnd(14)} ${redact(url)} ${detail}`;
if (kind === 'private-bytes') console.log(line);
else {
badUrls += 1;
console.warn(line);
}
if (needs) {
console.log(' this one is correct and it is the one that needs the '
+ 'header');
}
}
if (!savedFile) {
if (badUrls) {
console.warn(`verdict ${badUrls} stored URL(s) are not the file`);
process.exitCode = 1;
} else {
console.log('verdict clean every stored URL is a private file URL');
}
return;
}
}
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (!token) {
console.error(`set ${tokenEnv} to a bot token with files:read`);
process.exitCode = 2;
return;
}
const headers = { Authorization: `Bearer ${token}` };
const who = await (await fetch(`${API}auth.test`, { headers })).json();
if (who.ok !== true) {
console.error(`auth.test unavailable ${who.error}`);
process.exitCode = 2;
return;
}
console.log(`identity ${who.user_id} in ${who.team}`);
const rows = loadSaved(JSON.parse(await readFile(savedFile, 'utf8')));
if (!rows.length) {
console.error(`no {id: path} records found in ${savedFile}`);
process.exitCode = 2;
return;
}
let broken = 0;
for (const [fid, where] of rows) {
const params = new URLSearchParams({ file: fid });
const body = await (await fetch(`${API}files.info?${params}`, { headers })).json();
if (body.ok !== true) {
console.warn(`info unavailable ${fid} ${body.error}`);
continue;
}
const info = body.file ?? {};
const target = info.url_private_download ?? info.url_private;
const [kind, , detail] = urlClass(target);
console.log(`url ${kind.padEnd(14)} ${fid} ${redact(target)} ${detail}`);
const [head, savedSize] = await headAndSize(where);
if (savedSize === null) continue;
const [verdict, why] = savedVerdict(info, savedSize, head);
if (verdict === 'ok') {
console.log(`saved ok ${fid} ${why}`);
continue;
}
broken += 1;
console.warn(`saved ${verdict.padEnd(14)} ${fid} ${why}`);
}
if (!broken) {
console.log('verdict clean every saved file matches what files.info '
+ 'reports');
return;
}
console.warn(`verdict ${broken} of ${rows.length} saved file(s) are not what the `
+ 'API says they are');
console.warn(' repair: send Authorization: Bearer on every fetch of '
+ 'url_private_download, and never use permalink as a download URL');
console.warn(' repair: compare the response Content-Type against the API mimetype '
+ 'before writing, and refuse the write on a mismatch');
console.warn(' repair: if the header is provably sent, check redirects; several HTTP '
+ 'clients drop Authorization when a redirect changes host');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main() and fail the file on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The load-bearing assertion is an ordering one: a three-kilobyte HTML document standing in for a four-megabyte PDF must come back as login-page and not as truncated, even though it is also the wrong size, because the two verdicts send you to different files in your own codebase. redact is tested for what it removes rather than what it keeps, with a pub_secret query string that must not survive. And a .docx is tested as a zip archive, since a check that reports every Office document as a type mismatch is a check nobody reads twice.
from slack_download_audit import compatible, redact, saved_verdict, sniff, url_class
SIGNIN = b"<!DOCTYPE html>\n<html lang=\"en\"><head><title>Slack</title>"
PDF = b"%PDF-1.7\n%\xe2\xe3\xcf\xd3\n"
def test_the_sign_in_page_is_recognised_as_html():
assert sniff(SIGNIN) == "text/html"
assert sniff(b" <html>") == "text/html"
def test_the_common_formats_are_recognised_by_their_magic_numbers():
assert sniff(PDF) == "application/pdf"
assert sniff(b"\x89PNG\r\n\x1a\nrest") == "image/png"
assert sniff(b"\xff\xd8\xff\xe0jpeg") == "image/jpeg"
assert sniff(b"PK\x03\x04docx") == "application/zip"
def test_unknown_bytes_are_not_guessed_at():
assert sniff(b"\x07\x07\x07nothing") == ""
assert sniff(b"") == ""
assert sniff(None) == ""
def test_an_office_document_is_a_zip_and_that_is_not_a_mismatch():
docx = ("application/vnd.openxmlformats-officedocument."
"wordprocessingml.document")
assert compatible("application/zip", docx) is True
def test_an_unknown_sniff_never_manufactures_a_mismatch():
assert compatible("", "application/pdf") is True
assert compatible("application/pdf", "") is True
def test_html_against_a_pdf_is_a_mismatch():
assert compatible("text/html", "application/pdf") is False
def test_the_private_file_url_is_the_only_one_that_needs_a_header():
kind, needs, _ = url_class("https://files.slack.com/files-pri/T1-F1/report.pdf")
assert kind == "private-bytes"
assert needs is True
def test_a_permalink_is_a_page_and_was_never_going_to_be_a_file():
kind, needs, detail = url_class(
"https://northwind.slack.com/files/U07BOT9QD/F06AAA1QQ/report.pdf")
assert kind == "web-page"
assert needs is False
assert "page in the Slack client" in detail
def test_a_public_link_is_named_as_an_exposure_rather_than_a_download_bug():
kind, _, detail = url_class(
"https://files.slack.com/files-pri/T1-F1/report.pdf?pub_secret=abc123")
assert kind == "public-link"
assert "exposure" in detail
def test_a_missing_or_broken_url_has_its_own_verdicts():
assert url_class("")[0] == "missing"
assert url_class(None)[0] == "missing"
assert url_class("files.slack.com/files-pri/x")[0] == "not-a-url"
def test_an_external_file_url_is_somebody_elses_problem():
assert url_class("https://example.com/report.pdf")[0] == "elsewhere"
def test_redact_keeps_the_host_and_the_path_kind_and_nothing_else():
assert redact("https://files.slack.com/files-pri/T1-F1/report.pdf") \
== "https://files.slack.com/files-pri/..."
def test_redact_removes_the_query_string_because_that_is_where_the_secret_is():
out = redact("https://files.slack.com/files-pri/T1-F1/r.pdf?pub_secret=abc123")
assert "pub_secret" not in out
assert "abc123" not in out
def test_the_sign_in_page_saved_over_a_pdf_is_the_finding():
verdict, detail = saved_verdict(
{"mimetype": "application/pdf", "size": 4194304}, len(SIGNIN), SIGNIN)
assert verdict == "login-page"
assert "no Authorization header" in detail
def test_a_login_page_is_never_reported_as_a_size_problem():
verdict, _ = saved_verdict({"mimetype": "application/pdf", "size": 4194304},
3000, SIGNIN)
assert verdict == "login-page"
assert verdict != "truncated"
def test_a_genuine_html_file_is_not_a_login_page():
assert saved_verdict({"mimetype": "text/html", "size": len(SIGNIN)},
len(SIGNIN), SIGNIN)[0] == "ok"
def test_a_short_write_is_truncated_rather_than_a_header_problem():
verdict, detail = saved_verdict({"mimetype": "application/pdf", "size": 4194304},
3000, PDF)
assert verdict == "truncated"
assert "not the same bug" in detail
def test_a_small_difference_is_a_size_mismatch_and_not_a_truncation():
assert saved_verdict({"mimetype": "application/pdf", "size": 1000},
900, PDF)[0] == "size-mismatch"
def test_nothing_on_disk_is_its_own_verdict():
assert saved_verdict({"mimetype": "application/pdf", "size": 10}, 0, b"")[0] \
== "empty"
def test_the_right_length_and_the_wrong_format_is_a_type_mismatch():
assert saved_verdict({"mimetype": "image/png", "size": len(PDF)},
len(PDF), PDF)[0] == "type-mismatch"
def test_a_matching_file_is_ok():
verdict, detail = saved_verdict({"mimetype": "application/pdf", "size": len(PDF)},
len(PDF), PDF)
assert verdict == "ok"
assert "matching what files.info reports" in detail
def test_a_file_the_api_reports_no_size_for_is_still_judged_on_its_bytes():
assert saved_verdict({"mimetype": "application/pdf"}, 1234, PDF)[0] == "ok"
assert saved_verdict({"mimetype": "application/pdf"}, 1234, SIGNIN)[0] \
== "login-page"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
compatible, redact, savedVerdict, sniff, urlClass,
} from './slack-download-audit.mjs';
const bytes = (s) => Uint8Array.from([...s].map((c) => c.charCodeAt(0)));
const SIGNIN = bytes('<!DOCTYPE html>\n<html lang="en"><head><title>Slack</title>');
const PDF = bytes('%PDF-1.7\n%\xe2\xe3\xcf\xd3\n');
test('the sign in page is recognised as HTML', () => {
assert.equal(sniff(SIGNIN), 'text/html');
assert.equal(sniff(bytes(' <html>')), 'text/html');
});
test('the common formats are recognised by their magic numbers', () => {
assert.equal(sniff(PDF), 'application/pdf');
assert.equal(sniff(bytes('\x89PNG\r\n\x1a\nrest')), 'image/png');
assert.equal(sniff(bytes('\xff\xd8\xff\xe0jpeg')), 'image/jpeg');
assert.equal(sniff(bytes('PK\x03\x04docx')), 'application/zip');
});
test('unknown bytes are not guessed at', () => {
assert.equal(sniff(bytes('\x07\x07\x07nothing')), '');
assert.equal(sniff(bytes('')), '');
assert.equal(sniff(null), '');
});
test('an Office document is a zip and that is not a mismatch', () => {
const docx = 'application/vnd.openxmlformats-officedocument.'
+ 'wordprocessingml.document';
assert.equal(compatible('application/zip', docx), true);
});
test('an unknown sniff never manufactures a mismatch', () => {
assert.equal(compatible('', 'application/pdf'), true);
assert.equal(compatible('application/pdf', ''), true);
});
test('HTML against a PDF is a mismatch', () => {
assert.equal(compatible('text/html', 'application/pdf'), false);
});
test('the private file URL is the only one that needs a header', () => {
const [kind, needs] = urlClass('https://files.slack.com/files-pri/T1-F1/report.pdf');
assert.equal(kind, 'private-bytes');
assert.equal(needs, true);
});
test('a permalink is a page and was never going to be a file', () => {
const [kind, needs, detail] = urlClass(
'https://northwind.slack.com/files/U07BOT9QD/F06AAA1QQ/report.pdf');
assert.equal(kind, 'web-page');
assert.equal(needs, false);
assert.match(detail, /page in the Slack client/);
});
test('a public link is named as an exposure rather than a download bug', () => {
const [kind, , detail] = urlClass(
'https://files.slack.com/files-pri/T1-F1/report.pdf?pub_secret=abc123');
assert.equal(kind, 'public-link');
assert.match(detail, /exposure/);
});
test('a missing or broken URL has its own verdicts', () => {
assert.equal(urlClass('')[0], 'missing');
assert.equal(urlClass(null)[0], 'missing');
assert.equal(urlClass('files.slack.com/files-pri/x')[0], 'not-a-url');
});
test('an external file URL is somebody elses problem', () => {
assert.equal(urlClass('https://example.com/report.pdf')[0], 'elsewhere');
});
test('redact keeps the host and the path kind and nothing else', () => {
assert.equal(redact('https://files.slack.com/files-pri/T1-F1/report.pdf'),
'https://files.slack.com/files-pri/...');
});
test('redact removes the query string because that is where the secret is', () => {
const out = redact('https://files.slack.com/files-pri/T1-F1/r.pdf?pub_secret=abc123');
assert.doesNotMatch(out, /pub_secret/);
assert.doesNotMatch(out, /abc123/);
});
test('the sign in page saved over a PDF is the finding', () => {
const [verdict, detail] = savedVerdict(
{ mimetype: 'application/pdf', size: 4194304 }, SIGNIN.length, SIGNIN);
assert.equal(verdict, 'login-page');
assert.match(detail, /no Authorization header/);
});
test('a login page is never reported as a size problem', () => {
const [verdict] = savedVerdict({ mimetype: 'application/pdf', size: 4194304 },
3000, SIGNIN);
assert.equal(verdict, 'login-page');
assert.notEqual(verdict, 'truncated');
});
test('a genuine HTML file is not a login page', () => {
assert.equal(savedVerdict({ mimetype: 'text/html', size: SIGNIN.length },
SIGNIN.length, SIGNIN)[0], 'ok');
});
test('a short write is truncated rather than a header problem', () => {
const [verdict, detail] = savedVerdict({ mimetype: 'application/pdf', size: 4194304 },
3000, PDF);
assert.equal(verdict, 'truncated');
assert.match(detail, /not the same bug/);
});
test('a small difference is a size mismatch and not a truncation', () => {
assert.equal(savedVerdict({ mimetype: 'application/pdf', size: 1000 },
900, PDF)[0], 'size-mismatch');
});
test('nothing on disk is its own verdict', () => {
assert.equal(savedVerdict({ mimetype: 'application/pdf', size: 10 }, 0,
bytes(''))[0], 'empty');
});
test('the right length and the wrong format is a type mismatch', () => {
assert.equal(savedVerdict({ mimetype: 'image/png', size: PDF.length },
PDF.length, PDF)[0], 'type-mismatch');
});
test('a matching file is ok', () => {
const [verdict, detail] = savedVerdict({ mimetype: 'application/pdf',
size: PDF.length }, PDF.length, PDF);
assert.equal(verdict, 'ok');
assert.match(detail, /matching what files\.info reports/);
});
test('a file the API reports no size for is still judged on its bytes', () => {
assert.equal(savedVerdict({ mimetype: 'application/pdf' }, 1234, PDF)[0], 'ok');
assert.equal(savedVerdict({ mimetype: 'application/pdf' }, 1234, SIGNIN)[0],
'login-page');
});
FAQ
Why does Slack return 200 instead of 401 for an unauthenticated file fetch?
Because it is answering the request it thinks it received. An anonymous request for a file URL looks like a person clicking a link in a browser, and the correct response to a person who is not signed in is a sign-in page, served with 200 like any other page. It is the same shape as the Web API putting ok: false inside a 200, moved to a different host, and it defeats the same reflex: checking the status code and moving on.
What is the difference between url_private, permalink and permalink_public?
url_private and url_private_download are the file, on files.slack.com, served only to a request carrying a bearer token. permalink is a page in the Slack client that shows the file to a signed-in human; it is not a download URL and never returns the bytes. permalink_public exists only for files that were deliberately made public and is a wrapper page rather than raw bytes - and its existence is a separate and more serious finding than a broken download.
We definitely send the Authorization header and we still get HTML. What else could it be?
Redirects. Several HTTP clients drop the Authorization header when a redirect changes host, as a deliberate defence against leaking credentials to third parties - Python's requests does exactly this. The first hop carries the header, the second does not, and the response body is the sign-in page again. Log the final URL and whether the header survived the last hop; the symptom is identical and the fix is in your client configuration rather than your code.
Why does this script not just download one file and check it?
Two reasons, and both are about what an audit should cost. It would need the working credential, which is precisely the thing under investigation, so the check would fail for the same reason the downloader did and tell you nothing new. And it would pull the content of every audited file through one more process, which for an archive of exports and screenshots is a real cost. files.info reports the mimetype and size, your disk holds the bytes, and the comparison needs neither a transfer nor the token being questioned.
Everything downloaded a year ago is the login page. Can I re-fetch it?
Usually yes, if the files still exist - fix the header, delete the local copies, and run the archiver again over the same ids. The part to check first is how many of those ids still resolve, because a year is long enough for deletions and retention to have removed some of them, and a re-fetch will report those separately. Whatever you do, add the Content-Type and length validation at write time before the second run, so the retry cannot fail the same way.
Related field notes
- the file URL that works for everyone instead of nobody
- the same 200-means-nothing trap on the Web API
- how many of those ids still resolve a year later
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- files.info method reference - Slack Docs
- Working with files - Slack Docs
- Stack Overflow: downloading a Slack file returns the sign-in page
- Stack Overflow: url_private, permalink and permalink_public
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.