Diagnostic Slack
next_cursor is ignored so only the first page is ever seen
The channel inventory has exactly 100 entries. So does the user directory. Nobody questions either number until somebody reports that a channel which plainly exists is missing from the report — and by then the sync has been dropping four fifths of the workspace every night for a year, with ok: true on every single response.
Every Slack list method returns a page, defaulting to 100 items, with the continuation token in response_metadata.next_cursor. When that string is non-empty there is more data. Nothing else in the response says so.
Check for the shape: a first page of exactly your limit with a non-empty next_cursor is a truncation bug with near-certainty. Then paginate fully and compare the totals — the delta is exactly the data the application has never seen.
The problem in plain words
This is silent data loss dressed as a healthy job. There is no error, no warning, no ok: false, no exception. The call succeeded; it simply answered a smaller question than the one that was asked. Everything downstream then behaves perfectly on 100 items out of 412.
What makes it survive review is that it looks right and, at first, is right. A workspace with 60 channels returns 60 and no cursor, so the code that reads response.channels and stops is correct on the day it is written and stays correct through every test. It breaks on the day the workspace crosses 100 channels, which is nobody's deploy and nobody's incident.
The consequences are unevenly distributed too. A user sync that misses a fifth of the directory silently fails to alert a fifth of the company; a channel audit that stops at 100 declares the rest of the workspace compliant without looking at it.
Why it happens
The page size is a default, not a total. conversations.list, users.list, conversations.members, conversations.history, files.list and users.conversations all return 100 items unless you say otherwise, and all of them carry the continuation token in the same place.
Only the cursor is authoritative. Slack explicitly does not guarantee a full page: a response can come back with fewer items than the limit and still have more pages behind it. Stopping when a page looks short is a heuristic that fails silently, and it fails in exactly the direction that loses data.
Raising the limit is not a fix. limit=1000 moves the cliff, it does not remove it — and asking for more than the documented maximum returns invalid_limit, which is at least loud. A workspace grows past whatever number you pick.
Nothing about it looks like an error. ok is true, so error handling never triggers; this is the same structural fact behind every other note in this section, seen from the one angle where the body is not lying to you but merely incomplete.
The fix, as a flow
The script probes with the page size your application uses, then walks the whole list once. The delta between those two numbers is the finding, because a cursor on its own persuades nobody.
How to fix it
Call the list method the way your application calls it
Same method, same limit, same filters. The finding you want is about the code in production, so a probe with a different page size answers a different question.
Read response_metadata.next_cursor, not the array length
A non-empty cursor means there is more, full page or not. Treat an absent response_metadata, a null cursor and an empty string as the same thing: the end. Anything else is a page boundary you have not crossed yet.
Flag a full first page with a cursor as a bug, not a maybe
Exactly limit items plus a cursor is the signature. It is possible for that to be a coincidence, and it almost never is: it is what every truncated read looks like from the outside.
Walk the whole thing once and measure the gap
Paginate to the end with limit=200 and count. The difference between that total and the first page is the number of channels, users or messages the application has never seen, and it is the only number that makes anyone fix this today rather than next quarter.
Replace the read with a loop, or with the SDK iterator
while (cursor) { ... cursor = r.response_metadata?.next_cursor || null; }. Both official SDKs ship this: for await (const page of client.paginate('conversations.list', {...})) in Node, for page in client.conversations_list(limit=200) in Python. Bound the loop by pages as well, so a cursor bug cannot spin forever.
How to check it worked
Re-run after the loop is in place. The full walk and the application's own count should agree, and every probed method should report complete.
python3 slack_pagination_audit.py --full
# 3 list method(s) probed, 0 truncated by a first-page-only readThe full code
Paginated GETs and nothing else, with the page walk bounded by both a page count and a hard item cap so a large workspace cannot turn an audit into a rate-limit incident. Two pure functions: one that reads the cursor defensively, because it is absent, null and empty in three different situations that all mean the same thing, and one that separates the four shapes a first page can have.
"""Report Slack list calls whose first page is not the whole answer.
Read only. GET requests and nothing else: give this a bot token with read
scopes. The repair is printed, never performed, because this token can post into
your workspace.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_pagination_audit")
API = "https://slack.com/api/"
# (method, params, key holding the items). Every one of these is cursor
# paginated and every one of them defaults to 100 items per page.
PAGED = [
("conversations.list", {"types": "public_channel,private_channel"}, "channels"),
("users.list", {}, "members"),
("users.conversations", {"types": "public_channel,private_channel"}, "channels"),
]
def cursor_of(body):
"""The continuation token, or "" when this page is the last one. Pure.
Absent response_metadata, a null cursor and an empty string all mean the
same thing, and only one of the three is obvious.
"""
meta = body.get("response_metadata") or {}
return (meta.get("next_cursor") or "").strip()
def verdict(count, limit, cursor, total=None):
"""Classify one first page. Pure, so it runs offline.
`count` is the length of the first page, `limit` the page size the
application asked for, `cursor` the value cursor_of() returned, and `total`
the size of the full walk when one was performed.
"""
delta = ""
if total is not None:
delta = (" Full walk: %d item(s), so a first-page-only read misses %d."
% (total, max(total - count, 0)))
if cursor:
if count >= limit:
return ("truncated",
"a full page of %d with a cursor set. The application is "
"seeing %d of a larger number it never asked for.%s"
% (count, count, delta))
return ("more-pages",
"only %d item(s) but the cursor is set, so more pages follow. A "
"short page is not the last page.%s" % (count, delta))
if count >= limit:
return ("complete-at-limit",
"exactly %d item(s) and no cursor: complete today. Code that "
"stops on a short page is right here by luck, and wrong on the "
"next item added.%s" % (count, delta))
return ("complete", "%d item(s), no cursor: this is the whole set.%s"
% (count, delta))
def get(session, method, params):
r = session.get(API + method, params=params, timeout=30)
return r.json()
def walk(session, method, params, key, max_pages, max_items):
"""Follow every cursor to the end, bounded twice."""
total, cursor, pages = 0, "", 0
while True:
page = dict(params, limit="200")
if cursor:
page["cursor"] = cursor
body = get(session, method, page)
if body.get("ok") is not True:
log.warning(" walk stopped: ok: false, error=%s", body.get("error"))
return total
total += len(body.get(key) or [])
pages += 1
cursor = cursor_of(body)
if not cursor or pages >= max_pages or total >= max_items:
return total
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--limit", type=int, default=100,
help="the page size your application asks for (default 100)")
ap.add_argument("--full", action="store_true",
help="follow every cursor and report how much is being missed")
ap.add_argument("--max-pages", type=int, default=50, help="cap on the full walk")
ap.add_argument("--max-items", type=int, default=10000, help="cap on the full walk")
args = ap.parse_args()
token = os.environ.get("SLACK_BOT_TOKEN")
if not token:
log.error("set SLACK_BOT_TOKEN (a bot token with read scopes is enough)")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + token})
bad = 0
for method, params, key in PAGED:
body = get(s, method, dict(params, limit=str(args.limit)))
if body.get("ok") is not True:
log.warning("%-18s %-22s ok: false, error=%s", "unreadable", method,
body.get("error"))
bad += 1
continue
count = len(body.get(key) or [])
cursor = cursor_of(body)
total = walk(s, method, params, key, args.max_pages, args.max_items) \
if (args.full and cursor) else None
state, detail = verdict(count, args.limit, cursor, total)
line = "%-18s %-22s %s" % (state, method, detail)
if state.startswith("complete"):
log.info(line)
continue
bad += 1
log.warning(line)
log.warning(" repair: loop on response_metadata.next_cursor until it is "
"empty, or use the SDK paginator")
log.info("%d list method(s) probed, %d truncated by a first-page-only read",
len(PAGED), bad)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report Slack list calls whose first page is not the whole answer.
*
* Read only. GET requests and nothing else: give this a bot token with read
* scopes. The repair is printed, never performed.
*/
const API = 'https://slack.com/api/';
// [method, params, key holding the items]. Every one of these is cursor
// paginated and every one of them defaults to 100 items per page.
const PAGED = [
['conversations.list', { types: 'public_channel,private_channel' }, 'channels'],
['users.list', {}, 'members'],
['users.conversations', { types: 'public_channel,private_channel' }, 'channels'],
];
/**
* The continuation token, or '' when this page is the last one. Pure.
* Absent response_metadata, a null cursor and an empty string all mean the same
* thing, and only one of the three is obvious.
*/
export function cursorOf(body) {
return (body.response_metadata?.next_cursor ?? '').trim();
}
/**
* Classify one first page. Pure, so it runs offline.
*/
export function verdict(count, limit, cursor, total = null) {
const delta = total === null ? ''
: ` Full walk: ${total} item(s), so a first-page-only read misses ` +
`${Math.max(total - count, 0)}.`;
if (cursor) {
if (count >= limit) {
return ['truncated',
`a full page of ${count} with a cursor set. The application is seeing ` +
`${count} of a larger number it never asked for.${delta}`];
}
return ['more-pages',
`only ${count} item(s) but the cursor is set, so more pages follow. A short ` +
`page is not the last page.${delta}`];
}
if (count >= limit) {
return ['complete-at-limit',
`exactly ${count} item(s) and no cursor: complete today. Code that stops on ` +
`a short page is right here by luck, and wrong on the next item added.${delta}`];
}
return ['complete', `${count} item(s), no cursor: this is the whole set.${delta}`];
}
async function get(token, method, params) {
const url = new URL(API + method);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${token}` } });
return res.json();
}
async function walk(token, method, params, key, maxPages, maxItems) {
let total = 0; let cursor = ''; let pages = 0;
for (;;) {
const page = { ...params, limit: '200' };
if (cursor) page.cursor = cursor;
const body = await get(token, method, page);
if (body.ok !== true) {
console.warn(` walk stopped: ok: false, error=${body.error}`);
return total;
}
total += (body[key] ?? []).length;
pages += 1;
cursor = cursorOf(body);
if (!cursor || pages >= maxPages || total >= maxItems) return total;
}
}
async function main() {
const token = process.env.SLACK_BOT_TOKEN;
if (!token) {
console.error('set SLACK_BOT_TOKEN (a bot token with read scopes is enough)');
process.exitCode = 2;
return;
}
const argv = process.argv.slice(2);
const limit = Number(argv.find((a) => a.startsWith('--limit='))?.split('=')[1] ?? 100);
const full = argv.includes('--full');
const maxPages = 50;
const maxItems = 10000;
let bad = 0;
for (const [method, params, key] of PAGED) {
const body = await get(token, method, { ...params, limit: String(limit) });
if (body.ok !== true) {
console.warn(`unreadable ${method.padEnd(22)} ok: false, error=${body.error}`);
bad += 1;
continue;
}
const count = (body[key] ?? []).length;
const cursor = cursorOf(body);
const total = (full && cursor)
? await walk(token, method, params, key, maxPages, maxItems) : null;
const [state, detail] = verdict(count, limit, cursor, total);
const line = `${state.padEnd(18)} ${method.padEnd(22)} ${detail}`;
if (state.startsWith('complete')) { console.log(line); continue; }
bad += 1;
console.warn(line);
console.warn(' repair: loop on response_metadata.next_cursor until it is empty, ' +
'or use the SDK paginator');
}
console.log(`${PAGED.length} list method(s) probed, ${bad} truncated by a ` +
'first-page-only read');
process.exitCode = bad ? 1 : 0;
}
// Only run when invoked directly, so importing this module from the test file
// does not execute main() and fail the suite on a missing token.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The two states worth keeping apart are the ones that look identical to a length check. A short page with a cursor still has pages behind it, and a full page with no cursor really is the end — which is why stopping on a short page is not a safe shortcut, and why being right about it once is not evidence of anything.
from slack_pagination_audit import cursor_of, verdict
def test_absent_response_metadata_means_the_end():
assert cursor_of({"ok": True, "channels": []}) == ""
def test_null_and_empty_cursors_mean_the_end_too():
assert cursor_of({"response_metadata": {"next_cursor": None}}) == ""
assert cursor_of({"response_metadata": {"next_cursor": " "}}) == ""
def test_a_real_cursor_survives():
assert cursor_of({"response_metadata": {"next_cursor": "dGVhbTpDMDYx"}}) == "dGVhbTpDMDYx"
def test_full_page_with_a_cursor_is_the_truncation_signature():
state, detail = verdict(100, 100, "dGVhbTpD")
assert state == "truncated"
assert "100" in detail
def test_short_page_with_a_cursor_still_has_more():
state, detail = verdict(37, 100, "dGVhbTpD")
assert state == "more-pages"
assert "not the last page" in detail
def test_full_page_without_a_cursor_is_complete_but_not_reassuring():
state, detail = verdict(100, 100, "")
assert state == "complete-at-limit"
assert "luck" in detail
def test_short_page_without_a_cursor_is_the_whole_set():
state, _ = verdict(42, 100, "")
assert state == "complete"
def test_the_full_walk_reports_what_is_being_missed():
_, detail = verdict(100, 100, "dGVhbTpD", total=412)
assert "412" in detail
assert "misses 312" in detail
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cursorOf, verdict } from './slack-pagination-audit.mjs';
test('absent response_metadata means the end', () => {
assert.equal(cursorOf({ ok: true, channels: [] }), '');
});
test('null and empty cursors mean the end too', () => {
assert.equal(cursorOf({ response_metadata: { next_cursor: null } }), '');
assert.equal(cursorOf({ response_metadata: { next_cursor: ' ' } }), '');
});
test('a real cursor survives', () => {
assert.equal(cursorOf({ response_metadata: { next_cursor: 'dGVhbTpDMDYx' } }),
'dGVhbTpDMDYx');
});
test('full page with a cursor is the truncation signature', () => {
const [state, detail] = verdict(100, 100, 'dGVhbTpD');
assert.equal(state, 'truncated');
assert.match(detail, /100/);
});
test('short page with a cursor still has more', () => {
const [state, detail] = verdict(37, 100, 'dGVhbTpD');
assert.equal(state, 'more-pages');
assert.match(detail, /not the last page/);
});
test('full page without a cursor is complete but not reassuring', () => {
const [state, detail] = verdict(100, 100, '');
assert.equal(state, 'complete-at-limit');
assert.match(detail, /luck/);
});
test('short page without a cursor is the whole set', () => {
assert.equal(verdict(42, 100, '')[0], 'complete');
});
test('the full walk reports what is being missed', () => {
const [, detail] = verdict(100, 100, 'dGVhbTpD', 412);
assert.match(detail, /412/);
assert.match(detail, /misses 312/);
});
FAQ
Why does conversations.list only return 100 channels?
Because 100 is the default page size, not the total. The rest is behind response_metadata.next_cursor, and Slack returns ok: true either way, so nothing about the response signals that you have seen a fraction of the workspace.
Can I just set limit=1000 and skip pagination?
No. A larger limit moves the boundary rather than removing it, Slack recommends staying well below the maximum for reliability, and asking for more than the documented ceiling returns invalid_limit. The workspace will eventually be larger than whatever number you chose.
Is a page with fewer items than the limit always the last page?
No, and this is the trap. Slack does not guarantee a full page, so a response can hold 37 items and still have a cursor. The cursor is the only authoritative signal; page length is a heuristic that fails quietly in the direction of losing data.
Which methods are cursor paginated?
conversations.list, users.list, conversations.members, conversations.history, conversations.replies, users.conversations and files.list among others. They all carry the token in the same place, so one loop written once handles every one of them.
How do I know how much data I have been missing?
Paginate to the end once and compare the total against the first page. That delta is the number of channels or users the application has never seen, and it is far more persuasive in a bug report than the observation that a cursor exists.
Related field notes
- Slack answers 200 and hides the failure in the body
- missing_scope names the scope you need
- not_in_channel: the bot was never invited
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Pagination in the Web API — Slack Docs
- conversations.list method reference — Slack Docs
- users.list method reference — Slack Docs
- Using the Slack Web API — Slack Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.