Diagnostic GitHub API
the endpoint ignores page and returns page one forever
The collector increments a page counter, exactly as it does against a dozen other endpoints, and every request comes back 200 with a full page of rows. It never finishes. Either it runs until something kills it, or it stops at an arbitrary page cap and hands over a dataset that is the same thirty records repeated forty times. The endpoint has been returning page one to every request since the first one.
A minority of GitHub list endpoints do not support offset pagination. They page with before and after cursors instead, or they do not page at all, and they ignore page and per_page rather than rejecting them. An unsupported parameter is not an error here; it is silence.
Two signals identify it, and you want both. Fetch page=1 and page=2 and compare the identifiers on each: identical rows are the symptom. Then read the endpoint's own Link header and look at what the next URL is built from — if it carries after= or before= rather than page=, the endpoint has told you plainly what it supports. The first signal alone can be a busy feed that reordered between two calls; the second does not depend on timing. The repair is to use the cursor parameters the endpoint's own next link uses, or the GraphQL equivalent with after: $cursor and pageInfo { hasNextPage endCursor }. Never synthesise page numbers.
The problem in plain words
Every other pagination bug on this API is an under-count. This one is an over-count, and the difference matters because of where the damage lands. A job that reads too little produces a small number in a report. A job that reads the same page forever produces rows — thousands of them, all duplicates of thirty — and then writes them somewhere: a warehouse table, a search index, a spreadsheet somebody is about to make a decision from.
It also does not stop. The usual terminating condition for a hand-rolled pager is a short page, and there is never a short page, because page forty is page one and page one is full. So the loop runs until a timeout, an out-of-memory kill or a page cap somebody added for exactly this reason without ever finding out why it was needed. In the meantime it is spending quota at full speed on a single page of data.
The investigation goes to the wrong layer almost every time. Duplicate rows read as an upsert bug, so the first day goes into deduplication keys and idempotency. The fix — dedupe on ingest — even works, in the sense that the duplicates stop appearing. What remains is a job that spends forty requests to collect thirty records and reports the first page as the entire dataset, which is a data-loss bug now wearing the costume of a resolved ticket.
Why it happens
An unsupported query parameter is ignored, not rejected. Sending page=2 to an endpoint that does not implement offset paging produces a perfectly ordinary 200. There is no 422, no Warning, and nothing in the body to say the parameter was dropped. This is consistent behaviour across the API and it is the reason the bug is silent: the client's only evidence that its parameter did anything is that the results changed, and they did not.
Cursor endpoints exist and are growing. Repository activity and several newer list endpoints page with before and after cursors, because an offset into a live feed is not a stable thing to hand out. Those endpoints advertise their cursors in the Link header exactly as the offset ones advertise page numbers, which means the header is a reliable, timing-independent statement of what the endpoint supports.
Identical rows on two pages are evidence, not proof. A feed sorted by recency can genuinely return overlapping rows across two requests made a second apart, because items moved. That is why this check reads the Link header as well: identical ids plus a cursor-shaped next link is conclusive, identical ids plus a page-shaped next link is a re-run and a closer look. A diagnostic that reported the first case and the second case identically would be wrong often enough to be ignored, which is worse than not existing.
The absence of a short page removes the terminating condition. Hand-rolled pagers usually stop when a page comes back shorter than the page size. Against an endpoint that ignores page, every page is the first page and every page is full, so that condition is never met. This is the one pagination bug in the section where the loop does not merely stop in the wrong place; it does not stop.
The repair is to stop generating URLs. The next URL in the Link header already contains whatever the endpoint needs, cursor or page. Following it verbatim is correct on both kinds of endpoint and needs no knowledge of which kind you are on, which is why it is the pattern worth standardising: it makes the entire class of bug unreachable rather than fixing one instance of it.
The check costs almost nothing and the script says so. Two requests per path at per_page=1, four by default. The irony of a diagnostic for a runaway loop being itself expensive is not lost, so the number is computed and printed before anything is fetched.
The fix, as a flow
Two GETs per probed path, both at the smallest page size, and the comparison is between the identifiers on page one and the identifiers on page two. Identical ids are suggestive rather than conclusive, because a feed sorted by recency can move between two requests, so the script reads a second and independent signal: the query parameters on the endpoint's own next link. An endpoint whose next link carries a cursor instead of a page number has told you what it supports, and that answer does not depend on timing at all.
How to fix it
Fetch page one and page two at the smallest page size
per_page=1 on both, so the comparison is between two single rows and the whole check costs two requests per path. The small page size is not cosmetic: it also makes the two responses arrive close together, which narrows the window in which a live feed could legitimately reorder underneath you.
Compare identifiers rather than payloads
Take id, falling back to node_id, sha or url, and compare the lists. Comparing whole objects produces false negatives, because timestamps and counters inside an item change between two requests while the item itself is the same row. An identifier is the only field that means what you want it to mean here.
Read what the endpoint's own next link is built from
Parse the Link header and look at the query parameters on the next URL. page= means offset paging is what this endpoint speaks. after= or before= means it speaks cursors and your page numbers were never being read. This signal does not depend on timing at all, which is what makes it the one that settles the question.
Keep the ambiguous case ambiguous
Identical rows with a page-shaped next link is a busy feed, a re-run, or a genuinely broken endpoint, in that order of likelihood. Reporting that as a definite finding is how a check gets switched off. The script gives it its own state, says what would settle it, and does not overclaim.
Follow the next URL instead of building one
Take the next URL from the header verbatim, cursors and all, and stop when there is no next. That loop is correct on offset endpoints and cursor endpoints alike without knowing which it is on. Where you need the GraphQL equivalent, it is after: $cursor with pageInfo { hasNextPage endCursor }, and the same rule holds: the cursor is opaque and is not to be constructed.
How to check it worked
After the collector follows the header instead of counting, the same probe reports the endpoint as a cursor endpoint that pages correctly, and the row count stops being a multiple of the page size.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_page_param_ignored.py --repo acme/monorepo
# read cost: 4 request(s) against the core hourly quota
# /repos/acme/monorepo/activity: page=1 and page=2 returned the same id(s)
# ignores-page: page=2 returned the same row(s) as page=1 and the next link is
# built from after=, so this endpoint does not read page at all. A loop that
# stops on a short page has no terminating condition here
# repair: follow the next URL from the Link header verbatim, using after=
# after the collector is changed
# cursor-pagination: the next link is built from after= and the rows differ,
# so this endpoint pages correctly and simply not by numberThe full code
Two GETs per probed path and one comparison, but the comparison is deliberately built from two independent signals because either one alone produces a diagnostic somebody would learn to ignore. Identifiers are extracted defensively, since not every list on this API keys its items the same way, and the endpoint's own next link is inspected for the parameter names it carries. Their agreement is the definitive state; their disagreement gets its own honest, weaker verdict. There is also a pure predicate that answers the question people actually care about, which is whether a page-counting loop against this endpoint would ever terminate.
"""Find GitHub endpoints that ignore the page parameter instead of rejecting it.
Read only. Two GETs per probed path at per_page=1, and nothing is written.
A minority of endpoints do not implement offset pagination. They page with
before/after cursors, or they do not page at all, and they ignore page and
per_page rather than answering 422. So page=2 returns page one, a hand-rolled
loop that stops on a short page never stops, and the same rows are collected
until something kills the job.
Two independent signals are used, because either alone is unsafe. Identical
identifiers across page 1 and page 2 is the symptom, but a feed sorted by
recency can genuinely move between two requests. The parameter names on the
endpoint's own next link do not depend on timing at all: a next URL built from
after= or before= is the endpoint saying which kind of pagination it speaks.
Environment:
GITHUB_TOKEN a token with read access to the repository
"""
import argparse
import json
import logging
import os
import re
import sys
from urllib.parse import parse_qs, urlparse
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_page_param_ignored")
API = "https://api.github.com"
UA = "github-page-param-ignored/1.0"
LINK = re.compile(r'<([^>]+)>\s*;\s*rel="([^"]+)"')
# The parameter names that mean each kind of pagination. Cursor names first
# because an endpoint that offers both is a cursor endpoint being polite.
CURSOR_PARAMS = ("after", "before", "cursor")
OFFSET_PARAMS = ("page",)
# Tried in order. Not every list on this API keys its items the same way, and
# comparing whole objects is useless because counters inside an item change
# between two requests while the item stays the same row.
ID_FIELDS = ("id", "node_id", "sha", "url")
PROBES = ["activity", "events"]
def parse_link(header):
"""Parse a Link header into {rel: url}. Pure."""
if not header:
return {}
return {rel: url for url, rel in LINK.findall(header)}
def link_params(links):
"""The query parameter names on the next URL, sorted. Pure."""
url = (links or {}).get("next")
if not url:
return []
return sorted(parse_qs(urlparse(url).query))
def link_style(links):
"""What the endpoint's own next link is built from. Pure.
Returns cursor, offset or none. This is the signal that does not depend on
timing, which is why the definitive verdict requires it.
"""
names = set(link_params(links))
if names & set(CURSOR_PARAMS):
return "cursor"
if names & set(OFFSET_PARAMS):
return "offset"
return "none"
def cursor_hint(links):
"""The cursor parameter this endpoint actually uses, or None. Pure."""
names = set(link_params(links))
for name in CURSOR_PARAMS:
if name in names:
return name
return None
def identity(item):
"""A stable identifier for one list item, or None. Pure."""
if not isinstance(item, dict):
return None
for field in ID_FIELDS:
value = item.get(field)
if value not in (None, ""):
return str(value)
return None
def identities(items):
"""Identifiers for a page, dropping items that have none. Pure."""
if not isinstance(items, list):
return []
return [i for i in (identity(x) for x in items) if i is not None]
def same_rows(first, second):
"""Whether page two is page one, exactly. Pure."""
return bool(first) and bool(second) and list(first) == list(second)
def overlaps(first, second):
"""Whether the two pages share any row at all. Pure."""
return bool(set(first or []) & set(second or []))
def verdict(style, first_ids, second_ids):
"""Classify one endpoint from both signals. Pure. Returns (state, detail).
The two conclusive states require the signals to agree. Where they do not,
the weaker verdict is returned deliberately: a check that cries wolf on a
busy feed is a check somebody turns off.
"""
if not first_ids:
return ("inconclusive-empty",
"page 1 returned nothing this check could identify, so there is "
"no comparison to make. Point it at a path with rows in it.")
if not second_ids:
return ("offset-honoured",
"page 2 came back empty, so the collection ends inside page 1 "
"and the page parameter is being read.")
if same_rows(first_ids, second_ids):
if style in ("cursor", "none"):
shape = ("built from a cursor" if style == "cursor"
else "absent, so there is no next page to follow")
return ("ignores-page",
"page=2 returned the same row(s) as page=1 and the next "
"link is %s, so this endpoint does not read page at all. A "
"loop that stops on a short page has no terminating "
"condition here." % shape)
return ("suspect-ignores-page",
"page=2 returned the same row(s) as page=1, but the next link "
"is still built from page=, so this may be a feed that moved "
"between the two requests. Re-run it, or add a stable sort, "
"before treating it as a finding.")
if overlaps(first_ids, second_ids):
return ("overlapping-pages",
"page 1 and page 2 share rows without being identical, which is "
"an unstable sort rather than an ignored parameter. Paging this "
"endpoint will double-count and skip.")
if style == "cursor":
return ("cursor-pagination",
"the rows differ and the next link is built from a cursor, so "
"this endpoint pages correctly and simply not by number. Follow "
"its next URL rather than incrementing anything.")
return ("offset-honoured",
"page 2 returned different rows and the next link is built from "
"page=, so offset pagination works here.")
def loop_terminates(state):
"""Whether a page-counting loop against this endpoint would ever end. Pure."""
return state not in ("ignores-page", "suspect-ignores-page")
def repair(state, links=None):
"""The sentence a reader has to act on. Pure."""
if state == "ignores-page":
cursor = cursor_hint(links)
if cursor:
return ("follow the next URL from the Link header verbatim, using "
"%s=. Do not construct it, and do not send page: the value "
"is opaque and incrementing anything here is meaningless."
% cursor)
return ("stop paging this endpoint by number. It advertises no next "
"page, so one request is what it offers, and the GraphQL "
"equivalent with after: $cursor is the way to walk more.")
if state == "suspect-ignores-page":
return ("re-run the check, or add a deterministic sort, before changing "
"any code. Identical rows on a recency-ordered feed can be two "
"requests a second apart rather than an ignored parameter.")
if state == "overlapping-pages":
return ("sort deterministically before paging, or switch to the cursor "
"form. Offset paging over a feed that reorders will double-count "
"some rows and miss others whatever the page size.")
if state == "cursor-pagination":
return ("follow the next URL from the Link header verbatim. It already "
"carries the cursor, and building it yourself is the only way "
"to get this wrong.")
if state == "inconclusive-empty":
return "point the check at a path that has rows in it."
return "nothing."
def read_cost(paths):
"""Requests this run will spend against the core quota. Pure."""
return 2 * len(paths or [])
def get(session, path, params):
"""One GET. Returns (status, items-or-None, links)."""
r = session.get(API + path, params=params, timeout=30)
links = parse_link(r.headers.get("Link"))
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
if r.status_code == 403 and "rate limit" in r.text.lower():
raise SystemExit("403 rate limited. GET /rate_limit reports the reset "
"time and does not itself consume quota")
if r.status_code != 200:
return r.status_code, None, links
try:
body = r.json()
except ValueError:
return r.status_code, None, links
return r.status_code, body if isinstance(body, list) else None, links
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--path", action="append",
help="probe this API path instead of the defaults. Repeatable.")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
paths = args.path or ["/repos/%s/%s" % (args.repo, name) for name in PROBES]
log.info("read cost: %d request(s) against the core hourly quota",
read_cost(paths))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
findings = []
for path in paths:
status1, first, links = get(session, path, {"per_page": 1, "page": 1})
if first is None:
log.info("%s returned %d; skipping it", path, status1)
continue
status2, second, _links2 = get(session, path, {"per_page": 1, "page": 2})
first_ids, second_ids = identities(first), identities(second)
style = link_style(links)
state, detail = verdict(style, first_ids, second_ids)
if same_rows(first_ids, second_ids):
log.info("%s: page=1 and page=2 returned the same id(s)", path)
log.info("%s: %s", state, detail)
log.info("a page-counting loop here %s",
"terminates" if loop_terminates(state) else "never terminates")
log.info("repair: %s", repair(state, links))
findings.append({
"path": path,
"status": [status1, status2],
"next_link_params": link_params(links),
"link_style": style,
"cursor_parameter": cursor_hint(links),
"page_1_ids": first_ids,
"page_2_ids": second_ids,
"identical": same_rows(first_ids, second_ids),
"loop_terminates": loop_terminates(state),
"state": state,
"detail": detail,
})
print(json.dumps({"requests_spent": read_cost(paths),
"findings": findings}, indent=2, default=str))
bad = {"ignores-page", "suspect-ignores-page", "overlapping-pages"}
return 1 if any(f["state"] in bad for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find GitHub endpoints that ignore the page parameter instead of rejecting it.
*
* Read only. Two GETs per probed path at per_page=1, and nothing is written.
*
* Two independent signals are used, because either alone is unsafe: identical
* identifiers across page 1 and page 2, and the parameter names on the
* endpoint's own next link. The second does not depend on timing.
*
* Environment:
* GITHUB_TOKEN a token with read access to the repository
* GITHUB_REPO owner/name
*/
const API = 'https://api.github.com';
const UA = 'github-page-param-ignored/1.0';
const LINK = /<([^>]+)>\s*;\s*rel="([^"]+)"/g;
/** Cursor names first: an endpoint offering both is a cursor endpoint. */
export const CURSOR_PARAMS = ['after', 'before', 'cursor'];
export const OFFSET_PARAMS = ['page'];
/** Tried in order. Not every list on this API keys its items the same way. */
export const ID_FIELDS = ['id', 'node_id', 'sha', 'url'];
const PROBES = ['activity', 'events'];
/** Parse a Link header into {rel: url}. Pure. */
export function parseLink(header) {
const out = {};
if (!header) return out;
for (const m of String(header).matchAll(LINK)) out[m[2]] = m[1];
return out;
}
/** The query parameter names on the next URL, sorted. Pure. */
export function linkParams(links) {
const url = (links || {}).next;
if (!url) return [];
try {
return [...new Set([...new URL(url, API).searchParams.keys()])].sort();
} catch {
return [];
}
}
/** What the endpoint's own next link is built from: cursor, offset or none. */
export function linkStyle(links) {
const names = new Set(linkParams(links));
if (CURSOR_PARAMS.some((n) => names.has(n))) return 'cursor';
if (OFFSET_PARAMS.some((n) => names.has(n))) return 'offset';
return 'none';
}
/** The cursor parameter this endpoint actually uses, or null. Pure. */
export function cursorHint(links) {
const names = new Set(linkParams(links));
return CURSOR_PARAMS.find((n) => names.has(n)) ?? null;
}
/** A stable identifier for one list item, or null. Pure. */
export function identity(item) {
if (!item || typeof item !== 'object' || Array.isArray(item)) return null;
for (const field of ID_FIELDS) {
const value = item[field];
if (value !== null && value !== undefined && value !== '') return String(value);
}
return null;
}
/** Identifiers for a page, dropping items that have none. Pure. */
export function identities(items) {
if (!Array.isArray(items)) return [];
return items.map(identity).filter((v) => v !== null);
}
/** Whether page two is page one, exactly. Pure. */
export function sameRows(first, second) {
if (!first || !second || !first.length || !second.length) return false;
return first.length === second.length && first.every((v, i) => v === second[i]);
}
/** Whether the two pages share any row at all. Pure. */
export function overlaps(first, second) {
const set = new Set(first || []);
return (second || []).some((v) => set.has(v));
}
/** Classify one endpoint from both signals. Pure. Returns [state, detail]. */
export function verdict(style, firstIds, secondIds) {
if (!firstIds || !firstIds.length) {
return ['inconclusive-empty',
'page 1 returned nothing this check could identify, so there is no '
+ 'comparison to make. Point it at a path with rows in it.'];
}
if (!secondIds || !secondIds.length) {
return ['offset-honoured',
'page 2 came back empty, so the collection ends inside page 1 and the '
+ 'page parameter is being read.'];
}
if (sameRows(firstIds, secondIds)) {
if (style === 'cursor' || style === 'none') {
const shape = style === 'cursor'
? 'built from a cursor'
: 'absent, so there is no next page to follow';
return ['ignores-page',
`page=2 returned the same row(s) as page=1 and the next link is ${shape}, `
+ 'so this endpoint does not read page at all. A loop that stops on a '
+ 'short page has no terminating condition here.'];
}
return ['suspect-ignores-page',
'page=2 returned the same row(s) as page=1, but the next link is still '
+ 'built from page=, so this may be a feed that moved between the two '
+ 'requests. Re-run it, or add a stable sort, before treating it as a finding.'];
}
if (overlaps(firstIds, secondIds)) {
return ['overlapping-pages',
'page 1 and page 2 share rows without being identical, which is an '
+ 'unstable sort rather than an ignored parameter. Paging this endpoint '
+ 'will double-count and skip.'];
}
if (style === 'cursor') {
return ['cursor-pagination',
'the rows differ and the next link is built from a cursor, so this '
+ 'endpoint pages correctly and simply not by number. Follow its next URL '
+ 'rather than incrementing anything.'];
}
return ['offset-honoured',
'page 2 returned different rows and the next link is built from page=, so '
+ 'offset pagination works here.'];
}
/** Whether a page-counting loop against this endpoint would ever end. Pure. */
export function loopTerminates(state) {
return !['ignores-page', 'suspect-ignores-page'].includes(state);
}
/** The sentence a reader has to act on. Pure. */
export function repair(state, links = null) {
if (state === 'ignores-page') {
const cursor = cursorHint(links);
if (cursor) {
return 'follow the next URL from the Link header verbatim, using '
+ `${cursor}=. Do not construct it, and do not send page: the value is `
+ 'opaque and incrementing anything here is meaningless.';
}
return 'stop paging this endpoint by number. It advertises no next page, so '
+ 'one request is what it offers, and the GraphQL equivalent with '
+ 'after: $cursor is the way to walk more.';
}
if (state === 'suspect-ignores-page') {
return 're-run the check, or add a deterministic sort, before changing any '
+ 'code. Identical rows on a recency-ordered feed can be two requests a '
+ 'second apart rather than an ignored parameter.';
}
if (state === 'overlapping-pages') {
return 'sort deterministically before paging, or switch to the cursor form. '
+ 'Offset paging over a feed that reorders will double-count some rows and '
+ 'miss others whatever the page size.';
}
if (state === 'cursor-pagination') {
return 'follow the next URL from the Link header verbatim. It already '
+ 'carries the cursor, and building it yourself is the only way to get '
+ 'this wrong.';
}
if (state === 'inconclusive-empty') return 'point the check at a path that has rows in it.';
return 'nothing.';
}
/** Requests this run will spend against the core quota. Pure. */
export function readCost(paths) {
return 2 * (Array.isArray(paths) ? paths.length : 0);
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
};
}
async function get(token, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
const res = await fetch(url, { headers: headers(token) });
const links = parseLink(res.headers.get('link'));
if (res.status !== 200) return { status: res.status, items: null, links };
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, items: Array.isArray(body) ? body : null, links };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const repo = process.env.GITHUB_REPO;
if (!token || !repo) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_REPO=owner/name');
process.exitCode = 2;
return;
}
const paths = PROBES.map((name) => `/repos/${repo}/${name}`);
console.log(`read cost: ${readCost(paths)} request(s) against the core hourly quota`);
const findings = [];
for (const path of paths) {
const one = await get(token, path, { per_page: 1, page: 1 });
if (one.items === null) {
console.log(`${path} returned ${one.status}; skipping it`);
continue;
}
const two = await get(token, path, { per_page: 1, page: 2 });
const firstIds = identities(one.items);
const secondIds = identities(two.items);
const style = linkStyle(one.links);
const [state, detail] = verdict(style, firstIds, secondIds);
if (sameRows(firstIds, secondIds)) {
console.log(`${path}: page=1 and page=2 returned the same id(s)`);
}
console.log(`${state}: ${detail}`);
console.log(`a page-counting loop here ${loopTerminates(state) ? 'terminates' : 'never terminates'}`);
console.log(`repair: ${repair(state, one.links)}`);
findings.push({
path,
status: [one.status, two.status],
next_link_params: linkParams(one.links),
link_style: style,
cursor_parameter: cursorHint(one.links),
page_1_ids: firstIds,
page_2_ids: secondIds,
identical: sameRows(firstIds, secondIds),
loop_terminates: loopTerminates(state),
state,
detail,
});
}
console.log(JSON.stringify({ requests_spent: readCost(paths), findings }, null, 2));
const bad = ['ignores-page', 'suspect-ignores-page', 'overlapping-pages'];
process.exitCode = findings.some((f) => bad.includes(f.state)) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Most of these tests exist to stop the check overclaiming. Identical rows with a cursor-shaped next link must be the definite finding; identical rows with a page-shaped next link must be the softer one, because that is a busy feed as often as it is a bug, and a diagnostic that confuses the two gets switched off within a week. Beyond that: identifiers extracted from items that key themselves differently, partial overlap given its own state rather than being rounded to either neighbour, and the predicate that answers whether a page-counting loop here would ever terminate.
from github_page_param_ignored import (
cursor_hint, identities, identity, link_params, link_style, loop_terminates,
overlaps, parse_link, read_cost, repair, same_rows, verdict,
)
BASE = "https://api.github.com/repos/o/n"
OFFSET_LINK = {"next": BASE + "/issues?per_page=1&page=2"}
CURSOR_LINK = {"next": BASE + "/activity?per_page=1&after=Y3Vyc29yOjE="}
BEFORE_LINK = {"next": BASE + "/activity?per_page=1&before=Y3Vyc29yOjk="}
NO_LINK = {}
def test_the_link_style_is_read_from_the_parameter_names():
assert link_style(CURSOR_LINK) == "cursor"
assert link_style(BEFORE_LINK) == "cursor"
assert link_style(OFFSET_LINK) == "offset"
assert link_style(NO_LINK) == "none"
assert link_style(None) == "none"
def test_the_cursor_parameter_is_named_so_the_repair_can_be_concrete():
assert cursor_hint(CURSOR_LINK) == "after"
assert cursor_hint(BEFORE_LINK) == "before"
assert cursor_hint(OFFSET_LINK) is None
assert link_params(OFFSET_LINK) == ["page", "per_page"]
def test_identifiers_fall_back_through_the_fields_a_list_might_use():
assert identity({"id": 41, "node_id": "MDQ6"}) == "41"
assert identity({"node_id": "MDQ6"}) == "MDQ6"
assert identity({"sha": "9f2c1ab"}) == "9f2c1ab"
assert identity({"url": BASE + "/pulls/3"}) == BASE + "/pulls/3"
assert identity({"title": "no identifier here"}) is None
assert identity(None) is None
def test_a_page_of_unidentifiable_items_does_not_become_a_finding():
assert identities([{"title": "a"}, {"title": "b"}]) == []
assert identities([{"id": 1}, {"title": "b"}]) == ["1"]
assert identities("not a list") == []
def test_identical_rows_with_a_cursor_link_is_the_definite_finding():
state, detail = verdict("cursor", ["9"], ["9"])
assert state == "ignores-page"
assert "does not read page at all" in detail
assert "no terminating condition" in detail
assert not loop_terminates(state)
def test_identical_rows_with_a_page_link_is_only_a_suspicion():
state, detail = verdict("offset", ["9"], ["9"])
assert state == "suspect-ignores-page"
assert "may be a feed that moved" in detail
assert "Re-run it" in detail
def test_a_partial_overlap_is_its_own_answer():
state, detail = verdict("offset", ["9", "8"], ["8", "7"])
assert state == "overlapping-pages"
assert "unstable sort" in detail
assert loop_terminates(state)
def test_a_cursor_endpoint_that_pages_properly_is_not_a_finding():
state, detail = verdict("cursor", ["9"], ["8"])
assert state == "cursor-pagination"
assert "not by number" in detail
def test_offset_pagination_that_works_is_reported_as_working():
assert verdict("offset", ["9"], ["8"])[0] == "offset-honoured"
assert verdict("offset", ["9"], [])[0] == "offset-honoured"
def test_an_empty_first_page_proves_nothing():
state, detail = verdict("none", [], [])
assert state == "inconclusive-empty"
assert "no comparison to make" in detail
def test_the_row_comparisons_are_order_sensitive_and_set_based_in_turn():
assert same_rows(["1", "2"], ["1", "2"])
assert not same_rows(["1", "2"], ["2", "1"])
assert not same_rows([], [])
assert overlaps(["1", "2"], ["2", "3"])
assert not overlaps(["1"], ["2"])
def test_the_repair_names_the_cursor_the_endpoint_actually_uses():
assert "after=" in repair("ignores-page", CURSOR_LINK)
assert "before=" in repair("ignores-page", BEFORE_LINK)
assert "no next page" in repair("ignores-page", NO_LINK)
assert "Re-run" not in repair("cursor-pagination")
def test_the_repair_for_a_suspicion_changes_no_code():
fix = repair("suspect-ignores-page")
assert "re-run the check" in fix
assert "before changing any code" in fix
def test_the_header_is_parsed_around_commas_inside_urls():
header = '<%s/issues?labels=bug,ci&page=2>; rel="next"' % BASE
assert link_style(parse_link(header)) == "offset"
def test_the_run_says_what_it_will_spend():
assert read_cost(["/a", "/b"]) == 4
assert read_cost([]) == 0
assert read_cost(None) == 0
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
cursorHint, identities, identity, linkParams, linkStyle, loopTerminates,
overlaps, parseLink, readCost, repair, sameRows, verdict,
} from './github-page-param-ignored.mjs';
const BASE = 'https://api.github.com/repos/o/n';
const OFFSET_LINK = { next: `${BASE}/issues?per_page=1&page=2` };
const CURSOR_LINK = { next: `${BASE}/activity?per_page=1&after=Y3Vyc29yOjE=` };
const BEFORE_LINK = { next: `${BASE}/activity?per_page=1&before=Y3Vyc29yOjk=` };
const NO_LINK = {};
test('the link style is read from the parameter names', () => {
assert.equal(linkStyle(CURSOR_LINK), 'cursor');
assert.equal(linkStyle(BEFORE_LINK), 'cursor');
assert.equal(linkStyle(OFFSET_LINK), 'offset');
assert.equal(linkStyle(NO_LINK), 'none');
assert.equal(linkStyle(null), 'none');
});
test('the cursor parameter is named so the repair can be concrete', () => {
assert.equal(cursorHint(CURSOR_LINK), 'after');
assert.equal(cursorHint(BEFORE_LINK), 'before');
assert.equal(cursorHint(OFFSET_LINK), null);
assert.deepEqual(linkParams(OFFSET_LINK), ['page', 'per_page']);
});
test('identifiers fall back through the fields a list might use', () => {
assert.equal(identity({ id: 41, node_id: 'MDQ6' }), '41');
assert.equal(identity({ node_id: 'MDQ6' }), 'MDQ6');
assert.equal(identity({ sha: '9f2c1ab' }), '9f2c1ab');
assert.equal(identity({ url: `${BASE}/pulls/3` }), `${BASE}/pulls/3`);
assert.equal(identity({ title: 'no identifier here' }), null);
assert.equal(identity(null), null);
});
test('a page of unidentifiable items does not become a finding', () => {
assert.deepEqual(identities([{ title: 'a' }, { title: 'b' }]), []);
assert.deepEqual(identities([{ id: 1 }, { title: 'b' }]), ['1']);
assert.deepEqual(identities('not a list'), []);
});
test('identical rows with a cursor link is the definite finding', () => {
const [state, detail] = verdict('cursor', ['9'], ['9']);
assert.equal(state, 'ignores-page');
assert.match(detail, /does not read page at all/);
assert.match(detail, /no terminating condition/);
assert.ok(!loopTerminates(state));
});
test('identical rows with a page link is only a suspicion', () => {
const [state, detail] = verdict('offset', ['9'], ['9']);
assert.equal(state, 'suspect-ignores-page');
assert.match(detail, /may be a feed that moved/);
assert.match(detail, /Re-run it/);
});
test('a partial overlap is its own answer', () => {
const [state, detail] = verdict('offset', ['9', '8'], ['8', '7']);
assert.equal(state, 'overlapping-pages');
assert.match(detail, /unstable sort/);
assert.ok(loopTerminates(state));
});
test('a cursor endpoint that pages properly is not a finding', () => {
const [state, detail] = verdict('cursor', ['9'], ['8']);
assert.equal(state, 'cursor-pagination');
assert.match(detail, /not by number/);
});
test('offset pagination that works is reported as working', () => {
assert.equal(verdict('offset', ['9'], ['8'])[0], 'offset-honoured');
assert.equal(verdict('offset', ['9'], [])[0], 'offset-honoured');
});
test('an empty first page proves nothing', () => {
const [state, detail] = verdict('none', [], []);
assert.equal(state, 'inconclusive-empty');
assert.match(detail, /no comparison to make/);
});
test('the row comparisons are order sensitive and set based in turn', () => {
assert.ok(sameRows(['1', '2'], ['1', '2']));
assert.ok(!sameRows(['1', '2'], ['2', '1']));
assert.ok(!sameRows([], []));
assert.ok(overlaps(['1', '2'], ['2', '3']));
assert.ok(!overlaps(['1'], ['2']));
});
test('the repair names the cursor the endpoint actually uses', () => {
assert.match(repair('ignores-page', CURSOR_LINK), /after=/);
assert.match(repair('ignores-page', BEFORE_LINK), /before=/);
assert.match(repair('ignores-page', NO_LINK), /no next page/);
assert.ok(!repair('cursor-pagination').includes('Re-run'));
});
test('the repair for a suspicion changes no code', () => {
const fix = repair('suspect-ignores-page');
assert.match(fix, /re-run the check/);
assert.match(fix, /before changing any code/);
});
test('the header is parsed around commas inside URLs', () => {
const header = `<${BASE}/issues?labels=bug,ci&page=2>; rel="next"`;
assert.equal(linkStyle(parseLink(header)), 'offset');
});
test('the run says what it will spend', () => {
assert.equal(readCost(['/a', '/b']), 4);
assert.equal(readCost([]), 0);
assert.equal(readCost(null), 0);
});
FAQ
Which endpoints ignore page?
There is no published list, and there does not need to be, because every endpoint tells you itself in its Link header. Repository activity is the well-known one and it pages with before and after cursors; several newer list endpoints do the same, on the reasonable grounds that an offset into a live feed is not a stable thing to hand out. Rather than memorising a set that changes, probe the exact calls your integration makes and read what their next links are built from. That is two requests per path.
Why compare identifiers instead of the whole response body?
Because a list item is not stable between two requests even when it is the same row. Comment counts, updated timestamps, reaction totals and mergeability all change under you, so comparing whole objects reports two identical rows as different and hides the bug you are looking for. An identifier is the only field that means what you need it to mean, and the script falls back through id, node_id, sha and url because not every list on this API keys its items the same way.
The check says suspect-ignores-page. What does that mean?
It means one signal fired and the other did not: page two returned the same rows as page one, but the endpoint's next link is still built from page=. A feed sorted by recency can legitimately return an overlapping row for two requests made a second apart, so the honest answer is that this needs a re-run or a deterministic sort before anybody changes code. The script keeps that state separate on purpose. A diagnostic that reports a busy feed as a definite bug is one that gets muted.
Why is this worse than a pager that stops too early?
Because it does not stop, and because it produces data rather than withholding it. An under-count puts a small number in a report. This puts thousands of duplicate rows into whatever the job writes to, at full request rate, until a timeout or a page cap ends it. And it is usually misdiagnosed one layer down: duplicates read as an upsert problem, so the deduplication gets fixed and the job settles into quietly reporting page one as the entire dataset.
How much quota does the check itself cost?
Two requests per path probed, four by default, against the hourly core bucket, and both are made at per_page=1 so nothing large is transferred. The script computes and prints the total before it fetches anything. Keeping it small matters here more than usual: the bug being diagnosed is a loop that spends quota at full speed on one page of data, and a diagnostic for it has no business doing the same.
Related field notes
- Only the first page is ever read
- The Link header has no rel=last
- The compare endpoint stops at 250 commits
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using pagination in the REST API — GitHub Docs
- Pagination parameters ignored on some endpoints — GitHub Community
- Repositories — GitHub REST API
- Best practices for using the REST API — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.