Diagnostic GitHub API
per_page above 100 is clamped and never rejected
Somebody set per_page=500 to cut the number of round trips, and the request came back with exactly one hundred items and a 200. One hundred is fewer than five hundred, so the loop concluded it had reached the end and stopped. There was no 422, no warning header and nothing in the logs. Four fifths of the collection was never read, and the report built on it looks entirely finished.
GitHub caps per_page at 100 and enforces the cap by reducing the value rather than rejecting it. A request for 500 is served as a request for 100, with a success status and no indication anywhere in the response that anything was adjusted.
On its own that is harmless. It turns into data loss the moment the client decides whether it has reached the last page by comparing the number of items it received against the number it asked for — a predicate that is only correct if you know the effective page size, and you do not, because it is the smallest of your number, the endpoint's own maximum and what is left in the collection. Send per_page=100 explicitly and terminate on the absence of rel="next" in the Link header. Never on a short page.
The problem in plain words
The number gets written once, by somebody being sensible. Fewer requests is the right instinct, 500 is a round number, and nothing about the request fails, so nobody revisits it. It survives review for the same reason: a reviewer looking at per_page=500 sees an intention rather than an error, and a reviewer looking at if len(items) < page_size: break sees a pattern they have written themselves, because it is correct in plenty of other APIs and in most cursor-based ones.
Then it survives testing, because the fixture repository has eleven issues. The short-page predicate is right on eleven items, right on ninety-nine, and wrong from the hundred-and-first onwards. The bug does not exist until the collection crosses a boundary nobody is watching, and it appears first in the busiest repository, which is the one whose numbers people rely on.
What makes it hard to find afterwards is that the truncation is not ragged. It is exactly one hundred, every time, in a report that has no other reason to contain round numbers. That reads as a limit somebody configured on purpose, and the search goes looking for a hard-coded 100 in your own code, where there isn't one. The 100 came from GitHub, quietly, in response to being asked for five hundred.
Why it happens
The cap is enforced by clamping, not by validation. The maximum page size on the REST API is 100. A larger value is not a client error; it is served as 100. There is no Warning header, no field in the body and no change to the status, so the only place the adjustment is visible is in the length of the array you got back — which is exactly the number the buggy predicate is reading.
The short-page predicate has a premise you do not control. "Fewer items than I asked for means the end" is sound only when the server serves what you asked for. The effective page size is the minimum of your requested value, the endpoint's own maximum and the number of items remaining. Two of those three are GitHub's to change, and one of them changed the moment you typed a number above 100.
Some endpoints enforce a smaller maximum than 100. The same clamping behaviour applies, at a lower number, so a client that has carefully hard-coded 100 as its page size can still receive a short full page and stop. This is why the check measures the effective page size against what came back rather than asserting a constant: a page that is shorter than requested and carries a rel="next" is the finding whatever the numbers happen to be.
The Link header is the only authority on whether there is more. A response with rel="next" has more data behind it, at any page size, whether or not the page you are holding is full. A response without it does not. That predicate has no premise about page sizes in it at all, which is why it is the one to terminate on. Where rel="last" is missing from the header as well, that has its own consequences.
This is not the note about the default. Leaving per_page unset gets you 30 items a page and costs 3.3 times the requests, which is a spending problem with a correct answer at the end of it. This one is a correctness problem: the answer is wrong and the job finished early and under budget. They are repaired by the same one-line change and they are not the same bug, and a team that has already set per_page has fixed the first while possibly introducing this one.
The API cannot see your client, so the check proves the trap. Nothing GitHub returns says whether your loop terminates on a short page or on a missing rel="next". What a read-only script can show is that the clamp happened on this endpoint, that the response is shorter than what was requested, and that there is more data on the far side of it. That is the trap being set, not you falling into it, and the script says so rather than accusing you of a bug it cannot see.
The fix, as a flow
One GET per probed path, and the request deliberately asks for a page size that cannot be served. What comes back is measured rather than assumed, because the effective page size is the smallest of three numbers and only one of them is yours. The whole finding is then a disagreement between two predicates evaluated over the same response: the one that asks whether the page was shorter than requested, and the one that asks whether the Link header still advertises a next page. Both are pure functions, so the rule can be tested without spending a request on it.
How to fix it
Ask for more than the maximum on purpose
Send the list request your integration sends, with per_page=500. One request. Count the items in the array that comes back: 100 is the clamp, plainly, in the one place it is ever visible. This is a deliberately abnormal request made once by a diagnostic, not something to leave in production code.
Read the Link header on that same response
The clamp alone is not a finding. The clamp plus rel="next" is: it means the page you were served is shorter than the page you asked for and there is more behind it, which is precisely the state in which a short-page check stops early. If there is no rel="next", this collection happens to end here and the trap is armed for whenever it grows.
Run both termination predicates side by side
Evaluate "received fewer than requested" and "there is no rel="next"" over the same response and print both. Where they disagree, the first one is the bug and the second one is the answer. Making the comparison explicit is worth more than the verdict, because it is the sentence that changes somebody's loop.
Set per_page to 100 and stop naming a bigger number
100 is the maximum and asking for more buys nothing at all — not a bigger page, not fewer requests, not a warning that you asked. Writing 100 makes the effective page size and the requested page size the same number, which removes one of the two ways the short-page predicate can lie to you. It does not remove the other, which is why the next step exists.
Terminate on the header, then keep the check cheap
Follow rel="next" until it is absent, and never construct the next URL yourself. The audit itself costs one request per path probed, three by default, against the hourly core quota — small enough to run on every deploy, and worth knowing before you point it at forty endpoints in a loop. GET /rate_limit will tell you what is left and does not itself consume any.
How to check it worked
After the loop terminates on the header instead of on the page length, the same audit reports a full page with a next link and nothing to do.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_per_page_clamp.py --repo acme/monorepo
# read cost: 3 request(s) against the core hourly quota
# /repos/acme/monorepo/issues: asked for 500, received 100
# clamped-and-truncated: per_page=500 was reduced to 100 and rel="next" is
# present, so a client that stops on a short page stops here with more to read
# repair: send per_page=100 and terminate on the absence of rel="next"
# after the loop is changed to follow the header
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_per_page_clamp.py --repo acme/monorepo --per-page 100
# within-cap-more-pages: asked for 100, received 100, rel="next" is present and
# the short-page check agrees with the header hereThe full code
One GET per probed path, asking on purpose for a page size that cannot be served. Everything that matters afterwards is two pure predicates evaluated over the same response: the one that asks whether the page was shorter than requested, and the one that asks whether the header still advertises a next page. The verdict is their disagreement, which means the entire rule can be tested offline and the network part is a thin shell around it. The cost of the run is computed and printed before any request is made, because a diagnostic in a section full of quota notes should say what it spends.
"""Show that per_page above the maximum is reduced rather than refused.
Read only. One GET per probed path, plus one more per path with --confirm.
Nothing is written and the repair is printed rather than performed.
GitHub caps per_page at 100 and enforces the cap by silently lowering the value:
a request for 500 is served as a request for 100 with a success status and no
warning anywhere in the response. That is harmless on its own. It becomes data
loss in a client that decides it has reached the last page by noticing it got
fewer items than it asked for, because 100 is fewer than 500 and the loop stops
with four fifths of the collection unread.
What this can and cannot see: the API has no idea which predicate your client
terminates on. It can show that the clamp happened here and that there is more
data behind the shortened page. That is the trap, not the fall.
Environment:
GITHUB_TOKEN a token with read access to the repository
"""
import argparse
import json
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_per_page_clamp")
API = "https://api.github.com"
UA = "github-per-page-clamp/1.0"
# The documented ceiling on a page. Named rather than inlined because it turns up
# in the output and a reader comparing this against the documentation should find
# it in one place.
MAX_PER_PAGE = 100
# Anchored on the angle brackets rather than split on commas. A pagination URL
# can contain a comma of its own, labels=bug,ci being the everyday case, and
# splitting the header on commas turns one good link into two broken ones.
LINK = re.compile(r'<([^>]+)>\s*;\s*rel="([^"]+)"')
PROBES = ["issues", "pulls", "branches"]
def parse_link(header):
"""Parse a Link header into {rel: url}. Pure."""
if not header:
return {}
return {rel: url for url, rel in LINK.findall(header)}
def clamped_to(requested):
"""The page size GitHub will actually use for this request. Pure.
None for anything that is not a usable page size, so a bad argument is
reported rather than silently treated as a finding.
"""
try:
n = int(requested)
except (TypeError, ValueError):
return None
if n < 1:
return None
return min(n, MAX_PER_PAGE)
def is_over_maximum(requested):
"""Whether this value will be lowered before it is served. Pure."""
try:
return int(requested) > MAX_PER_PAGE
except (TypeError, ValueError):
return False
def stops_on_short_page(requested, received):
"""The buggy predicate: fewer items than asked for, so that was the end.
Pure, and written out under its own name on purpose. It is the whole bug,
and it is much easier to argue about when it is a function with a name than
when it is an inequality inside somebody's while loop.
"""
size = clamped_to(requested)
try:
got = int(received)
except (TypeError, ValueError):
return False
if size is None:
return False
try:
return got < int(requested)
except (TypeError, ValueError):
return False
def stops_on_missing_next(links):
"""The correct predicate: the header no longer advertises a next page. Pure."""
return "next" not in (links or {})
def predicates_disagree(requested, received, links):
"""Whether the short-page check would stop while the header says otherwise."""
return stops_on_short_page(requested, received) and not stops_on_missing_next(links)
def verdict(requested, received, links):
"""Classify one response. Pure. Returns (state, detail).
The states keep two kinds of short page apart. One is losing data right now.
The other is a collection that happens to end on the boundary, which is the
same trap with the spring not yet released.
"""
size = clamped_to(requested)
if size is None or received is None:
return ("unknown",
"the request was not answered in a form this check can read.")
try:
got = int(received)
except (TypeError, ValueError):
return ("unknown",
"the request was not answered in a form this check can read.")
more = not stops_on_missing_next(links)
over = is_over_maximum(requested)
if predicates_disagree(requested, received, links):
if over and got == MAX_PER_PAGE:
return ("clamped-and-truncated",
'per_page=%s was reduced to %d and rel="next" is present, so '
"a client that stops on a short page stops here with more to "
"read." % (requested, MAX_PER_PAGE))
return ("smaller-maximum",
"per_page=%s was asked for and %d item(s) came back with "
'rel="next" still present, so this endpoint serves a smaller '
"page than you requested and a short-page check stops here too."
% (requested, got))
if over and got == MAX_PER_PAGE:
return ("clamped-at-boundary",
"per_page=%s was reduced to %d and there is no next page, so "
"this collection happens to end exactly on the boundary. The "
"clamp is real and the truncation starts on item %d."
% (requested, MAX_PER_PAGE, MAX_PER_PAGE + 1))
if over:
return ("clamped-untested",
"per_page=%s was reduced to %d, but only %d item(s) exist here, "
"so the truncation cannot be shown on this path. The clamp still "
"applies to every path that grows past %d."
% (requested, MAX_PER_PAGE, got, MAX_PER_PAGE))
if more:
return ("within-cap-more-pages",
'per_page=%s was served in full and rel="next" is present. The '
"short-page check agrees with the header here, which is luck "
"rather than correctness." % requested)
return ("within-cap-complete",
"per_page=%s was served in full and there is no next page. One "
"request really is the whole list here." % requested)
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state in ("clamped-and-truncated", "clamped-at-boundary", "clamped-untested"):
return ("send per_page=100 and terminate on the absence of rel=\"next\" "
"in the Link header. Asking for more than 100 buys nothing: not "
"a bigger page, not fewer requests, not an error telling you so.")
if state == "smaller-maximum":
return ("this endpoint serves a smaller page than 100, so hard-coding "
"any page size as your terminating condition is unsafe here. "
"Follow rel=\"next\" until it is absent.")
if state == "within-cap-more-pages":
return ("nothing on the page size. Check that the loop terminates on "
"the missing rel=\"next\" rather than on the page length: the "
"two agree on this response and will part company on a clamp.")
if state == "within-cap-complete":
return "nothing."
return "point the check at a path this token can list."
def read_cost(paths, confirm=False):
"""Requests this run will spend against the core quota. Pure."""
n = len(paths or [])
return n * (2 if confirm else 1)
def get(session, path, params):
"""One GET. Returns (status, items-or-None, links)."""
r = session.get(API + path, params=params, timeout=30)
links = parse_link(r.headers.get("Link"))
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
if r.status_code == 403 and "rate limit" in r.text.lower():
raise SystemExit("403 rate limited. GET /rate_limit reports the reset "
"time and does not itself consume quota")
if r.status_code != 200:
log.info("%s returned %d; skipping it", path, r.status_code)
return r.status_code, None, links
try:
body = r.json()
except ValueError:
return r.status_code, None, links
return r.status_code, body if isinstance(body, list) else None, links
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--path", action="append",
help="probe this API path instead of the defaults, e.g. "
"/repos/o/n/releases. Repeatable.")
ap.add_argument("--per-page", type=int, default=500,
help="the page size to ask for. The default is deliberately "
"above the maximum, which is the whole point.")
ap.add_argument("--confirm", action="store_true",
help="spend a second request per path at per_page=100 to "
"show the honest page size beside the clamped one")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
paths = args.path or ["/repos/%s/%s" % (args.repo, name) for name in PROBES]
log.info("read cost: %d request(s) against the core hourly quota",
read_cost(paths, args.confirm))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
findings = []
for path in paths:
status, items, links = get(session, path, {"per_page": args.per_page})
if items is None:
continue
state, detail = verdict(args.per_page, len(items), links)
log.info("%s: asked for %s, received %d", path, args.per_page, len(items))
log.info("%s: %s", state, detail)
log.info("repair: %s", repair(state))
honest = None
if args.confirm:
_s, confirmed, _l = get(session, path, {"per_page": MAX_PER_PAGE})
honest = len(confirmed) if confirmed is not None else None
if honest is not None:
log.info("%s: at per_page=%d the same call returns %d item(s)",
path, MAX_PER_PAGE, honest)
findings.append({
"path": path,
"status": status,
"requested": args.per_page,
"effective_page_size": clamped_to(args.per_page),
"received": len(items),
"rels": sorted(links),
"short_page_check_stops": stops_on_short_page(args.per_page, len(items)),
"header_check_stops": stops_on_missing_next(links),
"predicates_disagree": predicates_disagree(args.per_page, len(items), links),
"at_per_page_100": honest,
"state": state,
"detail": detail,
})
print(json.dumps({"requests_spent": read_cost(paths, args.confirm),
"findings": findings}, indent=2, default=str))
bad = {"clamped-and-truncated", "smaller-maximum", "clamped-at-boundary"}
return 1 if any(f["state"] in bad for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Show that per_page above the maximum is reduced rather than refused.
*
* Read only. One GET per probed path, plus one more per path when
* GITHUB_CONFIRM is set. Nothing is written and the repair is printed rather
* than performed.
*
* Environment:
* GITHUB_TOKEN a token with read access to the repository
* GITHUB_REPO owner/name
* GITHUB_PER_PAGE the page size to ask for, default 500
* GITHUB_CONFIRM set to spend a second request per path at per_page=100
*/
const API = 'https://api.github.com';
const UA = 'github-per-page-clamp/1.0';
/** The documented ceiling on a page. */
export const MAX_PER_PAGE = 100;
// Anchored on the angle brackets rather than split on commas: a pagination URL
// can contain a comma of its own, and splitting on it breaks the link.
const LINK = /<([^>]+)>\s*;\s*rel="([^"]+)"/g;
const PROBES = ['issues', 'pulls', 'branches'];
/** Parse a Link header into {rel: url}. Pure. */
export function parseLink(header) {
const out = {};
if (!header) return out;
for (const m of String(header).matchAll(LINK)) out[m[2]] = m[1];
return out;
}
/** The page size GitHub will actually use for this request. Pure. */
export function clampedTo(requested) {
const n = Number(requested);
if (!Number.isFinite(n) || Math.trunc(n) < 1) return null;
return Math.min(Math.trunc(n), MAX_PER_PAGE);
}
/** Whether this value will be lowered before it is served. Pure. */
export function isOverMaximum(requested) {
const n = Number(requested);
return Number.isFinite(n) && n > MAX_PER_PAGE;
}
/** The buggy predicate: fewer items than asked for, so that was the end. Pure. */
export function stopsOnShortPage(requested, received) {
const size = clampedTo(requested);
const got = Number(received);
if (size === null || !Number.isFinite(got)) return false;
return got < Number(requested);
}
/** The correct predicate: the header no longer advertises a next page. Pure. */
export function stopsOnMissingNext(links) {
return !(links && Object.prototype.hasOwnProperty.call(links, 'next'));
}
/** Whether the short-page check would stop while the header says otherwise. */
export function predicatesDisagree(requested, received, links) {
return stopsOnShortPage(requested, received) && !stopsOnMissingNext(links);
}
/** Classify one response. Pure. Returns [state, detail]. */
export function verdict(requested, received, links) {
const size = clampedTo(requested);
const got = Number(received);
if (size === null || received === null || received === undefined || !Number.isFinite(got)) {
return ['unknown', 'the request was not answered in a form this check can read.'];
}
const more = !stopsOnMissingNext(links);
const over = isOverMaximum(requested);
if (predicatesDisagree(requested, received, links)) {
if (over && got === MAX_PER_PAGE) {
return ['clamped-and-truncated',
`per_page=${requested} was reduced to ${MAX_PER_PAGE} and rel="next" is `
+ 'present, so a client that stops on a short page stops here with more '
+ 'to read.'];
}
return ['smaller-maximum',
`per_page=${requested} was asked for and ${got} item(s) came back with `
+ 'rel="next" still present, so this endpoint serves a smaller page than '
+ 'you requested and a short-page check stops here too.'];
}
if (over && got === MAX_PER_PAGE) {
return ['clamped-at-boundary',
`per_page=${requested} was reduced to ${MAX_PER_PAGE} and there is no next `
+ 'page, so this collection happens to end exactly on the boundary. The '
+ `clamp is real and the truncation starts on item ${MAX_PER_PAGE + 1}.`];
}
if (over) {
return ['clamped-untested',
`per_page=${requested} was reduced to ${MAX_PER_PAGE}, but only ${got} `
+ 'item(s) exist here, so the truncation cannot be shown on this path. The '
+ `clamp still applies to every path that grows past ${MAX_PER_PAGE}.`];
}
if (more) {
return ['within-cap-more-pages',
`per_page=${requested} was served in full and rel="next" is present. The `
+ 'short-page check agrees with the header here, which is luck rather than '
+ 'correctness.'];
}
return ['within-cap-complete',
`per_page=${requested} was served in full and there is no next page. One `
+ 'request really is the whole list here.'];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (['clamped-and-truncated', 'clamped-at-boundary', 'clamped-untested'].includes(state)) {
return 'send per_page=100 and terminate on the absence of rel="next" in the '
+ 'Link header. Asking for more than 100 buys nothing: not a bigger page, '
+ 'not fewer requests, not an error telling you so.';
}
if (state === 'smaller-maximum') {
return 'this endpoint serves a smaller page than 100, so hard-coding any '
+ 'page size as your terminating condition is unsafe here. Follow '
+ 'rel="next" until it is absent.';
}
if (state === 'within-cap-more-pages') {
return 'nothing on the page size. Check that the loop terminates on the '
+ 'missing rel="next" rather than on the page length: the two agree on '
+ 'this response and will part company on a clamp.';
}
if (state === 'within-cap-complete') return 'nothing.';
return 'point the check at a path this token can list.';
}
/** Requests this run will spend against the core quota. Pure. */
export function readCost(paths, confirm = false) {
const n = Array.isArray(paths) ? paths.length : 0;
return n * (confirm ? 2 : 1);
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
};
}
async function get(token, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
const res = await fetch(url, { headers: headers(token) });
const links = parseLink(res.headers.get('link'));
if (!res.ok) return { status: res.status, items: null, links };
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, items: Array.isArray(body) ? body : null, links };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const repo = process.env.GITHUB_REPO;
if (!token || !repo) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_REPO=owner/name');
process.exitCode = 2;
return;
}
const perPage = Number(process.env.GITHUB_PER_PAGE || 500);
const confirm = Boolean(process.env.GITHUB_CONFIRM);
const paths = PROBES.map((name) => `/repos/${repo}/${name}`);
console.log(`read cost: ${readCost(paths, confirm)} request(s) against the core hourly quota`);
const findings = [];
for (const path of paths) {
const { status, items, links } = await get(token, path, { per_page: perPage });
if (items === null) {
console.log(`${path} returned ${status}; skipping it`);
continue;
}
const [state, detail] = verdict(perPage, items.length, links);
console.log(`${path}: asked for ${perPage}, received ${items.length}`);
console.log(`${state}: ${detail}`);
console.log(`repair: ${repair(state)}`);
let honest = null;
if (confirm) {
const second = await get(token, path, { per_page: MAX_PER_PAGE });
honest = second.items === null ? null : second.items.length;
if (honest !== null) {
console.log(`${path}: at per_page=${MAX_PER_PAGE} the same call returns ${honest} item(s)`);
}
}
findings.push({
path,
status,
requested: perPage,
effective_page_size: clampedTo(perPage),
received: items.length,
rels: Object.keys(links).sort(),
short_page_check_stops: stopsOnShortPage(perPage, items.length),
header_check_stops: stopsOnMissingNext(links),
predicates_disagree: predicatesDisagree(perPage, items.length, links),
at_per_page_100: honest,
state,
detail,
});
}
console.log(JSON.stringify({ requests_spent: readCost(paths, confirm), findings }, null, 2));
const bad = ['clamped-and-truncated', 'smaller-maximum', 'clamped-at-boundary'];
process.exitCode = findings.some((f) => bad.includes(f.state)) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The two predicates get the most attention, because the note is entirely about the gap between them: the short-page check has to be wrong on a clamped response and right on a full one, the header check has to be indifferent to page sizes altogether, and their disagreement has to be exactly the finding. After that, the clamp arithmetic at and around the boundary, the distinction between a truncation happening now and one waiting for the collection to grow, an endpoint whose own maximum is below 100, and the read cost, which is asserted because a diagnostic that quietly doubles its request count is a bad citizen in a section full of quota notes.
from github_per_page_clamp import (
MAX_PER_PAGE, clamped_to, is_over_maximum, parse_link, predicates_disagree,
read_cost, repair, stops_on_missing_next, stops_on_short_page, verdict,
)
MORE = {"next": "https://api.github.com/repositories/1/issues?page=2"}
END = {"prev": "https://api.github.com/repositories/1/issues?page=3"}
def test_the_clamp_is_a_minimum_not_a_rejection():
assert clamped_to(500) == MAX_PER_PAGE
assert clamped_to(101) == MAX_PER_PAGE
assert clamped_to(100) == 100
assert clamped_to(30) == 30
assert clamped_to("50") == 50
def test_a_page_size_that_is_not_one_is_reported_rather_than_guessed():
assert clamped_to(0) is None
assert clamped_to(-5) is None
assert clamped_to(None) is None
assert clamped_to("many") is None
def test_only_values_above_the_maximum_are_lowered():
assert is_over_maximum(500)
assert is_over_maximum(101)
assert not is_over_maximum(100)
assert not is_over_maximum(None)
def test_the_short_page_check_is_wrong_on_a_clamped_response():
assert stops_on_short_page(500, 100)
assert not stops_on_short_page(100, 100)
assert stops_on_short_page(100, 42)
def test_the_header_check_does_not_care_about_page_sizes():
assert not stops_on_missing_next(MORE)
assert stops_on_missing_next(END)
assert stops_on_missing_next({})
assert stops_on_missing_next(None)
def test_the_finding_is_exactly_the_disagreement():
assert predicates_disagree(500, 100, MORE)
assert not predicates_disagree(500, 100, END)
assert not predicates_disagree(100, 100, MORE)
def test_a_clamped_page_with_more_behind_it_is_the_finding():
state, detail = verdict(500, 100, MORE)
assert state == "clamped-and-truncated"
assert "reduced to 100" in detail
assert "stops on a short page" in detail
def test_a_collection_that_ends_on_the_boundary_is_still_a_trap():
state, detail = verdict(500, 100, END)
assert state == "clamped-at-boundary"
assert "item 101" in detail
def test_a_small_collection_cannot_prove_the_clamp():
state, detail = verdict(500, 12, {})
assert state == "clamped-untested"
assert "cannot be shown on this path" in detail
def test_an_endpoint_with_a_smaller_maximum_is_named_separately():
state, detail = verdict(100, 50, MORE)
assert state == "smaller-maximum"
assert "smaller page than you requested" in detail
def test_a_full_page_within_the_cap_is_not_a_finding():
assert verdict(100, 100, MORE)[0] == "within-cap-more-pages"
assert verdict(100, 100, END)[0] == "within-cap-complete"
assert verdict(30, 11, {})[0] == "within-cap-complete"
def test_an_unreadable_response_is_not_reported_as_a_clamp():
assert verdict(500, None, MORE)[0] == "unknown"
assert verdict(None, 100, MORE)[0] == "unknown"
def test_the_link_header_survives_a_comma_inside_a_url():
header = ('<https://api.github.com/repos/o/n/issues?labels=bug,ci&page=2>; rel="next", '
'<https://api.github.com/repos/o/n/issues?labels=bug,ci&page=9>; rel="last"')
links = parse_link(header)
assert set(links) == {"next", "last"}
assert links["next"].endswith("page=2")
assert parse_link(None) == {}
def test_the_repair_never_suggests_asking_for_more_than_the_maximum():
for state in ("clamped-and-truncated", "clamped-at-boundary", "clamped-untested"):
assert "per_page=100" in repair(state)
assert "500" not in repair(state)
assert "smaller page than 100" in repair("smaller-maximum")
assert repair("within-cap-complete") == "nothing."
def test_the_run_says_what_it_will_spend():
assert read_cost(["/a", "/b", "/c"]) == 3
assert read_cost(["/a", "/b", "/c"], confirm=True) == 6
assert read_cost([]) == 0
assert read_cost(None) == 0
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
MAX_PER_PAGE, clampedTo, isOverMaximum, parseLink, predicatesDisagree,
readCost, repair, stopsOnMissingNext, stopsOnShortPage, verdict,
} from './github-per-page-clamp.mjs';
const MORE = { next: 'https://api.github.com/repositories/1/issues?page=2' };
const END = { prev: 'https://api.github.com/repositories/1/issues?page=3' };
test('the clamp is a minimum, not a rejection', () => {
assert.equal(clampedTo(500), MAX_PER_PAGE);
assert.equal(clampedTo(101), MAX_PER_PAGE);
assert.equal(clampedTo(100), 100);
assert.equal(clampedTo(30), 30);
assert.equal(clampedTo('50'), 50);
});
test('a page size that is not one is reported rather than guessed', () => {
assert.equal(clampedTo(0), null);
assert.equal(clampedTo(-5), null);
assert.equal(clampedTo(null), null);
assert.equal(clampedTo('many'), null);
});
test('only values above the maximum are lowered', () => {
assert.ok(isOverMaximum(500));
assert.ok(isOverMaximum(101));
assert.ok(!isOverMaximum(100));
assert.ok(!isOverMaximum(null));
});
test('the short-page check is wrong on a clamped response', () => {
assert.ok(stopsOnShortPage(500, 100));
assert.ok(!stopsOnShortPage(100, 100));
assert.ok(stopsOnShortPage(100, 42));
});
test('the header check does not care about page sizes', () => {
assert.ok(!stopsOnMissingNext(MORE));
assert.ok(stopsOnMissingNext(END));
assert.ok(stopsOnMissingNext({}));
assert.ok(stopsOnMissingNext(null));
});
test('the finding is exactly the disagreement', () => {
assert.ok(predicatesDisagree(500, 100, MORE));
assert.ok(!predicatesDisagree(500, 100, END));
assert.ok(!predicatesDisagree(100, 100, MORE));
});
test('a clamped page with more behind it is the finding', () => {
const [state, detail] = verdict(500, 100, MORE);
assert.equal(state, 'clamped-and-truncated');
assert.match(detail, /reduced to 100/);
assert.match(detail, /stops on a short page/);
});
test('a collection that ends on the boundary is still a trap', () => {
const [state, detail] = verdict(500, 100, END);
assert.equal(state, 'clamped-at-boundary');
assert.match(detail, /item 101/);
});
test('a small collection cannot prove the clamp', () => {
const [state, detail] = verdict(500, 12, {});
assert.equal(state, 'clamped-untested');
assert.match(detail, /cannot be shown on this path/);
});
test('an endpoint with a smaller maximum is named separately', () => {
const [state, detail] = verdict(100, 50, MORE);
assert.equal(state, 'smaller-maximum');
assert.match(detail, /smaller page than you requested/);
});
test('a full page within the cap is not a finding', () => {
assert.equal(verdict(100, 100, MORE)[0], 'within-cap-more-pages');
assert.equal(verdict(100, 100, END)[0], 'within-cap-complete');
assert.equal(verdict(30, 11, {})[0], 'within-cap-complete');
});
test('an unreadable response is not reported as a clamp', () => {
assert.equal(verdict(500, null, MORE)[0], 'unknown');
assert.equal(verdict(null, 100, MORE)[0], 'unknown');
});
test('the Link header survives a comma inside a URL', () => {
const header = '<https://api.github.com/repos/o/n/issues?labels=bug,ci&page=2>; rel="next", '
+ '<https://api.github.com/repos/o/n/issues?labels=bug,ci&page=9>; rel="last"';
const links = parseLink(header);
assert.deepEqual(Object.keys(links).sort(), ['last', 'next']);
assert.ok(links.next.endsWith('page=2'));
assert.deepEqual(parseLink(null), {});
});
test('the repair never suggests asking for more than the maximum', () => {
for (const state of ['clamped-and-truncated', 'clamped-at-boundary', 'clamped-untested']) {
assert.match(repair(state), /per_page=100/);
assert.ok(!repair(state).includes('500'));
}
assert.match(repair('smaller-maximum'), /smaller page than 100/);
assert.equal(repair('within-cap-complete'), 'nothing.');
});
test('the run says what it will spend', () => {
assert.equal(readCost(['/a', '/b', '/c']), 3);
assert.equal(readCost(['/a', '/b', '/c'], true), 6);
assert.equal(readCost([]), 0);
assert.equal(readCost(null), 0);
});
FAQ
Why does GitHub not return a 422 for per_page=500?
Because rejecting it would break clients that have been sending an oversized value harmlessly for years, and because the cap is a serving decision rather than a validation rule: the request is answerable, just not at the size asked for. Whatever the reasoning, the practical consequence for you is fixed. There is no signal in the response that the value was adjusted, so a client cannot detect the clamp except by counting items, which is exactly the operation the bug is hiding inside.
Is this the same problem as leaving per_page unset?
No, and it is worth keeping them apart because the symptoms are opposite. An unset per_page gives you pages of 30 and costs 3.3 times the requests for a correct answer; that is a spending problem and it has its own note. This one costs you nothing extra and gives you a wrong answer, because the job stops early and under budget. The same one-line change repairs both, and a team that has already set per_page has fixed the first while possibly walking into this one.
Our loop stops on a short page and it has always worked. Why?
Because your page size has always been at or below 100, so the effective page size and the requested page size were the same number and the predicate happened to be right. It is right by coincidence rather than by construction. It breaks in three ways: somebody raises the value above 100, an endpoint enforces its own lower maximum, or a future response is short for a reason nobody anticipated. Terminating on the missing rel="next" has no premise about page sizes in it at all.
How much quota does the check itself cost?
One request per path probed, three by default, against the hourly core bucket. With --confirm it is two per path, because the second request re-runs the same call at per_page=100 so the honest page size can be printed beside the clamped one. The script computes and prints that number before it makes any request, so pointing it at forty paths in a loop is a decision rather than an accident. GET /rate_limit will tell you what is left and does not itself consume any.
Can the script tell whether my client actually has this bug?
No, and it does not claim to. Nothing GitHub returns says which predicate your loop terminates on, or whether it has a loop at all. What the script proves is that on this endpoint the value was clamped, that the page came back shorter than requested, and that there is more data behind it. That is the trap being set. Whether your client falls into it is a question about your code, and the printed comparison of the two predicates is there so you can answer it in a minute by reading your own loop.
Related field notes
- per_page is unset and every list costs more
- Only the first page is ever read
- The Link header has no rel=last
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using pagination in the REST API — GitHub Docs
- Best practices for using the REST API — GitHub Docs
- Issues — GitHub REST API
- octokit/plugin-paginate-rest.js
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.