Diagnostic GitHub API
the Link header has no rel=last so the page count breaks
The pager was written properly. It reads the Link header, it pulls the page number out of rel="last", and it uses that number to size the job before it starts: a progress bar, a worker pool over the page range, an estimate in the log line. On one endpoint the header comes back with a rel="next" and no rel="last" at all, and the job reports that it read one page of a one-page collection. It read one page of nine hundred.
GitHub includes rel="last" in the Link header only when it can calculate the final page. On some endpoints it cannot, and the header comes back carrying rel="next" and nothing that tells you how far the list goes.
That is fine for a pager that walks the list, and fatal for one that indexes it. Any code that computes a page count, fans out over range(1, last + 1), draws a progress bar or jumps to the final page for the oldest record depends on a field the API does not promise. The repair is to drive pagination off rel="next" alone and treat its absence as the terminating condition, and to treat a page count as a display nicety that is sometimes unavailable rather than as an input to the loop.
The problem in plain words
This one hurts because the code is good. Reading the header is the correct instinct, and a developer who has got as far as parsing Link has already avoided the common bug. They then do the natural next thing, which is to use the richest field in the header, and build the whole shape of the job around a number that is usually there.
The two ways it fails are very different and only one of them is survivable. If the code does int(links["last"]) it raises a KeyError, someone sees a stack trace, and it is fixed the same afternoon. If it does last = page_of(links.get("last")) or 1 — and it usually does, because somebody hit the KeyError in staging and made it safe — then a missing page count silently becomes a page count of one. The job runs, reports success, and reads the first thirty items of a list that has no visible end.
It is endpoint-specific, so it survives every test you have. The pager works against issues, works against pull requests, works against branches, and is quietly wrong on the one endpoint in your set that cannot be indexed. Nobody suspects the pager, because the pager is the tested part. The suspicion lands on the endpoint, which gets described as returning nothing, and on the token, which gets rotated.
Why it happens
A final page is a computation, not a fact. To say which page is last, GitHub has to know the total size of the result set at the moment of the request. Where that number is expensive, unbounded or genuinely unknown — high-volume feeds, cursor-based endpoints, anything counted lazily — it does not produce one, and the honest response is to omit the field rather than to invent it. So the presence of rel="last" is a property of the endpoint and the query, not a guarantee of the API.
Walking and indexing are different capabilities. An endpoint that gives you rel="last" supports four things: a page count, a progress bar, a fan-out over page numbers, and a jump to the end for the oldest or newest record. An endpoint that gives you only rel="next" supports exactly one: walking it. Sorting your endpoints into those two groups is the whole of this check, and it is the thing to know before you design the job, not after it under-reports.
Even where it exists, the number moves. rel="last" is computed per request. Between page one and page nine, issues get opened and closed, branches get deleted, and the final page number changes underneath a loop that captured it once. Caching it as the size of the job is a smaller version of the same mistake: it is a snapshot presented as a bound.
The terminating condition never needed it. The absence of rel="next" is the end of the list, on every endpoint, at every page size, whether or not a final page can be computed. It is a strictly weaker requirement, it is always available, and a loop built on it does not need to know how long it is going to run.
This is not the note about ignoring the header. The first-page-only bug is a client that never looked at Link at all, and it is the most common silent failure on this API. This one is a client that does look, does follow, and requires a field the header does not always carry. Both end with an under-count, and the code that produces them looks nothing alike.
The check proves the endpoint's shape, not your loop's. A read-only script can say which of your endpoints hand back a final page and which do not. It cannot see whether your pager needs one. So it prints the capability list rather than a verdict about your code, and it computes the page count both the careful way and the careless way so you can see the two answers differ.
The fix, as a flow
One GET per probed path at the smallest page size there is, because the finding lives entirely in a response header and none of the items are read. The rel names on that header are turned into a capability list rather than a boolean: an endpoint that can be indexed supports a page count, a progress bar, a fan-out over page numbers and a jump to the end, and an endpoint that can only be walked supports none of them. The script also models the bug directly, computing the page count the careless way beside the careful way so the two can be shown disagreeing.
How to fix it
Ask each endpoint for one item and read only the header
per_page=1 is the cheapest possible probe, and none of the items are read. Where rel="last" is present, its page number at that page size is also the exact item count, which is a useful free byproduct. Where it is absent, that absence is the finding.
Sort the endpoints into indexable and walk-only
Three states, not two. rel="last" present means the endpoint can be indexed. rel="next" present with no rel="last" means it can only be walked. Neither present means one page really is the whole list. The middle state is the one this note exists for and the one a boolean check would swallow.
Compute the page count twice on purpose
Once carefully, returning nothing when the field is missing, and once the way tired code does it, defaulting a missing value to 1. Printing both next to each other turns an abstract warning into a number: the careful answer is unknown and the careless answer is 1 page, and only one of those will make somebody go and look at their loop.
Name the patterns that stop working
A progress bar with no denominator, a worker pool with no range to divide, a jump to the last page for the oldest record, and any log line that says page 3 of 40. On a walk-only endpoint all four are unavailable, and knowing that before the job is written is much cheaper than discovering it from a report that says one page.
Drive the loop off rel=next and nothing else
Follow the next URL exactly as given until there is no next, and never rebuild it by hand. In Octokit that is octokit.paginate(); in PyGithub it is iterating the PaginatedList rather than asking it for a length. The probe costs one request per path, five by default, against the hourly core quota.
How to check it worked
Once the pager terminates on the missing next, the walk-only endpoint stops being a special case and the audit becomes a description of the endpoints rather than a warning about them.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_rel_last_absent.py --repo acme/monorepo
# read cost: 5 request(s) against the core hourly quota
# /repos/acme/monorepo/issues: rels next, last -> indexable, 912 page(s) at per_page=1
# /repos/acme/monorepo/events: rels next -> walk-only
# walk-only: rel="next" is present and rel="last" is not, so the size of this
# list is only knowable by walking it. A careful page count says unknown here;
# code that defaults a missing count to 1 reports 1 page
# unavailable here: page count, progress bar, parallel fan-out, jump to last
# repair: terminate on the absence of rel="next" and never require rel="last"The full code
One GET per probed path at the smallest page size there is, because the entire finding lives in a response header and not one item is read. The header is turned into a capability list rather than a boolean, since walk-only is a real and workable state and not an error. The script also models the bug directly: it computes the page count the careful way, which returns nothing when the field is absent, beside the careless way, which returns one, so the two can be printed disagreeing. Both are pure, so the rule is tested without a network.
"""Sort GitHub list endpoints into the ones you can index and the ones you can only walk.
Read only. One GET per probed path at per_page=1, which is the cheapest request
that still produces a Link header. No items are read and nothing is written.
GitHub includes rel="last" only when it can calculate a final page. On some
endpoints it cannot, so the header carries rel="next" and no way to know how far
the list goes. A pager that walks the list is unaffected. A pager that indexes it
-- a progress bar, a fan-out over a page range, a jump to the last page -- either
raises on the missing key or, far more often, defaults the missing count to 1 and
reports a single page as the whole collection.
What this can and cannot see: the API cannot tell whether your pager needs a page
count. It can say which of your endpoints will give you one. That is why the
output is a capability list rather than an accusation.
Environment:
GITHUB_TOKEN a token with read access to the repository
"""
import argparse
import json
import logging
import os
import re
import sys
from urllib.parse import parse_qs, urlparse
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_rel_last_absent")
API = "https://api.github.com"
UA = "github-rel-last-absent/1.0"
# Anchored on the angle brackets rather than split on commas, because a
# pagination URL can carry a comma of its own inside a query parameter.
LINK = re.compile(r'<([^>]+)>\s*;\s*rel="([^"]+)"')
PROBES = ["issues", "pulls", "branches", "events", "commits"]
# What each shape of header actually supports. Written as data rather than as a
# chain of ifs because the point of the script is the table, not the verdict.
CAPABILITIES = {
"indexable": {"walk": True, "page_count": True, "progress_bar": True,
"parallel_fanout": True, "jump_to_last": True},
"walk-only": {"walk": True, "page_count": False, "progress_bar": False,
"parallel_fanout": False, "jump_to_last": False},
"single-page": {"walk": True, "page_count": True, "progress_bar": True,
"parallel_fanout": False, "jump_to_last": False},
}
PATTERN_NAMES = {
"page_count": "page count",
"progress_bar": "progress bar",
"parallel_fanout": "parallel fan-out",
"jump_to_last": "jump to last",
}
def parse_link(header):
"""Parse a Link header into {rel: url}. Pure."""
if not header:
return {}
return {rel: url for url, rel in LINK.findall(header)}
def rels(links):
"""The rel names present, sorted. Pure, and the whole evidence base."""
return sorted(links or {})
def page_param(url):
"""The page query parameter on a pagination URL, or None. Pure."""
if not url:
return None
values = parse_qs(urlparse(url).query).get("page") or []
try:
return int(values[0])
except (IndexError, TypeError, ValueError):
return None
def pagination_style(links):
"""One of indexable, walk-only, single-page. Pure.
Three states rather than two on purpose. A boolean check collapses walk-only
into single-page, which is exactly the mistake this note is about.
"""
links = links or {}
if "last" in links:
return "indexable"
if "next" in links:
return "walk-only"
return "single-page"
def page_count(links):
"""The number of pages, or None where it cannot be known. Pure.
None is the honest answer on a walk-only endpoint, and callers are expected
to handle it rather than coerce it.
"""
if pagination_style(links) == "single-page":
return 1
return page_param((links or {}).get("last"))
def naive_page_count(links):
"""The page count the careless way: a missing value becomes 1. Pure.
This is not a helper. It is the bug, kept under its own name so the script
can print it beside the careful answer and let the difference do the work.
"""
return page_count(links) or 1
def item_count(links, per_page):
"""Total items, but only where the endpoint can be indexed. Pure."""
pages = page_count(links)
try:
size = int(per_page)
except (TypeError, ValueError):
return None
if pages is None or size != 1:
return None
return pages
def capabilities(style):
"""What a pager may rely on against an endpoint of this shape. Pure."""
return dict(CAPABILITIES.get(style, CAPABILITIES["walk-only"]))
def unavailable(style):
"""The named patterns that do not work here, in a fixed order. Pure."""
caps = capabilities(style)
return [PATTERN_NAMES[k] for k in ("page_count", "progress_bar",
"parallel_fanout", "jump_to_last")
if not caps[k]]
def verdict(links, per_page=1):
"""Classify one endpoint's header. Pure. Returns (state, detail)."""
style = pagination_style(links)
if style == "walk-only":
return (style,
'rel="next" is present and rel="last" is not, so the size of '
"this list is only knowable by walking it. A careful page count "
"says unknown here; code that defaults a missing count to 1 "
"reports %d page." % naive_page_count(links))
if style == "indexable":
total = item_count(links, per_page)
return (style,
'rel="last" is present, so this endpoint can be indexed: %s '
"page(s) at per_page=%s%s. That number is computed per request "
"and moves between calls, so it is a display value rather than "
"a bound." % (page_count(links), per_page,
", which is %d item(s)" % total if total else ""))
return (style,
'neither rel="next" nor rel="last" is present. One request is the '
"whole list here, and nothing about paging applies.")
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state == "walk-only":
return ('terminate on the absence of rel="next" and never require '
'rel="last". Drop the progress bar or make it indeterminate, '
"and replace any fan-out over a page range with a sequential "
"walk that follows the next URL exactly as given.")
if state == "indexable":
return ('nothing, provided rel="last" is treated as a snapshot. Do not '
"cache it as the size of the job, and do not let its absence on "
"some other endpoint default to 1.")
return "nothing."
def read_cost(paths):
"""Requests this run will spend against the core quota. Pure."""
return len(paths or [])
def get(session, path, per_page):
"""One GET. Returns (status, links)."""
r = session.get(API + path, params={"per_page": per_page}, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
if r.status_code == 403 and "rate limit" in r.text.lower():
raise SystemExit("403 rate limited. GET /rate_limit reports the reset "
"time and does not itself consume quota")
return r.status_code, parse_link(r.headers.get("Link"))
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--path", action="append",
help="probe this API path instead of the defaults. Repeatable.")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
paths = args.path or ["/repos/%s/%s" % (args.repo, name) for name in PROBES]
log.info("read cost: %d request(s) against the core hourly quota",
read_cost(paths))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
findings = []
for path in paths:
status, links = get(session, path, 1)
if status != 200:
log.info("%s returned %d; skipping it", path, status)
continue
state, detail = verdict(links, 1)
log.info("%s: rels %s -> %s", path, ", ".join(rels(links)) or "none", state)
log.info("%s: %s", state, detail)
missing = unavailable(state)
if missing:
log.info("unavailable here: %s", ", ".join(missing))
log.info("repair: %s", repair(state))
findings.append({
"path": path,
"rels": rels(links),
"style": state,
"pages": page_count(links),
"pages_if_missing_defaults_to_one": naive_page_count(links),
"items": item_count(links, 1),
"capabilities": capabilities(state),
"unavailable": missing,
"detail": detail,
})
print(json.dumps({"requests_spent": read_cost(paths),
"findings": findings}, indent=2, default=str))
return 1 if any(f["style"] == "walk-only" for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Sort GitHub list endpoints into the ones you can index and the ones you can
* only walk.
*
* Read only. One GET per probed path at per_page=1, which is the cheapest
* request that still produces a Link header. No items are read, nothing is
* written, and the repair is printed rather than performed.
*
* Environment:
* GITHUB_TOKEN a token with read access to the repository
* GITHUB_REPO owner/name
*/
const API = 'https://api.github.com';
const UA = 'github-rel-last-absent/1.0';
const LINK = /<([^>]+)>\s*;\s*rel="([^"]+)"/g;
const PROBES = ['issues', 'pulls', 'branches', 'events', 'commits'];
/** What each shape of header actually supports. Data, not a chain of ifs. */
export const CAPABILITIES = {
indexable: {
walk: true, page_count: true, progress_bar: true, parallel_fanout: true, jump_to_last: true,
},
'walk-only': {
walk: true, page_count: false, progress_bar: false, parallel_fanout: false, jump_to_last: false,
},
'single-page': {
walk: true, page_count: true, progress_bar: true, parallel_fanout: false, jump_to_last: false,
},
};
const PATTERN_NAMES = {
page_count: 'page count',
progress_bar: 'progress bar',
parallel_fanout: 'parallel fan-out',
jump_to_last: 'jump to last',
};
/** Parse a Link header into {rel: url}. Pure. */
export function parseLink(header) {
const out = {};
if (!header) return out;
for (const m of String(header).matchAll(LINK)) out[m[2]] = m[1];
return out;
}
/** The rel names present, sorted. Pure. */
export function rels(links) {
return Object.keys(links || {}).sort();
}
/** The page query parameter on a pagination URL, or null. Pure. */
export function pageParam(url) {
if (!url) return null;
try {
const raw = new URL(url, API).searchParams.get('page');
const n = Number(raw);
return Number.isFinite(n) && raw !== null ? Math.trunc(n) : null;
} catch {
return null;
}
}
/** One of indexable, walk-only, single-page. Pure. */
export function paginationStyle(links) {
const l = links || {};
if (Object.prototype.hasOwnProperty.call(l, 'last')) return 'indexable';
if (Object.prototype.hasOwnProperty.call(l, 'next')) return 'walk-only';
return 'single-page';
}
/** The number of pages, or null where it cannot be known. Pure. */
export function pageCount(links) {
if (paginationStyle(links) === 'single-page') return 1;
return pageParam((links || {}).last);
}
/** The page count the careless way: a missing value becomes 1. Pure. */
export function naivePageCount(links) {
return pageCount(links) || 1;
}
/** Total items, but only where the endpoint can be indexed. Pure. */
export function itemCount(links, perPage) {
const pages = pageCount(links);
const size = Number(perPage);
if (pages === null || !Number.isFinite(size) || size !== 1) return null;
return pages;
}
/** What a pager may rely on against an endpoint of this shape. Pure. */
export function capabilities(style) {
return { ...(CAPABILITIES[style] || CAPABILITIES['walk-only']) };
}
/** The named patterns that do not work here, in a fixed order. Pure. */
export function unavailable(style) {
const caps = capabilities(style);
return ['page_count', 'progress_bar', 'parallel_fanout', 'jump_to_last']
.filter((k) => !caps[k]).map((k) => PATTERN_NAMES[k]);
}
/** Classify one endpoint's header. Pure. Returns [state, detail]. */
export function verdict(links, perPage = 1) {
const style = paginationStyle(links);
if (style === 'walk-only') {
return [style,
'rel="next" is present and rel="last" is not, so the size of this list is '
+ 'only knowable by walking it. A careful page count says unknown here; '
+ `code that defaults a missing count to 1 reports ${naivePageCount(links)} page.`];
}
if (style === 'indexable') {
const total = itemCount(links, perPage);
return [style,
`rel="last" is present, so this endpoint can be indexed: ${pageCount(links)} `
+ `page(s) at per_page=${perPage}${total ? `, which is ${total} item(s)` : ''}. `
+ 'That number is computed per request and moves between calls, so it is a '
+ 'display value rather than a bound.'];
}
return [style,
'neither rel="next" nor rel="last" is present. One request is the whole list '
+ 'here, and nothing about paging applies.'];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (state === 'walk-only') {
return 'terminate on the absence of rel="next" and never require rel="last". '
+ 'Drop the progress bar or make it indeterminate, and replace any fan-out '
+ 'over a page range with a sequential walk that follows the next URL '
+ 'exactly as given.';
}
if (state === 'indexable') {
return 'nothing, provided rel="last" is treated as a snapshot. Do not cache '
+ 'it as the size of the job, and do not let its absence on some other '
+ 'endpoint default to 1.';
}
return 'nothing.';
}
/** Requests this run will spend against the core quota. Pure. */
export function readCost(paths) {
return Array.isArray(paths) ? paths.length : 0;
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
};
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const repo = process.env.GITHUB_REPO;
if (!token || !repo) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_REPO=owner/name');
process.exitCode = 2;
return;
}
const paths = PROBES.map((name) => `/repos/${repo}/${name}`);
console.log(`read cost: ${readCost(paths)} request(s) against the core hourly quota`);
const findings = [];
for (const path of paths) {
const url = new URL(API + path);
url.searchParams.set('per_page', '1');
const res = await fetch(url, { headers: headers(token) });
if (res.status !== 200) {
console.log(`${path} returned ${res.status}; skipping it`);
continue;
}
const links = parseLink(res.headers.get('link'));
const [state, detail] = verdict(links, 1);
console.log(`${path}: rels ${rels(links).join(', ') || 'none'} -> ${state}`);
console.log(`${state}: ${detail}`);
const missing = unavailable(state);
if (missing.length) console.log(`unavailable here: ${missing.join(', ')}`);
console.log(`repair: ${repair(state)}`);
findings.push({
path,
rels: rels(links),
style: state,
pages: pageCount(links),
pages_if_missing_defaults_to_one: naivePageCount(links),
items: itemCount(links, 1),
capabilities: capabilities(state),
unavailable: missing,
detail,
});
}
console.log(JSON.stringify({ requests_spent: readCost(paths), findings }, null, 2));
process.exitCode = findings.some((f) => f.style === 'walk-only') ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The three-way classification is the thing under test, because collapsing it into a boolean is the mistake the note describes: walk-only has to be its own answer and must never be confused with a single-page list. Beside it, the careful page count and the careless one are asserted to differ on exactly the endpoint where it matters and to agree everywhere else, which is the sentence the script is built to print. The capability table is checked as data, so adding an endpoint shape later cannot quietly grant a pattern that does not work.
from github_rel_last_absent import (
capabilities, item_count, naive_page_count, page_count, page_param,
pagination_style, parse_link, read_cost, rels, repair, unavailable, verdict,
)
BASE = "https://api.github.com/repositories/1/issues"
INDEXABLE = {"next": BASE + "?page=2", "last": BASE + "?page=912"}
WALK_ONLY = {"next": BASE + "?page=2"}
DEEP = {"first": BASE + "?page=1", "prev": BASE + "?page=4",
"next": BASE + "?page=6", "last": BASE + "?page=40"}
SINGLE = {}
def test_the_classification_is_three_states_not_two():
assert pagination_style(INDEXABLE) == "indexable"
assert pagination_style(WALK_ONLY) == "walk-only"
assert pagination_style(SINGLE) == "single-page"
assert pagination_style(None) == "single-page"
def test_walk_only_is_never_mistaken_for_a_single_page():
assert pagination_style(WALK_ONLY) != pagination_style(SINGLE)
assert capabilities("walk-only") != capabilities("single-page")
def test_a_careful_page_count_refuses_to_answer_without_the_field():
assert page_count(INDEXABLE) == 912
assert page_count(DEEP) == 40
assert page_count(SINGLE) == 1
assert page_count(WALK_ONLY) is None
def test_the_careless_page_count_turns_the_gap_into_one():
assert naive_page_count(WALK_ONLY) == 1
assert naive_page_count(INDEXABLE) == 912
assert naive_page_count(WALK_ONLY) != page_count(WALK_ONLY)
assert naive_page_count(INDEXABLE) == page_count(INDEXABLE)
def test_the_item_count_is_only_offered_at_a_page_size_of_one():
assert item_count(INDEXABLE, 1) == 912
assert item_count(INDEXABLE, 100) is None
assert item_count(WALK_ONLY, 1) is None
def test_the_capability_table_says_what_a_pager_may_rely_on():
walk = capabilities("walk-only")
assert walk["walk"] is True
assert walk["page_count"] is False
assert walk["progress_bar"] is False
assert walk["parallel_fanout"] is False
assert walk["jump_to_last"] is False
assert capabilities("indexable")["parallel_fanout"] is True
def test_the_capability_table_is_a_copy_so_a_caller_cannot_edit_it():
capabilities("indexable")["page_count"] = False
assert capabilities("indexable")["page_count"] is True
def test_the_broken_patterns_are_named_in_a_fixed_order():
assert unavailable("walk-only") == ["page count", "progress bar",
"parallel fan-out", "jump to last"]
assert unavailable("indexable") == []
assert unavailable("single-page") == ["parallel fan-out", "jump to last"]
def test_the_walk_only_verdict_prints_the_number_that_moves_somebody():
state, detail = verdict(WALK_ONLY)
assert state == "walk-only"
assert "only knowable by walking it" in detail
assert "reports 1 page" in detail
def test_an_indexable_endpoint_is_reported_as_a_snapshot():
state, detail = verdict(INDEXABLE, 1)
assert state == "indexable"
assert "912" in detail
assert "moves between calls" in detail
def test_a_single_page_list_is_not_a_pagination_finding():
state, detail = verdict(SINGLE)
assert state == "single-page"
assert "nothing about paging applies" in detail
assert repair(state) == "nothing."
def test_the_page_parameter_is_read_out_of_the_url_defensively():
assert page_param(BASE + "?page=7&per_page=1") == 7
assert page_param(BASE + "?per_page=1") is None
assert page_param("") is None
assert page_param(None) is None
def test_the_header_is_parsed_around_commas_inside_urls():
header = ('<%s?labels=bug,ci&page=2>; rel="next", '
'<%s?labels=bug,ci&page=9>; rel="last"' % (BASE, BASE))
assert rels(parse_link(header)) == ["last", "next"]
assert parse_link("") == {}
def test_the_repair_for_walk_only_never_asks_for_a_page_count():
fix = repair("walk-only")
assert 'rel="next"' in fix
assert "never require" in fix
assert "cache it as the size of the job" in repair("indexable")
def test_the_run_says_what_it_will_spend():
assert read_cost(["/a", "/b", "/c", "/d", "/e"]) == 5
assert read_cost([]) == 0
assert read_cost(None) == 0
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
capabilities, itemCount, naivePageCount, pageCount, pageParam,
paginationStyle, parseLink, readCost, rels, repair, unavailable, verdict,
} from './github-rel-last-absent.mjs';
const BASE = 'https://api.github.com/repositories/1/issues';
const INDEXABLE = { next: `${BASE}?page=2`, last: `${BASE}?page=912` };
const WALK_ONLY = { next: `${BASE}?page=2` };
const DEEP = {
first: `${BASE}?page=1`, prev: `${BASE}?page=4`, next: `${BASE}?page=6`, last: `${BASE}?page=40`,
};
const SINGLE = {};
test('the classification is three states, not two', () => {
assert.equal(paginationStyle(INDEXABLE), 'indexable');
assert.equal(paginationStyle(WALK_ONLY), 'walk-only');
assert.equal(paginationStyle(SINGLE), 'single-page');
assert.equal(paginationStyle(null), 'single-page');
});
test('walk-only is never mistaken for a single page', () => {
assert.notEqual(paginationStyle(WALK_ONLY), paginationStyle(SINGLE));
assert.notDeepEqual(capabilities('walk-only'), capabilities('single-page'));
});
test('a careful page count refuses to answer without the field', () => {
assert.equal(pageCount(INDEXABLE), 912);
assert.equal(pageCount(DEEP), 40);
assert.equal(pageCount(SINGLE), 1);
assert.equal(pageCount(WALK_ONLY), null);
});
test('the careless page count turns the gap into one', () => {
assert.equal(naivePageCount(WALK_ONLY), 1);
assert.equal(naivePageCount(INDEXABLE), 912);
assert.notEqual(naivePageCount(WALK_ONLY), pageCount(WALK_ONLY));
assert.equal(naivePageCount(INDEXABLE), pageCount(INDEXABLE));
});
test('the item count is only offered at a page size of one', () => {
assert.equal(itemCount(INDEXABLE, 1), 912);
assert.equal(itemCount(INDEXABLE, 100), null);
assert.equal(itemCount(WALK_ONLY, 1), null);
});
test('the capability table says what a pager may rely on', () => {
const walk = capabilities('walk-only');
assert.equal(walk.walk, true);
assert.equal(walk.page_count, false);
assert.equal(walk.progress_bar, false);
assert.equal(walk.parallel_fanout, false);
assert.equal(walk.jump_to_last, false);
assert.equal(capabilities('indexable').parallel_fanout, true);
});
test('the capability table is a copy so a caller cannot edit it', () => {
capabilities('indexable').page_count = false;
assert.equal(capabilities('indexable').page_count, true);
});
test('the broken patterns are named in a fixed order', () => {
assert.deepEqual(unavailable('walk-only'),
['page count', 'progress bar', 'parallel fan-out', 'jump to last']);
assert.deepEqual(unavailable('indexable'), []);
assert.deepEqual(unavailable('single-page'), ['parallel fan-out', 'jump to last']);
});
test('the walk-only verdict prints the number that moves somebody', () => {
const [state, detail] = verdict(WALK_ONLY);
assert.equal(state, 'walk-only');
assert.match(detail, /only knowable by walking it/);
assert.match(detail, /reports 1 page/);
});
test('an indexable endpoint is reported as a snapshot', () => {
const [state, detail] = verdict(INDEXABLE, 1);
assert.equal(state, 'indexable');
assert.match(detail, /912/);
assert.match(detail, /moves between calls/);
});
test('a single-page list is not a pagination finding', () => {
const [state, detail] = verdict(SINGLE);
assert.equal(state, 'single-page');
assert.match(detail, /nothing about paging applies/);
assert.equal(repair(state), 'nothing.');
});
test('the page parameter is read out of the URL defensively', () => {
assert.equal(pageParam(`${BASE}?page=7&per_page=1`), 7);
assert.equal(pageParam(`${BASE}?per_page=1`), null);
assert.equal(pageParam(''), null);
assert.equal(pageParam(null), null);
});
test('the header is parsed around commas inside URLs', () => {
const header = `<${BASE}?labels=bug,ci&page=2>; rel="next", `
+ `<${BASE}?labels=bug,ci&page=9>; rel="last"`;
assert.deepEqual(rels(parseLink(header)), ['last', 'next']);
assert.deepEqual(parseLink(''), {});
});
test('the repair for walk-only never asks for a page count', () => {
const fix = repair('walk-only');
assert.match(fix, /rel="next"/);
assert.match(fix, /never require/);
assert.match(repair('indexable'), /cache it as the size of the job/);
});
test('the run says what it will spend', () => {
assert.equal(readCost(['/a', '/b', '/c', '/d', '/e']), 5);
assert.equal(readCost([]), 0);
assert.equal(readCost(null), 0);
});
FAQ
Which endpoints omit rel=last?
It is not a fixed list, which is the practical reason to measure rather than memorise. GitHub includes the field when it can compute a final page for that request, and the calculation depends on the endpoint, the filters and how the underlying result set is counted. High-volume feeds and endpoints that page by cursor are the usual ones, but a list that is indexable with one set of query parameters can stop being indexable with another. Probe the exact calls your integration makes, which is one request each.
Is this not just the Link header note again?
No. That note is about a client that never read the header and reported the first page as the whole list. This one is about a client that reads it, follows it correctly, and additionally requires a field the header does not always carry. The two produce a similar-looking under-count and the code that causes them looks nothing alike: one has no pagination at all, the other has pagination plus a dependency on a page count. If your pager already follows rel="next" to the end, that note is closed for you and this one may not be.
What should a progress bar do on a walk-only endpoint?
Be indeterminate, and say so. A spinner with a running item count is honest; a bar at 4% that was computed from a number defaulted to one is worse than no bar, because it will sit at 4% and then jump to done. If you truly need a denominator, the answer is usually a different endpoint or a search query with a total_count, and both of those come with caveats of their own about what that total actually describes.
Can I cache rel=last as the size of the job?
Not safely, even on an endpoint that provides it. The final page number is computed per request, so it is a snapshot of a collection that other people are still writing to. Between page one and page nine issues get opened, branches get deleted, and the number you captured stops being the number. Use it for display, terminate on the absence of rel="next", and let the loop discover the actual end.
How much quota does the check itself cost?
One request per path probed, five by default, against the hourly core bucket. The probe uses per_page=1 because the finding is entirely in a response header and none of the items are needed, and that also makes the page number in rel="last" equal to the exact item count wherever the field is present. The script prints the total before it spends anything, so widening the probe list is a decision you make with the number in front of you.
Related field notes
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using pagination in the REST API — GitHub Docs
- octokit/plugin-paginate-rest.js
- Repositories — GitHub REST API
- Best practices for using the REST API — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.