Diagnostic GitHub API
GraphQL search stops at the same 1,000 results as REST
The REST search was hitting the thousand-result ceiling, loudly, with a 422 on page eleven that nobody could miss. So the query got rewritten in GraphQL, which is the modern API, has cursors instead of page numbers, and does not document a page limit anywhere obvious. The rewrite works. It runs to completion with no error at all, hasNextPage turns false, the loop exits cleanly, and it has collected 1,000 of the 18,231 issues that issueCount is reporting in the very same response.
The ceiling belongs to the search index, not to the protocol. GraphQL's search connection is served by the same index as GET /search/* and inherits the same limit of about 1,000 retrievable results per query. Changing API changed nothing about how many results you can reach.
What did change is how the truncation announces itself. REST refuses page 11 with 422 Validation Failed and the words “Only the first 1000 search results are available”, which is impossible to miss. GraphQL just stops: pageInfo.hasNextPage becomes false after roughly ten pages of 100 and the walk terminates normally, with issueCount still reporting the full match count beside it. Compare those two numbers on every search you paginate, and where you need everything rather than the best matches, use a typed connection — repository.issues, organization.repositories — which paginates with no ceiling at all.
The problem in plain words
This one is a reward for doing the right thing. Somebody hit the REST ceiling, read the note, understood that partitioning the query is the answer, and then noticed that GitHub has a second API which is newer, is cursor-based, and whose search field says nothing about a thousand of anything. Migrating looks like the structural fix and partitioning looks like the workaround, so the migration gets scheduled and the workaround gets dropped.
The migrated code is then better in every visible way. It asks for exactly the fields it wants, it costs one point a page instead of a search-bucket request, it uses cursors so pages cannot slip, and it has a proper termination condition instead of a page counter. It also silently returns six per cent of the data, and every property that makes it better makes the loss harder to see: there is no error to catch, no status code to check, and the loop ends the way a correct loop ends.
The detail that keeps it hidden for months is that issueCount is right. The response tells you there are 18,231 matches and hands you 1,000 of them, in the same object, and nothing anywhere compares the two. Dashboards built on this do not look empty; they look plausible and stable, because the truncation is deterministic. The first person to notice is usually someone who knows a specific issue exists and cannot find it in the export.
Why it happens
The limit lives in the index. Search on GitHub is a separate system from the object store, and it serves ranked results with a cap on how deep into the ranking you may page. Both APIs are front ends to it. That is why the number is the same on both, why it is not a page-size problem, and why no combination of first and after gets past it: you are asking the index for its 1,001st best match and it does not serve one.
Silent truncation is the actual difference between the two notes. The REST ceiling has its own note and it is a note about an error: total_count is huge, page 11 returns 422, and the repair is to partition. This note is about the same ceiling arriving with no error at all. A client that treats hasNextPage: false as “there is no more data” — which is exactly what the pagination documentation tells you to do — cannot distinguish a complete answer from a truncated one without looking at issueCount.
issueCount is a match count, not a fetch count. It reports how many documents the index matched, the same way total_count does in REST, and it keeps reporting the full number after the ceiling has cut you off. That makes it the single most useful field in the response for this problem: the gap between issueCount and the number of nodes you actually collected is the size of what you did not get, and it is available on page one, before you spend anything.
GraphQL has one escape REST does not. Typed connections are not search: repository.issues, repository.pullRequests, organization.repositories, repository.discussions read the object store directly and paginate without a ceiling. If what you want is an inventory rather than a ranking — every issue in this repository, every repository in this org — the search connection is the wrong tool and swapping to a typed connection removes the limit rather than working around it. Only the queries that genuinely need cross-repository search terms have to be partitioned.
Partitioning works the same as it does in REST, with better ergonomics. Slice by created: date ranges, by repo:, by label, until every slice reports an issueCount under 1,000, then union them. GraphQL lets you run several slices in one document as aliased fields, which costs fewer requests than the REST equivalent, though each slice still counts toward the point cost and each still has its own ceiling.
This is not the incomplete-results flag and not the node limit. A timed-out search returns partial results with a flag set and is non-deterministic; this ceiling is exact and repeatable. And the node limit rejects a query for its shape before it runs, which is loud and happens at 500,000 rather than 1,000. The script names which of the three it is looking at so none of them gets the wrong repair.
The fix, as a flow
The walk itself is four lines and the arithmetic around it is the note. Reachable and unreachable halves of a match count, pages that fit under the ceiling, slices a partition needs, and the typed connection that answers the same question without a ceiling are all computed rather than fetched. The live part pages the search connection with after: endCursor until it stops, then compares what it collected against what the index said it matched. That comparison is the only thing separating a truncated answer from a complete one, because both of them end the same way and neither raises an error.
How to fix it
Read issueCount on the first page and compare it to nothing else
One page, one point. If issueCount is above 1,000 you already know the tail is unreachable through this connection, before you have paged anywhere. That single comparison is the cheapest version of this whole audit, and it is the one to put in the client permanently.
Walk the cursor until it stops and record where it stopped
The script pages with after: endCursor at 100 a page and keeps count. Ten or eleven pages is enough: the point is to watch hasNextPage turn false while issueCount is still enormous. It prints the point cost before it starts, because a page is a point and a long walk is a real spend.
Name the shape of the stop
Pagination ending with a full collection and a large issueCount is the ceiling. Pagination ending early with a small count is a complete answer. Pagination that never ends is a client bug. The script sorts the run into one of those rather than reporting “done”, and it says explicitly that no error was raised, because the absence of an error is the thing people find hardest to believe.
Decide between partitioning and abandoning search
If the goal is an inventory, the script names the typed connection that answers the same question with no ceiling, and that is the better repair. If the goal is genuinely a search, it computes how many slices the match count needs and suggests the axis to slice on. Those are different repairs and picking the wrong one costs a week.
Put the comparison in the client, not in a runbook
The permanent fix is two lines: request issueCount alongside nodes, and refuse to publish a result set whose length is below it without labelling it truncated. A walk that trusts hasNextPage alone will keep terminating cleanly on an incomplete answer for as long as the query is popular enough to match a thousand things.
How to check it worked
Once the walk compares what it collected against what the index matched, the run reports the ceiling instead of terminating quietly on top of it.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_search_ceiling.py \
--query "org:acme is:issue is:open" --type ISSUE --max-pages 11
# point cost: up to 11 point(s) against the 5,000/hour GraphQL budget
# page 1: 100 node(s), collected=100, matches=18231, hasNextPage=yes
# page 10: 100 node(s), collected=1000, matches=18231, hasNextPage=no
# ceiling-hit-silently: pagination stopped after 1000 node(s) with the index
# reporting 18231 match(es). No error was raised and hasNextPage simply turned
# false.
# reachable: 1000 unreachable: 17231
# the REST twin of this stop is 422 Validation Failed on page 11 [...]
# here it is no error at all. pageInfo.hasNextPage turns false and the walk
# terminates the way a complete walk terminates.
# repair: for an inventory use the typed connection repository.issues or
# repository.pullRequests, which has no ceiling. For a genuine search,
# partition into at least 19 slice(s) by created: date range and union them.The full code
The walk is small and the arithmetic around it is the note. Everything that decides anything is pure: the reachable and unreachable halves of a match count, the number of pages that fit under the ceiling, the classification of how a walk ended, the number of slices a partition needs, and the typed connection that answers the same question without a ceiling. The live part sends one search query per page with after: endCursor, refuses any document that is not a read, and prints the maximum it might spend before it spends any of it.
"""Show that GraphQL search stops at the same 1,000 results REST does.
Read only, and queries only. GitHub's GraphQL endpoint takes its document in
the request body, so a read travels by POST there exactly as a write would;
that is a transport detail, not a licence to write. This script parses the
document it is about to send and refuses to open a socket if it contains a
mutation or a subscription.
GraphQL's search connection is served by the same index as GET /search/*, and
inherits the same ceiling of roughly 1,000 retrievable results per query. The
difference is how it says so: REST returns 422 Validation Failed on page 11,
GraphQL just sets pageInfo.hasNextPage to false and lets the walk finish
normally with issueCount still reporting the full match count beside it.
What this can and cannot see: the API cannot tell you whether your client
compares the two numbers. What it can do is walk the connection until it stops,
report where it stopped against what the index matched, and say plainly that no
error was raised.
Environment:
GITHUB_TOKEN a token with read access to the GraphQL API
"""
import argparse
import json
import logging
import math
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_graphql_search_ceiling")
API = "https://api.github.com"
UA = "github-graphql-search-ceiling/1.0"
# The retrievable-result ceiling on the search index. Identical on both APIs
# because it is a property of the index rather than of the protocol.
SEARCH_RESULT_CEILING = 1000
# The largest page any GraphQL connection will serve.
MAX_PAGE_SIZE = 100
# One search connection at first: 100 costs one point.
POINTS_PER_PAGE = 1
SEARCH_QUERY = (
"query($q: String!, $type: SearchType!, $after: String) {"
" search(query: $q, type: $type, first: 100, after: $after) {"
" issueCount repositoryCount userCount"
" pageInfo { hasNextPage endCursor }"
" nodes { __typename } } }"
)
# The connections that answer an inventory question without going through the
# index at all, and therefore without a ceiling.
TYPED_CONNECTIONS = {
"ISSUE": "repository.issues or repository.pullRequests",
"REPOSITORY": "organization.repositories or user.repositories",
"USER": "organization.membersWithRole",
"DISCUSSION": "repository.discussions",
}
def strip_noise(document):
"""Remove GraphQL comments and string literals from a document. Pure."""
src = str(document or "")
out = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "#":
while i < n and src[i] != "\n":
i += 1
continue
if src.startswith('"""', i):
j = src.find('"""', i + 3)
i = n if j < 0 else j + 3
out.append(" ")
continue
if ch == '"':
i += 1
while i < n and src[i] != '"':
i += 2 if src[i] == "\\" else 1
i += 1
out.append(" ")
continue
out.append(ch)
i += 1
return "".join(out)
def operations(document):
"""The top-level operations in a document, in order. Pure."""
src = strip_noise(document)
ops, depth, word, declared = [], 0, "", None
for ch in src + " ":
if ch.isalnum() or ch == "_":
word += ch
continue
if word:
if depth == 0 and word in ("query", "mutation", "subscription", "fragment"):
declared = word
word = ""
if ch == "{":
if depth == 0:
ops.append(declared or "query")
declared = None
depth += 1
elif ch == "}":
depth = max(0, depth - 1)
return ops
def refusal(document):
"""Why this document will not be sent, or None if it is a read. Pure."""
ops = operations(document)
if not ops:
return "the document contains no operation to send."
for kind in ("mutation", "subscription"):
if kind in ops:
return ("the document contains a %s. This script sends queries only: "
"a query is a read, and the section it belongs to promises "
"its scripts never write." % kind)
return None
def reachable(total):
"""How many of the matches can actually be paged to. Pure."""
try:
total = max(0, int(total))
except (TypeError, ValueError):
return 0
return min(total, SEARCH_RESULT_CEILING)
def unreachable(total):
"""How many matches exist that no cursor will ever reach. Pure."""
try:
total = max(0, int(total))
except (TypeError, ValueError):
return 0
return max(0, total - SEARCH_RESULT_CEILING)
def pages_to_ceiling(page_size=MAX_PAGE_SIZE):
"""How many pages of this size fit under the ceiling. Pure."""
size = min(max(1, int(page_size or 1)), MAX_PAGE_SIZE)
return math.ceil(SEARCH_RESULT_CEILING / size)
def slices_needed(total):
"""How many under-the-ceiling slices a partition needs. Pure."""
try:
total = max(0, int(total))
except (TypeError, ValueError):
return 0
return math.ceil(total / SEARCH_RESULT_CEILING) if total else 0
def typed_connection_for(search_type):
"""The ceiling-free connection that answers the same question. Pure."""
return TYPED_CONNECTIONS.get(str(search_type or "").upper(),
"the typed connection for this object type")
def truncation_signal(protocol):
"""How each API announces the same ceiling. Pure.
Kept as data because the whole note is the difference between these two
sentences, and a reader who has only ever seen the first one does not
believe the second until it is written down next to it.
"""
if str(protocol).lower() == "rest":
return ("422 Validation Failed on page 11, with the message "
"\"Only the first 1000 search results are available\".")
return ("no error at all. pageInfo.hasNextPage turns false and the walk "
"terminates the way a complete walk terminates.")
def classify_walk(total, collected, has_next_page, pages_walked, max_pages):
"""How this walk ended, and whether that ending was honest. Pure.
Returns (state, detail). The two states that both end with hasNextPage
false are the point: one of them is a complete answer and one of them is
six per cent of an answer, and they are the same shape of response.
"""
total = reachable(total) + unreachable(total)
collected = max(0, int(collected or 0))
if has_next_page and pages_walked >= max_pages:
return ("stopped-early-by-request",
"the walk stopped at the --max-pages limit with more pages "
"available, so nothing about the ceiling is proved yet. "
"%d of %d node(s) collected." % (collected, total))
if has_next_page:
return ("still-paging",
"the connection still reports another page. Keep walking or "
"raise --max-pages.")
if collected >= SEARCH_RESULT_CEILING and total > collected:
return ("ceiling-hit-silently",
"pagination stopped after %d node(s) with the index reporting "
"%d match(es). No error was raised and hasNextPage simply "
"turned false." % (collected, total))
if total > collected:
return ("truncated-early",
"the walk ended with %d of %d match(es) and below the ceiling, "
"which is not this note: check for a timed-out search or a "
"filter applied after the count." % (collected, total))
return ("complete",
"%d node(s) collected against %d match(es). This query is under "
"the ceiling and the answer is whole." % (collected, total))
def repair(state, total, search_type):
"""The sentence a reader has to act on. Pure."""
if state == "ceiling-hit-silently":
return ("for an inventory use the typed connection %s, which has no "
"ceiling. For a genuine search, partition into at least %d "
"slice(s) by created: date range and union them."
% (typed_connection_for(search_type), slices_needed(total)))
if state == "truncated-early":
return ("see /github/search-incomplete-results/ -- a search that ends "
"below the ceiling ended for a different reason, and that one "
"is not deterministic.")
if state == "still-paging":
return ("nothing yet. Walk to the end, or read issueCount on page one "
"and compare it against %d." % SEARCH_RESULT_CEILING)
if state == "stopped-early-by-request":
return ("raise --max-pages to at least %d to reach the ceiling, or "
"trust issueCount, which already tells you."
% pages_to_ceiling())
return ("request issueCount alongside nodes and refuse to publish a result "
"set shorter than it without labelling it truncated.")
def point_cost(max_pages):
"""The most this run can spend against the hourly budget. Pure."""
return max(0, int(max_pages or 0)) * POINTS_PER_PAGE
def run_query(session, document, variables):
"""Send one search page. Returns (status, body-or-None).
A GraphQL query is a read; POST is only how the document reaches the
endpoint, which is why the verb is written here beside the URL rather than
hidden in a constant where it could be mistaken for a write path.
"""
r = session.post(API + "/graphql",
json={"query": document, "variables": variables or {}},
timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
try:
return r.status_code, r.json()
except ValueError:
return r.status_code, None
def match_count(search, search_type):
"""The index's own count for this search type. Pure."""
if not isinstance(search, dict):
return 0
key = {"ISSUE": "issueCount", "REPOSITORY": "repositoryCount",
"USER": "userCount"}.get(str(search_type or "").upper(), "issueCount")
return int(search.get(key) or 0)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--query", required=True,
help="the search string, e.g. 'org:acme is:issue is:open'")
ap.add_argument("--type", default="ISSUE",
choices=["ISSUE", "REPOSITORY", "USER", "DISCUSSION"],
help="the GraphQL SearchType to use")
ap.add_argument("--max-pages", type=int, default=11,
help="how many pages of 100 to walk. Ten reaches the "
"ceiling; eleven proves the walk stops there.")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
why_not = refusal(SEARCH_QUERY)
if why_not:
log.error("refusing to send: %s", why_not)
return 2
log.info("point cost: up to %d point(s) against the 5,000/hour GraphQL budget",
point_cost(args.max_pages))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
cursor, collected, total, pages = None, 0, 0, 0
has_next = False
while pages < args.max_pages:
status, body = run_query(session, SEARCH_QUERY,
{"q": args.query, "type": args.type,
"after": cursor})
if not isinstance(body, dict) or body.get("errors"):
log.error("the search itself failed: HTTP %s %s", status,
json.dumps((body or {}).get("errors", []))[:300])
return 2
search = (body.get("data") or {}).get("search") or {}
nodes = search.get("nodes") or []
info = search.get("pageInfo") or {}
pages += 1
collected += len(nodes)
total = match_count(search, args.type)
has_next = bool(info.get("hasNextPage"))
cursor = info.get("endCursor")
log.info("page %d: %d node(s), collected=%d, matches=%d, hasNextPage=%s",
pages, len(nodes), collected, total, "yes" if has_next else "no")
if not has_next:
break
state, detail = classify_walk(total, collected, has_next, pages, args.max_pages)
log.info("%s: %s", state, detail)
log.info("reachable: %d unreachable: %d", reachable(total), unreachable(total))
log.info("the REST twin of this stop is %s", truncation_signal("rest"))
log.info("here it is %s", truncation_signal("graphql"))
log.info("repair: %s", repair(state, total, args.type))
print(json.dumps({
"points_spent": pages * POINTS_PER_PAGE,
"search": args.query,
"type": args.type,
"matches": total,
"collected": collected,
"pages_walked": pages,
"has_next_page": has_next,
"reachable": reachable(total),
"unreachable": unreachable(total),
"slices_needed": slices_needed(total),
"typed_connection": typed_connection_for(args.type),
"state": state,
"detail": detail,
"repair": repair(state, total, args.type),
}, indent=2, default=str))
return 1 if state == "ceiling-hit-silently" else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Show that GraphQL search stops at the same 1,000 results REST does.
*
* Read only, and queries only. GitHub's GraphQL endpoint takes its document in
* the request body, so a read travels by POST there exactly as a write would.
* The document is parsed first and anything containing a mutation or a
* subscription is refused before a socket opens.
*
* The search connection is served by the same index as GET /search/* and
* inherits the same ceiling of roughly 1,000 retrievable results. REST says so
* with 422 Validation Failed on page 11; GraphQL just sets hasNextPage to
* false and lets the walk finish normally.
*
* Environment:
* GITHUB_TOKEN a token with read access to the GraphQL API
* GITHUB_SEARCH the search string
* GITHUB_SEARCH_TYPE ISSUE, REPOSITORY, USER or DISCUSSION
* GITHUB_MAX_PAGES how many pages of 100 to walk
*/
const API = 'https://api.github.com';
const UA = 'github-graphql-search-ceiling/1.0';
/** The retrievable-result ceiling on the search index, both APIs alike. */
export const SEARCH_RESULT_CEILING = 1000;
/** The largest page any GraphQL connection will serve. */
export const MAX_PAGE_SIZE = 100;
/** One search connection at first: 100 costs one point. */
export const POINTS_PER_PAGE = 1;
const SEARCH_QUERY = 'query($q: String!, $type: SearchType!, $after: String) {'
+ ' search(query: $q, type: $type, first: 100, after: $after) {'
+ ' issueCount repositoryCount userCount'
+ ' pageInfo { hasNextPage endCursor }'
+ ' nodes { __typename } } }';
/** Connections that answer an inventory question without the index. */
export const TYPED_CONNECTIONS = {
ISSUE: 'repository.issues or repository.pullRequests',
REPOSITORY: 'organization.repositories or user.repositories',
USER: 'organization.membersWithRole',
DISCUSSION: 'repository.discussions',
};
/** Remove GraphQL comments and string literals from a document. Pure. */
export function stripNoise(document) {
const src = String(document ?? '');
const out = [];
let i = 0;
while (i < src.length) {
const ch = src[i];
if (ch === '#') {
while (i < src.length && src[i] !== '\n') i += 1;
continue;
}
if (src.startsWith('"""', i)) {
const j = src.indexOf('"""', i + 3);
i = j < 0 ? src.length : j + 3;
out.push(' ');
continue;
}
if (ch === '"') {
i += 1;
while (i < src.length && src[i] !== '"') i += src[i] === '\\' ? 2 : 1;
i += 1;
out.push(' ');
continue;
}
out.push(ch);
i += 1;
}
return out.join('');
}
/** The top-level operations in a document, in order. Pure. */
export function operations(document) {
const src = `${stripNoise(document)} `;
const ops = [];
let depth = 0;
let word = '';
let declared = null;
for (const ch of src) {
if (/[A-Za-z0-9_]/.test(ch)) { word += ch; continue; }
if (word) {
if (depth === 0 && ['query', 'mutation', 'subscription', 'fragment'].includes(word)) {
declared = word;
}
word = '';
}
if (ch === '{') {
if (depth === 0) { ops.push(declared || 'query'); declared = null; }
depth += 1;
} else if (ch === '}') {
depth = Math.max(0, depth - 1);
}
}
return ops;
}
/** Why this document will not be sent, or null if it is a read. Pure. */
export function refusal(document) {
const ops = operations(document);
if (ops.length === 0) return 'the document contains no operation to send.';
for (const kind of ['mutation', 'subscription']) {
if (ops.includes(kind)) {
return `the document contains a ${kind}. This script sends queries only: `
+ 'a query is a read, and the section it belongs to promises its '
+ 'scripts never write.';
}
}
return null;
}
function asCount(total) {
const n = Number(total);
return Number.isFinite(n) ? Math.max(0, Math.trunc(n)) : 0;
}
/** How many of the matches can actually be paged to. Pure. */
export function reachable(total) {
return Math.min(asCount(total), SEARCH_RESULT_CEILING);
}
/** How many matches exist that no cursor will ever reach. Pure. */
export function unreachable(total) {
return Math.max(0, asCount(total) - SEARCH_RESULT_CEILING);
}
/** How many pages of this size fit under the ceiling. Pure. */
export function pagesToCeiling(pageSize = MAX_PAGE_SIZE) {
const size = Math.min(Math.max(1, Math.trunc(Number(pageSize) || 1)), MAX_PAGE_SIZE);
return Math.ceil(SEARCH_RESULT_CEILING / size);
}
/** How many under-the-ceiling slices a partition needs. Pure. */
export function slicesNeeded(total) {
const n = asCount(total);
return n ? Math.ceil(n / SEARCH_RESULT_CEILING) : 0;
}
/** The ceiling-free connection that answers the same question. Pure. */
export function typedConnectionFor(searchType) {
const key = String(searchType ?? '').toUpperCase();
return TYPED_CONNECTIONS[key] || 'the typed connection for this object type';
}
/** How each API announces the same ceiling. Pure. */
export function truncationSignal(protocol) {
if (String(protocol).toLowerCase() === 'rest') {
return '422 Validation Failed on page 11, with the message "Only the first '
+ '1000 search results are available".';
}
return 'no error at all. pageInfo.hasNextPage turns false and the walk '
+ 'terminates the way a complete walk terminates.';
}
/** How this walk ended, and whether that ending was honest. Pure. */
export function classifyWalk(total, collected, hasNextPage, pagesWalked, maxPages) {
const matches = asCount(total);
const got = asCount(collected);
if (hasNextPage && pagesWalked >= maxPages) {
return ['stopped-early-by-request', 'the walk stopped at the --max-pages '
+ 'limit with more pages available, so nothing about the ceiling is '
+ `proved yet. ${got} of ${matches} node(s) collected.`];
}
if (hasNextPage) {
return ['still-paging', 'the connection still reports another page. Keep '
+ 'walking or raise --max-pages.'];
}
if (got >= SEARCH_RESULT_CEILING && matches > got) {
return ['ceiling-hit-silently', `pagination stopped after ${got} node(s) `
+ `with the index reporting ${matches} match(es). No error was raised and `
+ 'hasNextPage simply turned false.'];
}
if (matches > got) {
return ['truncated-early', `the walk ended with ${got} of ${matches} `
+ 'match(es) and below the ceiling, which is not this note: check for a '
+ 'timed-out search or a filter applied after the count.'];
}
return ['complete', `${got} node(s) collected against ${matches} match(es). `
+ 'This query is under the ceiling and the answer is whole.'];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state, total, searchType) {
if (state === 'ceiling-hit-silently') {
return `for an inventory use the typed connection ${typedConnectionFor(searchType)}, `
+ `which has no ceiling. For a genuine search, partition into at least `
+ `${slicesNeeded(total)} slice(s) by created: date range and union them.`;
}
if (state === 'truncated-early') {
return 'see /github/search-incomplete-results/ -- a search that ends below '
+ 'the ceiling ended for a different reason, and that one is not '
+ 'deterministic.';
}
if (state === 'still-paging') {
return 'nothing yet. Walk to the end, or read issueCount on page one and '
+ `compare it against ${SEARCH_RESULT_CEILING}.`;
}
if (state === 'stopped-early-by-request') {
return `raise --max-pages to at least ${pagesToCeiling()} to reach the `
+ 'ceiling, or trust issueCount, which already tells you.';
}
return 'request issueCount alongside nodes and refuse to publish a result set '
+ 'shorter than it without labelling it truncated.';
}
/** The most this run can spend against the hourly budget. Pure. */
export function pointCost(maxPages) {
return asCount(maxPages) * POINTS_PER_PAGE;
}
/** The index's own count for this search type. Pure. */
export function matchCount(search, searchType) {
if (!search || typeof search !== 'object') return 0;
const key = { ISSUE: 'issueCount', REPOSITORY: 'repositoryCount', USER: 'userCount' }[
String(searchType ?? '').toUpperCase()] || 'issueCount';
return asCount(search[key]);
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
'User-Agent': UA,
};
}
async function runQuery(token, document, variables) {
const res = await fetch(`${API}/graphql`, {
// A GraphQL query is a read. POST is only how the document reaches the
// endpoint, and refusal() has already rejected anything that is not a read.
method: 'POST',
headers: headers(token),
body: JSON.stringify({ query: document, variables: variables || {} }),
});
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, body };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const search = process.env.GITHUB_SEARCH;
if (!token || !search) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_SEARCH');
process.exitCode = 2;
return;
}
const type = (process.env.GITHUB_SEARCH_TYPE || 'ISSUE').toUpperCase();
const maxPages = Number(process.env.GITHUB_MAX_PAGES || 11);
const whyNot = refusal(SEARCH_QUERY);
if (whyNot) {
console.error(`refusing to send: ${whyNot}`);
process.exitCode = 2;
return;
}
console.log(`point cost: up to ${pointCost(maxPages)} point(s) against the `
+ '5,000/hour GraphQL budget');
let cursor = null;
let collected = 0;
let total = 0;
let pages = 0;
let hasNext = false;
while (pages < maxPages) {
// eslint-disable-next-line no-await-in-loop
const { status, body } = await runQuery(token, SEARCH_QUERY,
{ q: search, type, after: cursor });
if (!body || body.errors) {
console.error(`the search itself failed: HTTP ${status} `
+ `${JSON.stringify((body || {}).errors || []).slice(0, 300)}`);
process.exitCode = 2;
return;
}
const node = ((body.data || {}).search) || {};
const nodes = node.nodes || [];
const info = node.pageInfo || {};
pages += 1;
collected += nodes.length;
total = matchCount(node, type);
hasNext = Boolean(info.hasNextPage);
cursor = info.endCursor;
console.log(`page ${pages}: ${nodes.length} node(s), collected=${collected}, `
+ `matches=${total}, hasNextPage=${hasNext ? 'yes' : 'no'}`);
if (!hasNext) break;
}
const [state, detail] = classifyWalk(total, collected, hasNext, pages, maxPages);
console.log(`${state}: ${detail}`);
console.log(`reachable: ${reachable(total)} unreachable: ${unreachable(total)}`);
console.log(`the REST twin of this stop is ${truncationSignal('rest')}`);
console.log(`here it is ${truncationSignal('graphql')}`);
console.log(`repair: ${repair(state, total, type)}`);
console.log(JSON.stringify({
points_spent: pages * POINTS_PER_PAGE,
search,
type,
matches: total,
collected,
pages_walked: pages,
has_next_page: hasNext,
reachable: reachable(total),
unreachable: unreachable(total),
slices_needed: slicesNeeded(total),
typed_connection: typedConnectionFor(type),
state,
detail,
}, null, 2));
process.exitCode = state === 'ceiling-hit-silently' ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The interesting assertions are all about endings that look the same. A walk that collected 1,000 of 18,231 and a walk that collected 40 of 40 both finish with hasNextPage false, and the suite pins the classifier on exactly that pair before it checks anything else. Around it sit the arithmetic — reachable and unreachable halves, pages to the ceiling, slices for a partition — the typed connection each search type should have used instead, and the two truncation signals kept side by side so the REST wording and the GraphQL silence stay distinguishable. The refusal guard is tested last, on the document this script actually sends.
from github_graphql_search_ceiling import (
MAX_PAGE_SIZE, SEARCH_RESULT_CEILING, classify_walk, match_count,
operations, pages_to_ceiling, point_cost, reachable, refusal, repair,
slices_needed, truncation_signal, typed_connection_for, unreachable,
)
def test_the_ceiling_is_a_property_of_the_index_not_the_page_size():
assert SEARCH_RESULT_CEILING == 1000
assert MAX_PAGE_SIZE == 100
assert pages_to_ceiling(100) == 10
assert pages_to_ceiling(30) == 34
assert pages_to_ceiling(1) == 1000
def test_a_match_count_splits_into_a_reachable_and_an_unreachable_half():
assert reachable(18231) == 1000
assert unreachable(18231) == 17231
assert reachable(400) == 400
assert unreachable(400) == 0
assert reachable(None) == 0
assert unreachable("not a number") == 0
def test_the_ceiling_stop_and_a_complete_walk_are_the_same_shape():
hit, detail = classify_walk(18231, 1000, False, 10, 11)
done, _ = classify_walk(40, 40, False, 1, 11)
assert hit == "ceiling-hit-silently"
assert done == "complete"
# Both ended with hasNextPage false and no error. That is the note.
assert "No error was raised" in detail
assert "18231" in detail
def test_a_walk_cut_short_by_the_operator_proves_nothing():
state, detail = classify_walk(18231, 500, True, 5, 5)
assert state == "stopped-early-by-request"
assert "nothing about the ceiling is proved" in detail
assert "at least 10" in repair(state, 18231, "ISSUE")
def test_a_walk_still_going_is_not_a_finding():
state, _ = classify_walk(18231, 300, True, 3, 11)
assert state == "still-paging"
def test_ending_below_the_ceiling_is_a_different_note():
state, detail = classify_walk(900, 640, False, 7, 11)
assert state == "truncated-early"
assert "not this note" in detail
assert "search-incomplete-results" in repair(state, 900, "ISSUE")
def test_the_repair_names_a_ceiling_free_connection_and_a_slice_count():
fix = repair("ceiling-hit-silently", 18231, "ISSUE")
assert "repository.issues" in fix
assert "19 slice(s)" in fix
assert "organization.repositories" in repair(
"ceiling-hit-silently", 4000, "REPOSITORY")
def test_a_partition_needs_one_slice_per_thousand_matches():
assert slices_needed(18231) == 19
assert slices_needed(1000) == 1
assert slices_needed(1001) == 2
assert slices_needed(0) == 0
def test_every_search_type_has_a_connection_that_has_no_ceiling():
assert "repository.issues" in typed_connection_for("ISSUE")
assert "organization.repositories" in typed_connection_for("repository")
assert "membersWithRole" in typed_connection_for("USER")
assert "discussions" in typed_connection_for("DISCUSSION")
assert typed_connection_for("SOMETHING_ELSE").startswith("the typed connection")
def test_the_two_apis_announce_the_same_ceiling_differently():
assert "422" in truncation_signal("rest")
assert "1000 search results" in truncation_signal("rest")
assert "no error at all" in truncation_signal("graphql")
assert "hasNextPage" in truncation_signal("graphql")
def test_the_count_field_follows_the_search_type():
search = {"issueCount": 18231, "repositoryCount": 12, "userCount": 3}
assert match_count(search, "ISSUE") == 18231
assert match_count(search, "REPOSITORY") == 12
assert match_count(search, "USER") == 3
assert match_count(None, "ISSUE") == 0
def test_the_run_says_the_most_it_can_spend():
assert point_cost(11) == 11
assert point_cost(0) == 0
assert point_cost(None) == 0
def test_the_document_this_script_sends_is_a_read():
assert operations("query($q: String!) { search(query: $q, type: ISSUE, first: 100) { issueCount } }") == ["query"]
assert refusal("mutation M { addStar(input: {}) { clientMutationId } }")
assert refusal("subscription S { thing { id } }")
assert refusal("") == "the document contains no operation to send."
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
MAX_PAGE_SIZE, SEARCH_RESULT_CEILING, classifyWalk, matchCount, operations,
pagesToCeiling, pointCost, reachable, refusal, repair, slicesNeeded,
truncationSignal, typedConnectionFor, unreachable,
} from './github-graphql-search-ceiling.mjs';
test('the ceiling is a property of the index not the page size', () => {
assert.equal(SEARCH_RESULT_CEILING, 1000);
assert.equal(MAX_PAGE_SIZE, 100);
assert.equal(pagesToCeiling(100), 10);
assert.equal(pagesToCeiling(30), 34);
assert.equal(pagesToCeiling(1), 1000);
});
test('a match count splits into a reachable and an unreachable half', () => {
assert.equal(reachable(18231), 1000);
assert.equal(unreachable(18231), 17231);
assert.equal(reachable(400), 400);
assert.equal(unreachable(400), 0);
assert.equal(reachable(null), 0);
assert.equal(unreachable('not a number'), 0);
});
test('the ceiling stop and a complete walk are the same shape', () => {
const [hit, detail] = classifyWalk(18231, 1000, false, 10, 11);
const [done] = classifyWalk(40, 40, false, 1, 11);
assert.equal(hit, 'ceiling-hit-silently');
assert.equal(done, 'complete');
assert.match(detail, /No error was raised/);
assert.match(detail, /18231/);
});
test('a walk cut short by the operator proves nothing', () => {
const [state, detail] = classifyWalk(18231, 500, true, 5, 5);
assert.equal(state, 'stopped-early-by-request');
assert.match(detail, /nothing about the ceiling is proved/);
assert.match(repair(state, 18231, 'ISSUE'), /at least 10/);
});
test('a walk still going is not a finding', () => {
assert.equal(classifyWalk(18231, 300, true, 3, 11)[0], 'still-paging');
});
test('ending below the ceiling is a different note', () => {
const [state, detail] = classifyWalk(900, 640, false, 7, 11);
assert.equal(state, 'truncated-early');
assert.match(detail, /not this note/);
assert.match(repair(state, 900, 'ISSUE'), /search-incomplete-results/);
});
test('the repair names a ceiling free connection and a slice count', () => {
const fix = repair('ceiling-hit-silently', 18231, 'ISSUE');
assert.match(fix, /repository\.issues/);
assert.match(fix, /19 slice\(s\)/);
assert.match(repair('ceiling-hit-silently', 4000, 'REPOSITORY'),
/organization\.repositories/);
});
test('a partition needs one slice per thousand matches', () => {
assert.equal(slicesNeeded(18231), 19);
assert.equal(slicesNeeded(1000), 1);
assert.equal(slicesNeeded(1001), 2);
assert.equal(slicesNeeded(0), 0);
});
test('every search type has a connection that has no ceiling', () => {
assert.match(typedConnectionFor('ISSUE'), /repository\.issues/);
assert.match(typedConnectionFor('repository'), /organization\.repositories/);
assert.match(typedConnectionFor('USER'), /membersWithRole/);
assert.match(typedConnectionFor('DISCUSSION'), /discussions/);
assert.ok(typedConnectionFor('SOMETHING_ELSE').startsWith('the typed connection'));
});
test('the two apis announce the same ceiling differently', () => {
assert.match(truncationSignal('rest'), /422/);
assert.match(truncationSignal('rest'), /1000 search results/);
assert.match(truncationSignal('graphql'), /no error at all/);
assert.match(truncationSignal('graphql'), /hasNextPage/);
});
test('the count field follows the search type', () => {
const search = { issueCount: 18231, repositoryCount: 12, userCount: 3 };
assert.equal(matchCount(search, 'ISSUE'), 18231);
assert.equal(matchCount(search, 'REPOSITORY'), 12);
assert.equal(matchCount(search, 'USER'), 3);
assert.equal(matchCount(null, 'ISSUE'), 0);
});
test('the run says the most it can spend', () => {
assert.equal(pointCost(11), 11);
assert.equal(pointCost(0), 0);
assert.equal(pointCost(null), 0);
});
test('the document this script sends is a read', () => {
assert.deepEqual(
operations('query($q: String!) { search(query: $q, type: ISSUE, first: 100) { issueCount } }'),
['query'],
);
assert.ok(refusal('mutation M { addStar(input: {}) { clientMutationId } }'));
assert.ok(refusal('subscription S { thing { id } }'));
assert.equal(refusal(''), 'the document contains no operation to send.');
});
FAQ
Is the GraphQL limit really the same 1,000, or is it just similar?
It is the same limit, because it is the same index. Search on GitHub is a separate ranked-retrieval system from the object store, and both APIs are front ends to it; the cap is on how deep into a ranking you may page, not on how a request is framed. So no page size, cursor strategy or query rewrite gets past it, and the number does not differ between the two APIs. What differs is only the announcement: REST refuses the request that would cross the line, GraphQL serves you up to the line and stops.
How do I tell a truncated walk from a complete one?
Compare the count the index reports against the number of nodes you actually collected. issueCount, repositoryCount and userCount describe the match set and keep reporting the full figure after the ceiling has cut you off, so a walk that ends with 1,000 nodes and a count of 18,231 is truncated and a walk that ends with 40 nodes and a count of 40 is complete. Both end with hasNextPage false and neither raises an error, which is why the comparison has to be explicit and cannot be inferred from how the loop terminated.
Can I get past it by sorting differently and paging from both ends?
You can double your reach that way and it is a genuinely used trick: run the query ascending for the first 1,000 and descending for the last 1,000. It is still a partition, just an awkward one with two slices, and it breaks down the moment the match set exceeds 2,000 or the sort field has ties at the boundary. Slicing by created: date ranges scales to any size, is easy to reason about, and gives you slices you can re-run independently when one of them fails.
When should I use a typed connection instead?
Whenever the question is an inventory rather than a ranking. repository.issues, repository.pullRequests, organization.repositories and repository.discussions read the object store directly, paginate with cursors and have no result ceiling at all, so “every open issue in this repository” is answerable in full and “the thousand most relevant issues mentioning a phrase across the org” is not. Search earns its place when the filter genuinely spans repositories or depends on full-text matching; for everything else it is a ceiling you did not have to accept.
Does each page of the walk cost a point?
Yes. A search connection asking for first: 100 costs one point per request against the hourly GraphQL budget, so a full walk to the ceiling is ten or eleven points and a nineteen-slice partition run to completion is closer to two hundred. That is small against 5,000 an hour but not free, which is why this script prints the maximum it might spend before the first request and stops at --max-pages. Reading issueCount on page one costs one point and answers the question on its own.
Related field notes
- The REST ceiling, which says so with a 422
- A search that timed out and returned part of an answer
- Only the first page of results is ever read
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Queries — GitHub GraphQL API reference
- Search — GitHub REST API
- Using pagination in the GraphQL API — GitHub Docs
- GitHub GraphQL search with filtering — Stack Overflow
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.