Diagnostic GitHub API
code search is billed to its own 10 a minute bucket
The script walks the org, calling GET /search/code once per repository. It gets through nine of them. The tenth returns 403, and GET /rate_limit says you have 4,987 requests left in the hour. Both statements are correct, because the request that was refused was never being counted in the bucket you just looked at.
Code search is the tightest allowance GitHub hands out: about 10 requests a minute, a third of what issue and repository search get and a rounding error against the 5,000 an hour for core. It is billed to resources.code_search, and every code-search response names that bucket in x-ratelimit-resource, so the allowance you spent is stated rather than inferred.
The repair is not caching. It is the shape of the scan. One org:-qualified query, paged, covers what a loop of one query per repository covers, and it costs pages instead of repositories: 600 repositories collapse into ten pages. The script below costs both shapes against the bucket the API actually reports and tells you how many minutes each takes.
The problem in plain words
What makes this one slow to work out is that the number everybody checks is the wrong number. A 403 sends you to GET /rate_limit, the response is 4,987 remaining, and that reading is accurate about the core bucket, which is not the bucket that stopped you. Nothing in the error says "you are looking at the wrong row".
It also survives the first fix. Adding a one-second sleep between repositories turns 10 a minute into 60 a minute, which is still six times the allowance, so the job now fails a little later and looks intermittent. Then someone adds ETags, which do nothing here, because the requests are not repeats: each one is a different query.
And it scales the wrong way. Iterating repositories means the cost of the scan is the number of repositories, so the tool that worked on the twelve-repo test org falls over on the day it is pointed at the real one. The output is worse than an error, too: a partial scan looks like a clean result with fewer hits.
Why it happens
The buckets are separate on purpose. GET /rate_limit reports core, search, code_search, graphql, integration_manifest and more as independent rows with their own limit, remaining and reset. Code search is the tightest of them because it is the most expensive query GitHub answers, and it is the only one metered per minute rather than per hour at that size.
x-ratelimit-resource removes the ambiguity. Every response names the bucket it was charged to. If you are ever unsure whether a call went to search or code_search, read that header on the call itself rather than reasoning about the path.
Code search refuses to work unauthenticated at all. Unlike most read endpoints, which fall back to 60 an hour for anonymous callers, code search requires authentication. A script that quietly lost its token does not get a smaller allowance here, it gets nothing.
Per-repository iteration is the anti-pattern the qualifiers exist to prevent. q=addClass+org:acme is one query. repo:acme/api repeated 600 times is 600 queries for the same coverage, and it spends 600 units of a 10-a-minute allowance to do it.
A single query still cannot return more than 1,000 results. Collapsing the loop does not remove that ceiling, it just moves the problem: past 1,000 matches you have to narrow by path, extension or date rather than page further. That is its own note, and the costing below counts pages only up to the cap so it does not promise you results the API will not serve.
The fix, as a flow
The script reads the whole resources table rather than the one row everybody checks, then costs the scan twice: once as the loop that is running and once as the qualified query that would replace it. The gap between those two numbers is the finding, and it is measured in minutes rather than in requests.
How to fix it
Read the whole resources table, not just core
GET /rate_limit is free — it does not consume quota from any bucket — and it returns every row at once. Print resources.code_search next to resources.search and resources.core so the difference in scale is on the screen. If the row is missing entirely, you are on a deployment that does not report it, and the documented default applies.
Confirm the bucket on a live call with x-ratelimit-resource
One GET /search/code?q=...&per_page=1 and read x-ratelimit-resource on the response. It will say code_search. This is the cheapest possible way to settle which allowance an endpoint spends, and it works for any endpoint you are unsure about.
Cost the scan you are actually running
Requests equals repositories times queries per repository, and wall clock equals that divided by ten. Six hundred repositories with one query each is 600 requests and an hour of waiting even if nothing is refused. Put that number in front of whoever is asking why the scan is slow.
Collapse the loop into one qualified query and page it
Replace repo: per repository with a single org:acme or user:someone query and follow the Link header. At per_page=100 the whole reachable result set is at most ten requests, so the entire scan fits in one minute of the allowance rather than an hour of it.
For an exhaustive scan, stop using the search API
If you genuinely need every occurrence in every file, the search index is the wrong tool: it is capped, ranked and rate limited. Shallow-clone the repositories and grep locally. The API is for finding where something is; the clone is for auditing everywhere it is.
How to check it worked
Re-cost the scan after the loop is collapsed. The report should put the whole thing inside a single minute of the code-search allowance.
python3 github_code_search_budget.py --repos 600 --queries 1 --results 800
# per-repo-scan: 600 request(s) is 60 minute(s) at 10 a minute; the same
# coverage as 1 qualified query is 8 request(s) and 1 minute(s)
The full code
The only network call the script needs is GET /rate_limit, which spends nothing, and the optional probe is one search with per_page=1. Everything that produces a finding is arithmetic: a normaliser for the resources table that tells a missing row apart from an empty one, two costings, and a verdict. All four are pure, so the tests can hand them a bucket that is already exhausted instead of exhausting one.
"""Cost a code-search scan against the bucket code search is actually billed to.
Read only. Every request is a GET. GET /rate_limit consumes no quota from any
bucket, and the optional live probe is a single search with per_page=1.
Code search is metered by resources.code_search, which is roughly 10 requests a
minute. That is a different row from resources.search and a different row again
from resources.core, and reading the wrong row is most of why this failure takes
an afternoon.
"""
import argparse
import json
import logging
import math
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_code_search_budget")
API = "https://api.github.com"
UA = "github-code-search-budget/1.0"
# Documented defaults, used only to fill in a row GET /rate_limit did not return.
DEFAULTS = {"code_search": 10, "search": 30, "core": 5000}
# A single search query cannot return more than this many results however many
# pages are requested, and 100 is the largest page the API will serve.
RESULT_CAP = 1000
MAX_PAGE = 100
def buckets(payload):
"""Normalise the resources table from GET /rate_limit. Pure.
Returns {name: {"limit", "remaining", "reset", "present"}}. A row the
deployment does not report comes back with present False and the documented
default, because "the field is missing" and "the allowance is zero" are
different findings and only one of them is a problem you can wait out.
"""
resources = ((payload or {}).get("resources") or {})
out = {}
for name, default in DEFAULTS.items():
raw = resources.get(name)
if not isinstance(raw, dict):
out[name] = {"limit": default, "remaining": None,
"reset": None, "present": False}
continue
parsed = {}
for key in ("limit", "remaining", "reset"):
try:
parsed[key] = int(raw.get(key))
except (TypeError, ValueError):
parsed[key] = None
out[name] = {"limit": default if parsed["limit"] is None else parsed["limit"],
"remaining": parsed["remaining"],
"reset": parsed["reset"],
"present": True}
return out
def scan_cost(repos, queries_per_repo, per_minute):
"""Requests and wall-clock minutes for a scan that iterates repositories. Pure.
The number that surprises people is minutes, not requests: at ten a minute a
six hundred repository loop is an hour of waiting even when nothing is
refused.
"""
try:
repos = max(0, int(repos))
queries_per_repo = max(0, int(queries_per_repo))
except (TypeError, ValueError):
return {"requests": 0, "minutes": 0}
per_minute = max(1, int(per_minute or 1))
needed = repos * queries_per_repo
return {"requests": needed,
"minutes": math.ceil(needed / per_minute) if needed else 0}
def collapsed_cost(queries, results_per_query, per_minute,
page_size=MAX_PAGE, cap=RESULT_CAP):
"""Cost of the same coverage as one qualified query per concern, paged. Pure.
Capped at `cap` because a single query cannot return more than that many
results, so counting pages past it would promise results the API will not
serve. `truncated` says so out loud rather than quietly under-reporting.
"""
try:
queries = max(0, int(queries))
results = max(0, int(results_per_query))
except (TypeError, ValueError):
return {"requests": 0, "pages_per_query": 0, "minutes": 0, "truncated": False}
page_size = max(1, min(int(page_size or MAX_PAGE), MAX_PAGE))
reachable = min(results, cap)
# A query with no results still costs the one request that discovers that.
per_query = math.ceil(reachable / page_size) if reachable else 1
needed = queries * per_query
per_minute = max(1, int(per_minute or 1))
return {"requests": needed, "pages_per_query": per_query,
"minutes": math.ceil(needed / per_minute) if needed else 0,
"truncated": results > cap}
def seconds_until(reset, now):
"""Seconds until a bucket resets, floored at zero. Pure.
None rather than 0 when the value is unreadable: "resets right now" and "I
could not read the reset" should not print the same.
"""
try:
return max(0, int(reset) - int(now))
except (TypeError, ValueError):
return None
def verdict(bucket, iterating, collapsed):
"""Turn the bucket state and the two costings into a finding. Pure."""
limit = bucket.get("limit") or DEFAULTS["code_search"]
remaining = bucket.get("remaining")
note = "" if bucket.get("present") else (
" (GET /rate_limit did not report a code_search row, so this uses the "
"documented default of %d a minute)" % limit)
if remaining == 0:
return ("exhausted",
"the code_search bucket is empty. This is not the core quota, "
"which is why it can read as thousands remaining at the same "
"time. It refills on its own minute-long clock.%s" % note)
if not iterating.get("requests"):
return ("no-scan", "no scan described, so nothing to cost%s" % note)
ratio = iterating["requests"] / max(1, collapsed.get("requests") or 1)
if ratio >= 4:
return ("per-repo-scan",
"%d request(s) is %d minute(s) at %d a minute; the same coverage "
"as %d qualified quer(y/ies) is %d request(s) and %d minute(s). "
"The loop is the cost, not the caching.%s"
% (iterating["requests"], iterating["minutes"], limit,
max(1, collapsed.get("requests", 0) // max(1, collapsed.get("pages_per_query", 1))),
collapsed.get("requests", 0), collapsed.get("minutes", 0), note))
if iterating["minutes"] > 1:
return ("over-budget",
"%d request(s) at %d a minute is %d minute(s) of wall clock even "
"if nothing is refused.%s"
% (iterating["requests"], limit, iterating["minutes"], note))
return ("clear",
"%d request(s) fits inside one minute of a %d a minute allowance.%s"
% (iterating["requests"], limit, note))
def get(session, path, **kwargs):
"""One GET. Returns (status, json-or-None, headers)."""
url = API + path if path.startswith("/") else path
r = session.get(url, timeout=30, **kwargs)
try:
body = r.json()
except ValueError:
body = None
return r.status_code, body, dict(r.headers)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repos", type=int, default=0,
help="repositories the current scan iterates over")
ap.add_argument("--queries", type=int, default=1,
help="code-search queries issued per repository")
ap.add_argument("--results", type=int, default=200,
help="results you expect one qualified query to match")
ap.add_argument("--probe-query",
help="optional q= value; issues one search with per_page=1 "
"to read x-ratelimit-resource on a live response")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN. Code search refuses unauthenticated "
"callers outright, so there is no anonymous fallback here")
return 2
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
status, payload, _ = get(session, "/rate_limit")
if status != 200:
log.error("GET /rate_limit returned %d; cannot read the buckets", status)
return 2
table = buckets(payload)
for name in ("core", "search", "code_search"):
row = table[name]
wait = seconds_until(row["reset"], time.time())
log.info("%-12s limit %-5s remaining %-5s reset in %s",
name, row["limit"],
"?" if row["remaining"] is None else row["remaining"],
"unknown" if wait is None else "%ds" % wait)
if not row["present"]:
log.warning(" %s was not in the resources table; showing the "
"documented default", name)
if args.probe_query:
status, _, headers = get(session, "/search/code",
params={"q": args.probe_query, "per_page": 1})
lowered = {k.lower(): v for k, v in headers.items()}
log.info("probe: /search/code returned %d, billed to %s",
status, lowered.get("x-ratelimit-resource", "an unnamed bucket"))
if status == 403:
log.warning(" a 403 here with core headroom left is this bucket, "
"not the hourly quota and not your token scopes")
code = table["code_search"]
iterating = scan_cost(args.repos, args.queries, code["limit"])
collapsed = collapsed_cost(max(1, args.queries), args.results, code["limit"])
state, detail = verdict(code, iterating, collapsed)
log.info("%s: %s", state, detail)
if collapsed["truncated"]:
log.warning("one query cannot return more than %d results, so the "
"collapsed costing counts %d page(s) and stops. Narrow by "
"path, extension or date rather than paging further.",
RESULT_CAP, collapsed["pages_per_query"])
if state in ("per-repo-scan", "over-budget", "exhausted"):
log.info("repair: one qualified query instead of one per repository, "
"for example q=YOURTERM+org:YOURORG with per_page=100, and "
"follow the Link header.")
log.info("repair: for an exhaustive audit, shallow-clone and grep "
"locally. The search index is capped, ranked and metered; a "
"clone is none of those.")
print(json.dumps({"buckets": table, "iterating": iterating,
"collapsed": collapsed, "state": state}, indent=2))
return 1 if state in ("per-repo-scan", "over-budget", "exhausted") else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Cost a code-search scan against the bucket code search is actually billed to.
*
* Read only. Every request is a GET. GET /rate_limit consumes no quota, and the
* optional live probe is a single search with per_page=1.
*
* Code search is metered by resources.code_search, roughly 10 a minute. That is
* a different row from resources.search and from resources.core.
*/
const API = 'https://api.github.com';
const UA = 'github-code-search-budget/1.0';
// Documented defaults, used only to fill in a row GET /rate_limit did not return.
export const DEFAULTS = { code_search: 10, search: 30, core: 5000 };
// A single query cannot return more than this many results, and 100 is the
// largest page the API will serve.
export const RESULT_CAP = 1000;
export const MAX_PAGE = 100;
/**
* Normalise the resources table from GET /rate_limit. Pure.
* A missing row comes back with present false and the documented default:
* "the field is missing" and "the allowance is zero" are different findings.
*/
export function buckets(payload) {
const resources = (payload ?? {}).resources ?? {};
const out = {};
for (const [name, fallback] of Object.entries(DEFAULTS)) {
const raw = resources[name];
if (!raw || typeof raw !== 'object') {
out[name] = { limit: fallback, remaining: null, reset: null, present: false };
continue;
}
const num = (key) => {
const n = Number.parseInt(raw[key], 10);
return Number.isFinite(n) ? n : null;
};
const limit = num('limit');
out[name] = {
limit: limit === null ? fallback : limit,
remaining: num('remaining'),
reset: num('reset'),
present: true,
};
}
return out;
}
/** Requests and wall-clock minutes for a scan that iterates repositories. Pure. */
export function scanCost(repos, queriesPerRepo, perMinute) {
const r = Math.max(0, Number.parseInt(repos, 10) || 0);
const q = Math.max(0, Number.parseInt(queriesPerRepo, 10) || 0);
const rate = Math.max(1, Number.parseInt(perMinute, 10) || 1);
const needed = r * q;
return { requests: needed, minutes: needed ? Math.ceil(needed / rate) : 0 };
}
/**
* Cost of the same coverage as one qualified query per concern, paged. Pure.
* Capped at RESULT_CAP, because counting pages past it would promise results
* the API will not serve.
*/
export function collapsedCost(queries, resultsPerQuery, perMinute,
pageSize = MAX_PAGE, cap = RESULT_CAP) {
const q = Math.max(0, Number.parseInt(queries, 10) || 0);
const results = Math.max(0, Number.parseInt(resultsPerQuery, 10) || 0);
const size = Math.max(1, Math.min(Number.parseInt(pageSize, 10) || MAX_PAGE, MAX_PAGE));
const reachable = Math.min(results, cap);
// A query with no results still costs the one request that discovers that.
const perQuery = reachable ? Math.ceil(reachable / size) : 1;
const needed = q * perQuery;
const rate = Math.max(1, Number.parseInt(perMinute, 10) || 1);
return {
requests: needed,
pages_per_query: perQuery,
minutes: needed ? Math.ceil(needed / rate) : 0,
truncated: results > cap,
};
}
/** Seconds until a bucket resets, floored at zero; null when unreadable. Pure. */
export function secondsUntil(reset, now) {
const r = Number.parseInt(reset, 10);
const n = Number.parseInt(now, 10);
if (!Number.isFinite(r) || !Number.isFinite(n)) return null;
return Math.max(0, r - n);
}
/** Turn the bucket state and the two costings into a finding. Pure. */
export function verdict(bucket, iterating, collapsed) {
const limit = bucket.limit || DEFAULTS.code_search;
const note = bucket.present ? '' :
` (GET /rate_limit did not report a code_search row, so this uses the ` +
`documented default of ${limit} a minute)`;
if (bucket.remaining === 0) {
return ['exhausted',
'the code_search bucket is empty. This is not the core quota, which is ' +
'why it can read as thousands remaining at the same time. It refills on ' +
`its own minute-long clock.${note}`];
}
if (!iterating.requests) return ['no-scan', `no scan described, so nothing to cost${note}`];
const ratio = iterating.requests / Math.max(1, collapsed.requests || 1);
if (ratio >= 4) {
const queries = Math.max(1, Math.floor((collapsed.requests || 0) /
Math.max(1, collapsed.pages_per_query || 1)));
return ['per-repo-scan',
`${iterating.requests} request(s) is ${iterating.minutes} minute(s) at ` +
`${limit} a minute; the same coverage as ${queries} qualified quer(y/ies) ` +
`is ${collapsed.requests} request(s) and ${collapsed.minutes} minute(s). ` +
`The loop is the cost, not the caching.${note}`];
}
if (iterating.minutes > 1) {
return ['over-budget',
`${iterating.requests} request(s) at ${limit} a minute is ` +
`${iterating.minutes} minute(s) of wall clock even if nothing is refused.${note}`];
}
return ['clear',
`${iterating.requests} request(s) fits inside one minute of a ${limit} a ` +
`minute allowance.${note}`];
}
async function get(token, path, params) {
const url = new URL(path.startsWith('/') ? API + path : path);
for (const [k, v] of Object.entries(params ?? {})) url.searchParams.set(k, v);
const res = await fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
let body = null;
try { body = await res.json(); } catch { body = null; }
const headers = {};
for (const [k, v] of res.headers.entries()) headers[k.toLowerCase()] = v;
return { status: res.status, body, headers };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN. Code search refuses unauthenticated ' +
'callers outright, so there is no anonymous fallback here');
process.exitCode = 2;
return;
}
const repos = Number.parseInt(process.argv[2] ?? '0', 10) || 0;
const queries = Number.parseInt(process.argv[3] ?? '1', 10) || 1;
const results = Number.parseInt(process.argv[4] ?? '200', 10) || 200;
const probeQuery = process.argv[5];
const rate = await get(token, '/rate_limit');
if (rate.status !== 200) {
console.error(`GET /rate_limit returned ${rate.status}; cannot read the buckets`);
process.exitCode = 2;
return;
}
const table = buckets(rate.body);
const now = Math.floor(Date.now() / 1000);
for (const name of ['core', 'search', 'code_search']) {
const row = table[name];
const wait = secondsUntil(row.reset, now);
console.log(`${name.padEnd(12)} limit ${row.limit} remaining ` +
`${row.remaining ?? '?'} reset in ${wait === null ? 'unknown' : `${wait}s`}`);
if (!row.present) {
console.warn(` ${name} was not in the resources table; showing the ` +
'documented default');
}
}
if (probeQuery) {
const probe = await get(token, '/search/code', { q: probeQuery, per_page: 1 });
console.log(`probe: /search/code returned ${probe.status}, billed to ` +
`${probe.headers['x-ratelimit-resource'] ?? 'an unnamed bucket'}`);
if (probe.status === 403) {
console.warn(' a 403 here with core headroom left is this bucket, not ' +
'the hourly quota and not your token scopes');
}
}
const code = table.code_search;
const iterating = scanCost(repos, queries, code.limit);
const collapsed = collapsedCost(Math.max(1, queries), results, code.limit);
const [state, detail] = verdict(code, iterating, collapsed);
console.log(`${state}: ${detail}`);
if (collapsed.truncated) {
console.warn(`one query cannot return more than ${RESULT_CAP} results, so ` +
`the collapsed costing counts ${collapsed.pages_per_query} page(s) and ` +
'stops. Narrow by path, extension or date rather than paging further.');
}
if (state === 'per-repo-scan' || state === 'over-budget' || state === 'exhausted') {
console.log('repair: one qualified query instead of one per repository, ' +
'for example q=YOURTERM+org:YOURORG with per_page=100, and follow the ' +
'Link header.');
console.log('repair: for an exhaustive audit, shallow-clone and grep ' +
'locally. The search index is capped, ranked and metered; a clone is none ' +
'of those.');
}
console.log(JSON.stringify({ buckets: table, iterating, collapsed, state }, null, 2));
process.exitCode = (state === 'per-repo-scan' || state === 'over-budget' ||
state === 'exhausted') ? 1 : 0;
}
// Only run when invoked directly, so importing this module from the test file
// does not execute main(), fail on the missing token and set an exit code that
// fails the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The interesting cases are the ones you cannot arrange on demand: a code_search row that is already at zero, a deployment that does not return the row at all, and a query whose result count is past the 1,000 ceiling so the page count has to stop rather than keep multiplying. All four functions take plain values and return plain values, so every one of those is a two-line test.
from github_code_search_budget import (
buckets, collapsed_cost, scan_cost, seconds_until, verdict)
PAYLOAD = {"resources": {
"core": {"limit": 5000, "remaining": 4987, "reset": 1700000000},
"search": {"limit": 30, "remaining": 30, "reset": 1700000060},
"code_search": {"limit": 10, "remaining": 0, "reset": 1700000060},
}}
def test_every_documented_bucket_is_reported_separately():
table = buckets(PAYLOAD)
assert table["core"]["remaining"] == 4987
assert table["code_search"]["remaining"] == 0
assert table["code_search"]["limit"] == 10
def test_a_missing_row_is_flagged_rather_than_read_as_zero():
table = buckets({"resources": {"core": {"limit": 5000, "remaining": 10}}})
assert table["code_search"]["present"] is False
assert table["code_search"]["remaining"] is None
assert table["code_search"]["limit"] == 10
def test_an_empty_payload_still_returns_the_full_table():
table = buckets(None)
assert set(table) == {"core", "search", "code_search"}
assert all(row["present"] is False for row in table.values())
def test_unreadable_numbers_do_not_become_zero():
table = buckets({"resources": {"code_search": {"limit": "ten", "remaining": None}}})
assert table["code_search"]["limit"] == 10
assert table["code_search"]["remaining"] is None
def test_a_per_repo_scan_costs_repositories_not_pages():
cost = scan_cost(600, 1, 10)
assert cost["requests"] == 600
assert cost["minutes"] == 60
def test_minutes_round_up_because_a_partial_minute_still_waits():
assert scan_cost(11, 1, 10)["minutes"] == 2
assert scan_cost(0, 3, 10) == {"requests": 0, "minutes": 0}
def test_the_collapsed_scan_costs_pages():
cost = collapsed_cost(1, 800, 10)
assert cost["pages_per_query"] == 8
assert cost["requests"] == 8
assert cost["minutes"] == 1
assert cost["truncated"] is False
def test_paging_stops_at_the_thousand_result_ceiling():
cost = collapsed_cost(1, 50000, 10)
assert cost["pages_per_query"] == 10
assert cost["truncated"] is True
def test_a_query_with_no_matches_still_costs_one_request():
assert collapsed_cost(3, 0, 10)["requests"] == 3
def test_the_page_size_cannot_be_raised_past_a_hundred():
assert collapsed_cost(1, 500, 10, page_size=500)["pages_per_query"] == 5
def test_seconds_until_floors_at_zero_and_reports_junk_as_unknown():
assert seconds_until(1700000060, 1700000000) == 60
assert seconds_until(1700000000, 1700000060) == 0
assert seconds_until(None, 1700000000) is None
def test_an_empty_code_search_bucket_is_not_the_hourly_quota():
state, detail = verdict(buckets(PAYLOAD)["code_search"],
scan_cost(600, 1, 10), collapsed_cost(1, 800, 10))
assert state == "exhausted"
assert "not the core quota" in detail
def test_the_loop_is_named_as_the_cost_when_it_dwarfs_the_query():
bucket = {"limit": 10, "remaining": 10, "reset": 0, "present": True}
state, detail = verdict(bucket, scan_cost(600, 1, 10), collapsed_cost(1, 800, 10))
assert state == "per-repo-scan"
assert "600 request(s)" in detail
assert "8 request(s)" in detail
def test_a_scan_inside_one_minute_is_clear():
bucket = {"limit": 10, "remaining": 10, "reset": 0, "present": True}
state, _ = verdict(bucket, scan_cost(5, 1, 10), collapsed_cost(1, 200, 10))
assert state == "clear"
def test_a_missing_row_is_said_out_loud_in_the_verdict():
bucket = {"limit": 10, "remaining": None, "reset": None, "present": False}
_, detail = verdict(bucket, scan_cost(5, 1, 10), collapsed_cost(1, 200, 10))
assert "documented default" in detail
def test_nothing_to_cost_is_its_own_state():
bucket = {"limit": 10, "remaining": 10, "reset": 0, "present": True}
assert verdict(bucket, scan_cost(0, 0, 10), collapsed_cost(1, 200, 10))[0] == "no-scan"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
buckets, collapsedCost, scanCost, secondsUntil, verdict,
} from './github-code-search-budget.mjs';
const PAYLOAD = {
resources: {
core: { limit: 5000, remaining: 4987, reset: 1700000000 },
search: { limit: 30, remaining: 30, reset: 1700000060 },
code_search: { limit: 10, remaining: 0, reset: 1700000060 },
},
};
test('every documented bucket is reported separately', () => {
const table = buckets(PAYLOAD);
assert.equal(table.core.remaining, 4987);
assert.equal(table.code_search.remaining, 0);
assert.equal(table.code_search.limit, 10);
});
test('a missing row is flagged rather than read as zero', () => {
const table = buckets({ resources: { core: { limit: 5000, remaining: 10 } } });
assert.equal(table.code_search.present, false);
assert.equal(table.code_search.remaining, null);
assert.equal(table.code_search.limit, 10);
});
test('an empty payload still returns the full table', () => {
const table = buckets(null);
assert.deepEqual(Object.keys(table).sort(), ['code_search', 'core', 'search']);
assert.ok(Object.values(table).every((row) => row.present === false));
});
test('unreadable numbers do not become zero', () => {
const table = buckets({ resources: { code_search: { limit: 'ten', remaining: null } } });
assert.equal(table.code_search.limit, 10);
assert.equal(table.code_search.remaining, null);
});
test('a per-repo scan costs repositories, not pages', () => {
const cost = scanCost(600, 1, 10);
assert.equal(cost.requests, 600);
assert.equal(cost.minutes, 60);
});
test('minutes round up because a partial minute still waits', () => {
assert.equal(scanCost(11, 1, 10).minutes, 2);
assert.deepEqual(scanCost(0, 3, 10), { requests: 0, minutes: 0 });
});
test('the collapsed scan costs pages', () => {
const cost = collapsedCost(1, 800, 10);
assert.equal(cost.pages_per_query, 8);
assert.equal(cost.requests, 8);
assert.equal(cost.minutes, 1);
assert.equal(cost.truncated, false);
});
test('paging stops at the thousand-result ceiling', () => {
const cost = collapsedCost(1, 50000, 10);
assert.equal(cost.pages_per_query, 10);
assert.equal(cost.truncated, true);
});
test('a query with no matches still costs one request', () => {
assert.equal(collapsedCost(3, 0, 10).requests, 3);
});
test('the page size cannot be raised past a hundred', () => {
assert.equal(collapsedCost(1, 500, 10, 500).pages_per_query, 5);
});
test('secondsUntil floors at zero and reports junk as unknown', () => {
assert.equal(secondsUntil(1700000060, 1700000000), 60);
assert.equal(secondsUntil(1700000000, 1700000060), 0);
assert.equal(secondsUntil(null, 1700000000), null);
});
test('an empty code_search bucket is not the hourly quota', () => {
const [state, detail] = verdict(buckets(PAYLOAD).code_search,
scanCost(600, 1, 10), collapsedCost(1, 800, 10));
assert.equal(state, 'exhausted');
assert.match(detail, /not the core quota/);
});
test('the loop is named as the cost when it dwarfs the query', () => {
const bucket = { limit: 10, remaining: 10, reset: 0, present: true };
const [state, detail] = verdict(bucket, scanCost(600, 1, 10), collapsedCost(1, 800, 10));
assert.equal(state, 'per-repo-scan');
assert.match(detail, /600 request\(s\)/);
assert.match(detail, /8 request\(s\)/);
});
test('a scan inside one minute is clear', () => {
const bucket = { limit: 10, remaining: 10, reset: 0, present: true };
assert.equal(verdict(bucket, scanCost(5, 1, 10), collapsedCost(1, 200, 10))[0], 'clear');
});
test('a missing row is said out loud in the verdict', () => {
const bucket = { limit: 10, remaining: null, reset: null, present: false };
const [, detail] = verdict(bucket, scanCost(5, 1, 10), collapsedCost(1, 200, 10));
assert.match(detail, /documented default/);
});
test('nothing to cost is its own state', () => {
const bucket = { limit: 10, remaining: 10, reset: 0, present: true };
assert.equal(verdict(bucket, scanCost(0, 0, 10), collapsedCost(1, 200, 10))[0], 'no-scan');
});
FAQ
Why does GET /rate_limit say I have thousands of requests left when code search is refusing me?
Because you are reading the core row. The response is a table of independent buckets: core, search, code_search, graphql and others, each with its own limit, remaining and reset. Code search is billed to code_search, at roughly 10 a minute, and nothing you do there moves the core number. Print the whole resources table rather than the one field, and read x-ratelimit-resource on the refused response to confirm which allowance was spent.
Is the code search limit the same as the search limit?
No. General search is around 30 requests a minute for an authenticated caller; code search is around 10. They are separate rows in the same table, so spending one does not spend the other. This matters when a tool mixes them: a scan that issues one code search and one issue search per repository is draining two different buckets at two different rates and will hit the code-search one first.
Would caching with ETags fix this?
Not here, and that is worth being clear about because it is the reflex. Conditional requests save you when you ask the same question repeatedly and the answer has not changed. A per-repository scan asks a different question every time: repo:acme/api, then repo:acme/web, then repo:acme/jobs. There is no cache hit to be had. The fix is asking one question instead of six hundred.
Can I run code search without a token?
No. Most read endpoints degrade to 60 requests an hour for an anonymous caller, but code search requires authentication outright. That makes a lost token look different here than elsewhere: instead of a smaller allowance you get a refusal, which is worth knowing when a job that used to work stops working after a credential change.
What should I do when one query genuinely matches more than 1,000 results?
Split the query rather than paging further, because the 1,000-result ceiling applies per query and no amount of pagination gets past it. Narrow by language, by path, by extension, or by pushed date, and run the narrower queries as separate searches. If you need genuine completeness rather than a ranked sample, clone the repositories and grep, which has no ceiling and no meter.
Related field notes
- Search stops at 1,000 results per query
- per_page is unset so every list costs more
- Polling without ETags spends full quota
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limit — GitHub REST API
- Search — GitHub REST API
- Searching code — GitHub Docs
- Rate limits for the REST API — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.