Diagnostic GitHub API
over 100 concurrent requests trips a secondary rate limit
Someone replaces a for loop with a Promise.all and the job goes from nine minutes to forty seconds, once. The next run returns 403 on two thirds of the requests. You check the quota, because a 403 from GitHub means the quota, and the quota says four thousand eight hundred requests left. Both numbers are true. They are about different limits.
Read the body, not just the status. A 403 or 429 whose JSON message contains "You have exceeded a secondary rate limit" while x-ratelimit-remaining is still non-zero is a secondary limit. GitHub caps you at 100 concurrent requests across REST and GraphQL, and that cap is entirely separate from the hourly bucket.
There is no header for it. No x-ratelimit-* field tracks secondary limits and GET /rate_limit reports primary quota only, so nothing can tell you how close you are before you arrive. What a script can do is measure the concurrency your own client actually achieved, which is usually not the number you configured, and classify a throttled response correctly when one comes back.
The problem in plain words
The parallel version works in development and fails in production, and the difference is not the code. It is that development ran it against six repositories and production runs it against six hundred, so the fan-out that peaked at six in flight now peaks at two hundred. Nothing in the code changed; the shape of the input did.
What makes this expensive to diagnose is that the error looks exactly like the one you already know how to fix. A 403 from GitHub with the word "rate" in it sends everyone to GET /rate_limit, which reports a healthy bucket, which makes the 403 look like a permissions problem instead, which sends the next hour into checking token scopes. The quota is fine. The scopes are fine. The client sent 200 requests at once.
And the failure is partial, which is worse than total. Some requests in the batch succeed, so the job completes and writes a result. The result is missing whatever the throttled requests would have contributed, and nothing marks it as incomplete.
Why it happens
Secondary limits protect burst behaviour, not volume. The hourly bucket is about how much you ask for over an hour. Secondary limits are about how hard you ask at any instant: no more than 100 concurrent requests, no more than 900 points per minute on a single endpoint, no more than 90 seconds of CPU per 60 seconds of wall clock. A script can spend two hundred requests in one second and still have 4,800 of its 5,000 left.
There is no headroom API, and that is the honest answer. This is the one place in these notes where detection genuinely cannot come first. No x-ratelimit-* header reports a secondary bucket, and GET /rate_limit documents itself as covering primary quota. A secondary limit becomes observable at the moment you exceed it and not one request before, so any tool that claims to warn you in advance is inferring, not measuring.
The two limits are told apart by one field. On a primary exhaustion, x-ratelimit-remaining is 0: that is what exhausted means. On a secondary limit it is whatever it was, usually thousands. So 403 with headroom is the signature, and it is reliable enough to branch on even when the message wording changes.
Your configured concurrency is not your actual concurrency. A pool of 50 workers against an endpoint that answers in 40 ms rarely has 50 requests in flight; a pool of 20 against an endpoint that answers in four seconds reliably does. The number that matters is the peak overlap of the request spans, and it is measurable from timestamps the client already has.
Retrying immediately extends the window. The response carries retry-after. A client that treats the 403 as a generic transient error and retries in a second keeps the limit engaged, which is how a two-minute pause turns into a twenty-minute one.
The fix, as a flow
The script measures the overlap of its own request spans rather than trusting the pool size, because the pool size is a ceiling and the overlap is what actually happened. Then it classifies any refusal on the one field that separates the two limits.
How to fix it
Read the body of the 403 before you read anything else
GitHub returns JSON on every error. {"message": "You have exceeded a secondary rate limit..."} settles it immediately. A permissions 403 says "Resource not accessible by integration" or names a missing scope, and a primary exhaustion says "API rate limit exceeded for user ID ...". Three different repairs, one status code.
Cross-check x-ratelimit-remaining on the same response
The throttled response still carries the primary headers. If x-ratelimit-remaining is a large number and you were refused anyway, the refusal did not come from the bucket those headers describe. This check works even when the message wording is one you have not seen, which matters because the wording has changed before.
Measure the peak overlap your client actually reaches
Record a start and end timestamp for every request and sweep them: the peak number of spans open at once is your real concurrency. Run this against a cheap endpoint at your production settings. GET /rate_limit is the right probe because it does not consume primary quota, so the measurement costs nothing except the requests themselves.
Bound the pool instead of fanning out over the input
Promise.all(repos.map(fetch)) has a concurrency equal to repos.length, which is an input, not a setting. Replace it with a worker pool of a fixed small size — five to ten for reads is plenty, one for anything that writes — so the ceiling belongs to your code rather than to whoever added the six hundredth repository.
Honour retry-after and pause the whole pool, not one request
When one request in a batch is throttled, every other request in that batch is about to be. Sleep retry-after seconds before resuming anything; where the header is absent, wait at least 60 seconds and then back off exponentially. Retrying the single failed item while the other 99 keep going is why the window never closes.
How to check it worked
Re-run the probe at your production concurrency. The peak overlap should sit well under the ceiling and no response should classify as a secondary limit.
python3 github_concurrency_probe.py --requests 24 --concurrency 6
# peak overlap 6 of a 100 ceiling, 0 throttled: clearThe full code
The probe defaults to GET /rate_limit, which is the only endpoint in the API that does not spend what it measures. Two pure functions carry the diagnosis: one classifies a response into primary, secondary, permission or fine, and one sweeps request spans into a peak overlap. Neither touches the network, because both need to be exercised against responses you cannot conveniently produce on demand.
"""Measure the concurrency a client actually reaches, and classify any throttling.
Read only. Every request is a GET, and the default probe endpoint is
GET /rate_limit, which does not count against the primary rate limit.
There is no API for secondary-limit headroom: no x-ratelimit-* field tracks one
and GET /rate_limit covers primary quota only. So this script does not predict a
secondary limit. It measures the fan-out this client reaches and reports
correctly if one fires.
"""
import argparse
import concurrent.futures
import json
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_concurrency_probe")
API = "https://api.github.com"
UA = "github-concurrency-probe/1.0"
# Documented ceiling on requests in flight at once, across REST and GraphQL.
CONCURRENCY_CEILING = 100
# The wording has changed over the years, so match on the stable part of both
# the current phrasing and the one that predates it.
SECONDARY_MARKERS = ("secondary rate limit", "abuse detection")
def classify(status, body, headers):
"""Sort one response into primary, secondary, permission or fine. Pure.
Returns (state, detail). The distinguishing field is x-ratelimit-remaining on
the refused response itself: a primary exhaustion reports 0 there because that
is what exhausted means, while a secondary limit leaves the primary bucket
untouched. So "403 with headroom left" is the signature, and it still holds
when the message wording is one this code has never seen.
"""
lowered = {str(k).lower(): v for k, v in (headers or {}).items()}
text = str(body or "").lower()
try:
remaining = int(lowered.get("x-ratelimit-remaining"))
except (TypeError, ValueError):
remaining = None
try:
status = int(status)
except (TypeError, ValueError):
status = 0
if 200 <= status < 400:
seen = "unknown" if remaining is None else str(remaining)
return ("ok", "%d, primary bucket reports %s left" % (status, seen))
if status not in (403, 429):
return ("other", "%d is not a throttle at all" % status)
if any(marker in text for marker in SECONDARY_MARKERS):
return ("secondary",
"%d and the body names a secondary rate limit. The hourly quota "
"is not involved: it still reports %s remaining."
% (status, "an unknown number" if remaining is None else remaining))
if remaining == 0:
return ("primary",
"%d with x-ratelimit-remaining at 0. This is the hourly quota, "
"not a secondary limit, and it clears at x-ratelimit-reset."
% status)
if remaining is not None and remaining > 0:
return ("secondary-suspected",
"%d while %d request(s) remain in the primary bucket. The body "
"does not say secondary, but a refusal with headroom left did "
"not come from the bucket these headers describe."
% (status, remaining))
return ("forbidden",
"%d with no rate-limit headers to read. Treat this as permissions "
"until something proves otherwise." % status)
def peak_overlap(spans):
"""Peak number of requests in flight at once, from (start, end) pairs. Pure.
A sweep rather than a max of the pool size, because the pool size is a
ceiling and this is the number that was actually reached. Twenty workers
against a 40 ms endpoint rarely overlap; six against a four-second one
always do.
"""
events = []
for span in spans or []:
start, end = float(span[0]), float(span[1])
if end < start:
start, end = end, start
events.append((start, 1))
events.append((end, -1))
# A request that ended at the exact instant another began was never beside
# it, so ends are ordered before starts at an equal timestamp.
events.sort(key=lambda e: (e[0], e[1]))
peak = current = 0
for _, delta in events:
current += delta
if current > peak:
peak = current
return peak
def verdict(peak, states, ceiling=CONCURRENCY_CEILING):
"""Turn a peak overlap and a list of response states into a finding. Pure.
"clear" deliberately does not say the client is safe. Nothing can say that:
the limit has no headroom API, so a probe that did not trip it has shown
only that this run, at this moment, did not trip it.
"""
throttled = [s for s in (states or []) if s in ("secondary", "secondary-suspected")]
if throttled:
return ("tripped",
"%d of %d response(s) were refused with the primary bucket still "
"healthy. Peak overlap was %d. Bound the pool and honour "
"retry-after." % (len(throttled), len(states or []), peak))
if peak >= ceiling:
return ("over-ceiling",
"peak overlap %d at or above the documented ceiling of %d. This "
"run happened not to be refused; a slower endpoint or a busier "
"moment will be." % (peak, ceiling))
if peak >= ceiling * 0.8:
return ("near-ceiling",
"peak overlap %d against a ceiling of %d. One more worker or one "
"slow response is the difference." % (peak, ceiling))
return ("clear",
"peak overlap %d of a %d ceiling, nothing throttled. This proves the "
"run was fine, not that the client is: secondary limits have no "
"headroom API to check against." % (peak, ceiling))
def probe(session, url, index):
"""One timed GET. Returns a record; never raises, because a failed request
is data here rather than an error."""
start = time.monotonic()
try:
r = session.get(url, timeout=30)
end = time.monotonic()
return {"i": index, "start": start, "end": end, "status": r.status_code,
"body": r.text[:400], "headers": dict(r.headers)}
except requests.RequestException as exc:
return {"i": index, "start": start, "end": time.monotonic(), "status": 0,
"body": str(exc), "headers": {}}
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--endpoint", default="/rate_limit",
help="path to probe (default /rate_limit, which is free)")
ap.add_argument("--requests", type=int, default=12,
help="how many GETs to issue in total")
ap.add_argument("--concurrency", type=int, default=6,
help="worker pool size; the ceiling, not the achieved peak")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
workers = max(1, min(args.concurrency, CONCURRENCY_CEILING))
if workers != args.concurrency:
log.warning("clamping concurrency to %d: going past the documented "
"ceiling on purpose spends a shared quota to learn nothing "
"new", workers)
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
url = API + args.endpoint if args.endpoint.startswith("/") else args.endpoint
log.info("probing %s: %d request(s), pool of %d", url, args.requests, workers)
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
results = list(pool.map(lambda i: probe(session, url, i),
range(max(1, args.requests))))
states = []
for r in sorted(results, key=lambda r: r["i"]):
state, detail = classify(r["status"], r["body"], r["headers"])
states.append(state)
if state in ("ok", "other"):
log.debug("request %d: %s %s", r["i"], state, detail)
else:
log.warning("request %d: %-20s %s", r["i"], state, detail)
retry_after = {k.lower(): v for k, v in r["headers"].items()}.get("retry-after")
if retry_after:
log.warning(" retry-after: %s second(s). Pause the whole pool "
"for that long, not just this request.", retry_after)
peak = peak_overlap([(r["start"], r["end"]) for r in results])
state, detail = verdict(peak, states)
log.info("%s: %s", state, detail)
if state != "clear":
log.info("repair: replace the fan-out with a bounded pool. Python: "
"ThreadPoolExecutor(max_workers=6). Node: a queue of 6 rather "
"than Promise.all over the whole input list.")
log.info("repair: on a throttled response sleep retry-after seconds "
"before resuming any worker, and where the header is absent "
"wait 60 seconds and then back off exponentially.")
print(json.dumps({"peak_overlap": peak, "ceiling": CONCURRENCY_CEILING,
"requests": len(results), "state": state,
"states": states}, indent=2))
return 1 if state in ("tripped", "over-ceiling") else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Measure the concurrency a client actually reaches, and classify any throttling.
*
* Read only. Every request is a GET, and the default probe endpoint is
* GET /rate_limit, which does not count against the primary rate limit.
*
* Secondary limits have no headroom API, so nothing here predicts one.
*/
const API = 'https://api.github.com';
const UA = 'github-concurrency-probe/1.0';
// Documented ceiling on requests in flight at once, across REST and GraphQL.
export const CONCURRENCY_CEILING = 100;
// The wording has changed over the years; match the stable part of both forms.
const SECONDARY_MARKERS = ['secondary rate limit', 'abuse detection'];
/**
* Sort one response into primary, secondary, permission or fine. Pure.
* The distinguishing field is x-ratelimit-remaining on the refused response:
* a primary exhaustion reports 0, a secondary limit leaves the bucket alone.
*/
export function classify(status, body, headers) {
const lowered = {};
for (const [k, v] of Object.entries(headers ?? {})) lowered[k.toLowerCase()] = v;
const text = String(body ?? '').toLowerCase();
const rawRemaining = lowered['x-ratelimit-remaining'];
const parsed = Number.parseInt(rawRemaining, 10);
const remaining = Number.isFinite(parsed) ? parsed : null;
const code = Number.parseInt(status, 10) || 0;
if (code >= 200 && code < 400) {
return ['ok', `${code}, primary bucket reports ${remaining ?? 'unknown'} left`];
}
if (code !== 403 && code !== 429) return ['other', `${code} is not a throttle at all`];
if (SECONDARY_MARKERS.some((m) => text.includes(m))) {
return ['secondary',
`${code} and the body names a secondary rate limit. The hourly quota is ` +
`not involved: it still reports ${remaining ?? 'an unknown number'} remaining.`];
}
if (remaining === 0) {
return ['primary',
`${code} with x-ratelimit-remaining at 0. This is the hourly quota, not a ` +
'secondary limit, and it clears at x-ratelimit-reset.'];
}
if (remaining !== null && remaining > 0) {
return ['secondary-suspected',
`${code} while ${remaining} request(s) remain in the primary bucket. The ` +
'body does not say secondary, but a refusal with headroom left did not ' +
'come from the bucket these headers describe.'];
}
return ['forbidden',
`${code} with no rate-limit headers to read. Treat this as permissions ` +
'until something proves otherwise.'];
}
/**
* Peak number of requests in flight at once, from [start, end] pairs. Pure.
* A sweep, because the pool size is a ceiling and this is what was reached.
*/
export function peakOverlap(spans) {
const events = [];
for (const span of spans ?? []) {
let start = Number(span[0]);
let end = Number(span[1]);
if (end < start) [start, end] = [end, start];
events.push([start, 1], [end, -1]);
}
// Ends sort before starts at an equal timestamp: a request that ended as
// another began was never beside it.
events.sort((a, b) => (a[0] - b[0]) || (a[1] - b[1]));
let peak = 0;
let current = 0;
for (const [, delta] of events) {
current += delta;
if (current > peak) peak = current;
}
return peak;
}
/**
* Turn a peak overlap and a list of response states into a finding. Pure.
* "clear" says this run was fine, never that the client is: the limit has no
* headroom API to check against.
*/
export function verdict(peak, states, ceiling = CONCURRENCY_CEILING) {
const list = states ?? [];
const throttled = list.filter((s) => s === 'secondary' || s === 'secondary-suspected');
if (throttled.length) {
return ['tripped',
`${throttled.length} of ${list.length} response(s) were refused with the ` +
`primary bucket still healthy. Peak overlap was ${peak}. Bound the pool ` +
'and honour retry-after.'];
}
if (peak >= ceiling) {
return ['over-ceiling',
`peak overlap ${peak} at or above the documented ceiling of ${ceiling}. ` +
'This run happened not to be refused; a slower endpoint or a busier ' +
'moment will be.'];
}
if (peak >= ceiling * 0.8) {
return ['near-ceiling',
`peak overlap ${peak} against a ceiling of ${ceiling}. One more worker or ` +
'one slow response is the difference.'];
}
return ['clear',
`peak overlap ${peak} of a ${ceiling} ceiling, nothing throttled. This ` +
'proves the run was fine, not that the client is: secondary limits have no ' +
'headroom API to check against.'];
}
async function probe(token, url, index) {
const start = performance.now() / 1000;
try {
const res = await fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
const body = (await res.text()).slice(0, 400);
const headers = Object.fromEntries(res.headers.entries());
return { i: index, start, end: performance.now() / 1000, status: res.status, body, headers };
} catch (err) {
return {
i: index, start, end: performance.now() / 1000, status: 0,
body: err.message, headers: {},
};
}
}
async function main() {
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN (a read-only token is enough)');
process.exitCode = 2;
return;
}
const endpoint = process.argv[2] ?? '/rate_limit';
const total = Math.max(1, Number.parseInt(process.argv[3] ?? '12', 10) || 12);
const wanted = Number.parseInt(process.argv[4] ?? '6', 10) || 6;
const workers = Math.max(1, Math.min(wanted, CONCURRENCY_CEILING));
const url = endpoint.startsWith('/') ? API + endpoint : endpoint;
console.log(`probing ${url}: ${total} request(s), pool of ${workers}`);
// A queue, not Promise.all over the input: the whole point of the note is
// that Promise.all borrows its concurrency from the length of the list.
const results = [];
let next = 0;
await Promise.all(Array.from({ length: workers }, async () => {
while (next < total) {
const index = next;
next += 1;
results.push(await probe(token, url, index));
}
}));
const states = [];
for (const r of results.sort((a, b) => a.i - b.i)) {
const [state, detail] = classify(r.status, r.body, r.headers);
states.push(state);
if (state !== 'ok' && state !== 'other') {
console.warn(`request ${r.i}: ${state.padEnd(20)} ${detail}`);
const lowered = {};
for (const [k, v] of Object.entries(r.headers)) lowered[k.toLowerCase()] = v;
if (lowered['retry-after']) {
console.warn(` retry-after: ${lowered['retry-after']} second(s). Pause ` +
'the whole pool for that long, not just this request.');
}
}
}
const peak = peakOverlap(results.map((r) => [r.start, r.end]));
const [state, detail] = verdict(peak, states);
console.log(`${state}: ${detail}`);
if (state !== 'clear') {
console.log('repair: replace the fan-out with a bounded queue of 6 rather ' +
'than Promise.all over the whole input list.');
console.log('repair: on a throttled response sleep retry-after seconds ' +
'before resuming any worker; where the header is absent wait 60 seconds ' +
'and then back off exponentially.');
}
console.log(JSON.stringify({
peak_overlap: peak, ceiling: CONCURRENCY_CEILING,
requests: results.length, state, states,
}, null, 2));
process.exitCode = (state === 'tripped' || state === 'over-ceiling') ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing token, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The cases worth pinning are the ones the API deliberately makes ambiguous. A 403 with headroom left and a 403 with an empty bucket are the same status code and opposite repairs. A 403 with no rate-limit headers at all is neither, and calling it a secondary limit sends someone to add backoff to a permissions problem. And the overlap sweep has one edge that decides whether the number means anything: a request that ends exactly when the next begins was never in flight beside it.
from github_concurrency_probe import classify, peak_overlap, verdict
SECONDARY = ('{"message":"You have exceeded a secondary rate limit. '
'Please wait a few minutes before you try again."}')
PRIMARY = '{"message":"API rate limit exceeded for user ID 12345."}'
DENIED = '{"message":"Resource not accessible by integration"}'
def headers(remaining=4800, **extra):
h = {"X-RateLimit-Limit": "5000", "X-RateLimit-Used": str(5000 - remaining)}
if remaining is not None:
h["X-RateLimit-Remaining"] = str(remaining)
h.update(extra)
return h
def test_a_secondary_limit_is_named_in_the_body():
state, detail = classify(403, SECONDARY, headers(4800))
assert state == "secondary"
assert "4800" in detail
def test_the_same_message_on_a_429_classifies_identically():
assert classify(429, SECONDARY, headers(4800))[0] == "secondary"
def test_an_empty_bucket_is_the_primary_quota_not_a_secondary_limit():
state, detail = classify(403, PRIMARY, headers(0))
assert state == "primary"
assert "x-ratelimit-reset" in detail
def test_headroom_left_is_enough_to_suspect_a_secondary_limit():
# The wording has changed before, so the fallback must not need it.
state, detail = classify(403, '{"message":"Something new"}', headers(4321))
assert state == "secondary-suspected"
assert "4321" in detail
def test_a_403_with_no_rate_limit_headers_is_a_permissions_problem():
state, _ = classify(403, DENIED, {})
assert state == "forbidden"
def test_header_case_does_not_change_the_verdict():
lower = {"x-ratelimit-remaining": "0"}
assert classify(403, PRIMARY, lower)[0] == "primary"
def test_a_404_is_not_a_throttle():
assert classify(404, '{"message":"Not Found"}', headers())[0] == "other"
def test_a_success_is_reported_with_its_headroom():
state, detail = classify(200, "{}", headers(4999))
assert state == "ok"
assert "4999" in detail
def test_overlap_of_sequential_requests_is_one():
assert peak_overlap([(0.0, 1.0), (1.0, 2.0), (2.0, 3.0)]) == 1
def test_overlap_counts_only_spans_open_at_the_same_instant():
assert peak_overlap([(0.0, 3.0), (1.0, 2.0), (1.5, 4.0)]) == 3
assert peak_overlap([(0.0, 1.0), (0.5, 2.0)]) == 2
def test_an_empty_probe_has_no_overlap():
assert peak_overlap([]) == 0
assert peak_overlap(None) == 0
def test_a_reversed_span_is_still_measured():
assert peak_overlap([(2.0, 0.0), (1.0, 1.5)]) == 2
def test_any_throttled_response_beats_a_low_peak():
state, detail = verdict(3, ["ok", "secondary", "ok"])
assert state == "tripped"
assert "1 of 3" in detail
def test_a_peak_at_the_ceiling_is_reported_even_when_nothing_failed():
state, _ = verdict(100, ["ok"] * 100)
assert state == "over-ceiling"
assert verdict(85, ["ok"])[0] == "near-ceiling"
def test_clear_does_not_claim_the_client_is_safe():
state, detail = verdict(6, ["ok", "ok"])
assert state == "clear"
assert "headroom API" in detail
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, peakOverlap, verdict } from './github-concurrency-probe.mjs';
const SECONDARY = '{"message":"You have exceeded a secondary rate limit. ' +
'Please wait a few minutes before you try again."}';
const PRIMARY = '{"message":"API rate limit exceeded for user ID 12345."}';
const DENIED = '{"message":"Resource not accessible by integration"}';
const headers = (remaining = 4800, extra = {}) => ({
'X-RateLimit-Limit': '5000',
'X-RateLimit-Used': String(5000 - remaining),
'X-RateLimit-Remaining': String(remaining),
...extra,
});
test('a secondary limit is named in the body', () => {
const [state, detail] = classify(403, SECONDARY, headers(4800));
assert.equal(state, 'secondary');
assert.match(detail, /4800/);
});
test('the same message on a 429 classifies identically', () => {
assert.equal(classify(429, SECONDARY, headers(4800))[0], 'secondary');
});
test('an empty bucket is the primary quota, not a secondary limit', () => {
const [state, detail] = classify(403, PRIMARY, headers(0));
assert.equal(state, 'primary');
assert.match(detail, /x-ratelimit-reset/);
});
test('headroom left is enough to suspect a secondary limit', () => {
const [state, detail] = classify(403, '{"message":"Something new"}', headers(4321));
assert.equal(state, 'secondary-suspected');
assert.match(detail, /4321/);
});
test('a 403 with no rate-limit headers is a permissions problem', () => {
assert.equal(classify(403, DENIED, {})[0], 'forbidden');
});
test('header case does not change the verdict', () => {
assert.equal(classify(403, PRIMARY, { 'x-ratelimit-remaining': '0' })[0], 'primary');
});
test('a 404 is not a throttle', () => {
assert.equal(classify(404, '{"message":"Not Found"}', headers())[0], 'other');
});
test('a success is reported with its headroom', () => {
const [state, detail] = classify(200, '{}', headers(4999));
assert.equal(state, 'ok');
assert.match(detail, /4999/);
});
test('overlap of sequential requests is one', () => {
assert.equal(peakOverlap([[0, 1], [1, 2], [2, 3]]), 1);
});
test('overlap counts only spans open at the same instant', () => {
assert.equal(peakOverlap([[0, 3], [1, 2], [1.5, 4]]), 3);
assert.equal(peakOverlap([[0, 1], [0.5, 2]]), 2);
});
test('an empty probe has no overlap', () => {
assert.equal(peakOverlap([]), 0);
assert.equal(peakOverlap(null), 0);
});
test('a reversed span is still measured', () => {
assert.equal(peakOverlap([[2, 0], [1, 1.5]]), 2);
});
test('any throttled response beats a low peak', () => {
const [state, detail] = verdict(3, ['ok', 'secondary', 'ok']);
assert.equal(state, 'tripped');
assert.match(detail, /1 of 3/);
});
test('a peak at the ceiling is reported even when nothing failed', () => {
assert.equal(verdict(100, new Array(100).fill('ok'))[0], 'over-ceiling');
assert.equal(verdict(85, ['ok'])[0], 'near-ceiling');
});
test('clear does not claim the client is safe', () => {
const [state, detail] = verdict(6, ['ok', 'ok']);
assert.equal(state, 'clear');
assert.match(detail, /headroom API/);
});
FAQ
Can I check how close I am to a secondary rate limit before I hit one?
No, and this is worth being blunt about. There is no x-ratelimit-* header for secondary limits and GET /rate_limit documents itself as reporting primary quota only. A secondary limit becomes observable at the moment you exceed it, in the body of the 403 or 429 and in the retry-after header on it. Anything that claims to show you secondary headroom is inferring from your own request pattern, not reading a number GitHub published.
Why does x-ratelimit-remaining still show thousands when I am being refused?
Because it is describing a different limit. The hourly bucket counts requests over an hour; the secondary limits count bursts, concurrency and CPU time. You can spend two hundred requests in one second and still hold 4,800 of your 5,000. That mismatch is not a bug in the headers, it is the single most reliable way to tell the two failures apart, which is why the classifier branches on it.
Does GraphQL have its own concurrency allowance?
No. The 100-concurrent-request ceiling is shared across REST and GraphQL, even though the two have entirely separate primary buckets. A job that runs REST calls and GraphQL queries in parallel is spending one concurrency budget from two places, which is a common way to trip this while both point counters look healthy.
Is the probe safe to run against production credentials?
It only issues GETs, and its default endpoint is GET /rate_limit, which does not count against the primary rate limit. The concurrency argument is clamped at the documented ceiling, because deliberately exceeding it spends a quota that is shared with every other process using that token in order to learn something the documentation already states.
What concurrency should I actually use?
Lower than you think, and fixed rather than derived from the input. Five to ten in flight is comfortable for reads and leaves room for whatever else holds the same token; anything that creates content should be serialised to one at a time with a pause between items. The important change is not the number, it is that the number stops being repos.length.
Related field notes
- Ignoring retry-after extends the throttle
- Bulk creation exceeds 80 a minute
- per_page is unset so every list costs more
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits for the REST API — GitHub Docs
- Best practices for using the REST API — GitHub Docs
- Rate limit — GitHub REST API
- Troubleshooting the REST API — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.