Diagnostic GitHub API
expensive requests are killed at ten seconds with a 502
One call in the whole integration returns 502 Bad Gateway. Everything else is fine, the token is fine, the status page is green, and the call works on every repository except the big one. So the retry wrapper does what retry wrappers do: it waits a second and issues the identical expensive request, which takes the identical ten seconds and fails in the identical way, three times, before the job gives up and pages somebody.
GitHub gives up on a request it cannot serve in about ten seconds and answers with a gateway error rather than a 4xx. A 502 or 504 that arrives reliably, on one path, after roughly ten seconds, while cheap calls on the same token are instant, is not an outage and not a throttle. It is the API telling you the query is too expensive to answer.
The repair is to make the request smaller, not to send it again: lower per_page, add a date or path filter, split a comparison into ranges, or ask GraphQL for only the fields you need. Capture x-github-request-id from the failing response before you change anything, because it is the first thing support asks for and it does not survive the retry.
The problem in plain words
Every retry library in existence treats 5xx as transient, and it is right almost all of the time. A gateway error normally means something behind the proxy fell over and will be back shortly, so the correct response is to wait and try again. That reasoning is what makes this failure so durable: the code is doing the textbook thing, and the textbook thing is precisely wrong here, because the request will cost exactly as much the second time.
It also fails to look like a bug. It is one endpoint out of forty, it works in staging where the repository has eleven branches, and it works against every repository except the monorepo that everyone actually cares about. So it gets logged as flakiness, filed as a network issue, and worked around with a longer backoff and more attempts, which is the one change guaranteed to make it slower without making it work.
The thing that finally gets it fixed is usually the clock. Somebody notices the failures all take about the same time, and that the time is suspiciously round. Until then the evidence is unhelpfully empty: the body is a short generic message, there is no rate-limit header to read, no error code to look up, and x-github-request-id is sitting in a response object that the retry wrapper threw away in order to try again.
Why it happens
The cutoff is on the server and it applies per request. GitHub terminates a request it cannot complete in roughly ten seconds. Your own client timeout is a separate number, and if yours is shorter you never see the gateway error at all — you see a client-side timeout instead and blame the network. Raising your timeout above the server's cutoff is the first step, because you cannot diagnose a response you never waited for.
A gateway error carries no instructions. This is what separates it from every throttling failure in this section. A secondary rate limit hands you retry-after, and an exhausted primary quota hands you x-ratelimit-reset; in both cases the response tells you exactly how long to wait, and waiting is the repair. A 502 hands you nothing, because there is no amount of waiting that makes the query cheaper. Distinguishing the two is the first branch the script takes, by name, so a throttle never gets misfiled as a timeout.
Retrying is not neutral, it is the wrong direction. Each attempt spends ten seconds of wall clock, one request against your hourly quota, and real work on GitHub's side. Three attempts is thirty seconds to arrive at the same answer with less quota than you started with. The only retry worth making is a retry of a different, smaller request, which is why the script prints the narrowed version of your call rather than a backoff schedule.
Speed is the tell. A gateway error that comes back in 300 milliseconds is not your query running out of time; that is an incident, a proxy in front of you, or a load balancer with nothing to talk to, and the repair is to check the status page and retry. A gateway error at ten seconds is yours. Same status code, opposite diagnosis, and elapsed time is the only thing in the response that tells them apart — so the check times every attempt and reports the number.
The success right below the line is the interesting one. A call that returns 200 after nine seconds is not healthy. It is one busy afternoon, one more branch or one more merge away from crossing the cutoff, and it will cross it in production first. Treating a slow success as a finding is the only way this gets fixed before it becomes an incident, so the check reports it as its own state rather than as a pass.
This is not a partial answer. A search that runs out of time comes back 200 with incomplete_results set and gives you what it found. That is the same underlying constraint expressed as a shortfall you can read in the body. Here the request produces nothing at all, which is at least honest: you cannot mistake a 502 for data, and no cache will quietly keep it.
The fix, as a flow
Two timed GETs against the path under test plus one free baseline against GET /rate_limit, which is the only endpoint that answers without consuming quota and therefore the only honest control here. Everything else is a classifier over three inputs: a status code, an elapsed time and the response headers. Rate-limit headers are checked first and hard, because a throttle misfiled as a timeout sends somebody off rewriting a query that was never the problem, and the two repairs have nothing in common.
How to fix it
Raise your own timeout above the server's cutoff
If your client gives up at five seconds you will never see the gateway error, only a client-side timeout, and you will spend the afternoon looking at the network. Set the client timeout to thirty seconds for the duration of the investigation. The script reports a client-side timeout as its own state so the two are never confused.
Time the failing call, twice
Reproducibility is most of the diagnosis. Two attempts that both fail at about ten seconds is a cost problem. Two attempts where one fails in 300 milliseconds and the other succeeds is an incident, and no amount of rewriting your query will help. The script makes both attempts, prints each elapsed time, and classifies on the pair.
Take a free baseline against GET /rate_limit
GET /rate_limit is instant and does not consume quota, which makes it the ideal control. If the baseline is also slow, your problem is the network or the machine and nothing about the endpoint matters yet. If the baseline is 80 milliseconds and the call under test is ten seconds, the endpoint is the whole story.
Rule out the throttle before you blame the cost
A 403 or 429 carrying retry-after, or a response with x-ratelimit-remaining: 0, is a rate limit and belongs to a different note with a different repair. The script checks for those headers first and refuses to call anything a timeout while they are present, because a misfiled throttle sends you off rewriting a query that was never the problem.
Keep the request id, then shrink the request
Read x-github-request-id off the failing response and record it; it is what support will ask for and the retry destroys it. Then halve per_page, add a since or a path filter, split the comparison into date ranges, or move the call to GraphQL and ask for fewer fields. The script prints the narrowed parameters for you to try; it never issues them itself.
How to check it worked
Once the call is narrowed, the same probe answers in well under the cutoff and the state changes from a finding to a description.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_timeout_502.py \
--path /repos/acme/monorepo/compare/v1.0.0...main --param per_page=100
# read cost: 2 request(s) against the core hourly quota (the baseline is free)
# baseline: GET /rate_limit answered in 0.09s and consumed no quota
# attempt 1: 502 after 10.3s (x-github-request-id C4E2:1F03:...)
# attempt 2: 502 after 10.1s
# timeout: 502 came back after 10.3s, at the cutoff GitHub applies to a single
# request. The query is too expensive to serve, not unlucky
# repair: make the request cheaper rather than sending it again
# try instead: per_page=50
# after the comparison was split into two ranges
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_timeout_502.py \
--path /repos/acme/monorepo/compare/v1.0.0...v1.1.0 --param per_page=50
# attempt 1: 200 after 1.4s
# ok: the call answered in 1.4s, comfortably inside the cutoffThe full code
Two timed GETs against the path under test, plus one free baseline against GET /rate_limit, which is the only endpoint that answers without consuming quota and therefore the only honest control. Everything else is a pure classifier over three inputs: a status code, an elapsed time and the response headers. That is deliberate, because the whole note lives in the difference between a 502 at ten seconds and a 502 at three hundred milliseconds, and a rule that important should be testable without a network.
"""Tell an expensive request that GitHub gave up on from an incident.
Read only. Two timed GETs against the path under test, plus one free baseline
against GET /rate_limit. Nothing is written and the repair is printed rather
than performed.
GitHub terminates a request it cannot serve in about ten seconds and answers
with a gateway error rather than a 4xx. Retry logic reads 5xx as transient and
sends the identical expensive request again, which costs the same ten seconds
and fails the same way. The repair is to make the request smaller, so this
script prints a narrowed version of your parameters rather than a backoff.
What this can and cannot see: the API does not report a per-request time budget
anywhere, so the cutoff is inferred from elapsed time. A gateway error that
arrives quickly is an incident rather than a cost problem, and the script says
so rather than sending you to rewrite a query that was fine.
Environment:
GITHUB_TOKEN a token with read access to the repository
"""
import argparse
import json
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_timeout_502")
API = "https://api.github.com"
UA = "github-timeout-502/1.0"
# The server-side budget for a single request, in seconds. Approximate by
# nature: it is not published as a header and cannot be read from a response.
CUTOFF_SECONDS = 10.0
# How close to the cutoff still counts as "this ran out of time". A success at
# nine seconds is not healthy, it is next week's failure.
TOLERANCE = 2.0
# The statuses a killed request comes back as. 500 is deliberately not here: it
# is a different failure and calling it a timeout would be a guess.
GATEWAY = (502, 503, 504)
# The largest page size worth suggesting when nobody set one.
MAX_PER_PAGE = 100
def lower_headers(headers):
"""Headers keyed by lowercase name. Pure.
HTTP header names are case-insensitive and every client returns them in a
different case, so a check that reads Retry-After from a dict typed by hand
finds nothing and reports a timeout that was really a throttle.
"""
return {str(k).lower(): v for k, v in (headers or {}).items()}
def request_id(headers):
"""The value support will ask for, or None. Pure."""
return lower_headers(headers).get("x-github-request-id")
def is_gateway(status):
"""Whether this status is the shape a killed request comes back as. Pure."""
try:
return int(status) in GATEWAY
except (TypeError, ValueError):
return False
def is_throttled(status, headers):
"""Whether the response is a rate limit rather than a timeout. Pure.
Checked before anything else, because a throttle misfiled as a timeout
sends you off rewriting a query that was never the problem. The two are
separated by headers, not by status: a secondary limit answers 403 or 429
with retry-after, and an exhausted primary quota answers with a remaining
count of zero.
"""
h = lower_headers(headers)
try:
code = int(status)
except (TypeError, ValueError):
return False
if code not in (403, 429):
return False
return "retry-after" in h or str(h.get("x-ratelimit-remaining", "")).strip() == "0"
def near_cutoff(elapsed, cutoff=CUTOFF_SECONDS, tolerance=TOLERANCE):
"""Whether this call ran long enough to have been killed for it. Pure."""
try:
return float(elapsed) >= float(cutoff) - float(tolerance)
except (TypeError, ValueError):
return False
def classify(status, elapsed, headers=None):
"""Classify one timed attempt. Pure. Returns (state, detail)."""
try:
secs = float(elapsed)
except (TypeError, ValueError):
secs = None
if status is None:
if secs is not None and secs >= CUTOFF_SECONDS:
return ("client-timeout",
"your own client gave up after %.1fs, which is at or past "
"the server's own budget, so there is no response to read."
% secs)
return ("unknown", "the attempt produced neither a status nor a usable "
"elapsed time.")
try:
code = int(status)
except (TypeError, ValueError):
return ("unknown", "the attempt produced no readable status.")
if is_throttled(code, headers):
return ("throttled",
"%d carries rate-limit headers, so this is a throttle and not a "
"timeout. The response says how long to wait and waiting is the "
"repair." % code)
if is_gateway(code):
if secs is not None and near_cutoff(secs):
return ("timeout",
"%d came back after %.1fs, at the cutoff GitHub applies to "
"a single request. The query is too expensive to serve, not "
"unlucky." % (code, secs))
return ("gateway-early",
"%d came back after %.1fs, far short of the cutoff, so this is "
"not your query running out of time. Check the status page "
"before rewriting anything." % (code, secs if secs is not None else -1.0))
if 500 <= code < 600:
return ("server-other",
"%d is a server error of a different shape. It is not the "
"per-request cutoff and it is not a throttle." % code)
if 400 <= code < 500:
return ("client-error",
"%d is a client error, so the request was understood and "
"refused rather than abandoned partway through." % code)
if secs is not None and near_cutoff(secs):
return ("slow-success",
"the call answered %d in %.1fs, inside the tolerance of the "
"%.0fs cutoff. It works today and fails on the week the "
"repository grows." % (code, secs, CUTOFF_SECONDS))
return ("ok",
"the call answered %d in %.1fs, comfortably inside the cutoff."
% (code, secs if secs is not None else -1.0))
def retry_repeats_it(state):
"""Whether sending the identical request again reproduces this. Pure."""
return state in ("timeout", "client-timeout")
def wasted_retries(state, retries):
"""Attempts a retry wrapper would spend to no purpose at all. Pure."""
try:
n = int(retries)
except (TypeError, ValueError):
return 0
return max(0, n) if retry_repeats_it(state) else 0
def narrow(params):
"""A cheaper version of the same request. Pure.
Halving the page size is the one narrowing that applies to every list
endpoint without knowing anything about the query. Everything else worth
trying is specific to the call and is printed as prose instead.
"""
out = dict(params or {})
try:
size = int(out.get("per_page", MAX_PER_PAGE))
except (TypeError, ValueError):
size = MAX_PER_PAGE
out["per_page"] = max(1, size // 2)
return out
def narrowing_exhausted(params):
"""Whether the page size can no longer be halved. Pure."""
try:
return int((params or {}).get("per_page", MAX_PER_PAGE)) <= 1
except (TypeError, ValueError):
return False
def repair(state, params=None):
"""The sentence a reader has to act on. Pure."""
if state == "timeout":
base = ("make the request cheaper rather than sending it again: halve "
"per_page, add a date or path filter, split a comparison into "
"ranges, or ask GraphQL for only the fields you need. Record "
"x-github-request-id from the failing response first, because "
"the retry destroys it.")
if narrowing_exhausted(params):
return base + " The page size is already at 1, so the request has "\
"to be split by range or path instead."
return base
if state == "client-timeout":
return ("raise your own client timeout above the server's budget and "
"run this again. Until you wait longer than GitHub does you are "
"diagnosing your own deadline, not GitHub's.")
if state == "gateway-early":
return ("retry this one and check the status page. A gateway error that "
"arrives in a fraction of a second is not your query running "
"out of time.")
if state == "throttled":
return ("wait exactly as long as the response tells you to. This is the "
"rate-limit path, it has its own repair, and rewriting the "
"query will not change it.")
if state == "slow-success":
return ("narrow it now, while it still works. A call this close to the "
"cutoff crosses it on the busiest day of the quarter.")
if state == "server-other":
return ("retry once, then take x-github-request-id to support. This is "
"neither the per-request cutoff nor a throttle.")
if state == "client-error":
return "read the status: the request was refused, not abandoned."
if state == "ok":
return "nothing."
return "give the probe a path it can reach and a timeout longer than 10s."
def read_cost(paths, attempts=2):
"""Requests this run will spend against the core quota. Pure.
The baseline against GET /rate_limit is deliberately not counted: that
endpoint answers without consuming any, which is what makes it usable as a
control in a section full of notes about running out.
"""
try:
n, tries = len(paths or []), int(attempts)
except (TypeError, ValueError):
return 0
return n * max(0, tries)
def timed_get(session, path, params, timeout):
"""One timed GET. Returns (status, elapsed, headers)."""
started = time.monotonic()
try:
r = session.get(API + path, params=params, timeout=timeout)
except requests.exceptions.RequestException:
return None, time.monotonic() - started, {}
return r.status_code, time.monotonic() - started, dict(r.headers)
def parse_params(pairs):
"""key=value strings into a dict. Pure."""
out = {}
for pair in pairs or []:
if "=" in pair:
key, value = pair.split("=", 1)
out[key.strip()] = value.strip()
return out
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--path", action="append", required=True,
help="the expensive API path, e.g. "
"/repos/o/n/compare/v1...main. Repeatable.")
ap.add_argument("--param", action="append",
help="key=value query parameter. Repeatable.")
ap.add_argument("--attempts", type=int, default=2,
help="timed attempts per path. Two is enough to tell a "
"repeatable cost problem from an incident.")
ap.add_argument("--timeout", type=float, default=30.0,
help="client timeout in seconds. Keep it above the "
"server's own budget or you will only ever measure "
"your own deadline.")
ap.add_argument("--no-baseline", action="store_true",
help="skip the free GET /rate_limit control")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
params = parse_params(args.param)
log.info("read cost: %d request(s) against the core hourly quota "
"(the baseline is free)", read_cost(args.path, args.attempts))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
baseline = None
if not args.no_baseline:
_s, baseline, _h = timed_get(session, "/rate_limit", {}, args.timeout)
log.info("baseline: GET /rate_limit answered in %.2fs and consumed no "
"quota", baseline)
findings = []
for path in args.path:
attempts = []
for i in range(max(1, args.attempts)):
status, elapsed, headers = timed_get(session, path, params, args.timeout)
state, detail = classify(status, elapsed, headers)
rid = request_id(headers)
log.info("attempt %d: %s after %.1fs%s", i + 1, status, elapsed,
" (x-github-request-id %s)" % rid if rid else "")
attempts.append({"status": status, "elapsed": round(elapsed, 2),
"request_id": rid, "state": state, "detail": detail})
worst = attempts[0]
for a in attempts:
if retry_repeats_it(a["state"]):
worst = a
break
log.info("%s: %s", worst["state"], worst["detail"])
log.info("repair: %s", repair(worst["state"], params))
if worst["state"] in ("timeout", "slow-success"):
log.info("try instead: %s",
", ".join("%s=%s" % kv for kv in sorted(narrow(params).items())))
findings.append({
"path": path,
"baseline_seconds": round(baseline, 3) if baseline is not None else None,
"attempts": attempts,
"state": worst["state"],
"detail": worst["detail"],
"retry_reproduces_it": retry_repeats_it(worst["state"]),
"retries_wasted_on_three": wasted_retries(worst["state"], 3),
"narrowed_params": narrow(params),
"repair": repair(worst["state"], params),
})
print(json.dumps({"requests_spent": read_cost(args.path, args.attempts),
"findings": findings}, indent=2, default=str))
bad = {"timeout", "client-timeout", "slow-success"}
return 1 if any(f["state"] in bad for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Tell an expensive request that GitHub gave up on from an incident.
*
* Read only. Two timed GETs against the path under test, plus one free
* baseline against GET /rate_limit. Nothing is written and the repair is
* printed rather than performed.
*
* Environment:
* GITHUB_TOKEN a token with read access to the repository
* GITHUB_PATH the expensive API path, e.g. /repos/o/n/compare/v1...main
* GITHUB_PARAMS key=value pairs separated by commas, optional
* GITHUB_ATTEMPTS timed attempts, default 2
* GITHUB_TIMEOUT client timeout in seconds, default 30
*/
const API = 'https://api.github.com';
const UA = 'github-timeout-502/1.0';
/** The server-side budget for a single request, in seconds. */
export const CUTOFF_SECONDS = 10.0;
/** How close to the cutoff still counts as having run out of time. */
export const TOLERANCE = 2.0;
/** The statuses a killed request comes back as. 500 is deliberately not here. */
export const GATEWAY = [502, 503, 504];
export const MAX_PER_PAGE = 100;
/** A finite number, or null. Pure. Number(null) is 0, which would lie here. */
function toNumber(value) {
if (value === null || value === undefined || value === '') return null;
const n = Number(value);
return Number.isFinite(n) ? n : null;
}
/** Headers keyed by lowercase name. Pure. */
export function lowerHeaders(headers) {
const out = {};
for (const [k, v] of Object.entries(headers || {})) out[String(k).toLowerCase()] = v;
return out;
}
/** The value support will ask for, or null. Pure. */
export function requestId(headers) {
const h = lowerHeaders(headers);
return Object.prototype.hasOwnProperty.call(h, 'x-github-request-id')
? h['x-github-request-id'] : null;
}
/** Whether this status is the shape a killed request comes back as. Pure. */
export function isGateway(status) {
const n = toNumber(status);
return n !== null && GATEWAY.includes(n);
}
/** Whether the response is a rate limit rather than a timeout. Pure. */
export function isThrottled(status, headers) {
const h = lowerHeaders(headers);
const code = toNumber(status);
if (code === null || ![403, 429].includes(code)) return false;
if (Object.prototype.hasOwnProperty.call(h, 'retry-after')) return true;
return String(h['x-ratelimit-remaining'] ?? '').trim() === '0';
}
/** Whether this call ran long enough to have been killed for it. Pure. */
export function nearCutoff(elapsed, cutoff = CUTOFF_SECONDS, tolerance = TOLERANCE) {
const secs = toNumber(elapsed);
return secs !== null && secs >= cutoff - tolerance;
}
/** Classify one timed attempt. Pure. Returns [state, detail]. */
export function classify(status, elapsed, headers = null) {
const secs = toNumber(elapsed);
if (status === null || status === undefined) {
if (secs !== null && secs >= CUTOFF_SECONDS) {
return ['client-timeout',
`your own client gave up after ${secs.toFixed(1)}s, which is at or past `
+ "the server's own budget, so there is no response to read."];
}
return ['unknown', 'the attempt produced neither a status nor a usable elapsed time.'];
}
const code = toNumber(status);
if (code === null) return ['unknown', 'the attempt produced no readable status.'];
if (isThrottled(code, headers)) {
return ['throttled',
`${code} carries rate-limit headers, so this is a throttle and not a `
+ 'timeout. The response says how long to wait and waiting is the repair.'];
}
if (isGateway(code)) {
if (secs !== null && nearCutoff(secs)) {
return ['timeout',
`${code} came back after ${secs.toFixed(1)}s, at the cutoff GitHub `
+ 'applies to a single request. The query is too expensive to serve, '
+ 'not unlucky.'];
}
return ['gateway-early',
`${code} came back after ${(secs === null ? -1 : secs).toFixed(1)}s, far `
+ 'short of the cutoff, so this is not your query running out of time. '
+ 'Check the status page before rewriting anything.'];
}
if (code >= 500 && code < 600) {
return ['server-other',
`${code} is a server error of a different shape. It is not the `
+ 'per-request cutoff and it is not a throttle.'];
}
if (code >= 400 && code < 500) {
return ['client-error',
`${code} is a client error, so the request was understood and refused `
+ 'rather than abandoned partway through.'];
}
if (secs !== null && nearCutoff(secs)) {
return ['slow-success',
`the call answered ${code} in ${secs.toFixed(1)}s, inside the tolerance `
+ `of the ${CUTOFF_SECONDS.toFixed(0)}s cutoff. It works today and fails `
+ 'on the week the repository grows.'];
}
return ['ok',
`the call answered ${code} in ${(secs === null ? -1 : secs).toFixed(1)}s, `
+ 'comfortably inside the cutoff.'];
}
/** Whether sending the identical request again reproduces this. Pure. */
export function retryRepeatsIt(state) {
return state === 'timeout' || state === 'client-timeout';
}
/** Attempts a retry wrapper would spend to no purpose at all. Pure. */
export function wastedRetries(state, retries) {
const n = toNumber(retries);
if (n === null) return 0;
return retryRepeatsIt(state) ? Math.max(0, Math.trunc(n)) : 0;
}
/** A cheaper version of the same request. Pure. */
export function narrow(params) {
const out = { ...(params || {}) };
const size = toNumber(out.per_page) ?? MAX_PER_PAGE;
out.per_page = Math.max(1, Math.trunc(size / 2));
return out;
}
/** Whether the page size can no longer be halved. Pure. */
export function narrowingExhausted(params) {
const size = toNumber((params || {}).per_page);
return size === null ? false : size <= 1;
}
/** The sentence a reader has to act on. Pure. */
export function repair(state, params = null) {
if (state === 'timeout') {
const base = 'make the request cheaper rather than sending it again: halve '
+ 'per_page, add a date or path filter, split a comparison into ranges, or '
+ 'ask GraphQL for only the fields you need. Record x-github-request-id '
+ 'from the failing response first, because the retry destroys it.';
if (narrowingExhausted(params)) {
return `${base} The page size is already at 1, so the request has to be `
+ 'split by range or path instead.';
}
return base;
}
if (state === 'client-timeout') {
return "raise your own client timeout above the server's budget and run this "
+ 'again. Until you wait longer than GitHub does you are diagnosing your '
+ "own deadline, not GitHub's.";
}
if (state === 'gateway-early') {
return 'retry this one and check the status page. A gateway error that '
+ 'arrives in a fraction of a second is not your query running out of time.';
}
if (state === 'throttled') {
return 'wait exactly as long as the response tells you to. This is the '
+ 'rate-limit path, it has its own repair, and rewriting the query will '
+ 'not change it.';
}
if (state === 'slow-success') {
return 'narrow it now, while it still works. A call this close to the cutoff '
+ 'crosses it on the busiest day of the quarter.';
}
if (state === 'server-other') {
return 'retry once, then take x-github-request-id to support. This is '
+ 'neither the per-request cutoff nor a throttle.';
}
if (state === 'client-error') {
return 'read the status: the request was refused, not abandoned.';
}
if (state === 'ok') return 'nothing.';
return 'give the probe a path it can reach and a timeout longer than 10s.';
}
/** Requests this run will spend against the core quota. Pure. */
export function readCost(paths, attempts = 2) {
const n = Array.isArray(paths) ? paths.length : 0;
const tries = toNumber(attempts);
if (tries === null) return 0;
return n * Math.max(0, Math.trunc(tries));
}
/** key=value strings into an object. Pure. */
export function parseParams(pairs) {
const out = {};
for (const pair of pairs || []) {
const i = String(pair).indexOf('=');
if (i > 0) out[String(pair).slice(0, i).trim()] = String(pair).slice(i + 1).trim();
}
return out;
}
function headersFor(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
};
}
async function timedGet(token, path, params, timeoutSeconds) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params || {})) url.searchParams.set(k, String(v));
const started = process.hrtime.bigint();
const elapsed = () => Number(process.hrtime.bigint() - started) / 1e9;
try {
const res = await fetch(url, {
headers: headersFor(token),
signal: AbortSignal.timeout(timeoutSeconds * 1000),
});
return { status: res.status, elapsed: elapsed(), headers: Object.fromEntries(res.headers) };
} catch {
return { status: null, elapsed: elapsed(), headers: {} };
}
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const path = process.env.GITHUB_PATH;
if (!token || !path) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_PATH');
process.exitCode = 2;
return;
}
const params = parseParams((process.env.GITHUB_PARAMS || '').split(',').filter(Boolean));
const attempts = Number(process.env.GITHUB_ATTEMPTS || 2);
const timeoutSeconds = Number(process.env.GITHUB_TIMEOUT || 30);
console.log(`read cost: ${readCost([path], attempts)} request(s) against the core `
+ 'hourly quota (the baseline is free)');
const base = await timedGet(token, '/rate_limit', {}, timeoutSeconds);
console.log(`baseline: GET /rate_limit answered in ${base.elapsed.toFixed(2)}s `
+ 'and consumed no quota');
const tried = [];
for (let i = 0; i < Math.max(1, attempts); i += 1) {
const { status, elapsed, headers } = await timedGet(token, path, params, timeoutSeconds);
const [state, detail] = classify(status, elapsed, headers);
const rid = requestId(headers);
console.log(`attempt ${i + 1}: ${status} after ${elapsed.toFixed(1)}s`
+ (rid ? ` (x-github-request-id ${rid})` : ''));
tried.push({ status, elapsed: Number(elapsed.toFixed(2)), request_id: rid, state, detail });
}
const worst = tried.find((a) => retryRepeatsIt(a.state)) || tried[0];
console.log(`${worst.state}: ${worst.detail}`);
console.log(`repair: ${repair(worst.state, params)}`);
if (['timeout', 'slow-success'].includes(worst.state)) {
console.log(`try instead: ${Object.entries(narrow(params)).sort()
.map(([k, v]) => `${k}=${v}`).join(', ')}`);
}
console.log(JSON.stringify({
requests_spent: readCost([path], attempts),
findings: [{
path,
baseline_seconds: Number(base.elapsed.toFixed(3)),
attempts: tried,
state: worst.state,
detail: worst.detail,
retry_reproduces_it: retryRepeatsIt(worst.state),
retries_wasted_on_three: wastedRetries(worst.state, 3),
narrowed_params: narrow(params),
repair: repair(worst.state, params),
}],
}, null, 2));
process.exitCode = ['timeout', 'client-timeout', 'slow-success'].includes(worst.state) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The classifier is the note, so the tests are mostly about the boundaries it draws: a gateway error at ten seconds and the same status at three hundred milliseconds have to come out as different states, a throttle has to be caught before either of them however its headers are capitalised, and a success just under the cutoff has to be a finding rather than a pass. The rest pins the arithmetic a reader will act on: what a retry actually costs when the retry cannot work, what the narrowed request looks like when the page size is already at its floor, and a read cost that never counts the free baseline as spending.
from github_timeout_502 import (
CUTOFF_SECONDS, GATEWAY, classify, is_gateway, is_throttled, lower_headers,
narrow, narrowing_exhausted, near_cutoff, parse_params, read_cost, repair,
request_id, retry_repeats_it, wasted_retries,
)
THROTTLE = {"Retry-After": "60"}
EXHAUSTED = {"X-RateLimit-Remaining": "0"}
RID = {"X-GitHub-Request-Id": "C4E2:1F03:9AB"}
def test_only_gateway_shaped_statuses_count():
assert is_gateway(502)
assert is_gateway(504)
assert 500 not in GATEWAY
assert not is_gateway(500)
assert not is_gateway(200)
assert not is_gateway(None)
def test_headers_are_read_case_insensitively():
assert lower_headers(RID)["x-github-request-id"] == "C4E2:1F03:9AB"
assert request_id(RID) == "C4E2:1F03:9AB"
assert request_id({}) is None
assert request_id(None) is None
def test_a_throttle_is_recognised_before_anything_else():
assert is_throttled(403, THROTTLE)
assert is_throttled(429, EXHAUSTED)
assert not is_throttled(403, {})
assert not is_throttled(502, THROTTLE)
assert classify(403, 0.4, THROTTLE)[0] == "throttled"
assert classify(429, 0.2, EXHAUSTED)[0] == "throttled"
def test_the_cutoff_has_a_tolerance_and_it_is_generous():
assert near_cutoff(10.4)
assert near_cutoff(8.0)
assert not near_cutoff(7.9)
assert not near_cutoff(0.3)
assert not near_cutoff(None)
def test_a_gateway_error_at_the_cutoff_is_the_finding():
state, detail = classify(502, 10.4, RID)
assert state == "timeout"
assert "10.4s" in detail
assert "too expensive" in detail
def test_the_same_status_arriving_fast_is_a_different_diagnosis():
state, detail = classify(502, 0.3, {})
assert state == "gateway-early"
assert "status page" in detail
def test_a_success_just_under_the_line_is_not_a_pass():
state, detail = classify(200, 9.4, {})
assert state == "slow-success"
assert "fails on the week" in detail
assert classify(200, 0.4, {})[0] == "ok"
def test_the_other_failures_are_named_rather_than_lumped_in():
assert classify(500, 3.0, {})[0] == "server-other"
assert classify(404, 0.2, {})[0] == "client-error"
assert classify(None, 30.0, {})[0] == "client-timeout"
assert classify(None, None, {})[0] == "unknown"
assert classify("not a status", 1.0, {})[0] == "unknown"
def test_only_the_states_a_retry_cannot_fix_are_called_repeatable():
assert retry_repeats_it("timeout")
assert retry_repeats_it("client-timeout")
assert not retry_repeats_it("gateway-early")
assert not retry_repeats_it("throttled")
assert wasted_retries("timeout", 3) == 3
assert wasted_retries("gateway-early", 3) == 0
assert wasted_retries("timeout", None) == 0
def test_narrowing_halves_the_page_and_keeps_everything_else():
assert narrow({"per_page": 100})["per_page"] == 50
assert narrow({})["per_page"] == 50
assert narrow({"per_page": 1})["per_page"] == 1
assert narrow({"per_page": 40, "since": "2026-01-01"})["since"] == "2026-01-01"
assert not narrowing_exhausted({"per_page": 2})
assert narrowing_exhausted({"per_page": 1})
assert not narrowing_exhausted({})
def test_the_repair_for_a_timeout_never_says_retry():
text = repair("timeout", {"per_page": 100})
assert "cheaper" in text
assert "x-github-request-id" in text
assert "split by range" in repair("timeout", {"per_page": 1})
assert "wait exactly as long" in repair("throttled")
assert "status page" in repair("gateway-early")
assert repair("ok") == "nothing."
def test_the_baseline_is_never_counted_as_spending():
assert read_cost(["/a"], 2) == 2
assert read_cost(["/a", "/b"], 3) == 6
assert read_cost(["/a"], 0) == 0
assert read_cost([], 2) == 0
assert read_cost(None, 2) == 0
def test_parameters_survive_a_value_containing_an_equals_sign():
assert parse_params(["per_page=100", "q=repo:acme/x is:open"]) == {
"per_page": "100", "q": "repo:acme/x is:open"}
assert parse_params(["base=v1...main"])["base"] == "v1...main"
assert parse_params(None) == {}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
GATEWAY, classify, isGateway, isThrottled, lowerHeaders, narrow,
narrowingExhausted, nearCutoff, parseParams, readCost, repair, requestId,
retryRepeatsIt, wastedRetries,
} from './github-timeout-502.mjs';
const THROTTLE = { 'Retry-After': '60' };
const EXHAUSTED = { 'X-RateLimit-Remaining': '0' };
const RID = { 'X-GitHub-Request-Id': 'C4E2:1F03:9AB' };
test('only gateway-shaped statuses count', () => {
assert.ok(isGateway(502));
assert.ok(isGateway(504));
assert.ok(!GATEWAY.includes(500));
assert.ok(!isGateway(500));
assert.ok(!isGateway(200));
assert.ok(!isGateway(null));
});
test('headers are read case-insensitively', () => {
assert.equal(lowerHeaders(RID)['x-github-request-id'], 'C4E2:1F03:9AB');
assert.equal(requestId(RID), 'C4E2:1F03:9AB');
assert.equal(requestId({}), null);
assert.equal(requestId(null), null);
});
test('a throttle is recognised before anything else', () => {
assert.ok(isThrottled(403, THROTTLE));
assert.ok(isThrottled(429, EXHAUSTED));
assert.ok(!isThrottled(403, {}));
assert.ok(!isThrottled(502, THROTTLE));
assert.equal(classify(403, 0.4, THROTTLE)[0], 'throttled');
assert.equal(classify(429, 0.2, EXHAUSTED)[0], 'throttled');
});
test('the cutoff has a tolerance and it is generous', () => {
assert.ok(nearCutoff(10.4));
assert.ok(nearCutoff(8.0));
assert.ok(!nearCutoff(7.9));
assert.ok(!nearCutoff(0.3));
assert.ok(!nearCutoff(null));
});
test('a gateway error at the cutoff is the finding', () => {
const [state, detail] = classify(502, 10.4, RID);
assert.equal(state, 'timeout');
assert.match(detail, /10\.4s/);
assert.match(detail, /too expensive/);
});
test('the same status arriving fast is a different diagnosis', () => {
const [state, detail] = classify(502, 0.3, {});
assert.equal(state, 'gateway-early');
assert.match(detail, /status page/);
});
test('a success just under the line is not a pass', () => {
const [state, detail] = classify(200, 9.4, {});
assert.equal(state, 'slow-success');
assert.match(detail, /fails on the week/);
assert.equal(classify(200, 0.4, {})[0], 'ok');
});
test('the other failures are named rather than lumped in', () => {
assert.equal(classify(500, 3.0, {})[0], 'server-other');
assert.equal(classify(404, 0.2, {})[0], 'client-error');
assert.equal(classify(null, 30.0, {})[0], 'client-timeout');
assert.equal(classify(null, null, {})[0], 'unknown');
assert.equal(classify('not a status', 1.0, {})[0], 'unknown');
});
test('only the states a retry cannot fix are called repeatable', () => {
assert.ok(retryRepeatsIt('timeout'));
assert.ok(retryRepeatsIt('client-timeout'));
assert.ok(!retryRepeatsIt('gateway-early'));
assert.ok(!retryRepeatsIt('throttled'));
assert.equal(wastedRetries('timeout', 3), 3);
assert.equal(wastedRetries('gateway-early', 3), 0);
assert.equal(wastedRetries('timeout', null), 0);
});
test('narrowing halves the page and keeps everything else', () => {
assert.equal(narrow({ per_page: 100 }).per_page, 50);
assert.equal(narrow({}).per_page, 50);
assert.equal(narrow({ per_page: 1 }).per_page, 1);
assert.equal(narrow({ per_page: 40, since: '2026-01-01' }).since, '2026-01-01');
assert.ok(!narrowingExhausted({ per_page: 2 }));
assert.ok(narrowingExhausted({ per_page: 1 }));
assert.ok(!narrowingExhausted({}));
});
test('the repair for a timeout never says retry', () => {
const text = repair('timeout', { per_page: 100 });
assert.match(text, /cheaper/);
assert.match(text, /x-github-request-id/);
assert.match(repair('timeout', { per_page: 1 }), /split by range/);
assert.match(repair('throttled'), /wait exactly as long/);
assert.match(repair('gateway-early'), /status page/);
assert.equal(repair('ok'), 'nothing.');
});
test('the baseline is never counted as spending', () => {
assert.equal(readCost(['/a'], 2), 2);
assert.equal(readCost(['/a', '/b'], 3), 6);
assert.equal(readCost(['/a'], 0), 0);
assert.equal(readCost([], 2), 0);
assert.equal(readCost(null, 2), 0);
});
test('parameters survive a value containing an equals sign', () => {
assert.deepEqual(parseParams(['per_page=100', 'q=repo:acme/x is:open']),
{ per_page: '100', q: 'repo:acme/x is:open' });
assert.equal(parseParams(['base=v1...main']).base, 'v1...main');
assert.deepEqual(parseParams(null), {});
});
FAQ
Is a 502 from GitHub ever worth retrying?
Yes, when it comes back fast. A gateway error in a few hundred milliseconds is an incident, a proxy problem or a load balancer with nothing behind it, and a retry is exactly right. A gateway error at ten seconds is your query being killed for cost, and the retry costs another ten seconds, another request against your quota and arrives at the same place. The elapsed time is the only thing in the response that separates the two, which is why the script prints it on every attempt.
How is this different from being rate limited?
A throttle tells you what to do. A secondary limit answers with retry-after and an exhausted primary quota answers with x-ratelimit-reset, and in both cases waiting the stated interval is the whole repair. A timeout gives you no header, no interval and nothing to wait for, because time is not what is missing. The script checks for the rate-limit headers first and refuses to classify anything as a timeout while they are present.
Where exactly is the ten seconds documented?
GitHub documents that it terminates requests it cannot serve in about ten seconds and asks you to make the query cheaper rather than retry it. The number is approximate and it is not returned in any header, so a script can only infer it from elapsed time. That is why the check uses a tolerance rather than an equality, and why a success at nine seconds is reported as a finding: near the boundary, the exact value matters less than the distance from it.
What do I actually change to make the request cheaper?
Whatever makes the server do less work. Halve per_page, which the script prints for you. Add since or a path filter to a commits listing. Split a comparison of two distant tags into a few smaller ranges and stitch them. Move the call to GraphQL and request only the fields you use, which for wide REST resources is often the biggest saving available. If none of those are possible, the shape of the data has outgrown the endpoint and the answer is usually a different endpoint.
Why bother recording x-github-request-id?
Because it identifies your exact failing request in GitHub's logs and it is the first thing support asks for. It only exists on the response, and the response is what a retry wrapper throws away before trying again, so by the time a human is looking at the incident the identifier has usually been destroyed several times over. The script reads it off every attempt and prints it, which costs nothing and occasionally saves a week.
Related field notes
- The client ignores retry-after and hammers on
- incomplete_results is true and nobody checks
- A pull request's lists are both capped
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Troubleshooting the REST API — GitHub Docs
- Resource limitations — GitHub GraphQL API
- Best practices for using the REST API — GitHub Docs
- GitHub Status
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.