Diagnostic GitHub API
GraphQL points run out in a bucket separate from REST
Every GraphQL call is coming back 200 with errors[0].type set to RATE_LIMITED. The health check is green, because the health check is a REST call and REST calls with the same token are working perfectly. Somebody is going to spend an hour looking for a difference between the two code paths, and the difference is not in the code: they are billed from two different buckets, and only one of them is empty.
GraphQL has its own hourly budget, measured in points rather than requests: 5,000 points an hour for a user token, 1,000 for the GITHUB_TOKEN inside GitHub Actions, 10,000 on Enterprise Cloud. Draining it has no effect on the REST core bucket and draining core has no effect on it. That is why a REST-based health check reports green while every GraphQL query in the system is failing.
Read GET /rate_limit and look at resources.graphql, not resources.core. That endpoint is free and covers both. Then put rateLimit { cost remaining } into your real queries so every response tells you its own price, and throttle on remaining instead of finding out when it reaches zero. A query costing 12 points means the budget is 416 queries an hour, and that is the number to plan against.
The problem in plain words
The migration is what sets this up. Somebody replaces a hundred REST calls with one GraphQL query, which is the correct move and does exactly what it promised: the REST bucket stops being the constraint. Nobody adds a second gauge, because the first gauge is now reading comfortably and a gauge that reads comfortably does not invite questions. The monitoring, the alerting and the mental model all stay pointed at core.
What makes the failure so disorienting when it lands is the shape of the evidence. The token works. You can prove it works, in a terminal, in front of somebody, with a curl against a REST endpoint that returns instantly. The GraphQL calls from the same process with the same header fail every time. Every instinct says the problem is in the client, and hours go into the client.
The second thing that goes wrong is arithmetic. Even once somebody has found the right bucket, "5,000" gets read as five thousand queries, because that is what the number means on the REST side. A query that fans out over a few connections can cost ten or twenty points, so the real budget might be two or three hundred calls an hour. A job sized against the wrong unit does not fail gradually; it works fine in testing at ten queries and dies at four hundred.
Why it happens
Two buckets, no relationship. GET /rate_limit returns resources.core and resources.graphql as independent objects with their own limit, used, remaining and reset. Nothing you do to one moves the other. This script prints both side by side for exactly that reason: seeing 4,983 remaining next to 0 remaining ends the argument about whether the token is fine faster than any explanation.
The unit is points, and points are not requests. A query's cost is derived from how many connections it traverses and how many items it asks for, and it is reported back to you in the response if you ask for it. One query can cost one point or several dozen. The only honest way to state your budget is to measure the cost of the query you actually send and divide, which is what the script does rather than repeating the headline number.
The limit tells you which budget you are on. A user token gets 5,000 points an hour and the GITHUB_TOKEN issued to a GitHub Actions workflow gets 1,000. A job that works on a laptop and fails in CI at a fifth of the volume is usually this, not a difference in code, and reading resources.graphql.limit identifies it in one call. The script names which budget the observed limit implies.
This is not either of the REST buckets. The hourly REST quota running out is its own problem with its own note, and the per-minute point cap on a single hot REST endpoint is a third thing again, undocumented in the response and only visible after the fact. Three buckets, three notes, and this one owns the 5,000-points-an-hour GraphQL budget and nothing else. When the script sees REST in trouble and GraphQL healthy it says so and sends you to the right note rather than giving advice about points.
Asking for the budget in-band costs one point. Adding rateLimit { limit cost remaining resetAt } to a query you were sending anyway is free of extra round trips and tells you the exact price of that query. Sending it as a query of its own costs a point, which is why this script uses the free REST endpoint by default and only goes in-band when you ask it to.
Consumption is not attributable. The bucket is shared by every process holding that token, and the API reports the drain without ever saying who caused it. If the budget is vanishing and you cannot account for it, the answer is a separate token per workload rather than a better query, because separate tokens are the only way to make the number mean something per job.
The fix, as a flow
The default run spends nothing at all. GET /rate_limit reports the GraphQL bucket beside the REST one and is documented not to count against either, so the two can be printed side by side, which is what ends the argument about whether the token is broken. The observed limit is then read backwards to name the actor it belongs to, since a job that works on a laptop and dies in CI at a fifth of the volume is usually a 1,000-point budget rather than a code difference. Finally the points are divided by a measured query cost, because queries per hour is the only unit anybody can put in a scheduler.
How to fix it
Read both buckets in one free call
GET /rate_limit returns resources.core and resources.graphql together and consumes neither. Print them side by side. If GraphQL is at zero while core is nearly full, the investigation is over in one line and nobody has to look at the client at all.
Check which budget the limit implies
A limit of 5,000 is a user token, 1,000 is the GITHUB_TOKEN inside a workflow, 10,000 is Enterprise Cloud. If the number surprises you, that is the finding: the job is running as a different actor than you assumed, and no amount of query tuning fixes a fivefold difference in budget.
Measure what your query actually costs
Add rateLimit { cost remaining } to the query your integration sends and read the cost back. This is the only way to know the price; it depends on the connections traversed and the first values requested. The script will do it in-band for one point if you pass --in-band, and it says so before spending.
Convert points into queries, because that is what you schedule
Divide the budget by the measured cost. Five thousand points at twelve points a query is 416 queries an hour, or roughly one every nine seconds. That number is schedulable and "5,000 points" is not, and getting it wrong is what turns a job that passed testing at ten queries into one that dies at four hundred.
Throttle on remaining, and split the token if you cannot account for the drain
Read remaining from every response and slow down before it reaches zero rather than reacting to RATE_LIMITED. If the budget is disappearing faster than your own queries explain, the bucket is shared with something else holding the same token, and the repair is a separate token per workload. The API will never tell you which process spent it.
How to check it worked
With the right gauge in front of you, the two buckets stop looking like one broken token.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_points.py
# point cost: 0 point(s). GET /rate_limit reports the GraphQL bucket and does
# not consume any of it.
# core 4211 / 5000 remaining, resets in 41m
# graphql 0 / 5000 remaining, resets in 12m
# budget: a limit of 5000 points/hour is a user token
# graphql-exhausted-rest-healthy: the GraphQL bucket is empty while core is at
# 84% remaining, so a REST health check reports green on a dead integration
# repair: throttle on resources.graphql.remaining, not on core
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_points.py --in-band
# point cost: 1 point(s) against the 5,000/hour GraphQL budget
# measured cost: 12 point(s) for this query shape
# at 12 points a query the budget is 416 queries/hour, one every 8.7sThe full code
The default run spends nothing: GET /rate_limit is a plain GET, it reports both buckets, and it is documented not to count against either. The in-band probe is opt-in, costs one point and is the only way to learn what a particular query shape actually prices at. Everything else is arithmetic held in pure functions: used fractions, seconds to reset, which budget a limit implies, and the conversion from points into the unit anybody can schedule against, which is queries per hour at a measured cost.
"""Read the GraphQL point budget, which is not the REST one.
Read only. The default run spends nothing at all: GET /rate_limit reports both
buckets and is documented not to count against either. The optional in-band
probe sends one query and costs one point, and it says so first.
Queries only. GitHub's GraphQL endpoint takes a document in the request body, so
a read is carried by POST there just as a write would be; that is transport, not
intent. Any document containing a mutation or a subscription is refused before a
socket opens. Nothing is written and the repair is printed rather than performed.
GraphQL is billed in points from its own hourly budget: 5,000 for a user token,
1,000 for the GITHUB_TOKEN inside GitHub Actions, 10,000 on Enterprise Cloud.
The REST core bucket is untouched by GraphQL traffic and vice versa, which is why
a REST health check reports green while every GraphQL call is failing.
What this can and cannot see: the bucket is shared by every process using the
token and the API never says which one spent it. A drain you cannot account for
is a reason to issue a separate token per workload, not a query to tune.
Environment:
GITHUB_TOKEN a token with read access to the GraphQL API
"""
import argparse
import json
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_graphql_points")
API = "https://api.github.com"
UA = "github-graphql-points/1.0"
# The published hourly point budgets, keyed by the actor they belong to. Read
# backwards, from an observed limit to the actor, this identifies a job that is
# running as something other than what its author assumed.
BUDGETS = {
5000: "a user token",
1000: "the GITHUB_TOKEN issued to a GitHub Actions workflow",
10000: "an Enterprise Cloud token",
}
# The smallest useful in-band probe. Asking for cost alone would not report the
# node count, and nodeCount is the number the query-shape note needs.
BUDGET_QUERY = "query { rateLimit { limit cost remaining used resetAt nodeCount } }"
# Below this fraction of budget left, slow down rather than discover zero.
TIGHT = 0.2
def strip_noise(document):
"""Remove GraphQL comments and string literals from a document. Pure."""
src = str(document or "")
out = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "#":
while i < n and src[i] != "\n":
i += 1
continue
if src.startswith('"""', i):
j = src.find('"""', i + 3)
i = n if j < 0 else j + 3
out.append(" ")
continue
if ch == '"':
i += 1
while i < n and src[i] != '"':
i += 2 if src[i] == "\\" else 1
i += 1
out.append(" ")
continue
out.append(ch)
i += 1
return "".join(out)
def operations(document):
"""The top-level operations in a document, in order. Pure."""
src = strip_noise(document)
ops, depth, word, declared = [], 0, "", None
for ch in src + " ":
if ch.isalnum() or ch == "_":
word += ch
continue
if word:
if depth == 0 and word in ("query", "mutation", "subscription", "fragment"):
declared = word
word = ""
if ch == "{":
if depth == 0:
ops.append(declared or "query")
declared = None
depth += 1
elif ch == "}":
depth = max(0, depth - 1)
return ops
def refusal(document):
"""Why this document will not be sent, or None if it is a read. Pure."""
ops = operations(document)
if not ops:
return "the document contains no operation to send."
for kind in ("mutation", "subscription"):
if kind in ops:
return ("the document contains a %s. This script sends queries only: "
"a query is a read, and the section it belongs to promises "
"its scripts never write." % kind)
return None
def bucket(rate_limit_body, name):
"""One resource object out of GET /rate_limit. Pure. None if absent."""
if not isinstance(rate_limit_body, dict):
return None
res = rate_limit_body.get("resources")
if not isinstance(res, dict):
return None
b = res.get(name)
return b if isinstance(b, dict) else None
def used_fraction(b):
"""How much of a bucket is gone, 0.0 to 1.0. Pure. None if unreadable."""
if not isinstance(b, dict):
return None
try:
limit = int(b.get("limit"))
remaining = int(b.get("remaining"))
except (TypeError, ValueError):
return None
if limit <= 0:
return None
return max(0.0, min(1.0, (limit - remaining) / float(limit)))
def seconds_to_reset(b, now):
"""Seconds until this bucket refills. Pure. None if unreadable."""
if not isinstance(b, dict):
return None
try:
return max(0, int(b.get("reset")) - int(now))
except (TypeError, ValueError):
return None
def identify_budget(limit):
"""Which actor an observed hourly limit implies. Pure."""
try:
n = int(limit)
except (TypeError, ValueError):
return "an unreadable limit"
if n in BUDGETS:
return BUDGETS[n]
return ("a limit of %d, which matches none of the published budgets. Read "
"it as the truth and plan against it." % n)
def queries_left(remaining, cost_per_query):
"""How many more queries of this shape fit in what is left. Pure."""
try:
rem = int(remaining)
cost = int(cost_per_query)
except (TypeError, ValueError):
return None
if cost <= 0:
return None
return max(0, rem // cost)
def sustainable_rate(limit, cost_per_query):
"""Queries per hour this budget supports at a measured cost. Pure."""
try:
lim = int(limit)
cost = int(cost_per_query)
except (TypeError, ValueError):
return None
if cost <= 0 or lim <= 0:
return None
return lim // cost
def seconds_between(limit, cost_per_query):
"""The gap to leave between queries to stay inside the budget. Pure."""
rate = sustainable_rate(limit, cost_per_query)
if not rate:
return None
return round(3600.0 / rate, 1)
def error_types(body):
"""The type of every entry in a GraphQL errors array. Pure."""
if not isinstance(body, dict):
return []
return [(e.get("type") or "UNTYPED") if isinstance(e, dict) else "UNTYPED"
for e in (body.get("errors") or [])]
def is_rate_limited(body):
"""Whether a GraphQL envelope reports the budget as spent. Pure."""
return "RATE_LIMITED" in error_types(body)
def in_band_cost(body):
"""The cost this query reported for itself. Pure. None if not asked for."""
if not isinstance(body, dict):
return None
data = body.get("data")
if not isinstance(data, dict):
return None
rl = data.get("rateLimit")
if not isinstance(rl, dict):
return None
try:
return int(rl.get("cost"))
except (TypeError, ValueError):
return None
def classify(graphql_b, core_b):
"""Compare the two buckets. Pure. Returns (state, detail).
The point of comparing rather than reporting one is that the confusing case
has a shape: one bucket empty, the other nearly full, same token.
"""
g = used_fraction(graphql_b)
c = used_fraction(core_b)
if g is None:
return ("unreadable",
"resources.graphql was not present in the response, so the "
"GraphQL budget cannot be read from it.")
g_left = 1.0 - g
c_left = 1.0 - c if c is not None else None
g_empty = (graphql_b or {}).get("remaining") == 0
c_empty = c_left is not None and (core_b or {}).get("remaining") == 0
if g_empty and c_empty:
return ("both-exhausted",
"both buckets are empty, so this is not the confusing case: "
"everything fails and everything is meant to.")
if g_empty:
return ("graphql-exhausted-rest-healthy",
"the GraphQL bucket is empty while core is at %d%% remaining, so "
"a REST health check reports green on a dead integration."
% round((c_left or 0) * 100))
if c_empty:
return ("rest-exhausted-graphql-healthy",
"core is empty and the GraphQL budget is at %d%% remaining. That "
"is the REST hourly quota, not this one."
% round(g_left * 100))
if g_left < TIGHT:
return ("graphql-tight",
"%d%% of the GraphQL budget is left, which is close enough that "
"the next burst decides it." % round(g_left * 100))
return ("both-healthy",
"%d%% of the GraphQL budget and %s of core are left."
% (round(g_left * 100),
"an unknown amount" if c_left is None else "%d%%" % round(c_left * 100)))
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state == "graphql-exhausted-rest-healthy":
return ("throttle on resources.graphql.remaining, not on core. Add "
"rateLimit { cost remaining } to your real queries so each one "
"reports its own price, and point the health check at the bucket "
"the traffic actually spends.")
if state == "graphql-tight":
return ("slow down now rather than at zero. Divide the remaining points "
"by the measured cost of your query to get the number of calls "
"you have left, and space them out.")
if state == "rest-exhausted-graphql-healthy":
return ("see /github/rate-limit-core-exhausted/ -- this is the REST "
"hourly quota and the repair for it is conditional requests and "
"webhooks, not point budgeting.")
if state == "both-exhausted":
return ("wait for the resets and then fix them separately: they refill "
"on their own schedules and neither repair helps the other.")
if state == "both-healthy":
return ("nothing today. Measure the cost of your query anyway, because "
"the budget in queries is what you schedule against and it is "
"not 5,000.")
return "read GET /rate_limit with a token this API accepts."
def point_cost(in_band):
"""Points this run will spend. Pure. Zero unless the in-band probe is asked for."""
return 1 if in_band else 0
def fmt_reset(seconds):
"""A reset delay in something readable. Pure."""
if seconds is None:
return "unknown"
if seconds < 90:
return "%ds" % seconds
return "%dm" % round(seconds / 60.0)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--in-band", action="store_true",
help="spend one point sending { rateLimit { ... } } to "
"measure a query cost directly")
ap.add_argument("--query",
help="measure this document's cost instead of the minimal "
"probe. Add rateLimit { cost remaining } to it first. "
"Mutations are refused.")
ap.add_argument("--cost", type=int, default=1,
help="the cost of one of your queries, if you already know "
"it, used to convert points into queries")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
document = args.query or BUDGET_QUERY
if args.in_band:
why_not = refusal(document)
if why_not:
log.error("refusing to send: %s", why_not)
return 2
log.info("point cost: %d point(s) against the 5,000/hour GraphQL budget",
point_cost(True))
else:
log.info("point cost: %d point(s). GET /rate_limit reports the GraphQL "
"bucket and does not consume any of it.", point_cost(False))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"User-Agent": UA,
})
r = session.get(API + "/rate_limit", timeout=30)
if r.status_code == 401:
log.error("401 from GitHub: GITHUB_TOKEN is missing, malformed or revoked")
return 2
body = r.json() if r.status_code == 200 else None
graphql_b = bucket(body, "graphql")
core_b = bucket(body, "core")
now = int(time.time())
for name, b in (("core", core_b), ("graphql", graphql_b)):
if b is None:
log.info("%-8s not reported", name)
continue
log.info("%-8s %s / %s remaining, resets in %s",
name, b.get("remaining"), b.get("limit"),
fmt_reset(seconds_to_reset(b, now)))
if graphql_b:
log.info("budget: a limit of %s points/hour is %s",
graphql_b.get("limit"), identify_budget(graphql_b.get("limit")))
state, detail = classify(graphql_b, core_b)
log.info("%s: %s", state, detail)
measured = args.cost
envelope = None
if args.in_band:
# A GraphQL query is a read; POST is only how the document reaches the
# endpoint, which is why the verb sits here beside the URL rather than
# in a constant where it could be mistaken for a write path.
resp = session.post(API + "/graphql", json={"query": document}, timeout=30)
try:
envelope = resp.json()
except ValueError:
envelope = None
if is_rate_limited(envelope):
log.info("the in-band probe itself came back RATE_LIMITED, which is "
"the finding stated by the endpoint rather than inferred")
cost = in_band_cost(envelope)
if cost is not None:
measured = cost
log.info("measured cost: %d point(s) for this query shape", cost)
else:
log.info("the response carried no rateLimit.cost; add "
"rateLimit { cost remaining } to the document")
limit = (graphql_b or {}).get("limit")
remaining = (graphql_b or {}).get("remaining")
rate = sustainable_rate(limit, measured)
gap = seconds_between(limit, measured)
if rate:
log.info("at %d points a query the budget is %d queries/hour, one every %ss",
measured, rate, gap)
left = queries_left(remaining, measured)
if left is not None:
log.info("%s point(s) left is %d more quer%s of this shape",
remaining, left, "y" if left == 1 else "ies")
log.info("repair: %s", repair(state))
print(json.dumps({
"points_spent": point_cost(args.in_band),
"graphql": graphql_b,
"core": core_b,
"graphql_used_fraction": used_fraction(graphql_b),
"core_used_fraction": used_fraction(core_b),
"budget_identified_as": identify_budget(limit),
"measured_cost": measured,
"queries_per_hour": rate,
"seconds_between_queries": gap,
"queries_left": left,
"in_band_rate_limited": is_rate_limited(envelope),
"state": state,
"detail": detail,
}, indent=2, default=str))
bad = {"graphql-exhausted-rest-healthy", "graphql-tight", "both-exhausted"}
return 1 if state in bad else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Read the GraphQL point budget, which is not the REST one.
*
* Read only. The default run spends nothing: GET /rate_limit reports both
* buckets and is documented not to count against either. The optional in-band
* probe sends one query and costs one point, and it says so first.
*
* Queries only. GitHub's GraphQL endpoint takes a document in the request body,
* so a read is carried by POST there just as a write would be; that is
* transport, not intent. Any document containing a mutation or a subscription
* is refused before a socket opens.
*
* Environment:
* GITHUB_TOKEN a token with read access to the GraphQL API
* GITHUB_IN_BAND set to spend one point measuring a query cost directly
* GITHUB_QUERY measure this document instead of the minimal probe
* GITHUB_COST the cost of one of your queries, if you already know it
*/
const API = 'https://api.github.com';
const UA = 'github-graphql-points/1.0';
/** The published hourly point budgets, keyed by the actor they belong to. */
export const BUDGETS = {
5000: 'a user token',
1000: 'the GITHUB_TOKEN issued to a GitHub Actions workflow',
10000: 'an Enterprise Cloud token',
};
const BUDGET_QUERY = 'query { rateLimit { limit cost remaining used resetAt nodeCount } }';
/** Below this fraction of budget left, slow down rather than discover zero. */
export const TIGHT = 0.2;
/** Remove GraphQL comments and string literals from a document. Pure. */
export function stripNoise(document) {
const src = String(document ?? '');
const out = [];
let i = 0;
while (i < src.length) {
const ch = src[i];
if (ch === '#') {
while (i < src.length && src[i] !== '\n') i += 1;
continue;
}
if (src.startsWith('"""', i)) {
const j = src.indexOf('"""', i + 3);
i = j < 0 ? src.length : j + 3;
out.push(' ');
continue;
}
if (ch === '"') {
i += 1;
while (i < src.length && src[i] !== '"') i += src[i] === '\\' ? 2 : 1;
i += 1;
out.push(' ');
continue;
}
out.push(ch);
i += 1;
}
return out.join('');
}
/** The top-level operations in a document, in order. Pure. */
export function operations(document) {
const src = `${stripNoise(document)} `;
const ops = [];
let depth = 0;
let word = '';
let declared = null;
for (const ch of src) {
if (/[A-Za-z0-9_]/.test(ch)) { word += ch; continue; }
if (word) {
if (depth === 0 && ['query', 'mutation', 'subscription', 'fragment'].includes(word)) {
declared = word;
}
word = '';
}
if (ch === '{') {
if (depth === 0) { ops.push(declared || 'query'); declared = null; }
depth += 1;
} else if (ch === '}') {
depth = Math.max(0, depth - 1);
}
}
return ops;
}
/** Why this document will not be sent, or null if it is a read. Pure. */
export function refusal(document) {
const ops = operations(document);
if (ops.length === 0) return 'the document contains no operation to send.';
for (const kind of ['mutation', 'subscription']) {
if (ops.includes(kind)) {
return `the document contains a ${kind}. This script sends queries only: `
+ 'a query is a read, and the section it belongs to promises its '
+ 'scripts never write.';
}
}
return null;
}
/** One resource object out of GET /rate_limit. Pure. null if absent. */
export function bucket(rateLimitBody, name) {
if (!rateLimitBody || typeof rateLimitBody !== 'object') return null;
const res = rateLimitBody.resources;
if (!res || typeof res !== 'object') return null;
const b = res[name];
return b && typeof b === 'object' ? b : null;
}
/** How much of a bucket is gone, 0..1. Pure. null if unreadable. */
export function usedFraction(b) {
if (!b || typeof b !== 'object') return null;
const limit = Number(b.limit);
const remaining = Number(b.remaining);
if (!Number.isFinite(limit) || !Number.isFinite(remaining) || limit <= 0) return null;
return Math.max(0, Math.min(1, (limit - remaining) / limit));
}
/** Seconds until this bucket refills. Pure. null if unreadable. */
export function secondsToReset(b, now) {
if (!b || typeof b !== 'object') return null;
const reset = Number(b.reset);
const t = Number(now);
if (!Number.isFinite(reset) || !Number.isFinite(t)) return null;
return Math.max(0, Math.trunc(reset) - Math.trunc(t));
}
/** Which actor an observed hourly limit implies. Pure. */
export function identifyBudget(limit) {
const n = Number(limit);
if (!Number.isFinite(n)) return 'an unreadable limit';
if (Object.prototype.hasOwnProperty.call(BUDGETS, String(Math.trunc(n)))) {
return BUDGETS[String(Math.trunc(n))];
}
return `a limit of ${Math.trunc(n)}, which matches none of the published `
+ 'budgets. Read it as the truth and plan against it.';
}
/** How many more queries of this shape fit in what is left. Pure. */
export function queriesLeft(remaining, costPerQuery) {
const rem = Number(remaining);
const cost = Number(costPerQuery);
if (!Number.isFinite(rem) || !Number.isFinite(cost) || cost <= 0) return null;
return Math.max(0, Math.floor(rem / cost));
}
/** Queries per hour this budget supports at a measured cost. Pure. */
export function sustainableRate(limit, costPerQuery) {
const lim = Number(limit);
const cost = Number(costPerQuery);
if (!Number.isFinite(lim) || !Number.isFinite(cost) || cost <= 0 || lim <= 0) return null;
return Math.floor(lim / cost);
}
/** The gap to leave between queries to stay inside the budget. Pure. */
export function secondsBetween(limit, costPerQuery) {
const rate = sustainableRate(limit, costPerQuery);
if (!rate) return null;
return Math.round((3600 / rate) * 10) / 10;
}
/** The type of every entry in a GraphQL errors array. Pure. */
export function errorTypes(body) {
if (!body || typeof body !== 'object' || !Array.isArray(body.errors)) return [];
return body.errors.map((e) => (e && typeof e === 'object' && e.type) || 'UNTYPED');
}
/** Whether a GraphQL envelope reports the budget as spent. Pure. */
export function isRateLimited(body) {
return errorTypes(body).includes('RATE_LIMITED');
}
/** The cost this query reported for itself. Pure. null if not asked for. */
export function inBandCost(body) {
if (!body || typeof body !== 'object') return null;
const data = body.data;
if (!data || typeof data !== 'object') return null;
const rl = data.rateLimit;
if (!rl || typeof rl !== 'object') return null;
const cost = Number(rl.cost);
return Number.isFinite(cost) ? Math.trunc(cost) : null;
}
/** Compare the two buckets. Pure. Returns [state, detail]. */
export function classify(graphqlB, coreB) {
const g = usedFraction(graphqlB);
const c = usedFraction(coreB);
if (g === null) {
return ['unreadable', 'resources.graphql was not present in the response, '
+ 'so the GraphQL budget cannot be read from it.'];
}
const gLeft = 1 - g;
const cLeft = c === null ? null : 1 - c;
const gEmpty = Number(graphqlB.remaining) === 0;
const cEmpty = cLeft !== null && Number(coreB.remaining) === 0;
if (gEmpty && cEmpty) {
return ['both-exhausted', 'both buckets are empty, so this is not the '
+ 'confusing case: everything fails and everything is meant to.'];
}
if (gEmpty) {
return ['graphql-exhausted-rest-healthy',
`the GraphQL bucket is empty while core is at ${Math.round((cLeft || 0) * 100)}% `
+ 'remaining, so a REST health check reports green on a dead integration.'];
}
if (cEmpty) {
return ['rest-exhausted-graphql-healthy',
`core is empty and the GraphQL budget is at ${Math.round(gLeft * 100)}% `
+ 'remaining. That is the REST hourly quota, not this one.'];
}
if (gLeft < TIGHT) {
return ['graphql-tight', `${Math.round(gLeft * 100)}% of the GraphQL budget `
+ 'is left, which is close enough that the next burst decides it.'];
}
return ['both-healthy', `${Math.round(gLeft * 100)}% of the GraphQL budget and `
+ `${cLeft === null ? 'an unknown amount' : `${Math.round(cLeft * 100)}%`} of core are left.`];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (state === 'graphql-exhausted-rest-healthy') {
return 'throttle on resources.graphql.remaining, not on core. Add '
+ 'rateLimit { cost remaining } to your real queries so each one reports '
+ 'its own price, and point the health check at the bucket the traffic '
+ 'actually spends.';
}
if (state === 'graphql-tight') {
return 'slow down now rather than at zero. Divide the remaining points by '
+ 'the measured cost of your query to get the number of calls you have '
+ 'left, and space them out.';
}
if (state === 'rest-exhausted-graphql-healthy') {
return 'see /github/rate-limit-core-exhausted/ -- this is the REST hourly '
+ 'quota and the repair for it is conditional requests and webhooks, not '
+ 'point budgeting.';
}
if (state === 'both-exhausted') {
return 'wait for the resets and then fix them separately: they refill on '
+ 'their own schedules and neither repair helps the other.';
}
if (state === 'both-healthy') {
return 'nothing today. Measure the cost of your query anyway, because the '
+ 'budget in queries is what you schedule against and it is not 5,000.';
}
return 'read GET /rate_limit with a token this API accepts.';
}
/** Points this run will spend. Pure. Zero unless the in-band probe is asked for. */
export function pointCost(inBand) {
return inBand ? 1 : 0;
}
/** A reset delay in something readable. Pure. */
export function fmtReset(seconds) {
if (seconds === null || seconds === undefined) return 'unknown';
const n = Number(seconds);
if (!Number.isFinite(n)) return 'unknown';
return n < 90 ? `${Math.trunc(n)}s` : `${Math.round(n / 60)}m`;
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
'User-Agent': UA,
};
}
async function main() {
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN (read-only is enough)');
process.exitCode = 2;
return;
}
const inBand = Boolean(process.env.GITHUB_IN_BAND);
const document = process.env.GITHUB_QUERY || BUDGET_QUERY;
if (inBand) {
const whyNot = refusal(document);
if (whyNot) {
console.error(`refusing to send: ${whyNot}`);
process.exitCode = 2;
return;
}
console.log(`point cost: ${pointCost(true)} point(s) against the 5,000/hour GraphQL budget`);
} else {
console.log(`point cost: ${pointCost(false)} point(s). GET /rate_limit reports `
+ 'the GraphQL bucket and does not consume any of it.');
}
const res = await fetch(`${API}/rate_limit`, { headers: headers(token) });
if (res.status === 401) {
console.error('401 from GitHub: GITHUB_TOKEN is missing, malformed or revoked');
process.exitCode = 2;
return;
}
let body = null;
try { body = await res.json(); } catch { body = null; }
const graphqlB = bucket(body, 'graphql');
const coreB = bucket(body, 'core');
const now = Math.trunc(Date.now() / 1000);
for (const [name, b] of [['core', coreB], ['graphql', graphqlB]]) {
if (!b) { console.log(`${name.padEnd(8)} not reported`); continue; }
console.log(`${name.padEnd(8)} ${b.remaining} / ${b.limit} remaining, `
+ `resets in ${fmtReset(secondsToReset(b, now))}`);
}
if (graphqlB) {
console.log(`budget: a limit of ${graphqlB.limit} points/hour is ${identifyBudget(graphqlB.limit)}`);
}
const [state, detail] = classify(graphqlB, coreB);
console.log(`${state}: ${detail}`);
let measured = Number(process.env.GITHUB_COST || 1);
let envelope = null;
if (inBand) {
const probe = await fetch(`${API}/graphql`, {
// A GraphQL query is a read. POST is only how the document reaches the
// endpoint, and refusal() has already rejected anything that is not.
method: 'POST',
headers: headers(token),
body: JSON.stringify({ query: document }),
});
try { envelope = await probe.json(); } catch { envelope = null; }
if (isRateLimited(envelope)) {
console.log('the in-band probe itself came back RATE_LIMITED, which is the '
+ 'finding stated by the endpoint rather than inferred');
}
const cost = inBandCost(envelope);
if (cost !== null) {
measured = cost;
console.log(`measured cost: ${cost} point(s) for this query shape`);
} else {
console.log('the response carried no rateLimit.cost; add '
+ 'rateLimit { cost remaining } to the document');
}
}
const limit = graphqlB ? graphqlB.limit : null;
const remaining = graphqlB ? graphqlB.remaining : null;
const rate = sustainableRate(limit, measured);
const gap = secondsBetween(limit, measured);
if (rate) {
console.log(`at ${measured} points a query the budget is ${rate} queries/hour, one every ${gap}s`);
}
const left = queriesLeft(remaining, measured);
if (left !== null) {
console.log(`${remaining} point(s) left is ${left} more quer${left === 1 ? 'y' : 'ies'} of this shape`);
}
console.log(`repair: ${repair(state)}`);
console.log(JSON.stringify({
points_spent: pointCost(inBand),
graphql: graphqlB,
core: coreB,
graphql_used_fraction: usedFraction(graphqlB),
core_used_fraction: usedFraction(coreB),
budget_identified_as: identifyBudget(limit),
measured_cost: measured,
queries_per_hour: rate,
seconds_between_queries: gap,
queries_left: left,
in_band_rate_limited: isRateLimited(envelope),
state,
detail,
}, null, 2));
const bad = ['graphql-exhausted-rest-healthy', 'graphql-tight', 'both-exhausted'];
process.exitCode = bad.includes(state) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The comparison carries the note, so the suite spends most of its time on pairs of buckets: GraphQL empty beside a healthy core, which is the case that wastes an afternoon; core empty beside a healthy GraphQL, which belongs to a different note and is sent there by name; both empty, which is confusing in no way at all. After that, the arithmetic that turns points into something schedulable, asserted at a realistic cost rather than at one, the reverse lookup from an observed limit to the actor it implies, and the point cost, which has to be zero on the default run because the whole claim is that reading the budget is free.
from github_graphql_points import (
TIGHT, bucket, classify, fmt_reset, identify_budget, in_band_cost,
is_rate_limited, operations, point_cost, queries_left, refusal, repair,
seconds_between, seconds_to_reset, sustainable_rate, used_fraction,
)
def rl(core_remaining, graphql_remaining, core_limit=5000, graphql_limit=5000):
return {"resources": {
"core": {"limit": core_limit, "remaining": core_remaining,
"used": core_limit - core_remaining, "reset": 1_800_000_000},
"graphql": {"limit": graphql_limit, "remaining": graphql_remaining,
"used": graphql_limit - graphql_remaining, "reset": 1_800_000_000},
}}
def test_the_two_buckets_are_read_separately():
body = rl(4983, 0)
assert bucket(body, "core")["remaining"] == 4983
assert bucket(body, "graphql")["remaining"] == 0
assert bucket(body, "search") is None
assert bucket({}, "graphql") is None
def test_an_empty_graphql_bucket_beside_a_healthy_core_is_the_headline():
body = rl(4983, 0)
state, detail = classify(bucket(body, "graphql"), bucket(body, "core"))
assert state == "graphql-exhausted-rest-healthy"
assert "health check reports green" in detail
assert "resources.graphql.remaining" in repair(state)
def test_an_empty_core_beside_a_healthy_graphql_belongs_to_another_note():
body = rl(0, 4983)
state, _ = classify(bucket(body, "graphql"), bucket(body, "core"))
assert state == "rest-exhausted-graphql-healthy"
assert "rate-limit-core-exhausted" in repair(state)
assert "point budgeting" in repair(state)
def test_both_empty_is_not_the_confusing_case():
body = rl(0, 0)
state, detail = classify(bucket(body, "graphql"), bucket(body, "core"))
assert state == "both-exhausted"
assert "not the confusing case" in detail
def test_a_tight_budget_is_flagged_before_it_reaches_zero():
body = rl(4983, 500)
state, _ = classify(bucket(body, "graphql"), bucket(body, "core"))
assert state == "graphql-tight"
assert TIGHT == 0.2
healthy = rl(4983, 4000)
assert classify(bucket(healthy, "graphql"), bucket(healthy, "core"))[0] == "both-healthy"
def test_a_missing_graphql_bucket_is_reported_rather_than_assumed_full():
state, _ = classify(None, {"limit": 5000, "remaining": 4983})
assert state == "unreadable"
def test_used_fraction_survives_a_bucket_that_makes_no_sense():
assert used_fraction({"limit": 5000, "remaining": 2500}) == 0.5
assert used_fraction({"limit": 0, "remaining": 0}) is None
assert used_fraction({"limit": "many", "remaining": 1}) is None
assert used_fraction(None) is None
def test_points_are_converted_into_the_unit_you_can_schedule():
assert sustainable_rate(5000, 12) == 416
assert seconds_between(5000, 12) == 8.7
assert queries_left(1200, 12) == 100
assert queries_left(11, 12) == 0
def test_the_conversion_refuses_a_cost_that_cannot_be_divided_by():
assert sustainable_rate(5000, 0) is None
assert queries_left(1200, 0) is None
assert seconds_between(5000, "free") is None
def test_an_observed_limit_names_the_actor_it_belongs_to():
assert identify_budget(5000) == "a user token"
assert "GitHub Actions" in identify_budget(1000)
assert "Enterprise Cloud" in identify_budget(10000)
assert "matches none of the published budgets" in identify_budget(2500)
def test_the_error_type_is_read_from_the_envelope_not_the_status():
assert is_rate_limited({"errors": [{"type": "RATE_LIMITED"}]})
assert not is_rate_limited({"errors": [{"type": "NOT_FOUND"}]})
assert not is_rate_limited({"data": {"rateLimit": {"cost": 1}}})
assert not is_rate_limited(None)
def test_the_in_band_cost_is_read_only_when_it_was_asked_for():
assert in_band_cost({"data": {"rateLimit": {"cost": 12, "remaining": 4988}}}) == 12
assert in_band_cost({"data": {"viewer": {"login": "ada"}}}) is None
assert in_band_cost({"data": None}) is None
def test_the_reset_delay_is_readable_and_never_negative():
assert seconds_to_reset({"reset": 1000}, 940) == 60
assert seconds_to_reset({"reset": 1000}, 2000) == 0
assert seconds_to_reset({}, 100) is None
assert fmt_reset(45) == "45s"
assert fmt_reset(720) == "12m"
assert fmt_reset(None) == "unknown"
def test_the_default_run_spends_nothing():
assert point_cost(False) == 0
assert point_cost(True) == 1
def test_the_script_refuses_to_send_a_mutation():
assert operations("query { rateLimit { cost } }") == ["query"]
assert refusal("mutation M { addStar(input: {}) { clientMutationId } }")
assert refusal("query { rateLimit { cost remaining } }") is None
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
TIGHT, bucket, classify, fmtReset, identifyBudget, inBandCost, isRateLimited,
operations, pointCost, queriesLeft, refusal, repair, secondsBetween,
secondsToReset, sustainableRate, usedFraction,
} from './github-graphql-points.mjs';
function rl(coreRemaining, graphqlRemaining, coreLimit = 5000, graphqlLimit = 5000) {
return {
resources: {
core: {
limit: coreLimit,
remaining: coreRemaining,
used: coreLimit - coreRemaining,
reset: 1800000000,
},
graphql: {
limit: graphqlLimit,
remaining: graphqlRemaining,
used: graphqlLimit - graphqlRemaining,
reset: 1800000000,
},
},
};
}
test('the two buckets are read separately', () => {
const body = rl(4983, 0);
assert.equal(bucket(body, 'core').remaining, 4983);
assert.equal(bucket(body, 'graphql').remaining, 0);
assert.equal(bucket(body, 'search'), null);
assert.equal(bucket({}, 'graphql'), null);
});
test('an empty GraphQL bucket beside a healthy core is the headline', () => {
const body = rl(4983, 0);
const [state, detail] = classify(bucket(body, 'graphql'), bucket(body, 'core'));
assert.equal(state, 'graphql-exhausted-rest-healthy');
assert.match(detail, /health check reports green/);
assert.match(repair(state), /resources\.graphql\.remaining/);
});
test('an empty core beside a healthy GraphQL belongs to another note', () => {
const body = rl(0, 4983);
const [state] = classify(bucket(body, 'graphql'), bucket(body, 'core'));
assert.equal(state, 'rest-exhausted-graphql-healthy');
assert.match(repair(state), /rate-limit-core-exhausted/);
assert.match(repair(state), /point budgeting/);
});
test('both empty is not the confusing case', () => {
const body = rl(0, 0);
const [state, detail] = classify(bucket(body, 'graphql'), bucket(body, 'core'));
assert.equal(state, 'both-exhausted');
assert.match(detail, /not the confusing case/);
});
test('a tight budget is flagged before it reaches zero', () => {
const body = rl(4983, 500);
assert.equal(classify(bucket(body, 'graphql'), bucket(body, 'core'))[0], 'graphql-tight');
assert.equal(TIGHT, 0.2);
const healthy = rl(4983, 4000);
assert.equal(
classify(bucket(healthy, 'graphql'), bucket(healthy, 'core'))[0],
'both-healthy',
);
});
test('a missing GraphQL bucket is reported rather than assumed full', () => {
const [state] = classify(null, { limit: 5000, remaining: 4983 });
assert.equal(state, 'unreadable');
});
test('usedFraction survives a bucket that makes no sense', () => {
assert.equal(usedFraction({ limit: 5000, remaining: 2500 }), 0.5);
assert.equal(usedFraction({ limit: 0, remaining: 0 }), null);
assert.equal(usedFraction({ limit: 'many', remaining: 1 }), null);
assert.equal(usedFraction(null), null);
});
test('points are converted into the unit you can schedule', () => {
assert.equal(sustainableRate(5000, 12), 416);
assert.equal(secondsBetween(5000, 12), 8.7);
assert.equal(queriesLeft(1200, 12), 100);
assert.equal(queriesLeft(11, 12), 0);
});
test('the conversion refuses a cost that cannot be divided by', () => {
assert.equal(sustainableRate(5000, 0), null);
assert.equal(queriesLeft(1200, 0), null);
assert.equal(secondsBetween(5000, 'free'), null);
});
test('an observed limit names the actor it belongs to', () => {
assert.equal(identifyBudget(5000), 'a user token');
assert.match(identifyBudget(1000), /GitHub Actions/);
assert.match(identifyBudget(10000), /Enterprise Cloud/);
assert.match(identifyBudget(2500), /matches none of the published budgets/);
});
test('the error type is read from the envelope, not the status', () => {
assert.ok(isRateLimited({ errors: [{ type: 'RATE_LIMITED' }] }));
assert.ok(!isRateLimited({ errors: [{ type: 'NOT_FOUND' }] }));
assert.ok(!isRateLimited({ data: { rateLimit: { cost: 1 } } }));
assert.ok(!isRateLimited(null));
});
test('the in-band cost is read only when it was asked for', () => {
assert.equal(inBandCost({ data: { rateLimit: { cost: 12, remaining: 4988 } } }), 12);
assert.equal(inBandCost({ data: { viewer: { login: 'ada' } } }), null);
assert.equal(inBandCost({ data: null }), null);
});
test('the reset delay is readable and never negative', () => {
assert.equal(secondsToReset({ reset: 1000 }, 940), 60);
assert.equal(secondsToReset({ reset: 1000 }, 2000), 0);
assert.equal(secondsToReset({}, 100), null);
assert.equal(fmtReset(45), '45s');
assert.equal(fmtReset(720), '12m');
assert.equal(fmtReset(null), 'unknown');
});
test('the default run spends nothing', () => {
assert.equal(pointCost(false), 0);
assert.equal(pointCost(true), 1);
});
test('the script refuses to send a mutation', () => {
assert.deepEqual(operations('query { rateLimit { cost } }'), ['query']);
assert.ok(refusal('mutation M { addStar(input: {}) { clientMutationId } }'));
assert.equal(refusal('query { rateLimit { cost remaining } }'), null);
});
FAQ
Why does my REST health check say the token is fine?
Because it is fine, for REST. The GraphQL budget and the REST core quota are separate buckets with separate limits, separate counters and separate reset times, and GET /rate_limit returns them as two independent objects. A check that reads resources.core is reporting on traffic that has nothing to do with the traffic that is failing. Point the check at resources.graphql, or better, at both, and print them together so the difference is impossible to miss.
Is 5,000 points the same as 5,000 queries?
Almost never. A point cost is derived from the connections a query traverses and the number of items each one asks for, so a flat query costs a point or two and one that fans out over repositories and their pull requests can cost dozens. Measure yours by adding rateLimit { cost remaining } to the document and reading the number back. At twelve points a query the real budget is 416 calls an hour, which is a schedule; 5,000 points is not.
How is this different from the secondary rate limit?
Different bucket, different window, different evidence. The GraphQL budget is 5,000 points an hour, it is reported by GET /rate_limit before you hit it, and running out produces a 200 with RATE_LIMITED in the errors array. The secondary limit is a per-minute cap on a single REST endpoint, it appears in no response header at all, and it announces itself as a 403 or 429 after the fact. They have separate notes because they are found by separate means.
My budget disappears and my own queries do not account for it. Where did it go?
The API cannot tell you, and that is a documented blind spot rather than a gap in this script. The bucket belongs to the token and is shared by every process holding it, so a scheduled job, a developer's laptop and a bot can all be drawing on the same 5,000 points while each of them believes it is the only consumer. The only repair that makes the number mean something is a separate token per workload, after which the drain is attributable by construction.
Does asking for the budget cost anything?
GET /rate_limit is free and reports both buckets, which is why this script uses it by default and prints a point cost of zero. Adding rateLimit { cost remaining } to a query you were already sending is also effectively free, since it rides along on a request you were making anyway, and it is the only way to learn the price of that specific shape. Sending a query whose sole purpose is to ask about the budget costs one point, which is what --in-band does and what it announces before doing.
Related field notes
- The REST core quota is empty
- A hot endpoint burns 900 points a minute
- A query over the 500,000 node limit
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Rate limits and node limits for the GraphQL API — GitHub Docs
- Rate limit — GitHub REST API
- Rate limits for the REST API — GitHub Docs
- Resource limitations — GitHub GraphQL API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.