Diagnostic GitHub API
A mutation costs five secondary points, a query costs one
The read job runs all day without complaint. The write job, which sends fewer requests, dies eleven minutes in with 403 and You have exceeded a secondary rate limit. Somebody checks GET /rate_limit, sees four thousand GraphQL points still sitting there unspent, and concludes GitHub is wrong. GitHub is not wrong. The bucket that emptied is not the one being looked at, and the reason the write job reached it first is that every document containing a mutation is priced at five times a read.
The GraphQL endpoint has a per-minute secondary limit of 2,000 points that is entirely separate from the hourly point budget. Against that limit a request whose document contains a mutation counts as 5 points and a request whose document does not counts as 1. The arithmetic is the whole note: 2,000 read requests a minute, 400 mutation requests a minute, from one token.
So a write loop trips at roughly a fifth of the request rate a read loop survives, and it trips with the hourly budget almost untouched, which is what makes it look like a bug. Diagnose it by reading both facts at once: a 403 or 429 whose body says “secondary rate limit” while resources.graphql.remaining is still healthy is this limit and not the budget. Repair it by rate-limiting mutations against 2,000 points a minute rather than against a request count, serialising them instead of fanning them out, and honouring retry-after when it arrives.
The problem in plain words
The shape that produces this is a migration or a backfill: label eleven thousand issues, close six thousand stale pull requests, add a project item for every repository in the org. Somebody writes it as a loop, runs it against a hundred rows to check the logic, and it is fine. They raise the concurrency because a hundred rows took a while and eleven thousand will take all afternoon, and now it dies.
Every instinct after that points at the wrong bucket. The error says rate limit, so the first thing anybody does is look at the rate limit, and the rate limit is fine — thousands of points remaining, a reset an hour away, nothing near zero. The natural conclusion is that the 403 is spurious, which leads to a retry loop, which sends the same requests faster, which extends the throttle. There is no header to check, because the secondary limit has no bucket in GET /rate_limit and never reports headroom. It is only ever visible after you have crossed it.
The second thing that misleads is the comparison with the read job. The read job sends more requests than the write job and does not get throttled, so request rate looks like it is not the problem. It is the problem; it is just being counted in a unit nobody converted to. Four hundred is a much smaller number than two thousand, and the only place that difference is written down is a sentence in the resource-limitations documentation about how mutations are weighed.
Why it happens
Two buckets, two units, two time windows. The primary budget is 5,000 points an hour and is measurable before you spend it: GET /rate_limit reports resources.graphql for free, and that is its own note. The secondary limit is 2,000 points a minute, is not reported anywhere, and exists to stop bursts rather than to meter volume. A job can be comfortably inside its hourly budget and outside the per-minute one on every single minute of its run.
The weight is per request, not per mutation. A document that contains a mutation is priced at 5 points whether it contains one mutation or six. That cuts both ways: batching several mutations into one document is a real reduction in secondary cost, and it is the cheapest fix available, whereas splitting one document into six is a fivefold increase for the same work. A document with no mutation in it is 1 point regardless of how expensive it is against the hourly budget — the two prices are unrelated, and a query can be costly in one and trivial in the other.
The REST secondary limits are different limits with different numbers. REST has its own per-minute ceilings — 900 points a minute on a single endpoint, with a CPU-time cap beside it — and a separate allowance for content-generating requests, around 80 a minute. Those apply to the REST path. This one applies to /graphql, and a client that moved its writes to GraphQL to escape the REST content-creation limit landed on a different number rather than on none.
Concurrency is a separate ceiling on top of this one. No more than 100 concurrent requests are allowed across the REST and GraphQL APIs together, and GitHub asks for at least one second between mutations that affect the same resource. Those rules are not expressed in points, so satisfying the point arithmetic does not satisfy them; fanning 400 mutations out across 200 workers respects the per-minute ceiling and violates the concurrency one.
The differential diagnosis is two readings taken together. A GraphQL rate-limit failure that arrives as a 200 with errors[0].type: "RATE_LIMITED" is the hourly budget. A 403 or 429 with “secondary rate limit” in the body is this one. Reading resources.graphql.remaining at the same moment settles which: healthy remaining plus a secondary message is proof, and it is the reading that stops somebody rewriting the retry logic for a bug that is not there.
This script prices mutations and never sends one. The section promises its scripts never write, and the note is about writes, so the arithmetic is done on the document text. The parser that decides whether a document is a read — the guard every GraphQL script in this section runs before it opens a socket — is the same parser that decides whether it is worth 5 points or 1. Refusing to send it and pricing it are the same question asked twice.
The fix, as a flow
The parser earns its place twice. It is the guard that keeps a mutation off the wire, and it is also the pricer, because the question the secondary limit asks of a request is exactly the question the guard asks: does this document contain a mutation. Everything after it is arithmetic over integers, which is why the documents can be priced without being sent. The one live call is a single read query, and the hourly budget beside it comes from a free REST call, so a throttle you already recorded can be attributed to the per-minute ceiling rather than to the bucket that was never empty.
How to fix it
Price your own documents without sending them
Point the script at the query files your writer sends, or paste one in. It parses each document, lists its top-level operations, and prices it: 5 points if any operation is a mutation, 1 if not. Mutation documents are refused for sending in the same breath — they are read, priced and never transmitted, which is the only way a read-only tool can have an opinion about a write.
Convert the price into a rate you can put in a scheduler
2,000 points a minute divided by the weight is the ceiling: 400 mutation requests a minute, 2,000 read requests a minute. The script prints the ceiling for each document and the minimum gap between calls it implies, so the number that goes into your limiter is a number rather than a guess. Then it prints how long your actual batch will take at that rate, because the honest version of this repair is usually “this backfill takes forty minutes”.
Compare that against the rate the job actually runs at
Give the script the loop's real rate with --rate and it multiplies rate by weight to get points a minute, then says whether that number is over 2,000. A loop sending 500 mutation requests a minute is 2,500 points and will be throttled; the same loop sending queries is 500 points and will not. Seeing both lines next to each other is what explains why the read job never had this problem.
Separate a secondary throttle from an exhausted budget
The script reads the GraphQL budget for free from GET /rate_limit and sends exactly one read query of its own to confirm the endpoint answers. If your logs carry a 403 or 429 that says “secondary rate limit” while that budget was healthy, feed it in and the script names the limit rather than leaving you to infer it. This is the step that prevents the retry loop.
Fix it in the writer, not in the retry handler
Batch several mutations into one document where the API allows it, since the 5 points are charged per request. Serialise the writes rather than fanning them out, leave at least a second between mutations touching the same resource, and honour retry-after exactly rather than backing off by a guess. The whole audit costs one point, printed before it is spent, out of 5,000 an hour.
How to check it worked
Once the writer is limited against points a minute instead of requests a minute, the same arithmetic reports headroom instead of a breach.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_mutation_budget.py \
--document label_issue.graphql --document fetch_issues.graphql --rate 500
# point cost: 1 point(s) against the 5000/hour GraphQL budget
# graphql budget: 4863/5000 remaining
# probe read query: HTTP 200, 1 point(s) spent
#
# label_issue.graphql: operations=mutation -> 5 point(s) per request
# not sent: the document contains a mutation. This script prices documents,
# it does not send them.
# ceiling 400 request(s)/minute, minimum gap 0.150s on one worker
# over-ceiling: 500 request(s)/minute of this document is 2500 point(s)/minute
# against a limit of 2000.
# 11000 row(s) takes at least 28 minute(s) at the ceiling
# repair: batch mutations into one document, serialise the loop, and cap it
# at 400/minute or below.
#
# fetch_issues.graphql: operations=query -> 1 point(s) per request
# ceiling 2000 request(s)/minute, minimum gap 0.030s on one worker
# within-ceiling: 500 request(s)/minute is 500 point(s)/minute against a
# limit of 2000.The full code
The parser is doing double duty here. It is the guard that keeps a mutation off the wire, and it is also the pricer, because the question “does this document contain a mutation” is exactly the question the secondary limit asks. Everything downstream of it is arithmetic over integers: weight, ceiling, points per minute at a given rate, minutes to finish a batch. The one live call is a single read query, and the budget beside it comes from a free REST call, so the script can prove which bucket is which without spending anything meaningful in either.
"""Price GraphQL documents against the per-minute secondary limit.
Read only, and queries only. GitHub's GraphQL endpoint takes its document in
the request body, so a read travels by POST there exactly as a write would;
that is a transport detail, not a licence to write. This script parses every
document it is given, refuses to open a socket for anything containing a
mutation or a subscription, and sends exactly one read query of its own.
The point of the note: against the secondary rate limit of 2,000 points per
minute, a GraphQL request whose document contains a mutation counts as 5
points and one that does not counts as 1. So a write loop reaches the limit at
roughly a fifth of the request rate a read loop survives, and it does so with
the separate hourly point budget almost untouched.
What this can and cannot see: secondary limits have no bucket. GET /rate_limit
reports the hourly budget only, and nothing anywhere reports how close you are
to the per-minute ceiling. So the ceiling is computed from the documented
weights and compared against a rate you supply, and a throttle you already
recorded is classified after the fact.
Environment:
GITHUB_TOKEN a token with read access to the GraphQL API
"""
import argparse
import json
import logging
import math
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_graphql_mutation_budget")
API = "https://api.github.com"
UA = "github-graphql-mutation-budget/1.0"
# The secondary limit on the GraphQL endpoint, and the two weights it applies.
# Documented, not measurable: no response reports headroom against this.
SECONDARY_POINTS_PER_MINUTE = 2000
WEIGHT_WITH_MUTATION = 5
WEIGHT_WITHOUT_MUTATION = 1
# The other bucket entirely, quoted here only so the two are never confused.
PRIMARY_POINTS_PER_HOUR = 5000
# GitHub asks for at least this long between mutations affecting one resource.
# Not expressed in points, so satisfying the arithmetic does not satisfy it.
SAME_RESOURCE_GAP_SECONDS = 1.0
# The one document this script ever sends. A read, and it is put through the
# same refusal check as anything supplied on the command line.
PROBE_QUERY = "query { rateLimit { limit cost remaining used resetAt } }"
# This run's own cost against the hourly budget.
POINTS_PER_QUERY = 1
def strip_noise(document):
"""Remove GraphQL comments and string literals from a document. Pure.
A scanner rather than a regex: a hash inside a string literal is an
ordinary character and a comment marker outside one, and the word
"mutation" inside a search string is not a mutation. Getting that wrong
here would misprice a document as well as misjudge whether to send it.
"""
src = str(document or "")
out = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "#":
while i < n and src[i] != "\n":
i += 1
continue
if src.startswith('"""', i):
j = src.find('"""', i + 3)
i = n if j < 0 else j + 3
out.append(" ")
continue
if ch == '"':
i += 1
while i < n and src[i] != '"':
i += 2 if src[i] == "\\" else 1
i += 1
out.append(" ")
continue
out.append(ch)
i += 1
return "".join(out)
def operations(document):
"""The top-level operations in a document, in order. Pure.
One entry per brace group at depth zero: "query", "mutation",
"subscription" or "fragment". An anonymous document is query shorthand.
"""
src = strip_noise(document)
ops, depth, word, declared = [], 0, "", None
for ch in src + " ":
if ch.isalnum() or ch == "_":
word += ch
continue
if word:
if depth == 0 and word in ("query", "mutation", "subscription", "fragment"):
declared = word
word = ""
if ch == "{":
if depth == 0:
ops.append(declared or "query")
declared = None
depth += 1
elif ch == "}":
depth = max(0, depth - 1)
return ops
def refusal(document):
"""Why this document will not be sent, or None if it is a read. Pure.
The endpoint is the same one mutations go to, so the guard lives in code
rather than in a comment. This script has an opinion about mutations and
still never transmits one.
"""
ops = operations(document)
if not ops:
return "the document contains no operation to send."
for kind in ("mutation", "subscription"):
if kind in ops:
return ("the document contains a %s. This script prices documents, "
"it does not send them: a query is a read, and the section "
"it belongs to promises its scripts never write." % kind)
return None
def weight(document):
"""Secondary-limit points for one request carrying this document. Pure.
Per request, not per mutation: a document with six mutations in it is
priced at 5, the same as a document with one. That is why batching is a
real reduction and splitting is a fivefold increase.
"""
return WEIGHT_WITH_MUTATION if "mutation" in operations(document) else WEIGHT_WITHOUT_MUTATION
def ceiling_per_minute(points):
"""Requests a minute this weight allows before the limit binds. Pure."""
if not points or points <= 0:
return 0
return SECONDARY_POINTS_PER_MINUTE // int(points)
def min_gap_seconds(points):
"""Seconds between requests implied by the ceiling, one worker. Pure."""
ceiling = ceiling_per_minute(points)
return 0.0 if ceiling <= 0 else 60.0 / ceiling
def points_per_minute(rate, points):
"""What a given request rate costs against the per-minute limit. Pure."""
return max(0, int(rate or 0)) * int(points or 0)
def minutes_for_batch(count, rate):
"""How long a batch of this size takes at this rate, in minutes. Pure."""
rate = max(0, int(rate or 0))
if rate <= 0:
return None
return math.ceil(max(0, int(count or 0)) / rate)
def classify_rate(rate, points):
"""Judge a request rate against the per-minute limit. Pure.
Returns (state, detail). The middle state matters: a rate can be legal on
points and still wrong, because concurrency and the same-resource gap are
separate rules that points do not express.
"""
spend = points_per_minute(rate, points)
ceiling = ceiling_per_minute(points)
if not rate:
return ("not-measured",
"no rate given, so this document is priced but not judged. "
"Its ceiling is %d request(s)/minute." % ceiling)
if spend > SECONDARY_POINTS_PER_MINUTE:
return ("over-ceiling",
"%d request(s)/minute of this document is %d point(s)/minute "
"against a limit of %d." % (rate, spend, SECONDARY_POINTS_PER_MINUTE))
if spend > SECONDARY_POINTS_PER_MINUTE * 0.8:
return ("near-ceiling",
"%d request(s)/minute is %d point(s)/minute, inside the limit of "
"%d but with under a fifth of it left."
% (rate, spend, SECONDARY_POINTS_PER_MINUTE))
return ("within-ceiling",
"%d request(s)/minute is %d point(s)/minute against a limit of %d."
% (rate, spend, SECONDARY_POINTS_PER_MINUTE))
def classify_throttle(status, message, graphql_remaining):
"""Attribute a recorded failure to one bucket or the other. Pure.
Returns (state, detail). The whole diagnosis is two readings taken
together: which failure arrived, and what the hourly budget said at that
moment. Either on its own is ambiguous.
"""
text = str(message or "").lower()
secondary = "secondary rate limit" in text
try:
remaining = int(graphql_remaining)
except (TypeError, ValueError):
remaining = None
healthy = remaining is not None and remaining > PRIMARY_POINTS_PER_HOUR * 0.1
if secondary and healthy:
return ("secondary-not-budget",
"a secondary rate limit with %d point(s) still in the hourly "
"budget. This is the per-minute ceiling, and no amount of "
"waiting for the hourly reset will help." % remaining)
if secondary:
return ("secondary-limit",
"a secondary rate limit. The hourly budget was not readable or "
"was itself low, so slow down and check both.")
if "rate limit" in text and remaining == 0:
return ("primary-exhausted",
"the hourly point budget is spent. That is a different bucket "
"with a different note and it refills on a schedule.")
if "rate limit" in text:
return ("rate-limited-unclassified",
"a rate-limit message that does not name the secondary limit. "
"Read resources.graphql at the moment of failure to attribute it.")
if str(status) in ("403", "429"):
return ("forbidden-not-throttled",
"HTTP %s with no rate-limit wording, so this is a permission "
"problem rather than a throttle." % status)
return ("no-throttle", "nothing in this record names a rate limit.")
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state == "over-ceiling":
return ("batch mutations into one document, serialise the loop, and cap "
"it at %d/minute or below. The 5 points are charged per "
"request, so fewer requests is the whole lever."
% ceiling_per_minute(WEIGHT_WITH_MUTATION))
if state == "near-ceiling":
return ("leave headroom. A retry, a redeploy or one extra worker puts "
"this over, and the limit gives no warning before it binds.")
if state == "secondary-not-budget":
return ("rate-limit the writer against points a minute, not requests a "
"minute, and honour retry-after. Do not rewrite the retry "
"logic around the hourly budget; that bucket was fine.")
if state == "secondary-limit":
return ("slow the writer down and record resources.graphql at the "
"moment of failure so the next one can be attributed.")
if state == "primary-exhausted":
return ("see /github/graphql-rate-limited/ -- the hourly point budget "
"is a different bucket and this is not the note for it.")
if state == "within-ceiling":
return ("nothing on the point arithmetic. Check concurrency and the "
"one-second gap between mutations on the same resource "
"separately; points do not express either.")
return ("supply the rate the loop actually runs at, or the failure you "
"recorded, and the arithmetic becomes a verdict.")
def price(label, document, rate):
"""Everything this script knows about one document. Pure."""
ops = operations(document)
points = weight(document)
state, detail = classify_rate(rate, points)
return {
"document": label,
"operations": ops,
"points_per_request": points,
"ceiling_per_minute": ceiling_per_minute(points),
"min_gap_seconds": round(min_gap_seconds(points), 4),
"points_per_minute_at_rate": points_per_minute(rate, points),
"not_sent": refusal(document),
"state": state,
"detail": detail,
"repair": repair(state),
}
def run_query(session, document, variables=None):
"""Send one read query. Returns (status, body-or-None).
A GraphQL query is a read; POST is only how the document reaches the
endpoint, which is why the verb is written here beside the URL rather than
hidden in a constant where it could be mistaken for a write path.
"""
r = session.post(API + "/graphql",
json={"query": document, "variables": variables or {}},
timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
try:
return r.status_code, r.json()
except ValueError:
return r.status_code, None
def graphql_budget(session):
"""The hourly GraphQL bucket, read for free. Returns a dict or None."""
r = session.get(API + "/rate_limit", timeout=30)
if r.status_code != 200:
return None
try:
return (r.json().get("resources") or {}).get("graphql")
except ValueError:
return None
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--document", action="append", default=[],
help="path to a .graphql file to price. Repeatable. "
"Mutation documents are priced and never sent.")
ap.add_argument("--query",
help="price a document given inline instead of from a file")
ap.add_argument("--rate", type=int, default=0,
help="requests a minute the loop actually sends")
ap.add_argument("--batch", type=int, default=0,
help="how many rows the job has to get through")
ap.add_argument("--throttle-message", default="",
help="the error body you recorded, to attribute it")
ap.add_argument("--throttle-status", default="",
help="the status code you recorded alongside it")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
documents = []
for path in args.document:
try:
documents.append((path, open(path, encoding="utf-8").read()))
except OSError as exc:
log.error("cannot read %s: %s", path, exc)
return 2
if args.query:
documents.append(("--query", args.query))
if not documents:
log.error("give at least one --document or --query to price")
return 2
log.info("point cost: %d point(s) against the %d/hour GraphQL budget",
POINTS_PER_QUERY, PRIMARY_POINTS_PER_HOUR)
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
budget = graphql_budget(session)
if budget:
log.info("graphql budget: %s/%s remaining", budget.get("remaining"),
budget.get("limit"))
# The probe goes through the same guard as anything supplied by the caller.
if refusal(PROBE_QUERY) is None:
status, body = run_query(session, PROBE_QUERY)
log.info("probe read query: HTTP %s, %d point(s) spent", status,
POINTS_PER_QUERY)
if isinstance(body, dict) and body.get("errors"):
log.warning("the probe itself carried errors: %s",
json.dumps(body["errors"])[:200])
priced = [price(label, doc, args.rate) for label, doc in documents]
for p in priced:
log.info("%s: operations=%s -> %d point(s) per request",
p["document"], ", ".join(p["operations"]) or "none",
p["points_per_request"])
if p["not_sent"]:
log.info(" not sent: %s", p["not_sent"])
log.info(" ceiling %d request(s)/minute, minimum gap %.3fs on one worker",
p["ceiling_per_minute"], p["min_gap_seconds"])
log.info(" %s: %s", p["state"], p["detail"])
if args.batch:
at_ceiling = minutes_for_batch(args.batch, p["ceiling_per_minute"])
log.info(" %d row(s) takes at least %s minute(s) at the ceiling",
args.batch, at_ceiling)
log.info(" repair: %s", p["repair"])
throttle = None
if args.throttle_message or args.throttle_status:
state, detail = classify_throttle(
args.throttle_status, args.throttle_message,
(budget or {}).get("remaining"))
log.info("recorded failure -> %s: %s", state, detail)
log.info("repair: %s", repair(state))
throttle = {"state": state, "detail": detail}
print(json.dumps({
"points_spent": POINTS_PER_QUERY,
"secondary_points_per_minute": SECONDARY_POINTS_PER_MINUTE,
"same_resource_gap_seconds": SAME_RESOURCE_GAP_SECONDS,
"graphql_budget": budget,
"documents": priced,
"recorded_failure": throttle,
}, indent=2, default=str))
over = [p for p in priced if p["state"] in ("over-ceiling", "near-ceiling")]
return 1 if over or (throttle and throttle["state"].startswith("secondary")) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Price GraphQL documents against the per-minute secondary limit.
*
* Read only, and queries only. GitHub's GraphQL endpoint takes its document in
* the request body, so a read travels by POST there exactly as a write would.
* Every document is parsed first, anything containing a mutation or a
* subscription is refused before a socket opens, and the only document this
* script sends is a read query of its own.
*
* Against the secondary limit of 2,000 points a minute, a request whose
* document contains a mutation counts as 5 points and one that does not counts
* as 1. A write loop therefore reaches the wall at a fifth of the rate a read
* loop survives, with the separate hourly budget almost untouched.
*
* Environment:
* GITHUB_TOKEN a token with read access to the GraphQL API
* GITHUB_QUERY a document to price
* GITHUB_RATE requests a minute the loop actually sends
* GITHUB_BATCH how many rows the job has to get through
*/
const API = 'https://api.github.com';
const UA = 'github-graphql-mutation-budget/1.0';
/** The secondary limit on the GraphQL endpoint, and its two weights. */
export const SECONDARY_POINTS_PER_MINUTE = 2000;
export const WEIGHT_WITH_MUTATION = 5;
export const WEIGHT_WITHOUT_MUTATION = 1;
/** The other bucket entirely, named so the two are never confused. */
export const PRIMARY_POINTS_PER_HOUR = 5000;
/** GitHub asks for at least this long between mutations on one resource. */
export const SAME_RESOURCE_GAP_SECONDS = 1.0;
/** The one document this script ever sends, and it is guarded like any other. */
export const PROBE_QUERY = 'query { rateLimit { limit cost remaining used resetAt } }';
/** This run's own cost against the hourly budget. */
export const POINTS_PER_QUERY = 1;
/** Remove GraphQL comments and string literals from a document. Pure. */
export function stripNoise(document) {
const src = String(document ?? '');
const out = [];
let i = 0;
while (i < src.length) {
const ch = src[i];
if (ch === '#') {
while (i < src.length && src[i] !== '\n') i += 1;
continue;
}
if (src.startsWith('"""', i)) {
const j = src.indexOf('"""', i + 3);
i = j < 0 ? src.length : j + 3;
out.push(' ');
continue;
}
if (ch === '"') {
i += 1;
while (i < src.length && src[i] !== '"') i += src[i] === '\\' ? 2 : 1;
i += 1;
out.push(' ');
continue;
}
out.push(ch);
i += 1;
}
return out.join('');
}
/** The top-level operations in a document, in order. Pure. */
export function operations(document) {
const src = `${stripNoise(document)} `;
const ops = [];
let depth = 0;
let word = '';
let declared = null;
for (const ch of src) {
if (/[A-Za-z0-9_]/.test(ch)) { word += ch; continue; }
if (word) {
if (depth === 0 && ['query', 'mutation', 'subscription', 'fragment'].includes(word)) {
declared = word;
}
word = '';
}
if (ch === '{') {
if (depth === 0) { ops.push(declared || 'query'); declared = null; }
depth += 1;
} else if (ch === '}') {
depth = Math.max(0, depth - 1);
}
}
return ops;
}
/** Why this document will not be sent, or null if it is a read. Pure. */
export function refusal(document) {
const ops = operations(document);
if (ops.length === 0) return 'the document contains no operation to send.';
for (const kind of ['mutation', 'subscription']) {
if (ops.includes(kind)) {
return `the document contains a ${kind}. This script prices documents, it `
+ 'does not send them: a query is a read, and the section it belongs to '
+ 'promises its scripts never write.';
}
}
return null;
}
/** Secondary-limit points for one request carrying this document. Pure. */
export function weight(document) {
return operations(document).includes('mutation')
? WEIGHT_WITH_MUTATION : WEIGHT_WITHOUT_MUTATION;
}
/** Requests a minute this weight allows before the limit binds. Pure. */
export function ceilingPerMinute(points) {
const n = Number(points);
if (!Number.isFinite(n) || n <= 0) return 0;
return Math.floor(SECONDARY_POINTS_PER_MINUTE / n);
}
/** Seconds between requests implied by the ceiling, one worker. Pure. */
export function minGapSeconds(points) {
const ceiling = ceilingPerMinute(points);
return ceiling <= 0 ? 0 : 60 / ceiling;
}
/** What a given request rate costs against the per-minute limit. Pure. */
export function pointsPerMinute(rate, points) {
const r = Math.max(0, Math.trunc(Number(rate) || 0));
const p = Math.trunc(Number(points) || 0);
return r * p;
}
/** How long a batch of this size takes at this rate, in minutes. Pure. */
export function minutesForBatch(count, rate) {
const r = Math.max(0, Math.trunc(Number(rate) || 0));
if (r <= 0) return null;
return Math.ceil(Math.max(0, Math.trunc(Number(count) || 0)) / r);
}
/** Judge a request rate against the per-minute limit. Pure. [state, detail]. */
export function classifyRate(rate, points) {
const spend = pointsPerMinute(rate, points);
const ceiling = ceilingPerMinute(points);
if (!rate) {
return ['not-measured', 'no rate given, so this document is priced but not '
+ `judged. Its ceiling is ${ceiling} request(s)/minute.`];
}
if (spend > SECONDARY_POINTS_PER_MINUTE) {
return ['over-ceiling', `${rate} request(s)/minute of this document is `
+ `${spend} point(s)/minute against a limit of ${SECONDARY_POINTS_PER_MINUTE}.`];
}
if (spend > SECONDARY_POINTS_PER_MINUTE * 0.8) {
return ['near-ceiling', `${rate} request(s)/minute is ${spend} `
+ `point(s)/minute, inside the limit of ${SECONDARY_POINTS_PER_MINUTE} but `
+ 'with under a fifth of it left.'];
}
return ['within-ceiling', `${rate} request(s)/minute is ${spend} `
+ `point(s)/minute against a limit of ${SECONDARY_POINTS_PER_MINUTE}.`];
}
/** Attribute a recorded failure to one bucket or the other. Pure. */
export function classifyThrottle(status, message, graphqlRemaining) {
const text = String(message ?? '').toLowerCase();
const secondary = text.includes('secondary rate limit');
const parsed = Number(graphqlRemaining);
const remaining = Number.isFinite(parsed) && graphqlRemaining !== null
&& graphqlRemaining !== '' ? parsed : null;
const healthy = remaining !== null && remaining > PRIMARY_POINTS_PER_HOUR * 0.1;
if (secondary && healthy) {
return ['secondary-not-budget', `a secondary rate limit with ${remaining} `
+ 'point(s) still in the hourly budget. This is the per-minute ceiling, '
+ 'and no amount of waiting for the hourly reset will help.'];
}
if (secondary) {
return ['secondary-limit', 'a secondary rate limit. The hourly budget was '
+ 'not readable or was itself low, so slow down and check both.'];
}
if (text.includes('rate limit') && remaining === 0) {
return ['primary-exhausted', 'the hourly point budget is spent. That is a '
+ 'different bucket with a different note and it refills on a schedule.'];
}
if (text.includes('rate limit')) {
return ['rate-limited-unclassified', 'a rate-limit message that does not '
+ 'name the secondary limit. Read resources.graphql at the moment of '
+ 'failure to attribute it.'];
}
if (['403', '429'].includes(String(status))) {
return ['forbidden-not-throttled', `HTTP ${status} with no rate-limit `
+ 'wording, so this is a permission problem rather than a throttle.'];
}
return ['no-throttle', 'nothing in this record names a rate limit.'];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (state === 'over-ceiling') {
return 'batch mutations into one document, serialise the loop, and cap it '
+ `at ${ceilingPerMinute(WEIGHT_WITH_MUTATION)}/minute or below. The 5 `
+ 'points are charged per request, so fewer requests is the whole lever.';
}
if (state === 'near-ceiling') {
return 'leave headroom. A retry, a redeploy or one extra worker puts this '
+ 'over, and the limit gives no warning before it binds.';
}
if (state === 'secondary-not-budget') {
return 'rate-limit the writer against points a minute, not requests a '
+ 'minute, and honour retry-after. Do not rewrite the retry logic around '
+ 'the hourly budget; that bucket was fine.';
}
if (state === 'secondary-limit') {
return 'slow the writer down and record resources.graphql at the moment of '
+ 'failure so the next one can be attributed.';
}
if (state === 'primary-exhausted') {
return 'see /github/graphql-rate-limited/ -- the hourly point budget is a '
+ 'different bucket and this is not the note for it.';
}
if (state === 'within-ceiling') {
return 'nothing on the point arithmetic. Check concurrency and the '
+ 'one-second gap between mutations on the same resource separately; '
+ 'points do not express either.';
}
return 'supply the rate the loop actually runs at, or the failure you '
+ 'recorded, and the arithmetic becomes a verdict.';
}
/** Everything this script knows about one document. Pure. */
export function price(label, document, rate) {
const ops = operations(document);
const points = weight(document);
const [state, detail] = classifyRate(rate, points);
return {
document: label,
operations: ops,
points_per_request: points,
ceiling_per_minute: ceilingPerMinute(points),
min_gap_seconds: Number(minGapSeconds(points).toFixed(4)),
points_per_minute_at_rate: pointsPerMinute(rate, points),
not_sent: refusal(document),
state,
detail,
repair: repair(state),
};
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
'User-Agent': UA,
};
}
async function runQuery(token, document) {
const res = await fetch(`${API}/graphql`, {
// A GraphQL query is a read. POST is only how the document reaches the
// endpoint, and refusal() has already rejected anything that is not a read.
method: 'POST',
headers: headers(token),
body: JSON.stringify({ query: document, variables: {} }),
});
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, body };
}
async function graphqlBudget(token) {
const res = await fetch(`${API}/rate_limit`, { headers: headers(token) });
if (!res.ok) return null;
try {
const body = await res.json();
return (body.resources || {}).graphql || null;
} catch { return null; }
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const document = process.env.GITHUB_QUERY;
if (!token || !document) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_QUERY');
process.exitCode = 2;
return;
}
const rate = Number(process.env.GITHUB_RATE || 0);
const batch = Number(process.env.GITHUB_BATCH || 0);
console.log(`point cost: ${POINTS_PER_QUERY} point(s) against the `
+ `${PRIMARY_POINTS_PER_HOUR}/hour GraphQL budget`);
const budget = await graphqlBudget(token);
if (budget) {
console.log(`graphql budget: ${budget.remaining}/${budget.limit} remaining`);
}
if (refusal(PROBE_QUERY) === null) {
const { status } = await runQuery(token, PROBE_QUERY);
console.log(`probe read query: HTTP ${status}, ${POINTS_PER_QUERY} point(s) spent`);
}
const p = price('GITHUB_QUERY', document, rate);
console.log(`${p.document}: operations=${p.operations.join(', ') || 'none'} -> `
+ `${p.points_per_request} point(s) per request`);
if (p.not_sent) console.log(` not sent: ${p.not_sent}`);
console.log(` ceiling ${p.ceiling_per_minute} request(s)/minute, minimum gap `
+ `${p.min_gap_seconds.toFixed(3)}s on one worker`);
console.log(` ${p.state}: ${p.detail}`);
if (batch) {
console.log(` ${batch} row(s) takes at least `
+ `${minutesForBatch(batch, p.ceiling_per_minute)} minute(s) at the ceiling`);
}
console.log(` repair: ${p.repair}`);
console.log(JSON.stringify({
points_spent: POINTS_PER_QUERY,
secondary_points_per_minute: SECONDARY_POINTS_PER_MINUTE,
graphql_budget: budget,
documents: [p],
}, null, 2));
process.exitCode = ['over-ceiling', 'near-ceiling'].includes(p.state) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The weights are asserted directly, because the whole note is two numbers and a ratio: a document with one mutation and a document with six are both 5 points, a document with none is 1, and the word appearing in a string literal or a comment changes nothing. On top of that sit the derived figures — the ceiling, the gap, the points a given rate spends — and then the differential, which is asserted on both readings together, since a secondary message with a healthy budget and a secondary message with an empty one are different findings. The refusal is tested last, including on the script's own probe, which has to pass the same guard as anything a caller supplies.
from github_graphql_mutation_budget import (
PROBE_QUERY, SECONDARY_POINTS_PER_MINUTE, WEIGHT_WITHOUT_MUTATION,
WEIGHT_WITH_MUTATION, ceiling_per_minute, classify_rate, classify_throttle,
min_gap_seconds, minutes_for_batch, operations, points_per_minute, price,
refusal, repair, weight,
)
READ = "query Q($n: Int!) { repository(owner: \"a\", name: \"b\") { issues(first: $n) { nodes { id } } } }"
WRITE = "mutation M($id: ID!) { addLabelsToLabelable(input: {labelableId: $id, labelIds: []}) { clientMutationId } }"
THREE_WRITES = ("mutation A { one { clientMutationId } } "
"mutation B { two { clientMutationId } } "
"mutation C { three { clientMutationId } }")
def test_a_mutation_document_is_five_points_and_a_query_is_one():
assert weight(WRITE) == WEIGHT_WITH_MUTATION == 5
assert weight(READ) == WEIGHT_WITHOUT_MUTATION == 1
def test_the_weight_is_per_request_not_per_mutation():
assert operations(THREE_WRITES) == ["mutation", "mutation", "mutation"]
assert weight(THREE_WRITES) == 5
# Which is why batching is a real reduction: three separate requests would
# be 15 points, the same three in one document are 5.
assert weight(WRITE) * 3 > weight(THREE_WRITES)
def test_the_word_mutation_in_a_string_or_a_comment_is_not_one():
quoted = 'query Q { search(query: "mutation", type: ISSUE, first: 1) { issueCount } }'
assert weight(quoted) == 1
assert refusal(quoted) is None
commented = "# mutation M { addStar }\nquery Q { viewer { login } }"
assert weight(commented) == 1
assert refusal(commented) is None
def test_the_ceiling_is_the_limit_divided_by_the_weight():
assert SECONDARY_POINTS_PER_MINUTE == 2000
assert ceiling_per_minute(5) == 400
assert ceiling_per_minute(1) == 2000
assert ceiling_per_minute(0) == 0
assert ceiling_per_minute(None) == 0
def test_the_gap_falls_out_of_the_ceiling():
assert round(min_gap_seconds(5), 3) == 0.15
assert round(min_gap_seconds(1), 3) == 0.03
assert min_gap_seconds(0) == 0.0
def test_a_rate_is_priced_in_points_not_in_requests():
assert points_per_minute(500, 5) == 2500
assert points_per_minute(500, 1) == 500
assert points_per_minute(0, 5) == 0
assert points_per_minute(None, 5) == 0
def test_the_same_rate_breaks_the_writer_and_not_the_reader():
write_state, _ = classify_rate(500, weight(WRITE))
read_state, _ = classify_rate(500, weight(READ))
assert write_state == "over-ceiling"
assert read_state == "within-ceiling"
def test_a_rate_just_inside_the_limit_is_still_reported():
state, detail = classify_rate(340, 5)
assert state == "near-ceiling"
assert "1700" in detail
assert "headroom" in repair(state)
def test_an_unmeasured_rate_is_priced_but_not_judged():
state, detail = classify_rate(0, 5)
assert state == "not-measured"
assert "400" in detail
def test_a_secondary_message_with_a_healthy_budget_is_the_finding():
state, detail = classify_throttle(
403, "You have exceeded a secondary rate limit", 4863)
assert state == "secondary-not-budget"
assert "4863" in detail
assert "points a minute" in repair(state)
def test_a_secondary_message_with_an_empty_budget_is_not_conclusive():
state, _ = classify_throttle(
429, "You have exceeded a secondary rate limit", 0)
assert state == "secondary-limit"
def test_an_exhausted_hourly_budget_is_handed_to_the_other_note():
state, _ = classify_throttle(200, "API rate limit exceeded", 0)
assert state == "primary-exhausted"
assert "graphql-rate-limited" in repair(state)
def test_a_403_that_is_not_a_throttle_is_not_called_one():
state, _ = classify_throttle(403, "Resource not accessible", 4900)
assert state == "forbidden-not-throttled"
assert classify_throttle("", "", 4900)[0] == "no-throttle"
def test_a_batch_is_costed_in_minutes():
assert minutes_for_batch(11000, 400) == 28
assert minutes_for_batch(11000, 2000) == 6
assert minutes_for_batch(11000, 0) is None
def test_the_document_is_priced_and_refused_in_the_same_breath():
p = price("label_issue.graphql", WRITE, 500)
assert p["points_per_request"] == 5
assert p["ceiling_per_minute"] == 400
assert p["state"] == "over-ceiling"
assert p["not_sent"] and "does not send them" in p["not_sent"]
assert price("fetch.graphql", READ, 500)["not_sent"] is None
def test_the_scripts_own_probe_passes_its_own_guard():
assert refusal(PROBE_QUERY) is None
assert weight(PROBE_QUERY) == 1
assert refusal("subscription S { thing { id } }")
assert refusal("") == "the document contains no operation to send."
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
PROBE_QUERY, SECONDARY_POINTS_PER_MINUTE, WEIGHT_WITHOUT_MUTATION,
WEIGHT_WITH_MUTATION, ceilingPerMinute, classifyRate, classifyThrottle,
minGapSeconds, minutesForBatch, operations, pointsPerMinute, price, refusal,
repair, weight,
} from './github-graphql-mutation-budget.mjs';
const READ = 'query Q($n: Int!) { repository(owner: "a", name: "b") { issues(first: $n) { nodes { id } } } }';
const WRITE = 'mutation M($id: ID!) { addLabelsToLabelable(input: {labelableId: $id, labelIds: []}) { clientMutationId } }';
const THREE_WRITES = 'mutation A { one { clientMutationId } } '
+ 'mutation B { two { clientMutationId } } '
+ 'mutation C { three { clientMutationId } }';
test('a mutation document is five points and a query is one', () => {
assert.equal(weight(WRITE), WEIGHT_WITH_MUTATION);
assert.equal(WEIGHT_WITH_MUTATION, 5);
assert.equal(weight(READ), WEIGHT_WITHOUT_MUTATION);
assert.equal(WEIGHT_WITHOUT_MUTATION, 1);
});
test('the weight is per request not per mutation', () => {
assert.deepEqual(operations(THREE_WRITES), ['mutation', 'mutation', 'mutation']);
assert.equal(weight(THREE_WRITES), 5);
assert.ok(weight(WRITE) * 3 > weight(THREE_WRITES));
});
test('the word mutation in a string or a comment is not one', () => {
const quoted = 'query Q { search(query: "mutation", type: ISSUE, first: 1) { issueCount } }';
assert.equal(weight(quoted), 1);
assert.equal(refusal(quoted), null);
const commented = '# mutation M { addStar }\nquery Q { viewer { login } }';
assert.equal(weight(commented), 1);
assert.equal(refusal(commented), null);
});
test('the ceiling is the limit divided by the weight', () => {
assert.equal(SECONDARY_POINTS_PER_MINUTE, 2000);
assert.equal(ceilingPerMinute(5), 400);
assert.equal(ceilingPerMinute(1), 2000);
assert.equal(ceilingPerMinute(0), 0);
assert.equal(ceilingPerMinute(null), 0);
});
test('the gap falls out of the ceiling', () => {
assert.equal(Number(minGapSeconds(5).toFixed(3)), 0.15);
assert.equal(Number(minGapSeconds(1).toFixed(3)), 0.03);
assert.equal(minGapSeconds(0), 0);
});
test('a rate is priced in points not in requests', () => {
assert.equal(pointsPerMinute(500, 5), 2500);
assert.equal(pointsPerMinute(500, 1), 500);
assert.equal(pointsPerMinute(0, 5), 0);
assert.equal(pointsPerMinute(null, 5), 0);
});
test('the same rate breaks the writer and not the reader', () => {
assert.equal(classifyRate(500, weight(WRITE))[0], 'over-ceiling');
assert.equal(classifyRate(500, weight(READ))[0], 'within-ceiling');
});
test('a rate just inside the limit is still reported', () => {
const [state, detail] = classifyRate(340, 5);
assert.equal(state, 'near-ceiling');
assert.match(detail, /1700/);
assert.match(repair(state), /headroom/);
});
test('an unmeasured rate is priced but not judged', () => {
const [state, detail] = classifyRate(0, 5);
assert.equal(state, 'not-measured');
assert.match(detail, /400/);
});
test('a secondary message with a healthy budget is the finding', () => {
const [state, detail] = classifyThrottle(
403, 'You have exceeded a secondary rate limit', 4863);
assert.equal(state, 'secondary-not-budget');
assert.match(detail, /4863/);
assert.match(repair(state), /points a minute/);
});
test('a secondary message with an empty budget is not conclusive', () => {
assert.equal(
classifyThrottle(429, 'You have exceeded a secondary rate limit', 0)[0],
'secondary-limit',
);
});
test('an exhausted hourly budget is handed to the other note', () => {
const [state] = classifyThrottle(200, 'API rate limit exceeded', 0);
assert.equal(state, 'primary-exhausted');
assert.match(repair(state), /graphql-rate-limited/);
});
test('a 403 that is not a throttle is not called one', () => {
assert.equal(classifyThrottle(403, 'Resource not accessible', 4900)[0],
'forbidden-not-throttled');
assert.equal(classifyThrottle('', '', 4900)[0], 'no-throttle');
});
test('a batch is costed in minutes', () => {
assert.equal(minutesForBatch(11000, 400), 28);
assert.equal(minutesForBatch(11000, 2000), 6);
assert.equal(minutesForBatch(11000, 0), null);
});
test('the document is priced and refused in the same breath', () => {
const p = price('label_issue.graphql', WRITE, 500);
assert.equal(p.points_per_request, 5);
assert.equal(p.ceiling_per_minute, 400);
assert.equal(p.state, 'over-ceiling');
assert.match(p.not_sent, /does not send them/);
assert.equal(price('fetch.graphql', READ, 500).not_sent, null);
});
test('the scripts own probe passes its own guard', () => {
assert.equal(refusal(PROBE_QUERY), null);
assert.equal(weight(PROBE_QUERY), 1);
assert.ok(refusal('subscription S { thing { id } }'));
assert.equal(refusal(''), 'the document contains no operation to send.');
});
FAQ
Why does GET /rate_limit show plenty of points left when I am being throttled?
Because it reports a different bucket. GET /rate_limit describes the primary budget: 5,000 points an hour for a user token, refilling on a schedule, with a remaining count you can read before you spend it. The secondary limit is a per-minute burst control with no bucket, no header and no endpoint that reports headroom, and it is the one a fast mutation loop hits. Seeing a healthy remaining count next to a secondary-rate-limit message is not a contradiction, it is the diagnosis: those two readings together are what tell you which limit you crossed.
Does a document with ten mutations in it cost 50 points?
No. The weight is applied to the request, not to the operations inside it: any document containing a mutation is 5 points, whether that is one mutation or ten. That makes batching the cheapest fix available, because ten separate requests are 50 points and the same ten in one document are 5. It also means the reverse is a real regression — splitting one document into several to make the code tidier multiplies the secondary cost by the number of pieces, and nothing in the response will tell you that happened.
Is 400 mutations a minute actually safe to run at?
It is the ceiling, not a target. Two other rules sit on top of the point arithmetic and neither is expressed in points: no more than 100 concurrent requests across the REST and GraphQL APIs together, and at least one second between mutations that affect the same resource. A loop that satisfies the points and violates either of those is still throttled. Treat 400 as the number you must stay under and pick your real rate from the shape of the work, leaving room for retries, which cost the same as first attempts.
Should the script not just send a mutation to prove the limit exists?
It should not, and it cannot. Every script in this section holds a token that can reach real repositories and none of them writes, so a note about mutations has to be built out of the document text, the documented weights and failures you already recorded. That is enough: the weight is a property of the document, the ceiling is division, and the attribution of a past 403 is a comparison between its message and the budget at that moment. The script parses your mutation documents, prices them, and refuses to open a socket for them.
How is this different from the REST secondary limits?
Different endpoint, different numbers, same family of control. REST has a per-endpoint allowance of 900 points a minute alongside a CPU-time cap, and a separate ceiling of roughly 80 content-generating requests a minute; both have their own notes here. The GraphQL endpoint has this one: 2,000 points a minute, mutations at 5 and everything else at 1. A team that moved its writes to GraphQL to get away from the content-creation limit did not escape a limit, it changed which arithmetic applies, and 400 requests a minute is the number that matters afterwards.
Related field notes
- The hourly GraphQL point budget is a separate bucket
- The REST content-creation limit, around 80 a minute
- Ignoring retry-after extends the throttle
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Resource limitations — GitHub GraphQL API
- Rate limits and node limits for the GraphQL API — GitHub Docs
- Rate limits for the REST API — GitHub Docs
- Rate limit — GitHub REST API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.