Diagnostic GitHub API
GraphQL returns 200 with an errors array and null data
The call succeeded. response.ok is true, the status is 200, the body is valid JSON, and data.repository is null. Downstream something throws Cannot read property 'name' of null, or, far worse, nothing throws at all and the dashboard records that the repository has zero open pull requests. The reason is one line further down the body, in an errors array nobody wrote a branch for.
GraphQL reports application failures in the response body, not in the status line. A query that hit a missing repository, a permission your token does not have, an exhausted point budget or a query too large to run still comes back as 200 OK with {"data": ..., "errors": [{"type": "...", "message": "..."}]}. Error handling written around HTTP status codes sees a success and carries on.
Read body.errors before you touch body.data, on every single response, and branch on errors[].type rather than on the message text: RATE_LIMITED waits, FORBIDDEN alerts a human, NOT_FOUND marks the record missing, MAX_NODE_LIMIT_EXCEEDED means the query has to be reshaped and will never succeed on a retry. A non-empty errors array is the finding whatever the status says.
The problem in plain words
This one is written by good habits rather than bad ones. Every REST client in the codebase checks the status code, because for REST that is the right thing to check, and the GraphQL client gets written by the same person on the same afternoon with the same shape: send it, check res.ok, parse the JSON, use the data. Nothing about that reads as wrong in review. The errors key is not in the happy-path response at all, so nobody who has only ever seen a working call knows it can be there.
It then holds up beautifully in development, where the token is your own and has every permission, the repository definitely exists, and the point budget is nowhere near spent. The first response that carries an errors array arrives in production, from a token that is deliberately narrower than yours, against a repository that got renamed last week.
What it looks like from the outside is not an error at all. If you are lucky it is a TypeError on a null, thrown three frames away from the request, with a stack trace that points at your own code and a logged status of 200 that makes the API look innocent. If you are unlucky the code is defensive in the wrong direction: it treats a null repository as an empty one, writes zero into the metrics table, and the number is wrong for months in a way that looks like a real decline in activity.
Why it happens
The status code describes the transport, not the query. A 200 from /graphql means the endpoint received a document, parsed it and produced a response. It does not mean the response contains what you asked for. The GraphQL specification puts execution errors in a top-level errors entry in the response body precisely because a single document can fail in many places at once, which a single status code cannot express.
Every interesting failure arrives this way. A repository your token cannot see, a repository that does not exist, an exhausted point budget, a query over the node limit, an internal error on GitHub's side: all of them are 200 responses with a typed entry in errors. The failures that do change the status line are the boring ones — 401 for a bad token, 502 for an outage — and those are the ones your status check already handles.
The type field is the part to branch on. NOT_FOUND, FORBIDDEN, RATE_LIMITED, MAX_NODE_LIMIT_EXCEEDED and INTERNAL demand four genuinely different responses and one of them is "stop retrying, this query can never work". Matching on message text instead is fragile in the ordinary way, and it also loses the distinction between a failure that will clear on its own and one that will not.
A 404 does not exist here, and that matters. The REST API answers 404 for a repository you cannot see as well as for one that is not there, which is its own well-known trap. GraphQL is better: it tells you NOT_FOUND or FORBIDDEN in the type field and it will happily tell you both about different fields of the same query. You only get that resolution if you read the array.
Errors alongside surviving data are a different problem. When data is null and errors is non-empty, the whole query failed and this note is the one you want. When data is populated and errors is non-empty, some fields resolved and some did not, which is partial success and a different repair. This script separates the two and refuses to give advice about the second, because "the call failed, retry it" is the wrong instruction for a response that is nine tenths correct.
The API cannot see your error handling. Nothing GitHub returns says whether your client reads errors. What a script can do is make the endpoint produce the shape on demand, print the two predicates side by side, and show you which one your code is using. That is the trap being demonstrated, not your bug being found, and the script says so.
The fix, as a flow
One query per probe, deliberately aimed at something the endpoint will refuse, because the shape this note is about only exists on a response that failed. The document is parsed before anything is sent and a mutation or a subscription is refused outright, since the GraphQL endpoint is reached the same way for reads and writes. Everything after the request is two pure predicates over one body: the one a status-code client uses and the one a correct client uses. Where they disagree is the finding, and every response that carries errors alongside surviving data is named as a different problem rather than folded into this one.
How to fix it
Send one query that is guaranteed to fail in the body
Ask for a repository that does not exist, or one this token cannot see. One query, one point. What comes back is a 200, a valid JSON body, data.repository set to null and an errors array with a type in it. That is the whole phenomenon in a single response, and it is much more convincing than reading about it.
Print both predicates over that one response
Evaluate "the status is 200" and "the errors array is empty" against the same body and print both answers. On this response the first says success and the second says failure. Their disagreement is the finding, and it is the sentence that changes somebody's client, because it names the exact line that has to move.
Read the type, not the message
Pull errors[].type out and map it to a behaviour before you write a handler. The script prints the mapping it recommends for each type it sees, so that RATE_LIMITED becomes a wait, FORBIDDEN becomes an alert to a human, NOT_FOUND becomes a recorded absence, and MAX_NODE_LIMIT_EXCEEDED becomes a query change rather than a retry loop that can never terminate.
Split total failure from partial success
If data came back with real values in it alongside the errors, this is not a failed call and treating it as one throws away good data. The script labels that case separately and points at the partial-response note instead of pretending one rule covers both. Getting this boundary right is most of the value of reading the envelope at all.
Move the check to the front of the client and keep the run cheap
Put the errors check in the one function that sends queries, before any caller sees a body, so it cannot be forgotten per call site. The audit itself costs one point per probe, two by default, out of 5,000 an hour. GET /rate_limit reports the GraphQL budget and does not consume any of it.
How to check it worked
Once the client reads the envelope, the same probe reports the disagreement and the client no longer has it.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_envelope.py --repo acme/renamed-last-week
# point cost: 2 point(s) against the 5,000/hour GraphQL budget
# probe missing-repository: HTTP 200, errors=1, data present=no
# 200-with-errors-no-data: the status line says success and the body carries 1
# error(s) of type NOT_FOUND with no usable data
# status check passes: yes envelope check passes: no they disagree: yes
# repair: read body.errors before body.data and branch on errors[].type
# a probe that returns errors alongside real data is handed on, not absorbed
# 200-with-errors-and-data: 1 error(s) of type FORBIDDEN arrived with usable
# data, which is partial success and a different repair
# repair: see /github/graphql-partial-data-nulls/ -- do not retry this oneThe full code
One query per probe, sent to the GraphQL endpoint because that is the only way a document travels, and refused outright if the document contains a mutation or a subscription. Everything after the request is pure: the predicate a status-code client uses, the predicate a correct client uses, the disagreement between them, the error types lifted out of the body and the behaviour each one demands. The whole rule is therefore testable against recorded envelopes without a token, which is the point, because the envelopes you most need to handle are the ones you cannot conveniently produce.
"""Show that a GraphQL 200 can carry an errors array a status check walks past.
Read only, and queries only. GitHub's GraphQL endpoint takes a document in the
request body, so a read is carried by POST there just as a write would be; that
is a transport detail, not a licence to write. This script sends queries and
refuses any document containing a mutation or a subscription before it opens a
socket. Nothing is written and the repair is printed rather than performed.
GraphQL reports application failures in the response body. A query that hit a
missing repository, a permission the token lacks, an exhausted point budget or a
query too large to run still returns 200 OK with an errors array beside a null
data field. Error handling written around HTTP status codes sees a success.
What this can and cannot see: the API has no idea whether your client reads the
errors array. What it can do is make the endpoint produce the shape on demand
and print both predicates over the same response so you can compare them against
your own code. That is the trap, not the fall.
Environment:
GITHUB_TOKEN a token with read access to the GraphQL API
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_graphql_envelope")
API = "https://api.github.com"
UA = "github-graphql-envelope/1.0"
# A simple query costs one point. Named because it is printed before anything is
# spent, and a reader comparing this against the documentation should find it in
# one place.
POINTS_PER_QUERY = 1
DEFAULT_QUERY = (
"query($owner: String!, $name: String!) {"
" repository(owner: $owner, name: $name) { name isPrivate } }"
)
# The behaviours the five documented error types actually demand. Two of these
# are retryable, one is a permission change, one is a query change that no
# amount of retrying will fix, and one is a fact about the world.
BEHAVIOUR = {
"RATE_LIMITED": ("wait", "the point budget is spent. Wait for the reset that "
"GET /rate_limit reports and do not retry before it."),
"FORBIDDEN": ("alert", "the token cannot see this. Retrying changes nothing; "
"a human has to widen the permission or accept the gap."),
"NOT_FOUND": ("record-absent", "the resource is missing or invisible to this "
"token. Record the absence; do not treat it as zero."),
"MAX_NODE_LIMIT_EXCEEDED": ("reshape", "the query asks for too many nodes and "
"will fail identically every time. Lower "
"the first values and paginate."),
"INTERNAL": ("retry-once", "a failure on GitHub's side. Retry once with backoff, "
"then give up and log the query."),
"SERVICE_UNAVAILABLE": ("retry-once", "a transient failure on GitHub's side. "
"Retry once with backoff."),
}
def strip_noise(document):
"""Remove GraphQL comments and string literals from a document. Pure.
Written as a scanner rather than a regex because a hash inside a string
literal is a legitimate character and a comment marker outside one, and a
single pattern that gets that right is harder to read than this loop.
"""
src = str(document or "")
out = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "#":
while i < n and src[i] != "\n":
i += 1
continue
if src.startswith('"""', i):
j = src.find('"""', i + 3)
i = n if j < 0 else j + 3
out.append(" ")
continue
if ch == '"':
i += 1
while i < n and src[i] != '"':
i += 2 if src[i] == "\\" else 1
i += 1
out.append(" ")
continue
out.append(ch)
i += 1
return "".join(out)
def operations(document):
"""The top-level operations in a document, in order. Pure.
One entry per brace group at depth zero: "query", "mutation",
"subscription" or "fragment". An anonymous document is the query shorthand.
"""
src = strip_noise(document)
ops, depth, word, declared = [], 0, "", None
for ch in src + " ":
if ch.isalnum() or ch == "_":
word += ch
continue
if word:
if depth == 0 and word in ("query", "mutation", "subscription", "fragment"):
declared = word
word = ""
if ch == "{":
if depth == 0:
ops.append(declared or "query")
declared = None
depth += 1
elif ch == "}":
depth = max(0, depth - 1)
return ops
def refusal(document):
"""Why this document will not be sent, or None if it is a read. Pure.
The endpoint is the same one mutations go to, so the guard lives here rather
than in a comment. A section that promises its scripts never write has to
mean it on the one endpoint where writing is a body away.
"""
ops = operations(document)
if not ops:
return "the document contains no operation to send."
for kind in ("mutation", "subscription"):
if kind in ops:
return ("the document contains a %s. This script sends queries only: "
"a query is a read, and the section it belongs to promises "
"its scripts never write." % kind)
return None
def status_says_ok(status):
"""The predicate a status-code client uses. Pure."""
try:
return 200 <= int(status) < 300
except (TypeError, ValueError):
return False
def envelope_says_ok(body):
"""The predicate a correct client uses. Pure."""
if not isinstance(body, dict):
return False
return not body.get("errors")
def error_types(body):
"""The type of every entry in the errors array, in order. Pure.
An entry with no type is reported as UNTYPED rather than dropped, because a
handler keyed on type has to have something to fall through to.
"""
if not isinstance(body, dict):
return []
out = []
for err in body.get("errors") or []:
if isinstance(err, dict):
out.append(err.get("type") or "UNTYPED")
else:
out.append("UNTYPED")
return out
def has_usable_data(body):
"""Whether any field in data resolved to something other than null. Pure."""
if not isinstance(body, dict):
return False
data = body.get("data")
if not isinstance(data, dict):
return False
return any(v is not None for v in data.values())
def predicates_disagree(status, body):
"""Whether a status check would pass on a response the envelope fails."""
return status_says_ok(status) and not envelope_says_ok(body)
def behaviour_for(error_type):
"""What one error type demands of a client. Pure. Returns (action, detail)."""
if error_type in BEHAVIOUR:
return BEHAVIOUR[error_type]
return ("log-verbatim",
"an error type this script does not know. Log it verbatim and fail "
"the call rather than guessing; new types get added over time.")
def classify(status, body):
"""Classify one response envelope. Pure. Returns (state, detail).
The two states that both carry errors are kept apart on purpose. One is a
call that failed and the other is a call that mostly worked, and giving the
same advice for both throws away good data.
"""
if not isinstance(body, dict):
return ("unreadable",
"the response was not a JSON object, so neither predicate can be "
"evaluated over it.")
if not status_says_ok(status):
return ("transport-failure",
"HTTP %s, which a status check already catches. The errors array "
"is not where this one hides." % status)
types = error_types(body)
if not types:
return ("200-clean",
"the status line and the errors array agree that this worked. "
"Both predicates pass, which on this response is agreement rather "
"than proof that your client checks the second one.")
if has_usable_data(body):
return ("200-with-errors-and-data",
"%d error(s) of type %s arrived with usable data, which is partial "
"success and a different repair."
% (len(types), ", ".join(sorted(set(types)))))
return ("200-with-errors-no-data",
"the status line says success and the body carries %d error(s) of type "
"%s with no usable data."
% (len(types), ", ".join(sorted(set(types)))))
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state == "200-with-errors-no-data":
return ("read body.errors before body.data and branch on errors[].type. "
"Put the check in the function that sends queries so no caller "
"can skip it.")
if state == "200-with-errors-and-data":
return ("see /github/graphql-partial-data-nulls/ -- do not retry this "
"one. Some fields resolved and discarding them because the call "
"carried errors loses data that arrived correctly.")
if state == "transport-failure":
return ("handle the status code as you already do. This note is about "
"the failures that arrive as a 200.")
if state == "200-clean":
return ("nothing on this response. Check that the errors array is read "
"at all: the two predicates agree here and part company on the "
"first failure.")
return "point the check at a document this endpoint can answer."
def point_cost(probes):
"""Points this run will spend against the GraphQL budget. Pure."""
return len(probes or []) * POINTS_PER_QUERY
def run_query(session, document, variables):
"""Send one query. Returns (status, body-or-None).
A GraphQL query is a read; POST is only how the document reaches the
endpoint, which is why the verb is written here beside the URL rather than
tucked into a constant where it could be mistaken for a write path.
"""
r = session.post(API + "/graphql",
json={"query": document, "variables": variables or {}},
timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
try:
return r.status_code, r.json()
except ValueError:
return r.status_code, None
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True,
help="owner/name to probe. A repository that does not exist "
"is the cheapest way to see the shape.")
ap.add_argument("--query",
help="send your own query document instead of the default. "
"Mutations and subscriptions are refused.")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
try:
owner, name = args.repo.split("/", 1)
except ValueError:
log.error("--repo takes owner/name")
return 2
document = args.query or DEFAULT_QUERY
why_not = refusal(document)
if why_not:
log.error("refusing to send: %s", why_not)
return 2
probes = [
("missing-repository", document,
{"owner": owner, "name": name + "-does-not-exist-probe"}),
("as-configured", document, {"owner": owner, "name": name}),
]
log.info("point cost: %d point(s) against the 5,000/hour GraphQL budget",
point_cost(probes))
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
findings = []
for label, doc, variables in probes:
status, body = run_query(session, doc, variables)
state, detail = classify(status, body)
types = error_types(body)
log.info("probe %s: HTTP %s, errors=%d, data present=%s",
label, status, len(types), "yes" if has_usable_data(body) else "no")
log.info("%s: %s", state, detail)
log.info("status check passes: %s envelope check passes: %s "
"they disagree: %s",
"yes" if status_says_ok(status) else "no",
"yes" if envelope_says_ok(body) else "no",
"yes" if predicates_disagree(status, body) else "no")
for t in sorted(set(types)):
action, why = behaviour_for(t)
log.info(" %s -> %s: %s", t, action, why)
log.info("repair: %s", repair(state))
findings.append({
"probe": label,
"status": status,
"error_types": types,
"has_usable_data": has_usable_data(body),
"status_check_passes": status_says_ok(status),
"envelope_check_passes": envelope_says_ok(body),
"predicates_disagree": predicates_disagree(status, body),
"behaviours": {t: behaviour_for(t)[0] for t in sorted(set(types))},
"state": state,
"detail": detail,
})
print(json.dumps({"points_spent": point_cost(probes), "findings": findings},
indent=2, default=str))
return 1 if any(f["predicates_disagree"] for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Show that a GraphQL 200 can carry an errors array a status check walks past.
*
* Read only, and queries only. GitHub's GraphQL endpoint takes a document in
* the request body, so a read is carried by POST there just as a write would
* be; that is a transport detail, not a licence to write. Any document
* containing a mutation or a subscription is refused before a socket opens.
*
* Environment:
* GITHUB_TOKEN a token with read access to the GraphQL API
* GITHUB_REPO owner/name to probe
* GITHUB_QUERY send your own query document instead of the default
*/
const API = 'https://api.github.com';
const UA = 'github-graphql-envelope/1.0';
/** A simple query costs one point. */
export const POINTS_PER_QUERY = 1;
const DEFAULT_QUERY = 'query($owner: String!, $name: String!) {'
+ ' repository(owner: $owner, name: $name) { name isPrivate } }';
/** The behaviours the documented error types actually demand. */
export const BEHAVIOUR = {
RATE_LIMITED: ['wait', 'the point budget is spent. Wait for the reset that '
+ 'GET /rate_limit reports and do not retry before it.'],
FORBIDDEN: ['alert', 'the token cannot see this. Retrying changes nothing; a '
+ 'human has to widen the permission or accept the gap.'],
NOT_FOUND: ['record-absent', 'the resource is missing or invisible to this '
+ 'token. Record the absence; do not treat it as zero.'],
MAX_NODE_LIMIT_EXCEEDED: ['reshape', 'the query asks for too many nodes and '
+ 'will fail identically every time. Lower the first values and paginate.'],
INTERNAL: ['retry-once', "a failure on GitHub's side. Retry once with "
+ 'backoff, then give up and log the query.'],
SERVICE_UNAVAILABLE: ['retry-once', "a transient failure on GitHub's side. "
+ 'Retry once with backoff.'],
};
/** Remove GraphQL comments and string literals from a document. Pure. */
export function stripNoise(document) {
const src = String(document ?? '');
const out = [];
let i = 0;
while (i < src.length) {
const ch = src[i];
if (ch === '#') {
while (i < src.length && src[i] !== '\n') i += 1;
continue;
}
if (src.startsWith('"""', i)) {
const j = src.indexOf('"""', i + 3);
i = j < 0 ? src.length : j + 3;
out.push(' ');
continue;
}
if (ch === '"') {
i += 1;
while (i < src.length && src[i] !== '"') i += src[i] === '\\' ? 2 : 1;
i += 1;
out.push(' ');
continue;
}
out.push(ch);
i += 1;
}
return out.join('');
}
/** The top-level operations in a document, in order. Pure. */
export function operations(document) {
const src = `${stripNoise(document)} `;
const ops = [];
let depth = 0;
let word = '';
let declared = null;
for (const ch of src) {
if (/[A-Za-z0-9_]/.test(ch)) { word += ch; continue; }
if (word) {
if (depth === 0 && ['query', 'mutation', 'subscription', 'fragment'].includes(word)) {
declared = word;
}
word = '';
}
if (ch === '{') {
if (depth === 0) { ops.push(declared || 'query'); declared = null; }
depth += 1;
} else if (ch === '}') {
depth = Math.max(0, depth - 1);
}
}
return ops;
}
/** Why this document will not be sent, or null if it is a read. Pure. */
export function refusal(document) {
const ops = operations(document);
if (ops.length === 0) return 'the document contains no operation to send.';
for (const kind of ['mutation', 'subscription']) {
if (ops.includes(kind)) {
return `the document contains a ${kind}. This script sends queries only: `
+ 'a query is a read, and the section it belongs to promises its '
+ 'scripts never write.';
}
}
return null;
}
/** The predicate a status-code client uses. Pure. */
export function statusSaysOk(status) {
const n = Number(status);
return Number.isFinite(n) && n >= 200 && n < 300;
}
/** The predicate a correct client uses. Pure. */
export function envelopeSaysOk(body) {
if (!body || typeof body !== 'object') return false;
return !(Array.isArray(body.errors) && body.errors.length > 0);
}
/** The type of every entry in the errors array, in order. Pure. */
export function errorTypes(body) {
if (!body || typeof body !== 'object' || !Array.isArray(body.errors)) return [];
return body.errors.map((e) => (e && typeof e === 'object' && e.type) || 'UNTYPED');
}
/** Whether any field in data resolved to something other than null. Pure. */
export function hasUsableData(body) {
if (!body || typeof body !== 'object') return false;
const data = body.data;
if (!data || typeof data !== 'object' || Array.isArray(data)) return false;
return Object.values(data).some((v) => v !== null && v !== undefined);
}
/** Whether a status check would pass on a response the envelope fails. Pure. */
export function predicatesDisagree(status, body) {
return statusSaysOk(status) && !envelopeSaysOk(body);
}
/** What one error type demands of a client. Pure. Returns [action, detail]. */
export function behaviourFor(errorType) {
if (Object.prototype.hasOwnProperty.call(BEHAVIOUR, errorType)) {
return BEHAVIOUR[errorType];
}
return ['log-verbatim', 'an error type this script does not know. Log it '
+ 'verbatim and fail the call rather than guessing; new types get added '
+ 'over time.'];
}
/** Classify one response envelope. Pure. Returns [state, detail]. */
export function classify(status, body) {
if (!body || typeof body !== 'object' || Array.isArray(body)) {
return ['unreadable', 'the response was not a JSON object, so neither '
+ 'predicate can be evaluated over it.'];
}
if (!statusSaysOk(status)) {
return ['transport-failure', `HTTP ${status}, which a status check already `
+ 'catches. The errors array is not where this one hides.'];
}
const types = errorTypes(body);
if (types.length === 0) {
return ['200-clean', 'the status line and the errors array agree that this '
+ 'worked. Both predicates pass, which on this response is agreement '
+ 'rather than proof that your client checks the second one.'];
}
const named = [...new Set(types)].sort().join(', ');
if (hasUsableData(body)) {
return ['200-with-errors-and-data',
`${types.length} error(s) of type ${named} arrived with usable data, `
+ 'which is partial success and a different repair.'];
}
return ['200-with-errors-no-data',
`the status line says success and the body carries ${types.length} error(s) `
+ `of type ${named} with no usable data.`];
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (state === '200-with-errors-no-data') {
return 'read body.errors before body.data and branch on errors[].type. Put '
+ 'the check in the function that sends queries so no caller can skip it.';
}
if (state === '200-with-errors-and-data') {
return 'see /github/graphql-partial-data-nulls/ -- do not retry this one. '
+ 'Some fields resolved and discarding them because the call carried '
+ 'errors loses data that arrived correctly.';
}
if (state === 'transport-failure') {
return 'handle the status code as you already do. This note is about the '
+ 'failures that arrive as a 200.';
}
if (state === '200-clean') {
return 'nothing on this response. Check that the errors array is read at '
+ 'all: the two predicates agree here and part company on the first '
+ 'failure.';
}
return 'point the check at a document this endpoint can answer.';
}
/** Points this run will spend against the GraphQL budget. Pure. */
export function pointCost(probes) {
return (Array.isArray(probes) ? probes.length : 0) * POINTS_PER_QUERY;
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
'User-Agent': UA,
};
}
async function runQuery(token, document, variables) {
const res = await fetch(`${API}/graphql`, {
// A GraphQL query is a read. POST is only how the document reaches the
// endpoint, and refusal() has already rejected anything that is not a read.
method: 'POST',
headers: headers(token),
body: JSON.stringify({ query: document, variables: variables || {} }),
});
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, body };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const repo = process.env.GITHUB_REPO;
if (!token || !repo) {
console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_REPO=owner/name');
process.exitCode = 2;
return;
}
const [owner, name] = repo.split('/');
if (!owner || !name) {
console.error('GITHUB_REPO takes owner/name');
process.exitCode = 2;
return;
}
const document = process.env.GITHUB_QUERY || DEFAULT_QUERY;
const whyNot = refusal(document);
if (whyNot) {
console.error(`refusing to send: ${whyNot}`);
process.exitCode = 2;
return;
}
const probes = [
['missing-repository', { owner, name: `${name}-does-not-exist-probe` }],
['as-configured', { owner, name }],
];
console.log(`point cost: ${pointCost(probes)} point(s) against the 5,000/hour GraphQL budget`);
const findings = [];
for (const [label, variables] of probes) {
const { status, body } = await runQuery(token, document, variables);
const [state, detail] = classify(status, body);
const types = errorTypes(body);
console.log(`probe ${label}: HTTP ${status}, errors=${types.length}, `
+ `data present=${hasUsableData(body) ? 'yes' : 'no'}`);
console.log(`${state}: ${detail}`);
console.log(`status check passes: ${statusSaysOk(status) ? 'yes' : 'no'} `
+ `envelope check passes: ${envelopeSaysOk(body) ? 'yes' : 'no'} `
+ `they disagree: ${predicatesDisagree(status, body) ? 'yes' : 'no'}`);
for (const t of [...new Set(types)].sort()) {
const [action, why] = behaviourFor(t);
console.log(` ${t} -> ${action}: ${why}`);
}
console.log(`repair: ${repair(state)}`);
findings.push({
probe: label,
status,
error_types: types,
has_usable_data: hasUsableData(body),
status_check_passes: statusSaysOk(status),
envelope_check_passes: envelopeSaysOk(body),
predicates_disagree: predicatesDisagree(status, body),
state,
detail,
});
}
console.log(JSON.stringify({ points_spent: pointCost(probes), findings }, null, 2));
process.exitCode = findings.some((f) => f.predicates_disagree) ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Most of the suite is envelopes: a 200 with errors and no data, a 200 with errors and real data beside them, a clean 200, a genuine transport failure and a body that is not an object at all. The two predicates are asserted separately and then asserted to disagree on exactly one of those shapes, because the disagreement is the note. After that, the error-type mapping, including the fall-through for a type that did not exist when this was written, the refusal that keeps a mutation off the wire, and the point cost, which is asserted because a script that prints a number before spending it should be held to the number.
from github_graphql_envelope import (
POINTS_PER_QUERY, behaviour_for, classify, envelope_says_ok, error_types,
has_usable_data, operations, point_cost, predicates_disagree, refusal,
repair, status_says_ok,
)
FAILED = {"data": {"repository": None},
"errors": [{"type": "NOT_FOUND", "message": "Could not resolve to a Repository"}]}
PARTIAL = {"data": {"repository": {"name": "monorepo", "diskUsage": None}},
"errors": [{"type": "FORBIDDEN", "path": ["repository", "diskUsage"]}]}
CLEAN = {"data": {"repository": {"name": "monorepo"}}}
def test_the_status_line_says_success_on_a_failed_query():
assert status_says_ok(200)
assert status_says_ok("201")
assert not status_says_ok(403)
assert not status_says_ok(None)
def test_the_envelope_check_reads_the_body_instead():
assert not envelope_says_ok(FAILED)
assert not envelope_says_ok(PARTIAL)
assert envelope_says_ok(CLEAN)
assert envelope_says_ok({"data": {}, "errors": []})
assert not envelope_says_ok("not a body")
def test_the_finding_is_exactly_the_disagreement():
assert predicates_disagree(200, FAILED)
assert predicates_disagree(200, PARTIAL)
assert not predicates_disagree(200, CLEAN)
assert not predicates_disagree(502, FAILED)
def test_error_types_survive_an_entry_with_no_type():
assert error_types(FAILED) == ["NOT_FOUND"]
assert error_types({"errors": [{"message": "boom"}]}) == ["UNTYPED"]
assert error_types({"errors": ["a string"]}) == ["UNTYPED"]
assert error_types(CLEAN) == []
def test_usable_data_means_at_least_one_field_resolved():
assert not has_usable_data(FAILED)
assert has_usable_data(PARTIAL)
assert has_usable_data(CLEAN)
assert not has_usable_data({"data": None, "errors": [{"type": "RATE_LIMITED"}]})
def test_a_200_carrying_errors_and_no_data_is_the_headline():
state, detail = classify(200, FAILED)
assert state == "200-with-errors-no-data"
assert "NOT_FOUND" in detail
assert "read body.errors before body.data" in repair(state)
def test_errors_alongside_real_data_are_handed_on_rather_than_absorbed():
state, detail = classify(200, PARTIAL)
assert state == "200-with-errors-and-data"
assert "partial success" in detail
assert "graphql-partial-data-nulls" in repair(state)
assert "do not retry" in repair(state)
def test_a_real_transport_failure_is_not_this_note():
state, _detail = classify(502, {"errors": [{"type": "INTERNAL"}]})
assert state == "transport-failure"
assert "status code as you already do" in repair(state)
def test_a_clean_response_is_not_reported_as_proof_of_anything():
state, detail = classify(200, CLEAN)
assert state == "200-clean"
assert "agreement rather than proof" in detail
def test_an_unreadable_body_is_not_reported_as_success():
assert classify(200, None)[0] == "unreadable"
assert classify(200, [1, 2])[0] == "unreadable"
def test_each_error_type_gets_its_own_behaviour():
assert behaviour_for("RATE_LIMITED")[0] == "wait"
assert behaviour_for("FORBIDDEN")[0] == "alert"
assert behaviour_for("NOT_FOUND")[0] == "record-absent"
assert behaviour_for("MAX_NODE_LIMIT_EXCEEDED")[0] == "reshape"
assert behaviour_for("INTERNAL")[0] == "retry-once"
def test_a_node_limit_error_is_never_advised_to_retry():
action, detail = behaviour_for("MAX_NODE_LIMIT_EXCEEDED")
assert action == "reshape"
assert "fail identically every time" in detail
def test_an_unknown_error_type_falls_through_rather_than_being_guessed():
action, detail = behaviour_for("SOMETHING_NEW_IN_2027")
assert action == "log-verbatim"
assert "does not know" in detail
def test_the_script_refuses_to_send_a_mutation():
assert operations("query Q { viewer { login } }") == ["query"]
assert operations("{ viewer { login } }") == ["query"]
assert operations("mutation M { addStar(input: {}) { clientMutationId } }") == ["mutation"]
assert refusal("mutation M { addStar(input: {}) { clientMutationId } }")
assert refusal("subscription S { thing { id } }")
assert refusal("") == "the document contains no operation to send."
assert refusal("query Q { viewer { login } }") is None
def test_the_word_mutation_inside_a_string_is_not_a_mutation():
doc = 'query Q { search(query: "mutation", type: ISSUE, first: 1) { issueCount } }'
assert operations(doc) == ["query"]
assert refusal(doc) is None
def test_a_commented_out_mutation_is_not_sent_and_not_feared():
doc = "# mutation M { addStar }\nquery Q { viewer { login } }"
assert operations(doc) == ["query"]
assert refusal(doc) is None
def test_the_run_says_what_it_will_spend():
assert POINTS_PER_QUERY == 1
assert point_cost([1, 2]) == 2
assert point_cost([]) == 0
assert point_cost(None) == 0
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
POINTS_PER_QUERY, behaviourFor, classify, envelopeSaysOk, errorTypes,
hasUsableData, operations, pointCost, predicatesDisagree, refusal, repair,
statusSaysOk,
} from './github-graphql-envelope.mjs';
const FAILED = {
data: { repository: null },
errors: [{ type: 'NOT_FOUND', message: 'Could not resolve to a Repository' }],
};
const PARTIAL = {
data: { repository: { name: 'monorepo', diskUsage: null } },
errors: [{ type: 'FORBIDDEN', path: ['repository', 'diskUsage'] }],
};
const CLEAN = { data: { repository: { name: 'monorepo' } } };
test('the status line says success on a failed query', () => {
assert.ok(statusSaysOk(200));
assert.ok(statusSaysOk('201'));
assert.ok(!statusSaysOk(403));
assert.ok(!statusSaysOk(null));
});
test('the envelope check reads the body instead', () => {
assert.ok(!envelopeSaysOk(FAILED));
assert.ok(!envelopeSaysOk(PARTIAL));
assert.ok(envelopeSaysOk(CLEAN));
assert.ok(envelopeSaysOk({ data: {}, errors: [] }));
assert.ok(!envelopeSaysOk('not a body'));
});
test('the finding is exactly the disagreement', () => {
assert.ok(predicatesDisagree(200, FAILED));
assert.ok(predicatesDisagree(200, PARTIAL));
assert.ok(!predicatesDisagree(200, CLEAN));
assert.ok(!predicatesDisagree(502, FAILED));
});
test('error types survive an entry with no type', () => {
assert.deepEqual(errorTypes(FAILED), ['NOT_FOUND']);
assert.deepEqual(errorTypes({ errors: [{ message: 'boom' }] }), ['UNTYPED']);
assert.deepEqual(errorTypes({ errors: ['a string'] }), ['UNTYPED']);
assert.deepEqual(errorTypes(CLEAN), []);
});
test('usable data means at least one field resolved', () => {
assert.ok(!hasUsableData(FAILED));
assert.ok(hasUsableData(PARTIAL));
assert.ok(hasUsableData(CLEAN));
assert.ok(!hasUsableData({ data: null, errors: [{ type: 'RATE_LIMITED' }] }));
});
test('a 200 carrying errors and no data is the headline', () => {
const [state, detail] = classify(200, FAILED);
assert.equal(state, '200-with-errors-no-data');
assert.match(detail, /NOT_FOUND/);
assert.match(repair(state), /read body\.errors before body\.data/);
});
test('errors alongside real data are handed on rather than absorbed', () => {
const [state, detail] = classify(200, PARTIAL);
assert.equal(state, '200-with-errors-and-data');
assert.match(detail, /partial success/);
assert.match(repair(state), /graphql-partial-data-nulls/);
assert.match(repair(state), /do not retry/);
});
test('a real transport failure is not this note', () => {
const [state] = classify(502, { errors: [{ type: 'INTERNAL' }] });
assert.equal(state, 'transport-failure');
assert.match(repair(state), /status code as you already do/);
});
test('a clean response is not reported as proof of anything', () => {
const [state, detail] = classify(200, CLEAN);
assert.equal(state, '200-clean');
assert.match(detail, /agreement rather than proof/);
});
test('an unreadable body is not reported as success', () => {
assert.equal(classify(200, null)[0], 'unreadable');
assert.equal(classify(200, [1, 2])[0], 'unreadable');
});
test('each error type gets its own behaviour', () => {
assert.equal(behaviourFor('RATE_LIMITED')[0], 'wait');
assert.equal(behaviourFor('FORBIDDEN')[0], 'alert');
assert.equal(behaviourFor('NOT_FOUND')[0], 'record-absent');
assert.equal(behaviourFor('MAX_NODE_LIMIT_EXCEEDED')[0], 'reshape');
assert.equal(behaviourFor('INTERNAL')[0], 'retry-once');
});
test('a node limit error is never advised to retry', () => {
const [action, detail] = behaviourFor('MAX_NODE_LIMIT_EXCEEDED');
assert.equal(action, 'reshape');
assert.match(detail, /fail identically every time/);
});
test('an unknown error type falls through rather than being guessed', () => {
const [action, detail] = behaviourFor('SOMETHING_NEW_IN_2027');
assert.equal(action, 'log-verbatim');
assert.match(detail, /does not know/);
});
test('the script refuses to send a mutation', () => {
assert.deepEqual(operations('query Q { viewer { login } }'), ['query']);
assert.deepEqual(operations('{ viewer { login } }'), ['query']);
assert.deepEqual(
operations('mutation M { addStar(input: {}) { clientMutationId } }'),
['mutation'],
);
assert.ok(refusal('mutation M { addStar(input: {}) { clientMutationId } }'));
assert.ok(refusal('subscription S { thing { id } }'));
assert.equal(refusal(''), 'the document contains no operation to send.');
assert.equal(refusal('query Q { viewer { login } }'), null);
});
test('the word mutation inside a string is not a mutation', () => {
const doc = 'query Q { search(query: "mutation", type: ISSUE, first: 1) { issueCount } }';
assert.deepEqual(operations(doc), ['query']);
assert.equal(refusal(doc), null);
});
test('a commented out mutation is not sent and not feared', () => {
const doc = '# mutation M { addStar }\nquery Q { viewer { login } }';
assert.deepEqual(operations(doc), ['query']);
assert.equal(refusal(doc), null);
});
test('the run says what it will spend', () => {
assert.equal(POINTS_PER_QUERY, 1);
assert.equal(pointCost([1, 2]), 2);
assert.equal(pointCost([]), 0);
assert.equal(pointCost(null), 0);
});
FAQ
Why does GitHub not return a 4xx when the query fails?
Because one GraphQL document can succeed and fail at the same time. A query over fifty repositories can resolve forty-two of them and be forbidden on eight, and there is no status code that means that. The specification puts execution errors in the body so that a response can carry both outcomes at once, and GitHub follows it. The consequence for you is that the status line stops being the place to look: 401 and 502 still mean what they always meant, and everything else that goes wrong arrives as a 200.
Should I just throw whenever the errors array is non-empty?
For a total failure, yes. For a response that also carries usable data, no, and that is why this script separates the two states rather than treating them as one. Throwing away forty-two resolved repositories because eight were forbidden turns a partial answer into no answer, and re-running the query costs points and returns the same shape. The rule that holds in both cases is narrower: never touch data without having read errors first, and never aggregate over a response whose errors array is non-empty without recording that you did.
Can I branch on the error message instead of the type?
You can, and it will break. Messages are written for humans and get reworded; the type field is the machine-readable classification and it is what the documented behaviours attach to. The type also carries a distinction the message tends to blur, which is whether retrying can ever help: RATE_LIMITED clears by itself, INTERNAL might, and MAX_NODE_LIMIT_EXCEEDED never will, because the query is the problem and it has not changed. A retry loop keyed on message text will happily spin on that last one until the point budget is gone.
How much does this check cost?
One point per probe, two by default, out of a budget of 5,000 points an hour for a user token. The script computes and prints that number before it sends anything, so pointing it at a list of repositories is a decision rather than an accident. If you want to know what is left before you start, GET /rate_limit reports the GraphQL bucket and does not itself consume any of it, which is a separate bucket from the REST one and has its own note.
Does the script ever write anything?
No. The GraphQL endpoint is reached by POST for queries as well as mutations, because the document travels in the request body rather than in the path, so the verb here is transport rather than intent. The script makes that explicit rather than relying on the reader to know it: it parses the document, lists its top-level operations, and refuses to open a socket at all if any of them is a mutation or a subscription. The word appearing inside a string literal or a comment is not treated as one.
Related field notes
- GraphQL fields come back null one at a time
- GraphQL points run out in their own bucket
- A 404 that is really a permission problem
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using the GraphQL API — GitHub Docs
- GraphQL specification: Errors
- GraphQL specification: Response Format
- Forming calls with GraphQL — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.