Diagnostic GitHub API
GraphQL node ids get stored where REST ids are expected
The importer that reads through REST and the sync job that reads through GraphQL have been running side by side for a year. Neither has ever thrown. The join between their two tables returns nothing, so somebody widens it to a left join, sees a wall of nulls, and starts checking timestamps. There is nothing wrong with the timestamps. One table has 1347 in its id column and the other has MDU6SXNzdWUxMzQ3, and those are the same issue.
The two APIs give the same object two identifiers and each calls its own one id. GraphQL's id is the global node ID, an opaque string. REST's id is the numeric database ID. Neither is wrong and neither is the default in the other place: GraphQL exposes the number as databaseId and REST exposes the string as node_id, and you have to ask for them.
The crosswalk is exact in both directions. REST node_id equals GraphQL id; REST id equals GraphQL databaseId. Pick one key space per entity, request that field explicitly on whichever path does not return it by default, and migrate the rows you already have rather than joining across two spaces. And watch the third integer: an issue's number is not its databaseId, both are small integers, and they fit the same column perfectly.
The problem in plain words
Nothing fails here, which is why it survives review. Both writers are correct in isolation: the REST importer stores the id its response gave it, the GraphQL sync stores the id its response gave it, and both of them wrote id in the field mapping because both responses call it that. The column is a string in one migration and a bigint in another, or it is a string in both and nobody notices that half of it is digits.
It is usually introduced by a rewrite rather than by a mistake. A service starts on REST, somebody adds a GraphQL path for a screen that needed three round trips, and the new path writes into the same tables. From that day the table holds two key spaces for one entity type, growing in whichever proportion the two code paths run. Deduplication stops working — the same issue is inserted twice, once under each key — and the duplicate count looks like a retry bug.
The discovery is nearly always a join returning zero rows, which is a hard bug to read because zero is not a suspicious number when you are expecting a small result. The alternative discovery is a lookup that 404s: somebody takes an id out of the store and puts it in a REST URL, GitHub answers 404, and now it looks like a permissions or an existence problem rather than a type error carried a hundred lines from where it was made.
Why it happens
Two identifiers, one word. Every object in GitHub's GraphQL schema implements the Node interface, whose id is a global identifier unique across the whole API. REST predates it and identifies rows by an integer primary key, also called id. Both responses are self-consistent and neither warns you that the other exists. GraphQL adds databaseId for the integer; REST adds node_id for the global one. Those two extra fields are the entire bridge and both are easy to never notice.
Old node ids are decodable and new ones are not. The legacy format is base64 of <length>:<Type><databaseId>, so MDU6SXNzdWUxMzQ3 decodes to 05:Issue1347 and yields the number directly. GitHub has moved to a new opaque format — I_kwDO... — that carries no recoverable database ID at all. So a migration cannot be done by decoding the ids you already hold; the ones minted since the change have to be re-fetched. Any code that parses a node ID to get a number is a bug waiting for the next object to be created.
The third integer is the one that bites twice. An issue has a number, which is what appears in the URL and what REST paths address it by, and a databaseId, which is the global row key. Both are integers, both are small for a young repository, and they are not equal. GET /repos/o/r/issues/1347 means issue number 1347, not the row whose id is 1347, so storing databaseId and then using it as a path segment produces a confident 404 or, much worse, the wrong issue.
Each key space buys something. Node IDs are globally unique, are the only thing node(id:) accepts, and are what every GraphQL mutation wants as an input. Database IDs are what REST paths, webhook payloads and most exports carry. If your system is mostly REST, store databaseId and request it explicitly in every GraphQL query. If it is mostly GraphQL, store node_id from REST responses. What you cannot do is decide per code path.
databaseId is not on everything. Newer types added after the transition expose no databaseId at all, and on some types it is nullable. Where it is missing the node ID is the only key that exists, and a schema that requires an integer key has no row to write. The script says which of the two you got rather than reporting a null and letting you assume it was a permission problem, which is a different note about a different null.
The API cannot see your schema. Nothing GitHub returns knows what your store holds. What a script can do is fetch one object down both paths and prove the crosswalk on live data, then take a sample of the ids you already have and tell you which key space each belongs to. A sample that contains both is the finding, and it is a finding you can get in one query and one REST call.
The fix, as a flow
Almost all of this runs without a token. Which key space a string belongs to, what a legacy node id decodes to, how many rows a join recovers across two spaces and after normalising to one, and which stored ids can be rewritten offline are all decided locally. The live part is deliberately tiny: one REST read and one GraphQL read of the same issue, whose only job is to show on real data what the pure functions assert about fixtures. The four identifiers are printed together because seeing them lined up is what makes the crosswalk stick, and because two of them are integers that are not the same.
How to fix it
Fetch one object down both paths
One REST call and one GraphQL query, one point. The script asks REST for an issue and GraphQL for the same issue, then prints the four identifiers side by side. Seeing node_id and id match across the two responses, and id and databaseId match the other way, is what makes the crosswalk concrete rather than a rule you half remember.
Take a sample of what your store actually holds
Paste a handful of ids from the column in question. The script classifies each one as a database ID, a node ID or unrecognisable, and reports the mix. A column containing both spaces for one entity type is the bug, and the proportion tells you which code path has been winning.
Watch the join fail and then watch it work
Given two id lists the script counts how many rows join. Across two key spaces the answer is zero. Normalised to one space first, it is the number you expected. That before-and-after is worth more than any explanation, because zero-row joins are usually blamed on filters.
Find out which ids can be migrated offline
Legacy node IDs decode to a type and a database ID with no network call at all, so those rows can be rewritten in place. New-format ids cannot, and the script counts them separately so the migration plan is honest about how many objects have to be re-fetched. Do not write a decoder that guesses at the new format.
Pin one key space and request it everywhere
Add databaseId to every GraphQL selection or read node_id from every REST response, whichever direction your store points, and add the number field explicitly wherever a path is built so nobody reaches for the integer that happens to be nearby. The whole audit costs one point and one core request.
How to check it worked
Once one key space is chosen and requested everywhere, the crosswalk is a check rather than a surprise.
GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_graphql_id_crosswalk.py \
--repo acme/monorepo --issue 1347 --ids MDU6SXNzdWUxMzQ3,1347,I_kwDOAbCdEf4AbCdE
# point cost: 1 point(s) against the 5,000/hour GraphQL budget, plus 1 core request
# rest: id=1347 node_id=MDU6SXNzdWUxMzQ3 number=1347
# graphql: databaseId=1347 id=MDU6SXNzdWUxMzQ3 number=1347
# crosswalk-confirmed: REST node_id equals GraphQL id, and REST id equals
# GraphQL databaseId.
# number and databaseId are both integers and are equal by coincidence on this
# object; they are never the same field
#
# store sample: MDU6SXNzdWUxMzQ3 -> graphql-node-id, 1347 -> rest-database-id,
# I_kwDOAbCdEf4AbCdE -> graphql-node-id
# mixed-key-space: one entity type is keyed two ways in the same column: 1
# database id(s) and 2 node id(s).
# migratable offline: 1 needs re-fetching: 1 already numeric: 1
# repair: pick one key space, request databaseId in every GraphQL selection or
# node_id from every REST response, and migrate the rows you hold.The full code
Almost all of this is pure and runs without a token: which key space a string belongs to, what a legacy node ID decodes to, whether two objects crosswalk, how many rows a join across a mixed column recovers, and which ids can be migrated offline. The live part is deliberately tiny — one REST read and one GraphQL read of the same issue — because its only job is to demonstrate on real data what the pure functions assert about fixtures. The GraphQL document is parsed and refused if it is anything other than a read, as everywhere else in this section.
"""Prove the crosswalk between GraphQL node ids and REST database ids.
Read only, and queries only. GitHub's GraphQL endpoint takes its document in
the request body, so a read travels by POST there exactly as a write would;
that is a transport detail, not a licence to write. The document is parsed
before anything is sent and refused if it contains a mutation or a
subscription.
GraphQL's id is an opaque global node ID and REST's id is a numeric database
ID, and each response calls its own one "id". The mapping is exact: REST
node_id equals GraphQL id, and REST id equals GraphQL databaseId. A store that
takes whichever field arrived ends up with two key spaces for one entity and a
join that returns nothing.
What this can and cannot see: nothing GitHub returns knows what your schema
holds. This fetches one object down both paths to prove the crosswalk, then
classifies a sample of ids you supply so you can see which spaces your own
column contains.
Environment:
GITHUB_TOKEN a token with read access to the repository
"""
import argparse
import base64
import binascii
import json
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_graphql_id_crosswalk")
API = "https://api.github.com"
UA = "github-graphql-id-crosswalk/1.0"
# One repository plus one issue in a single document costs one point.
POINTS_PER_QUERY = 1
# The legacy global node ID is base64 of "<length>:<Type><databaseId>", so
# MDU6SXNzdWUxMzQ3 decodes to 05:Issue1347. The newer format is opaque and
# carries nothing recoverable, which is why decoding is never a migration plan.
LEGACY_DECODED = re.compile(r"^(\d+):([A-Za-z]+)(\d+)$")
NEW_NODE_ID = re.compile(r"^[A-Za-z]{1,4}_[A-Za-z0-9_-]{8,}$")
ALL_DIGITS = re.compile(r"^\d+$")
ISSUE_QUERY = (
"query($owner: String!, $name: String!, $number: Int!) {"
" repository(owner: $owner, name: $name) {"
" id databaseId"
" issue(number: $number) { id databaseId number } } }"
)
def strip_noise(document):
"""Remove GraphQL comments and string literals from a document. Pure."""
src = str(document or "")
out = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "#":
while i < n and src[i] != "\n":
i += 1
continue
if src.startswith('"""', i):
j = src.find('"""', i + 3)
i = n if j < 0 else j + 3
out.append(" ")
continue
if ch == '"':
i += 1
while i < n and src[i] != '"':
i += 2 if src[i] == "\\" else 1
i += 1
out.append(" ")
continue
out.append(ch)
i += 1
return "".join(out)
def operations(document):
"""The top-level operations in a document, in order. Pure."""
src = strip_noise(document)
ops, depth, word, declared = [], 0, "", None
for ch in src + " ":
if ch.isalnum() or ch == "_":
word += ch
continue
if word:
if depth == 0 and word in ("query", "mutation", "subscription", "fragment"):
declared = word
word = ""
if ch == "{":
if depth == 0:
ops.append(declared or "query")
declared = None
depth += 1
elif ch == "}":
depth = max(0, depth - 1)
return ops
def refusal(document):
"""Why this document will not be sent, or None if it is a read. Pure."""
ops = operations(document)
if not ops:
return "the document contains no operation to send."
for kind in ("mutation", "subscription"):
if kind in ops:
return ("the document contains a %s. This script sends queries only: "
"a query is a read, and the section it belongs to promises "
"its scripts never write." % kind)
return None
def decode_legacy_node_id(value):
"""(type, database_id) for a legacy node ID, or None. Pure.
The legacy format is base64 of "<length>:<Type><databaseId>" where the
length is the length of the type name. Checking that length is what stops
an ordinary base64 string being read as an identifier.
"""
text = str(value or "")
if not text or ALL_DIGITS.match(text):
return None
padded = text + "=" * (-len(text) % 4)
try:
raw = base64.b64decode(padded, validate=True).decode("utf-8")
except (binascii.Error, UnicodeDecodeError, ValueError):
return None
m = LEGACY_DECODED.match(raw)
if not m:
return None
declared_len, type_name, database_id = m.group(1), m.group(2), m.group(3)
if int(declared_len) != len(type_name):
return None
return (type_name, int(database_id))
def id_space(value):
"""Which key space a stored identifier belongs to. Pure.
Returns "rest-database-id", "graphql-node-id" or "unknown". An integer or
an all-digit string is a database ID; anything that decodes as a legacy
node ID or matches the newer opaque shape is a node ID.
"""
if isinstance(value, bool):
return "unknown"
if isinstance(value, int):
return "rest-database-id"
text = str(value or "").strip()
if not text:
return "unknown"
if ALL_DIGITS.match(text):
return "rest-database-id"
if decode_legacy_node_id(text) or NEW_NODE_ID.match(text):
return "graphql-node-id"
return "unknown"
def to_database_id(value):
"""The numeric key for this identifier without a network call, or None. Pure.
Legacy node IDs carry the number and can be migrated offline. New-format
ones carry nothing and have to be re-fetched, which is the difference that
decides how large the migration is.
"""
space = id_space(value)
if space == "rest-database-id":
return int(str(value).strip())
decoded = decode_legacy_node_id(value)
return decoded[1] if decoded else None
def crosswalk(rest_object, graphql_object):
"""Compare one object fetched both ways. Pure. Returns a dict of facts."""
rest_object = rest_object if isinstance(rest_object, dict) else {}
graphql_object = graphql_object if isinstance(graphql_object, dict) else {}
rest_id = rest_object.get("id")
rest_node_id = rest_object.get("node_id")
gql_id = graphql_object.get("id")
gql_database_id = graphql_object.get("databaseId")
return {
"rest_id": rest_id,
"rest_node_id": rest_node_id,
"rest_number": rest_object.get("number"),
"graphql_id": gql_id,
"graphql_database_id": gql_database_id,
"graphql_number": graphql_object.get("number"),
"node_ids_match": bool(rest_node_id) and rest_node_id == gql_id,
"database_ids_match": rest_id is not None and rest_id == gql_database_id,
"database_id_present": gql_database_id is not None,
}
def number_is_not_the_database_id(rest_object):
"""Whether an object's number and database id differ. Pure.
They usually do, and both are integers, so a column typed for one accepts
the other silently. Where they happen to be equal the warning still stands.
"""
rest_object = rest_object if isinstance(rest_object, dict) else {}
number, database_id = rest_object.get("number"), rest_object.get("id")
if number is None or database_id is None:
return None
return number != database_id
def classify_pair(rest_object, graphql_object):
"""Judge one crosswalk. Pure. Returns (state, detail)."""
facts = crosswalk(rest_object, graphql_object)
if facts["rest_id"] is None or facts["graphql_id"] is None:
return ("incomplete",
"one of the two responses did not carry an identifier, so "
"nothing can be compared.")
if not facts["database_id_present"]:
return ("database-id-absent",
"this type exposes no databaseId, so the node ID is the only "
"key it has. A store that requires an integer has no row to "
"write for it.")
if facts["node_ids_match"] and facts["database_ids_match"]:
return ("crosswalk-confirmed",
"REST node_id equals GraphQL id, and REST id equals GraphQL "
"databaseId.")
return ("crosswalk-broken",
"the two responses disagree, which means they are not the same "
"object. Check that the number and the query are pointing at one "
"thing before reading anything into the ids.")
def classify_store(values):
"""Judge a sample of stored identifiers. Pure. Returns (state, detail)."""
values = list(values or [])
if not values:
return ("no-sample", "no identifiers were supplied to classify.")
counts = {"rest-database-id": 0, "graphql-node-id": 0, "unknown": 0}
for v in values:
counts[id_space(v)] += 1
if counts["rest-database-id"] and counts["graphql-node-id"]:
return ("mixed-key-space",
"one entity type is keyed two ways in the same column: %d "
"database id(s) and %d node id(s)."
% (counts["rest-database-id"], counts["graphql-node-id"]))
if counts["unknown"] == len(values):
return ("unrecognised",
"none of these look like either key space. They may be your "
"own surrogate keys, which is fine and not this note.")
if counts["graphql-node-id"]:
return ("consistent-node-id",
"every identifier is a global node ID. Read node_id from REST "
"responses to keep it that way.")
return ("consistent-database-id",
"every identifier is a numeric database ID. Request databaseId "
"explicitly in every GraphQL selection to keep it that way.")
def join_rows(left, right):
"""How many identifiers appear in both lists, compared as given. Pure."""
return len({str(v) for v in (left or [])} & {str(v) for v in (right or [])})
def join_rows_normalised(left, right):
"""The same join after both sides are reduced to database ids. Pure.
New-format node IDs drop out, because nothing local can turn one into a
number. That is the honest answer and it is why the count can still be
short after normalising.
"""
def keys(values):
out = set()
for v in values or []:
k = to_database_id(v)
if k is not None:
out.add(k)
return out
return len(keys(left) & keys(right))
def migration_split(values):
"""How many stored ids can be rewritten offline, and how many cannot. Pure."""
offline, refetch, already = 0, 0, 0
for v in values or []:
space = id_space(v)
if space == "rest-database-id":
already += 1
elif space == "graphql-node-id":
if to_database_id(v) is None:
refetch += 1
else:
offline += 1
return {"already_numeric": already, "decodable_offline": offline,
"needs_refetching": refetch}
def repair(state):
"""The sentence a reader has to act on. Pure."""
if state == "mixed-key-space":
return ("pick one key space, request databaseId in every GraphQL "
"selection or node_id from every REST response, and migrate "
"the rows you hold. Do not join across the two.")
if state == "crosswalk-broken":
return ("stop and confirm both calls address the same object. An "
"issue's number is not its databaseId, and using one where "
"the other belongs is the usual cause.")
if state == "database-id-absent":
return ("key this entity by its node ID. There is no integer to store "
"and decoding the node ID will not produce one.")
if state == "consistent-node-id":
return ("nothing to migrate. Keep reading node_id on the REST side so "
"a new code path cannot introduce the other space.")
if state == "consistent-database-id":
return ("nothing to migrate. Keep asking for databaseId on the GraphQL "
"side so a new code path cannot introduce the other space.")
if state == "unrecognised":
return ("nothing here is a GitHub identifier. Point the sample at the "
"column that holds them.")
return ("fetch one object down both paths and compare the four fields "
"before changing any schema.")
def run_query(session, document, variables):
"""Send one query. Returns (status, body-or-None).
A GraphQL query is a read; POST is only how the document reaches the
endpoint, which is why the verb is written here beside the URL rather than
hidden in a constant where it could be mistaken for a write path.
"""
r = session.post(API + "/graphql",
json={"query": document, "variables": variables or {}},
timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, malformed "
"or revoked")
try:
return r.status_code, r.json()
except ValueError:
return r.status_code, None
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--issue", type=int, required=True,
help="an issue NUMBER, which is not its database id")
ap.add_argument("--ids", default="",
help="comma-separated identifiers from your own store")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
try:
owner, name = args.repo.split("/", 1)
except ValueError:
log.error("--repo takes owner/name")
return 2
why_not = refusal(ISSUE_QUERY)
if why_not:
log.error("refusing to send: %s", why_not)
return 2
log.info("point cost: %d point(s) against the 5,000/hour GraphQL budget, "
"plus 1 core request", POINTS_PER_QUERY)
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
# GitHub rejects requests with no User-Agent outright.
"User-Agent": UA,
})
r = session.get("%s/repos/%s/%s/issues/%d" % (API, owner, name, args.issue),
timeout=30)
rest_object = r.json() if r.status_code == 200 else {}
if r.status_code != 200:
log.error("REST read failed with HTTP %s; the crosswalk needs both "
"sides", r.status_code)
status, body = run_query(session, ISSUE_QUERY,
{"owner": owner, "name": name, "number": args.issue})
repository = ((body or {}).get("data") or {}).get("repository") or {}
graphql_object = repository.get("issue") or {}
if isinstance(body, dict) and body.get("errors"):
log.error("the query carried errors: %s",
json.dumps(body["errors"])[:300])
log.info("rest: id=%s node_id=%s number=%s", rest_object.get("id"),
rest_object.get("node_id"), rest_object.get("number"))
log.info("graphql: databaseId=%s id=%s number=%s",
graphql_object.get("databaseId"), graphql_object.get("id"),
graphql_object.get("number"))
state, detail = classify_pair(rest_object, graphql_object)
log.info("%s: %s", state, detail)
differs = number_is_not_the_database_id(rest_object)
if differs is not None:
log.info("number and databaseId are both integers and are %s on this "
"object; they are never the same field",
"different" if differs else "equal by coincidence")
log.info("repair: %s", repair(state))
sample = [v.strip() for v in args.ids.split(",") if v.strip()]
store_state, store_detail = classify_store(sample)
split = migration_split(sample)
if sample:
log.info("store sample: %s", ", ".join(
"%s -> %s" % (v, id_space(v)) for v in sample))
log.info("%s: %s", store_state, store_detail)
log.info("migratable offline: %d needs re-fetching: %d already "
"numeric: %d", split["decodable_offline"],
split["needs_refetching"], split["already_numeric"])
log.info("repair: %s", repair(store_state))
print(json.dumps({
"points_spent": POINTS_PER_QUERY,
"http_status": status,
"crosswalk": crosswalk(rest_object, graphql_object),
"state": state,
"detail": detail,
"store_state": store_state,
"store_detail": store_detail,
"migration": split,
}, indent=2, default=str))
return 1 if state in ("crosswalk-broken",) or store_state == "mixed-key-space" else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Prove the crosswalk between GraphQL node ids and REST database ids.
*
* Read only, and queries only. GitHub's GraphQL endpoint takes its document in
* the request body, so a read travels by POST there exactly as a write would.
* The document is parsed first and refused if it contains a mutation or a
* subscription.
*
* GraphQL's id is an opaque global node ID and REST's id is a numeric database
* ID, and each response calls its own one "id". REST node_id equals GraphQL
* id; REST id equals GraphQL databaseId. A store that takes whichever field
* arrived ends up with two key spaces for one entity.
*
* Environment:
* GITHUB_TOKEN a token with read access to the repository
* GITHUB_REPO owner/name
* GITHUB_ISSUE an issue NUMBER, which is not its database id
* GITHUB_IDS comma-separated identifiers from your own store
*/
const API = 'https://api.github.com';
const UA = 'github-graphql-id-crosswalk/1.0';
/** One repository plus one issue in a single document costs one point. */
export const POINTS_PER_QUERY = 1;
const LEGACY_DECODED = /^(\d+):([A-Za-z]+)(\d+)$/;
const NEW_NODE_ID = /^[A-Za-z]{1,4}_[A-Za-z0-9_-]{8,}$/;
const ALL_DIGITS = /^\d+$/;
const ISSUE_QUERY = 'query($owner: String!, $name: String!, $number: Int!) {'
+ ' repository(owner: $owner, name: $name) {'
+ ' id databaseId'
+ ' issue(number: $number) { id databaseId number } } }';
/** Remove GraphQL comments and string literals from a document. Pure. */
export function stripNoise(document) {
const src = String(document ?? '');
const out = [];
let i = 0;
while (i < src.length) {
const ch = src[i];
if (ch === '#') {
while (i < src.length && src[i] !== '\n') i += 1;
continue;
}
if (src.startsWith('"""', i)) {
const j = src.indexOf('"""', i + 3);
i = j < 0 ? src.length : j + 3;
out.push(' ');
continue;
}
if (ch === '"') {
i += 1;
while (i < src.length && src[i] !== '"') i += src[i] === '\\' ? 2 : 1;
i += 1;
out.push(' ');
continue;
}
out.push(ch);
i += 1;
}
return out.join('');
}
/** The top-level operations in a document, in order. Pure. */
export function operations(document) {
const src = `${stripNoise(document)} `;
const ops = [];
let depth = 0;
let word = '';
let declared = null;
for (const ch of src) {
if (/[A-Za-z0-9_]/.test(ch)) { word += ch; continue; }
if (word) {
if (depth === 0 && ['query', 'mutation', 'subscription', 'fragment'].includes(word)) {
declared = word;
}
word = '';
}
if (ch === '{') {
if (depth === 0) { ops.push(declared || 'query'); declared = null; }
depth += 1;
} else if (ch === '}') {
depth = Math.max(0, depth - 1);
}
}
return ops;
}
/** Why this document will not be sent, or null if it is a read. Pure. */
export function refusal(document) {
const ops = operations(document);
if (ops.length === 0) return 'the document contains no operation to send.';
for (const kind of ['mutation', 'subscription']) {
if (ops.includes(kind)) {
return `the document contains a ${kind}. This script sends queries only: `
+ 'a query is a read, and the section it belongs to promises its '
+ 'scripts never write.';
}
}
return null;
}
/** [type, databaseId] for a legacy node ID, or null. Pure. */
export function decodeLegacyNodeId(value) {
const text = String(value ?? '');
if (!text || ALL_DIGITS.test(text)) return null;
const padded = text + '='.repeat((4 - (text.length % 4)) % 4);
let raw;
try {
raw = Buffer.from(padded, 'base64').toString('utf8');
if (Buffer.from(raw, 'utf8').toString('base64').replace(/=+$/, '')
!== padded.replace(/=+$/, '')) return null;
} catch { return null; }
const m = LEGACY_DECODED.exec(raw);
if (!m) return null;
const [, declaredLen, typeName, databaseId] = m;
if (Number(declaredLen) !== typeName.length) return null;
return [typeName, Number(databaseId)];
}
/** Which key space a stored identifier belongs to. Pure. */
export function idSpace(value) {
if (typeof value === 'boolean') return 'unknown';
if (typeof value === 'number' && Number.isInteger(value)) return 'rest-database-id';
const text = String(value ?? '').trim();
if (!text) return 'unknown';
if (ALL_DIGITS.test(text)) return 'rest-database-id';
if (decodeLegacyNodeId(text) || NEW_NODE_ID.test(text)) return 'graphql-node-id';
return 'unknown';
}
/** The numeric key for this identifier without a network call, or null. Pure. */
export function toDatabaseId(value) {
const space = idSpace(value);
if (space === 'rest-database-id') return Number(String(value).trim());
const decoded = decodeLegacyNodeId(value);
return decoded ? decoded[1] : null;
}
/** Compare one object fetched both ways. Pure. */
export function crosswalk(restObject, graphqlObject) {
const rest = (restObject && typeof restObject === 'object') ? restObject : {};
const gql = (graphqlObject && typeof graphqlObject === 'object') ? graphqlObject : {};
const restId = rest.id ?? null;
const restNodeId = rest.node_id ?? null;
const gqlId = gql.id ?? null;
const gqlDatabaseId = gql.databaseId ?? null;
return {
rest_id: restId,
rest_node_id: restNodeId,
rest_number: rest.number ?? null,
graphql_id: gqlId,
graphql_database_id: gqlDatabaseId,
graphql_number: gql.number ?? null,
node_ids_match: Boolean(restNodeId) && restNodeId === gqlId,
database_ids_match: restId !== null && restId === gqlDatabaseId,
database_id_present: gqlDatabaseId !== null,
};
}
/** Whether an object's number and database id differ. Pure. */
export function numberIsNotTheDatabaseId(restObject) {
const rest = (restObject && typeof restObject === 'object') ? restObject : {};
const number = rest.number ?? null;
const databaseId = rest.id ?? null;
if (number === null || databaseId === null) return null;
return number !== databaseId;
}
/** Judge one crosswalk. Pure. Returns [state, detail]. */
export function classifyPair(restObject, graphqlObject) {
const facts = crosswalk(restObject, graphqlObject);
if (facts.rest_id === null || facts.graphql_id === null) {
return ['incomplete', 'one of the two responses did not carry an '
+ 'identifier, so nothing can be compared.'];
}
if (!facts.database_id_present) {
return ['database-id-absent', 'this type exposes no databaseId, so the node '
+ 'ID is the only key it has. A store that requires an integer has no row '
+ 'to write for it.'];
}
if (facts.node_ids_match && facts.database_ids_match) {
return ['crosswalk-confirmed', 'REST node_id equals GraphQL id, and REST id '
+ 'equals GraphQL databaseId.'];
}
return ['crosswalk-broken', 'the two responses disagree, which means they are '
+ 'not the same object. Check that the number and the query are pointing at '
+ 'one thing before reading anything into the ids.'];
}
/** Judge a sample of stored identifiers. Pure. Returns [state, detail]. */
export function classifyStore(values) {
const list = Array.isArray(values) ? values : [];
if (list.length === 0) return ['no-sample', 'no identifiers were supplied to classify.'];
const counts = { 'rest-database-id': 0, 'graphql-node-id': 0, unknown: 0 };
for (const v of list) counts[idSpace(v)] += 1;
if (counts['rest-database-id'] && counts['graphql-node-id']) {
return ['mixed-key-space', 'one entity type is keyed two ways in the same '
+ `column: ${counts['rest-database-id']} database id(s) and `
+ `${counts['graphql-node-id']} node id(s).`];
}
if (counts.unknown === list.length) {
return ['unrecognised', 'none of these look like either key space. They may '
+ 'be your own surrogate keys, which is fine and not this note.'];
}
if (counts['graphql-node-id']) {
return ['consistent-node-id', 'every identifier is a global node ID. Read '
+ 'node_id from REST responses to keep it that way.'];
}
return ['consistent-database-id', 'every identifier is a numeric database ID. '
+ 'Request databaseId explicitly in every GraphQL selection to keep it that way.'];
}
/** How many identifiers appear in both lists, compared as given. Pure. */
export function joinRows(left, right) {
const a = new Set((left || []).map((v) => String(v)));
const b = new Set((right || []).map((v) => String(v)));
let n = 0;
for (const v of a) if (b.has(v)) n += 1;
return n;
}
/** The same join after both sides are reduced to database ids. Pure. */
export function joinRowsNormalised(left, right) {
const keys = (values) => {
const out = new Set();
for (const v of values || []) {
const k = toDatabaseId(v);
if (k !== null) out.add(k);
}
return out;
};
const a = keys(left);
const b = keys(right);
let n = 0;
for (const v of a) if (b.has(v)) n += 1;
return n;
}
/** How many stored ids can be rewritten offline, and how many cannot. Pure. */
export function migrationSplit(values) {
let already = 0;
let offline = 0;
let refetch = 0;
for (const v of values || []) {
const space = idSpace(v);
if (space === 'rest-database-id') already += 1;
else if (space === 'graphql-node-id') {
if (toDatabaseId(v) === null) refetch += 1; else offline += 1;
}
}
return { already_numeric: already, decodable_offline: offline, needs_refetching: refetch };
}
/** The sentence a reader has to act on. Pure. */
export function repair(state) {
if (state === 'mixed-key-space') {
return 'pick one key space, request databaseId in every GraphQL selection '
+ 'or node_id from every REST response, and migrate the rows you hold. '
+ 'Do not join across the two.';
}
if (state === 'crosswalk-broken') {
return 'stop and confirm both calls address the same object. An issue\'s '
+ 'number is not its databaseId, and using one where the other belongs '
+ 'is the usual cause.';
}
if (state === 'database-id-absent') {
return 'key this entity by its node ID. There is no integer to store and '
+ 'decoding the node ID will not produce one.';
}
if (state === 'consistent-node-id') {
return 'nothing to migrate. Keep reading node_id on the REST side so a new '
+ 'code path cannot introduce the other space.';
}
if (state === 'consistent-database-id') {
return 'nothing to migrate. Keep asking for databaseId on the GraphQL side '
+ 'so a new code path cannot introduce the other space.';
}
if (state === 'unrecognised') {
return 'nothing here is a GitHub identifier. Point the sample at the column '
+ 'that holds them.';
}
return 'fetch one object down both paths and compare the four fields before '
+ 'changing any schema.';
}
function headers(token) {
return {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
'User-Agent': UA,
};
}
async function runQuery(token, document, variables) {
const res = await fetch(`${API}/graphql`, {
// A GraphQL query is a read. POST is only how the document reaches the
// endpoint, and refusal() has already rejected anything that is not a read.
method: 'POST',
headers: headers(token),
body: JSON.stringify({ query: document, variables: variables || {} }),
});
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, body };
}
async function main() {
const token = process.env.GITHUB_TOKEN;
const repo = process.env.GITHUB_REPO;
const number = Number(process.env.GITHUB_ISSUE || 0);
if (!token || !repo || !number) {
console.error('set GITHUB_TOKEN (read-only is enough), GITHUB_REPO=owner/name '
+ 'and GITHUB_ISSUE=<issue number>');
process.exitCode = 2;
return;
}
const [owner, name] = repo.split('/');
const whyNot = refusal(ISSUE_QUERY);
if (whyNot) {
console.error(`refusing to send: ${whyNot}`);
process.exitCode = 2;
return;
}
console.log(`point cost: ${POINTS_PER_QUERY} point(s) against the 5,000/hour `
+ 'GraphQL budget, plus 1 core request');
const restRes = await fetch(`${API}/repos/${owner}/${name}/issues/${number}`,
{ headers: headers(token) });
const restObject = restRes.ok ? await restRes.json() : {};
const { body } = await runQuery(token, ISSUE_QUERY, { owner, name, number });
const repository = ((body || {}).data || {}).repository || {};
const graphqlObject = repository.issue || {};
console.log(`rest: id=${restObject.id} node_id=${restObject.node_id} `
+ `number=${restObject.number}`);
console.log(`graphql: databaseId=${graphqlObject.databaseId} `
+ `id=${graphqlObject.id} number=${graphqlObject.number}`);
const [state, detail] = classifyPair(restObject, graphqlObject);
console.log(`${state}: ${detail}`);
console.log(`repair: ${repair(state)}`);
const sample = String(process.env.GITHUB_IDS || '').split(',')
.map((v) => v.trim()).filter(Boolean);
const [storeState, storeDetail] = classifyStore(sample);
const split = migrationSplit(sample);
if (sample.length) {
console.log(sample.map((v) => `${v} -> ${idSpace(v)}`).join(', '));
console.log(`${storeState}: ${storeDetail}`);
console.log(`migratable offline: ${split.decodable_offline} needs `
+ `re-fetching: ${split.needs_refetching} already numeric: `
+ `${split.already_numeric}`);
console.log(`repair: ${repair(storeState)}`);
}
console.log(JSON.stringify({
points_spent: POINTS_PER_QUERY,
crosswalk: crosswalk(restObject, graphqlObject),
state,
store_state: storeState,
migration: split,
}, null, 2));
process.exitCode = (state === 'crosswalk-broken' || storeState === 'mixed-key-space') ? 1 : 0;
}
// Guarded so importing this file from the test runner does not start a run and
// fail the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The decoder is the part worth pinning hardest, because it is the part somebody will be tempted to lean on: MDU6SXNzdWUxMzQ3 really does decode to Issue 1347, and the newer opaque format really does yield nothing, so a migration written around decoding covers only the rows minted before the change. Around it the suite asserts the crosswalk in both directions on recorded objects, the classifier that spots a column holding both key spaces, and the join that returns zero across them and the right number after normalising. The last group is the third integer: an issue's number and its databaseId, side by side, different, and both perfectly castable to the same column type.
from github_graphql_id_crosswalk import (
classify_pair, classify_store, crosswalk, decode_legacy_node_id, id_space,
join_rows, join_rows_normalised, migration_split,
number_is_not_the_database_id, operations, refusal, repair, to_database_id,
)
REST_ISSUE = {"id": 1347, "node_id": "MDU6SXNzdWUxMzQ3", "number": 1347,
"title": "Found a bug"}
GQL_ISSUE = {"id": "MDU6SXNzdWUxMzQ3", "databaseId": 1347, "number": 1347}
NEW_STYLE = "I_kwDOAbCdEf4AbCdE"
def test_a_legacy_node_id_carries_the_database_id_inside_it():
assert decode_legacy_node_id("MDU6SXNzdWUxMzQ3") == ("Issue", 1347)
assert decode_legacy_node_id("MDU6SXNzdWUx") == ("Issue", 1)
assert decode_legacy_node_id("MDEwOlJlcG9zaXRvcnkxMjk2MjY5") == ("Repository", 1296269)
def test_the_new_format_carries_nothing_and_must_be_refetched():
assert decode_legacy_node_id(NEW_STYLE) is None
assert id_space(NEW_STYLE) == "graphql-node-id"
assert to_database_id(NEW_STYLE) is None
def test_an_ordinary_string_is_not_mistaken_for_an_identifier():
assert decode_legacy_node_id("aGVsbG8gd29ybGQ=") is None
assert decode_legacy_node_id("not base64 at all") is None
assert decode_legacy_node_id("") is None
assert decode_legacy_node_id("1347") is None
def test_each_identifier_is_placed_in_exactly_one_key_space():
assert id_space(1347) == "rest-database-id"
assert id_space("1347") == "rest-database-id"
assert id_space("MDU6SXNzdWUxMzQ3") == "graphql-node-id"
assert id_space("acme/monorepo#1347") == "unknown"
assert id_space(None) == "unknown"
assert id_space(True) == "unknown"
def test_the_crosswalk_holds_in_both_directions():
facts = crosswalk(REST_ISSUE, GQL_ISSUE)
assert facts["node_ids_match"]
assert facts["database_ids_match"]
state, detail = classify_pair(REST_ISSUE, GQL_ISSUE)
assert state == "crosswalk-confirmed"
assert "REST node_id equals GraphQL id" in detail
def test_two_different_objects_are_not_reported_as_a_key_problem():
state, detail = classify_pair(REST_ISSUE, {"id": "MDU6SXNzdWUx", "databaseId": 1})
assert state == "crosswalk-broken"
assert "not the same object" in detail
assert "number is not its databaseId" in repair(state)
def test_a_type_with_no_database_id_has_only_one_key():
state, _ = classify_pair(REST_ISSUE, {"id": NEW_STYLE, "databaseId": None})
assert state == "database-id-absent"
assert "node ID" in repair(state)
assert classify_pair({}, {})[0] == "incomplete"
def test_a_column_holding_both_spaces_is_the_finding():
state, detail = classify_store(["1347", "MDU6SXNzdWUxMzQ3", NEW_STYLE])
assert state == "mixed-key-space"
assert "1 database id(s)" in detail
assert "2 node id(s)" in detail
assert "pick one key space" in repair(state)
def test_a_consistent_column_is_left_alone():
assert classify_store(["1347", "1348"])[0] == "consistent-database-id"
assert classify_store(["MDU6SXNzdWUxMzQ3", NEW_STYLE])[0] == "consistent-node-id"
assert classify_store(["acme/monorepo#1"])[0] == "unrecognised"
assert classify_store([])[0] == "no-sample"
def test_the_join_returns_nothing_across_two_key_spaces():
rest_side = ["1347", "1348"]
graphql_side = ["MDU6SXNzdWUxMzQ3", "MDU6SXNzdWUxMzQ4"]
assert join_rows(rest_side, graphql_side) == 0
assert join_rows_normalised(rest_side, graphql_side) == 2
def test_normalising_cannot_rescue_the_new_format():
assert join_rows_normalised(["1347"], ["MDU6SXNzdWUxMzQ3"]) == 1
assert join_rows_normalised(["1347"], [NEW_STYLE]) == 0
def test_the_migration_is_split_into_offline_and_refetch():
split = migration_split(["1347", "MDU6SXNzdWUxMzQ3", NEW_STYLE, "junk"])
assert split == {"already_numeric": 1, "decodable_offline": 1,
"needs_refetching": 1}
def test_the_number_is_a_third_integer_and_not_the_database_id():
other = {"id": 2136843289, "node_id": "MDU6SXNzdWUx", "number": 1347}
assert number_is_not_the_database_id(other) is True
assert number_is_not_the_database_id(REST_ISSUE) is False
assert number_is_not_the_database_id({}) is None
# Both are integers, so a column typed for one silently accepts the other.
assert id_space(other["number"]) == id_space(other["id"])
def test_the_document_this_script_sends_is_a_read():
assert operations("query Q { repository(owner: \"a\", name: \"b\") { id databaseId } }") == ["query"]
assert refusal("mutation M { addStar(input: {}) { clientMutationId } }")
assert refusal("subscription S { thing { id } }")
assert refusal("") == "the document contains no operation to send."
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
classifyPair, classifyStore, crosswalk, decodeLegacyNodeId, idSpace, joinRows,
joinRowsNormalised, migrationSplit, numberIsNotTheDatabaseId, operations,
refusal, repair, toDatabaseId,
} from './github-graphql-id-crosswalk.mjs';
const REST_ISSUE = {
id: 1347, node_id: 'MDU6SXNzdWUxMzQ3', number: 1347, title: 'Found a bug',
};
const GQL_ISSUE = { id: 'MDU6SXNzdWUxMzQ3', databaseId: 1347, number: 1347 };
const NEW_STYLE = 'I_kwDOAbCdEf4AbCdE';
test('a legacy node id carries the database id inside it', () => {
assert.deepEqual(decodeLegacyNodeId('MDU6SXNzdWUxMzQ3'), ['Issue', 1347]);
assert.deepEqual(decodeLegacyNodeId('MDU6SXNzdWUx'), ['Issue', 1]);
assert.deepEqual(decodeLegacyNodeId('MDEwOlJlcG9zaXRvcnkxMjk2MjY5'),
['Repository', 1296269]);
});
test('the new format carries nothing and must be refetched', () => {
assert.equal(decodeLegacyNodeId(NEW_STYLE), null);
assert.equal(idSpace(NEW_STYLE), 'graphql-node-id');
assert.equal(toDatabaseId(NEW_STYLE), null);
});
test('an ordinary string is not mistaken for an identifier', () => {
assert.equal(decodeLegacyNodeId('aGVsbG8gd29ybGQ='), null);
assert.equal(decodeLegacyNodeId('not base64 at all'), null);
assert.equal(decodeLegacyNodeId(''), null);
assert.equal(decodeLegacyNodeId('1347'), null);
});
test('each identifier is placed in exactly one key space', () => {
assert.equal(idSpace(1347), 'rest-database-id');
assert.equal(idSpace('1347'), 'rest-database-id');
assert.equal(idSpace('MDU6SXNzdWUxMzQ3'), 'graphql-node-id');
assert.equal(idSpace('acme/monorepo#1347'), 'unknown');
assert.equal(idSpace(null), 'unknown');
assert.equal(idSpace(true), 'unknown');
});
test('the crosswalk holds in both directions', () => {
const facts = crosswalk(REST_ISSUE, GQL_ISSUE);
assert.ok(facts.node_ids_match);
assert.ok(facts.database_ids_match);
const [state, detail] = classifyPair(REST_ISSUE, GQL_ISSUE);
assert.equal(state, 'crosswalk-confirmed');
assert.match(detail, /REST node_id equals GraphQL id/);
});
test('two different objects are not reported as a key problem', () => {
const [state, detail] = classifyPair(REST_ISSUE,
{ id: 'MDU6SXNzdWUx', databaseId: 1 });
assert.equal(state, 'crosswalk-broken');
assert.match(detail, /not the same object/);
assert.match(repair(state), /number is not its databaseId/);
});
test('a type with no database id has only one key', () => {
const [state] = classifyPair(REST_ISSUE, { id: NEW_STYLE, databaseId: null });
assert.equal(state, 'database-id-absent');
assert.match(repair(state), /node ID/);
assert.equal(classifyPair({}, {})[0], 'incomplete');
});
test('a column holding both spaces is the finding', () => {
const [state, detail] = classifyStore(['1347', 'MDU6SXNzdWUxMzQ3', NEW_STYLE]);
assert.equal(state, 'mixed-key-space');
assert.match(detail, /1 database id\(s\)/);
assert.match(detail, /2 node id\(s\)/);
assert.match(repair(state), /pick one key space/);
});
test('a consistent column is left alone', () => {
assert.equal(classifyStore(['1347', '1348'])[0], 'consistent-database-id');
assert.equal(classifyStore(['MDU6SXNzdWUxMzQ3', NEW_STYLE])[0], 'consistent-node-id');
assert.equal(classifyStore(['acme/monorepo#1'])[0], 'unrecognised');
assert.equal(classifyStore([])[0], 'no-sample');
});
test('the join returns nothing across two key spaces', () => {
const restSide = ['1347', '1348'];
const graphqlSide = ['MDU6SXNzdWUxMzQ3', 'MDU6SXNzdWUxMzQ4'];
assert.equal(joinRows(restSide, graphqlSide), 0);
assert.equal(joinRowsNormalised(restSide, graphqlSide), 2);
});
test('normalising cannot rescue the new format', () => {
assert.equal(joinRowsNormalised(['1347'], ['MDU6SXNzdWUxMzQ3']), 1);
assert.equal(joinRowsNormalised(['1347'], [NEW_STYLE]), 0);
});
test('the migration is split into offline and refetch', () => {
assert.deepEqual(migrationSplit(['1347', 'MDU6SXNzdWUxMzQ3', NEW_STYLE, 'junk']),
{ already_numeric: 1, decodable_offline: 1, needs_refetching: 1 });
});
test('the number is a third integer and not the database id', () => {
const other = { id: 2136843289, node_id: 'MDU6SXNzdWUx', number: 1347 };
assert.equal(numberIsNotTheDatabaseId(other), true);
assert.equal(numberIsNotTheDatabaseId(REST_ISSUE), false);
assert.equal(numberIsNotTheDatabaseId({}), null);
assert.equal(idSpace(other.number), idSpace(other.id));
});
test('the document this script sends is a read', () => {
assert.deepEqual(
operations('query Q { repository(owner: "a", name: "b") { id databaseId } }'),
['query'],
);
assert.ok(refusal('mutation M { addStar(input: {}) { clientMutationId } }'));
assert.ok(refusal('subscription S { thing { id } }'));
assert.equal(refusal(''), 'the document contains no operation to send.');
});
FAQ
Which one should I store?
Whichever your system is mostly built on, and then everywhere. If most of your code paths are REST, or you consume webhook payloads, or you export to something that expects integers, store the database ID and add databaseId to every GraphQL selection you write. If you are mostly on GraphQL, or you call mutations, which take node IDs as inputs, store node_id from every REST response instead. The decision that causes this bug is not choosing wrong, it is choosing per code path, so write the choice down somewhere a reviewer will see it.
Can I convert one to the other locally?
Only for the old ones. A legacy node ID is base64 of the type name and the database ID, so MDU6SXNzdWUxMzQ3 decodes to Issue 1347 with no network call and those rows can be migrated in place. GitHub's newer format is deliberately opaque and carries no recoverable number, so ids minted since that change have to be re-fetched from the API to learn their databaseId. Any decoder you write will therefore be correct on your historical rows and silently wrong on new ones, which is the worst possible failure mode; the script counts both groups so the migration plan can be sized honestly.
Why did my REST call 404 with a databaseId in the URL?
Because REST addresses issues and pull requests by number, not by database ID. GET /repos/owner/repo/issues/1347 means the issue numbered 1347 in that repository, and issue numbers restart at 1 in every repository while database IDs are global and large. Passing a databaseId there asks for an issue number that usually does not exist, which is a 404, or that does exist and is a completely different issue, which is worse and will not raise anything. Request number explicitly wherever you build a path.
Is databaseId available on everything?
No, and that is a real constraint rather than a permission problem. Types introduced after GitHub moved to opaque identifiers expose no databaseId at all, and on some types the field is nullable. Where it is absent, the node ID is the only identifier the object has, so a schema whose key column is an integer has nothing to write. The script names that case explicitly rather than reporting a null, because a null in a GraphQL response is otherwise ambiguous between an absent field and a withheld one, and the withheld kind is a different note.
Nothing is failing. Is this worth fixing?
The failure mode is silence, so “nothing is failing” is not evidence. What it looks like in practice is a join that quietly returns fewer rows than it should, duplicate rows accumulating because deduplication compares two different key spaces, and counts that drift apart between two systems that are supposedly reading the same data. Take fifty ids out of the column and classify them; if they are all in one space you are fine and it costs a minute, and if they are not you have found the reason for a discrepancy somebody is probably already investigating.
Related field notes
- A null field that was withheld rather than empty
- A 200 that carries an errors array
- Identity that moves when a repository is renamed
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Using global node IDs — GitHub Docs
- Interfaces — GitHub GraphQL API reference
- Migrating from REST to GraphQL — GitHub Docs
- Issues — GitHub REST API
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.