Diagnostic GitHub API
the App JWT is signed with the wrong key or algorithm
It worked until the key was rotated. Now every call is 401 {"message": "A JSON web token could not be decoded"}, or occasionally 404 {"message": "Integration not found"}, and the two arrive from what looks like the same deployment. The key is in the environment. The App exists. Somebody has already pasted the PEM into three different terminals to check it looks right.
GitHub verified nothing, because it could not. That message means the JWT did not survive verification against any key currently registered on the App named in iss — a key from a different App, a key that was revoked when a new one was generated, an HS256 default in the signing library, or, most often, a PEM whose newlines were destroyed on the way into an environment variable. Integration not found is the neighbouring case: iss does not name an App at all.
Look at the key before you look at the request. Its PEM label says whether it is even an RSA private key; its line count says whether the newlines survived; the SHA-256 of its DER body is a fingerprint you can compare between a laptop and a container without either of them printing a secret. Then one GET /app under the JWT settles the rest, because a 200 there returns the App you actually authenticated as — which is not always the App you meant.
The problem in plain words
The message is a dead end on purpose. GitHub will not say which part of the verification failed, because telling an unauthenticated caller the algorithm was wrong rather than the signature was wrong is a small oracle it has no reason to hand out. So one sentence covers a wrong key, a revoked key, a key from the staging App, an unexpected algorithm and a mangled PEM, and none of those five is visibly different from the others.
The environment variable is the usual culprit and it is invisible in exactly the way that wastes the most time. A PEM is meaningful only because of its line breaks, and every layer between a settings page and a running process is capable of turning those into the two characters backslash and n: a .env file, a CI secret editor, a Kubernetes manifest quoted one way instead of another, a shell that helpfully collapsed the value. The variable is set. Its length looks about right. It is not a key any more.
And the successful case is not proof either. Teams that run a staging App and a production App keep both keys in the same shape of variable with the same name, and a JWT signed with the staging key against the staging App succeeds. Everything works, against the wrong installation set, on the wrong account, and nothing in a 200 announces that. That is why the script asks which App answered rather than only whether one did.
Why it happens
The key has to be registered on the App named in iss. Those two facts travel separately: iss is a claim your code wrote, and the key is a file your deployment shipped. Nothing checks that they belong together until GitHub does, and when they do not match the failure is the generic decode message rather than anything that names either half.
Generating a new key does not retire the old one, but deleting it does. An App can hold several private keys at once, which is what makes rotation safe: add the new one, deploy, then remove the old. Rotations that skip the middle step produce exactly this 401 on every host that has not been redeployed yet, and a rotation done in the wrong order produces it on all of them.
Only RS256 is accepted. A JWT library whose default is HS256 will happily sign with the PEM's bytes treated as a shared secret and produce a token that is well formed and unverifiable. Pinning the algorithm explicitly costs one argument and removes a class of failure that otherwise looks identical to a wrong key.
A fingerprint is safe to print and a key is not. The SHA-256 of the DER body identifies a key file without revealing anything about it, which is what makes it the right thing to log: you can compare the fingerprint on a developer machine against the one in a container and settle whether they are the same file, in a channel where nobody would paste a key.
What a read-only script cannot see. GitHub does not publish an App's registered public keys, so nothing here can prove a key is registered except by using it. That is the blind spot, and it is the reason the live half of this check is a single GET /app: a 200 is the proof, and a 401 narrows the cause to the list above without picking one. The claim arithmetic that produces its own 401s belongs to the note on a JWT that lives too long and to the note on clock drift, and this script names those messages when it sees them rather than absorbing them.
The fix, as a flow
One sentence from GitHub covers five causes, so almost all of the work happens before the request. A PEM is a label, a run of base64 and an end line, and each of those can be wrong in a way that is visible locally and safe to print. The single GET adds the one fact no local check can reach: which App the key turned out to belong to.
How to fix it
Read the PEM's label before anything else
BEGIN RSA PRIVATE KEY is what GitHub hands you. BEGIN PRIVATE KEY is the same key in a PKCS#8 wrapper and every sensible library accepts it. BEGIN OPENSSH PRIVATE KEY means somebody ran ssh-keygen, BEGIN PUBLIC KEY means the wrong half of the pair got deployed, and BEGIN EC PRIVATE KEY means a key that cannot sign RS256 at all. Four of those never work and are visible in one line.
Count the lines
A real PEM is many short lines. One line means the newlines are gone, and the two characters backslash and n appearing anywhere in the value means they were escaped rather than embedded. Both are the same deployment fault with the same repair: base64-encode the whole PEM for transport and decode it in the process, so no layer in between ever has an opinion about newlines.
Fingerprint it and compare the fingerprint, not the key
Take the SHA-256 of the decoded DER body and print the first sixteen hex characters. That is enough to tell two keys apart and useless to anybody who intercepts it. Run the script on the machine that works and the machine that does not; if the fingerprints differ you have already found the problem and nobody had to paste a key anywhere.
Send one GET /app and read which App answered
A 200 returns the App: its numeric id, its client_id, its slug and its owner. Compare that against the App you meant, by name or by id. This is the step that catches a working credential for the wrong environment, which is the only failure in this note that produces no error at all.
Pin the algorithm and set iss to the client ID
Pass RS256 explicitly rather than accepting a library default. Use the App's client ID as iss, which is the current recommendation; the numeric App ID still works. Anything else — a slug, an installation ID, an owner name — produces Integration not found, which at least tells you the claim rather than the key is wrong.
How to check it worked
Re-run after the key is deployed with its newlines intact. The PEM state moves from escaped-newlines to pkcs1-rsa-key, and the live check names the App you expected.
python3 github_app_key_identity.py --expect acme-deploy-bot
# key: label=RSA PRIVATE KEY fingerprint=3f9a1c04e7b25d18 der=1192B lines=28
# pkcs1-rsa-key: this is the PKCS#1 RSA private key GitHub issues.
# GET /app returned 200
# key-accepted: the JWT verified against a key registered on this App.
# identity-matches: GET /app answered as acme-deploy-bot (id 123456).The full code
Two halves that never touch a secret in the output. The first is structural and offline: a PEM is a label, a run of base64 and an end line, so the label lookup, the line count, the base64 decode and the SHA-256 fingerprint are all pure functions over a string, and every one of them can fail in a way worth naming. The second is a single GET /app, whose only job is to say which App the JWT authenticated as. The key is read from the environment, never written to a file and never printed; what the report contains is a label, a line count, a byte count and sixteen hex characters.
"""Say which GitHub App a private key belongs to, without printing the key.
Read only. One request, GET /app, sent with a JWT you already hold. Nothing is
minted, rotated, registered or changed, and the script never signs anything
itself: it inspects the key file and asks GitHub who answered.
The output contains a PEM label, a line count, a byte count and a truncated
SHA-256 fingerprint. It never contains the key, any part of the key, or the
JWT. A fingerprint is the right thing to compare across machines precisely
because it can be pasted into a chat window without consequence.
The blind spot is stated rather than worked around: GitHub does not publish the
public keys registered on an App, so nothing here can prove a key is registered
except by using it. A 200 from GET /app is that proof. A 401 narrows the cause
to a short list without choosing between its entries.
"""
import argparse
import base64
import hashlib
import json
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_app_key_identity")
API = "https://api.github.com"
UA = "github-app-key-identity/1.0"
# The two characters backslash and n, which is what an environment variable
# holds when a PEM's newlines were escaped rather than embedded. This is the
# single most common way a working key stops being a key.
ESCAPED_NEWLINE = "\\n"
# A 2048-bit RSA private key is around 1200 bytes of DER. Anything much under
# this is a different kind of key or a truncated one.
MIN_RSA_DER = 500
# The labels GitHub's own key downloads carry, and the one PKCS#8 alternative
# that every library still accepts.
USABLE_LABELS = {"RSA PRIVATE KEY": "pkcs1-rsa-key", "PRIVATE KEY": "pkcs8-key"}
REPAIRS = {
"no-key-present":
"set GITHUB_APP_PRIVATE_KEY to the PEM downloaded from the App's "
"settings page. Nothing can be said about a key that is not there.",
"escaped-newlines":
"the value contains the two characters backslash and n where line "
"breaks belong, so some layer between the settings page and this "
"process escaped them. Base64-encode the whole PEM for transport and "
"decode it in the process; then no layer in between has an opinion.",
"single-line-pem":
"the PEM has lost its line breaks entirely. Same repair: carry it "
"base64-encoded rather than raw.",
"not-a-pem":
"there is no BEGIN line, so this is not a PEM at all. Check what the "
"secret store actually returned.",
"truncated-pem":
"there is a BEGIN line and no matching END line, so the value was cut "
"short. Secret stores with a length limit do this quietly.",
"encrypted-key":
"this key is passphrase-protected. GitHub does not issue encrypted "
"keys, so this one was re-encrypted locally; decrypt it or download a "
"fresh key from the App.",
"openssh-format":
"this is an OpenSSH key, which is what ssh-keygen produces. It is not "
"the key GitHub issued for the App. Download the App's private key "
"from its settings page.",
"public-key-not-private":
"this is the public half of a pair. The public key cannot sign, so no "
"JWT made with it will ever verify.",
"certificate-not-key":
"this is a certificate rather than a key. Something is reading the "
"wrong entry out of the secret store.",
"not-an-rsa-key":
"GitHub App JWTs must be signed RS256, which needs an RSA key. This "
"key uses a different algorithm family and cannot sign one.",
"unknown-pem-label":
"the PEM label is not one this check recognises, which usually means "
"the wrong file entirely.",
"body-not-base64":
"the body between the BEGIN and END lines is not valid base64, so the "
"PEM was corrupted in transit or edited by hand.",
"too-small-for-rsa":
"the decoded body is too small to be an RSA private key of any usable "
"size, so this is either truncated or a different kind of key.",
"pkcs1-rsa-key":
"this is the PKCS#1 RSA private key GitHub issues.",
"pkcs8-key":
"this is a PKCS#8 wrapper, which every sensible JWT library accepts.",
}
def unwrap(text):
"""Undo base64 transport if the value is a wrapped PEM. Pure.
Carrying a PEM base64-encoded is the recommended way to survive an
environment variable, so a value with no BEGIN line that decodes to one is
not an error. Returns the PEM and whether it was wrapped.
"""
raw = str(text or "").strip()
if not raw or "BEGIN" in raw:
return raw, False
try:
decoded = base64.b64decode(raw, validate=True).decode("utf-8")
except Exception:
return raw, False
if "BEGIN" in decoded:
return decoded, True
return raw, False
def inspect_pem(text):
"""Reduce a PEM to a label, a shape and a fingerprint. Pure.
Never returns any part of the key. The fingerprint is the SHA-256 of the
decoded DER body, truncated, which identifies the file without revealing
it and is therefore safe to compare in a chat window.
"""
raw = str(text or "")
out = {"state": None, "label": None, "fingerprint": None,
"der_bytes": None, "lines": 0}
if not raw.strip():
out["state"] = "no-key-present"
return out
out["lines"] = len(raw.strip().splitlines())
if ESCAPED_NEWLINE in raw:
out["state"] = "escaped-newlines"
return out
found = re.search(r"-----BEGIN ([A-Z0-9 ]+)-----", raw)
if not found:
out["state"] = "not-a-pem"
return out
label = found.group(1).strip()
out["label"] = label
if label == "ENCRYPTED PRIVATE KEY" or "Proc-Type: 4,ENCRYPTED" in raw:
out["state"] = "encrypted-key"
return out
if label == "OPENSSH PRIVATE KEY":
out["state"] = "openssh-format"
return out
if label.endswith("PUBLIC KEY"):
out["state"] = "public-key-not-private"
return out
if label == "CERTIFICATE":
out["state"] = "certificate-not-key"
return out
if label in ("EC PRIVATE KEY", "DSA PRIVATE KEY"):
out["state"] = "not-an-rsa-key"
return out
if label not in USABLE_LABELS:
out["state"] = "unknown-pem-label"
return out
if ("-----END " + label + "-----") not in raw:
out["state"] = "truncated-pem"
return out
if out["lines"] < 3:
out["state"] = "single-line-pem"
return out
body = "".join(line.strip() for line in raw.splitlines()
if line.strip() and not line.strip().startswith("-----"))
try:
der = base64.b64decode(body, validate=True)
except Exception:
out["state"] = "body-not-base64"
return out
out["der_bytes"] = len(der)
out["fingerprint"] = hashlib.sha256(der).hexdigest()[:16]
if len(der) < MIN_RSA_DER:
out["state"] = "too-small-for-rsa"
return out
out["state"] = USABLE_LABELS[label]
return out
def usable(state):
"""Whether a key in this state could sign an RS256 JWT at all. Pure."""
return state in ("pkcs1-rsa-key", "pkcs8-key")
def repair_for(state):
"""The one sentence worth printing under a PEM state. Pure."""
return REPAIRS.get(state, "this state has no stock repair; read the label.")
def issuer_form(value):
"""Classify what was put in the iss claim. Pure.
iss must be the App's client ID or its numeric App ID. A slug, an owner
name or an installation ID all produce Integration not found, which is a
different failure from a key that does not verify.
"""
text = str(value or "").strip()
if not text:
return "no-issuer"
if text.isdigit():
return "app-id"
if text.startswith("Iv1.") or text.startswith("Iv23"):
return "client-id"
return "unusable-issuer"
def interpret(status, message):
"""Map a GET /app response to the defect it names. Pure.
GitHub deliberately will not say which part of verification failed, so the
decode message covers five causes at once and this function says so rather
than picking one. The claim messages are named only to hand them off.
"""
if status == 200:
return ("key-accepted",
"the JWT verified against a key registered on this App.")
text = str(message or "").lower()
if "could not be decoded" in text:
return ("signature-rejected",
"GitHub could not verify the JWT. That one message covers a "
"key from another App, a key deleted during rotation, an "
"algorithm other than RS256, and a PEM whose newlines were "
"destroyed. Compare the fingerprint against a machine that "
"works to split the list.")
if "integration not found" in text:
return ("issuer-does-not-resolve",
"iss does not name an App GitHub can find, so the claim is "
"wrong rather than the key. It must be the client ID or the "
"numeric App ID.")
if "issued at" in text or "'iat'" in text:
return ("clock-problem-not-key",
"GitHub is complaining about iat, which is clock drift on the "
"signing host and a different repair entirely.")
if "too far in the future" in text:
return ("lifetime-problem-not-key",
"GitHub is complaining about exp, so the requested lifetime is "
"over the ceiling and the key is fine.")
if "bad credentials" in text:
return ("not-a-jwt",
"GitHub parsed the credential and refused it outright, which "
"is what happens when an installation access token is sent to "
"a route that wants the App JWT.")
return ("unrelated",
"the response does not name a key or a claim, so this failure has "
"another cause.")
def reconcile(app, expected):
"""Say whether GET /app answered as the App you meant. Pure.
The failure this catches makes no noise at all: a staging key against a
staging App works perfectly, on the wrong account, with the wrong
installations, and returns 200 the whole time.
"""
if not isinstance(app, dict):
return ("no-app-body",
"GET /app returned nothing that could be read as an App.")
label = "%s (id %s, client_id %s)" % (app.get("slug") or app.get("name"),
app.get("id"), app.get("client_id"))
known = {str(app.get(field) or "").lower()
for field in ("id", "client_id", "slug", "name")}
known.discard("")
want = str(expected or "").strip().lower()
if not want:
return ("no-expectation-given",
"GET /app answered as %s. Pass --expect to have that checked "
"rather than reported." % label)
if want in known:
return ("identity-matches", "GET /app answered as %s." % label)
return ("authenticated-as-another-app",
"you expected %s and the key authenticated as %s. The credential "
"works; it belongs to a different App, which is how a staging key "
"reaches production without anything failing." % (expected, label))
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--expect", default=None,
help="the App you believe this key belongs to, as a slug, "
"a name, a numeric id or a client id")
ap.add_argument("--iss", default=None,
help="the value your signing code puts in the iss claim, "
"checked for shape only")
ap.add_argument("--offline", action="store_true",
help="inspect the key and skip the confirming GET /app")
args = ap.parse_args()
pem, wrapped = unwrap(os.environ.get("GITHUB_APP_PRIVATE_KEY"))
if wrapped:
log.info("the key was carried base64-encoded, which is the shape that "
"survives an environment variable")
key = inspect_pem(pem)
log.info("key: label=%s fingerprint=%s der=%sB lines=%d",
key["label"] or "none", key["fingerprint"] or "none",
key["der_bytes"] if key["der_bytes"] is not None else "?",
key["lines"])
log.info("%s: %s", key["state"], repair_for(key["state"]))
if args.iss is not None:
form = issuer_form(args.iss)
log.info("iss form: %s", form)
if form == "unusable-issuer":
log.info("repair: iss must be the App's client ID or its numeric "
"App ID. Anything else returns Integration not found.")
live_state = None
identity_state = None
if not args.offline:
jwt = os.environ.get("GITHUB_APP_JWT")
if not jwt:
log.warning("set GITHUB_APP_JWT to the JWT your signing code "
"produces, or pass --offline to inspect the key only")
else:
# The JWT is sent and nothing else. It is not decoded, stored or
# logged, in whole or in part.
r = requests.get(API + "/app", timeout=30, headers={
"Authorization": "Bearer " + jwt,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
try:
body = r.json()
except ValueError:
body = None
message = body.get("message") if isinstance(body, dict) else None
log.info("GET /app returned %d", r.status_code)
live_state, live_detail = interpret(r.status_code, message)
log.info("%s: %s", live_state, live_detail)
if r.status_code == 200:
identity_state, identity_detail = reconcile(body, args.expect)
log.info("%s: %s", identity_state, identity_detail)
print(json.dumps({"label": key["label"], "fingerprint": key["fingerprint"],
"der_bytes": key["der_bytes"], "lines": key["lines"],
"key_state": key["state"], "live_state": live_state,
"identity_state": identity_state}, indent=2))
ok = usable(key["state"]) and live_state in (None, "key-accepted") \
and identity_state != "authenticated-as-another-app"
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())
/**
* Say which GitHub App a private key belongs to, without printing the key.
*
* Read only. One request, GET /app, sent with a JWT you already hold. The
* script never signs anything itself: it inspects the key file and asks
* GitHub who answered.
*
* The output contains a PEM label, a line count, a byte count and a truncated
* SHA-256 fingerprint. It never contains the key or the JWT.
*
* The blind spot is stated rather than worked around: GitHub does not publish
* the public keys registered on an App, so nothing here can prove a key is
* registered except by using it.
*/
import { createHash } from 'node:crypto';
const API = 'https://api.github.com';
const UA = 'github-app-key-identity/1.0';
/** The two characters backslash and n, which is what an escaped PEM holds. */
export const ESCAPED_NEWLINE = '\\n';
/** A 2048-bit RSA private key is around 1200 bytes of DER. */
export const MIN_RSA_DER = 500;
/** The labels GitHub issues, and the PKCS#8 alternative libraries accept. */
export const USABLE_LABELS = {
'RSA PRIVATE KEY': 'pkcs1-rsa-key',
'PRIVATE KEY': 'pkcs8-key',
};
const REPAIRS = {
'no-key-present':
'set GITHUB_APP_PRIVATE_KEY to the PEM downloaded from the App\'s ' +
'settings page. Nothing can be said about a key that is not there.',
'escaped-newlines':
'the value contains the two characters backslash and n where line breaks ' +
'belong, so some layer between the settings page and this process escaped ' +
'them. Base64-encode the whole PEM for transport and decode it in the ' +
'process; then no layer in between has an opinion.',
'single-line-pem':
'the PEM has lost its line breaks entirely. Same repair: carry it ' +
'base64-encoded rather than raw.',
'not-a-pem':
'there is no BEGIN line, so this is not a PEM at all. Check what the ' +
'secret store actually returned.',
'truncated-pem':
'there is a BEGIN line and no matching END line, so the value was cut ' +
'short. Secret stores with a length limit do this quietly.',
'encrypted-key':
'this key is passphrase-protected. GitHub does not issue encrypted keys, ' +
'so this one was re-encrypted locally; decrypt it or download a fresh key.',
'openssh-format':
'this is an OpenSSH key, which is what ssh-keygen produces. It is not the ' +
'key GitHub issued for the App.',
'public-key-not-private':
'this is the public half of a pair. The public key cannot sign, so no JWT ' +
'made with it will ever verify.',
'certificate-not-key':
'this is a certificate rather than a key. Something is reading the wrong ' +
'entry out of the secret store.',
'not-an-rsa-key':
'GitHub App JWTs must be signed RS256, which needs an RSA key. This key ' +
'uses a different algorithm family and cannot sign one.',
'unknown-pem-label':
'the PEM label is not one this check recognises, which usually means the ' +
'wrong file entirely.',
'body-not-base64':
'the body between the BEGIN and END lines is not valid base64, so the PEM ' +
'was corrupted in transit or edited by hand.',
'too-small-for-rsa':
'the decoded body is too small to be an RSA private key of any usable ' +
'size, so this is either truncated or a different kind of key.',
'pkcs1-rsa-key': 'this is the PKCS#1 RSA private key GitHub issues.',
'pkcs8-key': 'this is a PKCS#8 wrapper, which every sensible JWT library accepts.',
};
/**
* Undo base64 transport if the value is a wrapped PEM. Pure.
* Returns [pem, wasWrapped].
*/
export function unwrap(text) {
const raw = String(text ?? '').trim();
if (!raw || raw.includes('BEGIN')) return [raw, false];
try {
const decoded = Buffer.from(raw, 'base64').toString('utf8');
if (decoded.includes('BEGIN')) return [decoded, true];
} catch {
return [raw, false];
}
return [raw, false];
}
const isBase64 = (text) => /^[A-Za-z0-9+/]*={0,2}$/.test(text) && text.length % 4 === 0;
/**
* Reduce a PEM to a label, a shape and a fingerprint. Pure.
* Never returns any part of the key.
*/
export function inspectPem(text) {
const raw = String(text ?? '');
const out = { state: null, label: null, fingerprint: null, der_bytes: null, lines: 0 };
if (!raw.trim()) {
out.state = 'no-key-present';
return out;
}
out.lines = raw.trim().split('\n').length;
if (raw.includes(ESCAPED_NEWLINE)) {
out.state = 'escaped-newlines';
return out;
}
const found = /-----BEGIN ([A-Z0-9 ]+)-----/.exec(raw);
if (!found) {
out.state = 'not-a-pem';
return out;
}
const label = found[1].trim();
out.label = label;
if (label === 'ENCRYPTED PRIVATE KEY' || raw.includes('Proc-Type: 4,ENCRYPTED')) {
out.state = 'encrypted-key';
return out;
}
if (label === 'OPENSSH PRIVATE KEY') { out.state = 'openssh-format'; return out; }
if (label.endsWith('PUBLIC KEY')) { out.state = 'public-key-not-private'; return out; }
if (label === 'CERTIFICATE') { out.state = 'certificate-not-key'; return out; }
if (label === 'EC PRIVATE KEY' || label === 'DSA PRIVATE KEY') {
out.state = 'not-an-rsa-key';
return out;
}
if (!(label in USABLE_LABELS)) { out.state = 'unknown-pem-label'; return out; }
if (!raw.includes(`-----END ${label}-----`)) { out.state = 'truncated-pem'; return out; }
if (out.lines < 3) { out.state = 'single-line-pem'; return out; }
const body = raw.split('\n').map((l) => l.trim())
.filter((l) => l && !l.startsWith('-----')).join('');
if (!isBase64(body)) { out.state = 'body-not-base64'; return out; }
const der = Buffer.from(body, 'base64');
out.der_bytes = der.length;
out.fingerprint = createHash('sha256').update(der).digest('hex').slice(0, 16);
if (der.length < MIN_RSA_DER) { out.state = 'too-small-for-rsa'; return out; }
out.state = USABLE_LABELS[label];
return out;
}
/** Whether a key in this state could sign an RS256 JWT at all. Pure. */
export function usable(state) {
return state === 'pkcs1-rsa-key' || state === 'pkcs8-key';
}
/** The one sentence worth printing under a PEM state. Pure. */
export function repairFor(state) {
return REPAIRS[state] ?? 'this state has no stock repair; read the label.';
}
/**
* Classify what was put in the iss claim. Pure.
* It must be the client ID or the numeric App ID; anything else returns
* Integration not found, which is a different failure from a bad key.
*/
export function issuerForm(value) {
const text = String(value ?? '').trim();
if (!text) return 'no-issuer';
if (/^\d+$/.test(text)) return 'app-id';
if (text.startsWith('Iv1.') || text.startsWith('Iv23')) return 'client-id';
return 'unusable-issuer';
}
/** Map a GET /app response to the defect it names. Pure. */
export function interpret(status, message) {
if (status === 200) {
return ['key-accepted', 'the JWT verified against a key registered on this App.'];
}
const text = String(message ?? '').toLowerCase();
if (text.includes('could not be decoded')) {
return ['signature-rejected',
'GitHub could not verify the JWT. That one message covers a key from ' +
'another App, a key deleted during rotation, an algorithm other than ' +
'RS256, and a PEM whose newlines were destroyed. Compare the ' +
'fingerprint against a machine that works to split the list.'];
}
if (text.includes('integration not found')) {
return ['issuer-does-not-resolve',
'iss does not name an App GitHub can find, so the claim is wrong rather ' +
'than the key. It must be the client ID or the numeric App ID.'];
}
if (text.includes('issued at') || text.includes("'iat'")) {
return ['clock-problem-not-key',
'GitHub is complaining about iat, which is clock drift on the signing ' +
'host and a different repair entirely.'];
}
if (text.includes('too far in the future')) {
return ['lifetime-problem-not-key',
'GitHub is complaining about exp, so the requested lifetime is over the ' +
'ceiling and the key is fine.'];
}
if (text.includes('bad credentials')) {
return ['not-a-jwt',
'GitHub parsed the credential and refused it outright, which is what ' +
'happens when an installation access token is sent to a route that ' +
'wants the App JWT.'];
}
return ['unrelated',
'the response does not name a key or a claim, so this failure has another cause.'];
}
/** Say whether GET /app answered as the App you meant. Pure. */
export function reconcile(app, expected) {
if (!app || typeof app !== 'object' || Array.isArray(app)) {
return ['no-app-body', 'GET /app returned nothing that could be read as an App.'];
}
const label = `${app.slug ?? app.name} (id ${app.id}, client_id ${app.client_id})`;
const known = new Set(['id', 'client_id', 'slug', 'name']
.map((field) => String(app[field] ?? '').toLowerCase())
.filter(Boolean));
const want = String(expected ?? '').trim().toLowerCase();
if (!want) {
return ['no-expectation-given',
`GET /app answered as ${label}. Pass --expect to have that checked ` +
'rather than reported.'];
}
if (known.has(want)) return ['identity-matches', `GET /app answered as ${label}.`];
return ['authenticated-as-another-app',
`you expected ${expected} and the key authenticated as ${label}. The ` +
'credential works; it belongs to a different App, which is how a staging ' +
'key reaches production without anything failing.'];
}
function flag(name) {
const at = process.argv.indexOf(name);
return (at === -1 || at === process.argv.length - 1) ? null : process.argv[at + 1];
}
async function main() {
const [pem, wrapped] = unwrap(process.env.GITHUB_APP_PRIVATE_KEY);
if (wrapped) {
console.log('the key was carried base64-encoded, which is the shape that ' +
'survives an environment variable');
}
const key = inspectPem(pem);
console.log(`key: label=${key.label ?? 'none'} ` +
`fingerprint=${key.fingerprint ?? 'none'} ` +
`der=${key.der_bytes ?? '?'}B lines=${key.lines}`);
console.log(`${key.state}: ${repairFor(key.state)}`);
const iss = flag('--iss');
if (iss !== null) {
const form = issuerForm(iss);
console.log(`iss form: ${form}`);
if (form === 'unusable-issuer') {
console.log('repair: iss must be the App\'s client ID or its numeric ' +
'App ID. Anything else returns Integration not found.');
}
}
let liveState = null;
let identityState = null;
if (!process.argv.includes('--offline')) {
const jwt = process.env.GITHUB_APP_JWT;
if (!jwt) {
console.error('set GITHUB_APP_JWT to the JWT your signing code produces, ' +
'or pass --offline to inspect the key only');
} else {
// The JWT is sent and nothing else. Never decoded, stored or logged.
const res = await fetch(`${API}/app`, {
headers: {
Authorization: `Bearer ${jwt}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
let body = null;
try { body = await res.json(); } catch { body = null; }
const message = body && typeof body === 'object' ? body.message : null;
console.log(`GET /app returned ${res.status}`);
const [state, detail] = interpret(res.status, message);
liveState = state;
console.log(`${state}: ${detail}`);
if (res.status === 200) {
const [idState, idDetail] = reconcile(body, flag('--expect'));
identityState = idState;
console.log(`${idState}: ${idDetail}`);
}
}
}
console.log(JSON.stringify({
label: key.label,
fingerprint: key.fingerprint,
der_bytes: key.der_bytes,
lines: key.lines,
key_state: key.state,
live_state: liveState,
identity_state: identityState,
}, null, 2));
const ok = usable(key.state)
&& (liveState === null || liveState === 'key-accepted')
&& identityState !== 'authenticated-as-another-app';
process.exitCode = ok ? 0 : 1;
}
// Only run when invoked directly, so importing this module from the test file
// does not execute main() and set an exit code that fails a passing suite.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The fixtures are PEMs and none of them is a key: the body is a run of filler bytes long enough to pass the size check, so every case — an OpenSSH key, a public key, a certificate, a PEM whose newlines were escaped, one that was cut short — is three lines and carries no secret. The fingerprint is asserted to be stable and to differ between two different bodies, which is the only property it needs to have. Nothing signs, nothing verifies, and no test touches the network.
import base64
from github_app_key_identity import (
ESCAPED_NEWLINE, inspect_pem, interpret, issuer_form, reconcile, repair_for,
unwrap, usable,
)
FILLER = base64.b64encode(b"x" * 1200).decode()
def pem(label, body=None):
"""An obviously fake PEM: a real label and a run of filler bytes."""
chunk = body if body is not None else FILLER
rows = [chunk[i:i + 64] for i in range(0, len(chunk), 64)]
return "-----BEGIN %s-----\n%s\n-----END %s-----\n" % (
label, "\n".join(rows), label)
def test_the_key_github_issues_is_recognised_and_fingerprinted():
out = inspect_pem(pem("RSA PRIVATE KEY"))
assert out["state"] == "pkcs1-rsa-key"
assert out["label"] == "RSA PRIVATE KEY"
assert len(out["fingerprint"]) == 16
assert out["der_bytes"] >= 500
assert usable(out["state"])
def test_a_pkcs8_wrapper_is_the_same_key_and_is_accepted():
assert inspect_pem(pem("PRIVATE KEY"))["state"] == "pkcs8-key"
def test_the_fingerprint_identifies_the_file_and_nothing_else():
one = inspect_pem(pem("RSA PRIVATE KEY"))["fingerprint"]
again = inspect_pem(pem("RSA PRIVATE KEY"))["fingerprint"]
other = inspect_pem(pem("RSA PRIVATE KEY",
base64.b64encode(b"y" * 1200).decode()))["fingerprint"]
assert one == again
assert one != other
def test_escaped_newlines_are_the_headline_deployment_fault():
flattened = pem("RSA PRIVATE KEY").replace("\n", ESCAPED_NEWLINE)
out = inspect_pem(flattened)
assert out["state"] == "escaped-newlines"
assert "backslash and n" in repair_for(out["state"])
def test_a_pem_collapsed_onto_one_line_is_told_apart_from_an_escaped_one():
collapsed = pem("RSA PRIVATE KEY").replace("\n", " ")
assert inspect_pem(collapsed)["state"] == "single-line-pem"
def test_the_wrong_kind_of_key_is_named_rather_than_guessed_at():
assert inspect_pem(pem("OPENSSH PRIVATE KEY"))["state"] == "openssh-format"
assert inspect_pem(pem("PUBLIC KEY"))["state"] == "public-key-not-private"
assert inspect_pem(pem("EC PRIVATE KEY"))["state"] == "not-an-rsa-key"
assert inspect_pem(pem("CERTIFICATE"))["state"] == "certificate-not-key"
assert inspect_pem(pem("ENCRYPTED PRIVATE KEY"))["state"] == "encrypted-key"
assert inspect_pem(pem("DH PARAMETERS"))["state"] == "unknown-pem-label"
for state in ("openssh-format", "public-key-not-private", "not-an-rsa-key"):
assert not usable(state)
def test_a_truncated_pem_is_caught_before_its_body_is_read():
cut = pem("RSA PRIVATE KEY").split("-----END")[0]
assert inspect_pem(cut)["state"] == "truncated-pem"
def test_a_body_that_is_not_base64_says_so():
assert inspect_pem(pem("RSA PRIVATE KEY", "not base64 at all"))["state"] \
in ("body-not-base64", "too-small-for-rsa")
def test_something_far_too_small_to_be_an_rsa_key_is_rejected():
small = base64.b64encode(b"z" * 64).decode()
out = inspect_pem(pem("RSA PRIVATE KEY", small))
assert out["state"] == "too-small-for-rsa"
assert out["fingerprint"] is not None
def test_an_absent_key_is_a_state_and_not_a_crash():
assert inspect_pem("")["state"] == "no-key-present"
assert inspect_pem(None)["state"] == "no-key-present"
assert inspect_pem("just some text")["state"] == "not-a-pem"
def test_a_base64_wrapped_pem_is_unwrapped_rather_than_rejected():
raw = pem("RSA PRIVATE KEY")
wrapped = base64.b64encode(raw.encode()).decode()
text, was_wrapped = unwrap(wrapped)
assert was_wrapped is True
assert inspect_pem(text)["state"] == "pkcs1-rsa-key"
assert unwrap(raw) == (raw.strip(), False)
def test_the_issuer_claim_is_checked_for_shape_only():
assert issuer_form("123456") == "app-id"
assert issuer_form("Iv23liABCDEfghij") == "client-id"
assert issuer_form("acme-deploy-bot") == "unusable-issuer"
assert issuer_form("") == "no-issuer"
def test_one_decode_message_covers_five_causes_and_says_so():
state, detail = interpret(401, "A JSON web token could not be decoded")
assert state == "signature-rejected"
assert "another App" in detail
assert "RS256" in detail
def test_the_neighbouring_failures_are_handed_off_rather_than_absorbed():
assert interpret(200, None)[0] == "key-accepted"
assert interpret(404, "Integration not found")[0] == "issuer-does-not-resolve"
assert interpret(401, "'Issued at' claim ('iat') is in the "
"future")[0] == "clock-problem-not-key"
assert interpret(401, "'Expiration time' claim ('exp') is too far in the "
"future")[0] == "lifetime-problem-not-key"
assert interpret(401, "Bad credentials")[0] == "not-a-jwt"
assert interpret(403, "Resource not accessible by integration")[0] == "unrelated"
def test_a_working_key_for_the_wrong_app_is_the_finding_with_no_error():
app = {"id": 654321, "client_id": "Iv23liZZZZ", "slug": "acme-staging-bot",
"name": "Acme Staging Bot"}
state, detail = reconcile(app, "acme-deploy-bot")
assert state == "authenticated-as-another-app"
assert "staging key reaches production" in detail
assert reconcile(app, "acme-staging-bot")[0] == "identity-matches"
assert reconcile(app, "654321")[0] == "identity-matches"
assert reconcile(app, "Iv23liZZZZ")[0] == "identity-matches"
assert reconcile(app, None)[0] == "no-expectation-given"
assert reconcile(None, "acme")[0] == "no-app-body"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
ESCAPED_NEWLINE, inspectPem, interpret, issuerForm, reconcile, repairFor,
unwrap, usable,
} from './github-app-key-identity.mjs';
const FILLER = Buffer.from('x'.repeat(1200)).toString('base64');
/** An obviously fake PEM: a real label and a run of filler bytes. */
function pem(label, body = FILLER) {
const rows = [];
for (let i = 0; i < body.length; i += 64) {
rows.push(body.slice(i, i + 64));
}
return `-----BEGIN ${label}-----\n${rows.join('\n')}\n-----END ${label}-----\n`;
}
test('the key github issues is recognised and fingerprinted', () => {
const out = inspectPem(pem('RSA PRIVATE KEY'));
assert.equal(out.state, 'pkcs1-rsa-key');
assert.equal(out.label, 'RSA PRIVATE KEY');
assert.equal(out.fingerprint.length, 16);
assert.ok(out.der_bytes >= 500);
assert.ok(usable(out.state));
});
test('a pkcs8 wrapper is the same key and is accepted', () => {
assert.equal(inspectPem(pem('PRIVATE KEY')).state, 'pkcs8-key');
});
test('the fingerprint identifies the file and nothing else', () => {
const one = inspectPem(pem('RSA PRIVATE KEY')).fingerprint;
const again = inspectPem(pem('RSA PRIVATE KEY')).fingerprint;
const other = inspectPem(pem('RSA PRIVATE KEY',
Buffer.from('y'.repeat(1200)).toString('base64'))).fingerprint;
assert.equal(one, again);
assert.notEqual(one, other);
});
test('escaped newlines are the headline deployment fault', () => {
const flattened = pem('RSA PRIVATE KEY').split('\n').join(ESCAPED_NEWLINE);
const out = inspectPem(flattened);
assert.equal(out.state, 'escaped-newlines');
assert.match(repairFor(out.state), /backslash and n/);
});
test('a pem collapsed onto one line is told apart from an escaped one', () => {
const collapsed = pem('RSA PRIVATE KEY').split('\n').join(' ');
assert.equal(inspectPem(collapsed).state, 'single-line-pem');
});
test('the wrong kind of key is named rather than guessed at', () => {
assert.equal(inspectPem(pem('OPENSSH PRIVATE KEY')).state, 'openssh-format');
assert.equal(inspectPem(pem('PUBLIC KEY')).state, 'public-key-not-private');
assert.equal(inspectPem(pem('EC PRIVATE KEY')).state, 'not-an-rsa-key');
assert.equal(inspectPem(pem('CERTIFICATE')).state, 'certificate-not-key');
assert.equal(inspectPem(pem('ENCRYPTED PRIVATE KEY')).state, 'encrypted-key');
assert.equal(inspectPem(pem('DH PARAMETERS')).state, 'unknown-pem-label');
for (const state of ['openssh-format', 'public-key-not-private', 'not-an-rsa-key']) {
assert.ok(!usable(state));
}
});
test('a truncated pem is caught before its body is read', () => {
const cut = pem('RSA PRIVATE KEY').split('-----END')[0];
assert.equal(inspectPem(cut).state, 'truncated-pem');
});
test('a body that is not base64 says so', () => {
assert.ok(['body-not-base64', 'too-small-for-rsa']
.includes(inspectPem(pem('RSA PRIVATE KEY', 'not base64 at all')).state));
});
test('something far too small to be an rsa key is rejected', () => {
const small = Buffer.from('z'.repeat(64)).toString('base64');
const out = inspectPem(pem('RSA PRIVATE KEY', small));
assert.equal(out.state, 'too-small-for-rsa');
assert.notEqual(out.fingerprint, null);
});
test('an absent key is a state and not a crash', () => {
assert.equal(inspectPem('').state, 'no-key-present');
assert.equal(inspectPem(null).state, 'no-key-present');
assert.equal(inspectPem('just some text').state, 'not-a-pem');
});
test('a base64 wrapped pem is unwrapped rather than rejected', () => {
const raw = pem('RSA PRIVATE KEY');
const wrapped = Buffer.from(raw).toString('base64');
const [text, wasWrapped] = unwrap(wrapped);
assert.equal(wasWrapped, true);
assert.equal(inspectPem(text).state, 'pkcs1-rsa-key');
assert.deepEqual(unwrap(raw), [raw.trim(), false]);
});
test('the issuer claim is checked for shape only', () => {
assert.equal(issuerForm('123456'), 'app-id');
assert.equal(issuerForm('Iv23liABCDEfghij'), 'client-id');
assert.equal(issuerForm('acme-deploy-bot'), 'unusable-issuer');
assert.equal(issuerForm(''), 'no-issuer');
});
test('one decode message covers five causes and says so', () => {
const [state, detail] = interpret(401, 'A JSON web token could not be decoded');
assert.equal(state, 'signature-rejected');
assert.match(detail, /another App/);
assert.match(detail, /RS256/);
});
test('the neighbouring failures are handed off rather than absorbed', () => {
assert.equal(interpret(200, null)[0], 'key-accepted');
assert.equal(interpret(404, 'Integration not found')[0], 'issuer-does-not-resolve');
assert.equal(interpret(401, "'Issued at' claim ('iat') is in the future")[0],
'clock-problem-not-key');
assert.equal(interpret(401, "'Expiration time' claim ('exp') is too far in the future")[0],
'lifetime-problem-not-key');
assert.equal(interpret(401, 'Bad credentials')[0], 'not-a-jwt');
assert.equal(interpret(403, 'Resource not accessible by integration')[0], 'unrelated');
});
test('a working key for the wrong app is the finding with no error', () => {
const app = {
id: 654321, client_id: 'Iv23liZZZZ', slug: 'acme-staging-bot',
name: 'Acme Staging Bot',
};
const [state, detail] = reconcile(app, 'acme-deploy-bot');
assert.equal(state, 'authenticated-as-another-app');
assert.match(detail, /staging key reaches production/);
assert.equal(reconcile(app, 'acme-staging-bot')[0], 'identity-matches');
assert.equal(reconcile(app, '654321')[0], 'identity-matches');
assert.equal(reconcile(app, 'Iv23liZZZZ')[0], 'identity-matches');
assert.equal(reconcile(app, null)[0], 'no-expectation-given');
assert.equal(reconcile(null, 'acme')[0], 'no-app-body');
});
FAQ
Does GitHub ever say which of the five causes it was?
No, and that is deliberate rather than an oversight. Distinguishing wrong algorithm from wrong key for an unauthenticated caller would hand out information about a credential nobody has proved they own, so the same sentence covers all of it. That is exactly why the useful work happens before the request: the PEM's label, its line count and its fingerprint are all things you can read locally, and between them they eliminate four of the five without asking GitHub anything.
Why does the PEM keep losing its newlines?
Because a PEM is one of the few secrets whose meaning depends on whitespace, and almost nothing in a deployment pipeline is built to preserve it. A .env parser, a CI secret field, a YAML block scalar quoted the wrong way and a shell that collapsed the value can each turn line breaks into the two characters backslash and n, and none of them reports having done it. Carrying the PEM base64-encoded ends the argument: every layer sees one long token with no whitespace to have an opinion about, and the process decodes it once at startup.
Is it safe to log the fingerprint?
Yes, and that is most of the point. It is a truncated SHA-256 of the key's DER body, so it cannot be reversed and it carries no information about the key beyond identity. What it gives you is the ability to answer the only question that matters during an incident, which is whether the machine that works and the machine that does not are holding the same file, without either of them printing a key into a terminal that somebody is screen-sharing.
Can an App have more than one private key at a time?
Yes, and using that is how a rotation avoids this failure entirely. Generate the new key while the old one is still registered, deploy it everywhere, confirm every host is signing with the new fingerprint, and only then delete the old key from the App. Rotations that delete first produce this 401 on every host at once, and rotations that never delete leave a key in circulation that somebody downloaded onto a laptop two years ago.
The JWT is accepted but my calls still fail. Is the key wrong?
No. If GET /app returns 200 the key is registered on the App named in iss and this note is finished. What follows is a different layer: the JWT authenticates the App itself and cannot read repository data, so it has to be exchanged for an installation access token first, and that token then has its own permissions and its own repository coverage. A 403 mentioning permissions, a 404 on a repository, and a route that refuses installation tokens outright are three separate notes, and none of them is about the key.
Related field notes
- Clock drift puts iat in GitHub's future
- A JWT that asks to live for an hour
- Endpoints that refuse an installation token
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Generating a JSON Web Token (JWT) for a GitHub App — GitHub Docs
- Managing private keys for GitHub Apps — GitHub Docs
- JSON Web Signature (JWS) — RFC 7515
- PKCS #1: RSA Cryptography Specifications Version 2.2 — RFC 8017
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.