Diagnostic GitHub API
the installation token expired an hour into the job
The migration ran for fifty-eight minutes and processed eleven thousand repositories. Then every request started returning 401 Bad credentials — not some of them, all of them, from one second to the next. Restarting the job fixes it, which is the detail that sends everybody looking for a memory leak. It is not a leak. The process outlived its own credential.
An installation access token lasts exactly one hour from the moment it was minted, and nothing warns you. Code that fetches a token during startup and holds it in a variable is fine for the length of a request-response cycle and wrong for anything longer: a queue worker, a nightly migration, a daemon, a long backfill. The cliff arrives at sixty minutes of process life and takes every in-flight call with it.
The check is arithmetic, not a lookup. Take the moment the token was minted — your own code knows it, because your own code asked for it — add 3600, and compare against now. Then compare your refresh interval against the same 3600: an interval of an hour is not a refresh, it is a coin flip. Re-mint at fifty minutes, and re-mint on any 401 as well, so a slow batch cannot walk off the end of a token it is still using.
The problem in plain words
Everything about the shape of the failure argues against the real cause. It is sudden, it is total, and it is fixed by a restart, which is the signature of a resource leak rather than an expiry. So the first hour goes on heap graphs and connection pools, and the fact that it happens at almost exactly sixty minutes gets noticed late because nobody is looking at the clock, they are looking at the memory.
The message is no help either. 401 Bad credentials is what GitHub says about a token that expired, a token that was revoked, a token that was truncated on the way into the process and a token that was never valid. It is the same sentence in all four cases, so the response cannot tell you that the credential used to work, which is the single most useful fact about this failure.
And it hides in the small. A web service that mints a token per request never sees it, so the same codebase can carry the defect for years while only the batch jobs fail. When it finally shows up it shows up in the worst place: the long-running thing, halfway through, having already done fifty minutes of work that now has to be repeated.
Why it happens
One hour, fixed, from mint rather than from first use. The lifetime is not extended by activity and there is no refresh handshake. A token minted at startup and first used forty minutes later has twenty minutes left, which is why the burn-down has to be measured from the mint and not from anything the process did afterwards.
The mint is a write, so this check cannot make one. Tokens come from a POST to the App installation endpoint, and nothing in this section writes. That is a real constraint and it shapes the script: the moment of minting has to come from your own record of it, or from the expiry GitHub states on a response to the token you already hold. The script says which of the two it used rather than pretending it minted anything.
A refresh interval is only a refresh if it is shorter than the lifetime. An hourly timer against an hourly token means the window closes at the same moment it is meant to reopen, and whichever loses the race that day decides whether the job survives. Fifty minutes leaves ten minutes of margin for a slow mint, a retry and a batch that overran.
The two clocks can disagree about which token you hold. If your recorded mint time says forty minutes remain and the expiry GitHub reports says four, you are not holding the token you think you are — something cached an older one, or two workers are sharing a variable. That disagreement is worth more than either number alone, and it is the reason the script reads both.
This is the hour-scale question, not the day-scale one. Whether a credential expires next Tuesday, and what to do about the ones that carry no expiry at all, belongs to the note on a token expiring soon. Here the expiry is certain, it is an hour away, and the finding is whether your refresh schedule beats it. Nothing needs watching; something needs a timer.
The fix, as a flow
One GET against the route only an installation token can answer, and the rest is arithmetic on timestamps. The useful half runs on a healthy process: a refresh interval compared against a fixed hour finds the defect in the afternoon rather than at three in the morning, and the mint endpoint is a write, so the numbers come from your own record or from GitHub's stated expiry and the report says which.
How to fix it
Record the moment you minted, and keep it next to the token
Whatever holds the token should hold the timestamp too. It is the only number from which the burn-down can be computed without another request, and it is free: the code that asked for the token was there when the answer came back. A token in a variable with no mint time attached is a credential nobody can reason about.
Compute how much of the hour is left, not how long the process has run
Uptime is a proxy and it lies as soon as anything re-mints. Subtract the mint time from now, take that off 3600, and you have the remaining life of the credential you are actually holding. Under five minutes is the band where a long batch will cross the line mid-flight.
Compare your refresh interval against the lifetime
No interval at all means you minted once at startup and the cliff is at sixty minutes. An interval of an hour or more means the refresh cannot win. Anything over fifty minutes has no room for a retry. This is the part of the check that finds the bug before it fires, on a healthy process, in the middle of the afternoon.
Cross-check your record against what GitHub says
An authenticated response for a credential with an expiry carries GitHub's own view of when it ends. Read it and compare. Agreement is reassurance; a large disagreement means the process is holding a different token from the one it recorded, which is a sharing or caching bug and not an expiry one.
Re-mint on a timer and on any 401
Fifty minutes on a timer covers the ordinary case. Re-minting on a 401 covers the rest: a paused container, a clock correction, a batch that took longer than anyone modelled. Octokit's App authentication strategy does both for you, and a hand-rolled client has to do both explicitly, because doing only the first still fails on the day something stalls.
How to check it worked
Re-run against the process once it refreshes on a timer. The state moves from minted-once-at-startup to refresh-healthy, and the remaining life stops falling towards zero between runs.
python3 github_installation_token_age.py --refresh-interval 3000
# GET /installation/repositories returned 200
# minted 640s ago, 2960s of the hour remain
# fresh: 2960s remain of the 3600s lifetime.
# refresh-healthy: re-minting every 3000s leaves 600s of margin.
# record-agrees: GitHub's expiry and your recorded mint time are 2s apart.
The full code
One GET, against the route only an installation access token can answer, and everything else is arithmetic over timestamps. The pure half is the interesting half: parsing the two different time formats GitHub uses, turning a mint time into a remaining life, turning a remaining life into a band, and turning a refresh interval into a verdict about a schedule that has not failed yet. The token is read from the environment and never printed. The mint endpoint is a write and this script does not call it, so where the numbers come from is stated explicitly in the output rather than assumed.
"""Say how much of its hour a GitHub App installation token has left.
Read only. One request, GET /installation/repositories, which is the route an
installation access token can answer and almost nothing else can. Nothing is
minted, refreshed or changed.
Minting is a write - it is a POST to the App installation endpoint - so this
script never does it. The mint moment comes from your own record of it, or the
expiry comes from the header GitHub attaches to a response for a credential
that has one. The report says which source it used rather than implying it
learned the number from a fresh mint.
The token is read from the environment and never printed, in whole or in part.
What the report contains is seconds.
"""
import argparse
import json
import logging
import os
import re
import sys
import time
from datetime import datetime, timezone
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_installation_token_age")
API = "https://api.github.com"
UA = "github-installation-token-age/1.0"
# Fixed, from the moment of minting, and not extended by use.
LIFETIME = 3600
# Re-mint with this much of the hour still unspent, so a slow mint, a retry and
# an overrunning batch all fit inside the margin.
SAFE_MARGIN = 600
RECOMMENDED_INTERVAL = LIFETIME - SAFE_MARGIN
# Under this, a long batch will cross the line while it is still working.
DANGER_BAND = 300
# Two records of the same token should agree to about this. More than this and
# they are records of different tokens.
RECONCILE_TOLERANCE = 60
def parse_moment(value):
"""Parse an epoch or an ISO-8601 timestamp into epoch seconds. Pure.
Accepts what a program is likely to have written down: the integer it got
from time(), or the string GitHub uses in expires_at. Returns None rather
than raising, because an unparseable record is a finding of its own.
"""
text = str(value or "").strip()
if not text:
return None
if re.fullmatch(r"\d{9,11}", text):
return float(text)
try:
moment = datetime.fromisoformat(text.replace("Z", "+00:00"))
except ValueError:
return None
if moment.tzinfo is None:
moment = moment.replace(tzinfo=timezone.utc)
return moment.timestamp()
def parse_expiry_header(value):
"""Parse the expiry GitHub puts on a response. Pure.
It is not ISO-8601: the format is a space-separated date and time followed
by a zone name, so it needs its own small normalisation before the general
parser can take it.
"""
text = str(value or "").strip()
if not text:
return None
if text.endswith(" UTC"):
text = text[:-4].strip().replace(" ", "T") + "+00:00"
return parse_moment(text)
def remaining(minted_at, expires_at, now):
"""Seconds of life left in the token, and where that number came from. Pure.
GitHub's own expiry wins when both are available, because your record is a
record of a mint and GitHub's is a statement about the credential in your
hand. They should agree; when they do not, that is the finding.
"""
if expires_at is not None:
return int(expires_at - now), "github"
if minted_at is not None:
return int(minted_at + LIFETIME - now), "record"
return None, "nothing"
def classify(left):
"""Turn a remaining life into a band. Pure."""
if left is None:
return ("no-record",
"there is no mint time recorded and no expiry on the response, "
"so nothing can be said about how much of the hour is left. "
"Record the moment you mint, next to the token.")
if left <= 0:
return ("expired",
"this token ran out %ds ago. Every call made with it returns "
"401 Bad credentials, all at once, which is why a restart "
"appears to fix it." % -left)
if left < DANGER_BAND:
return ("inside-the-danger-band",
"%ds remain of the %ds lifetime. A batch that runs longer than "
"that will cross the line while it is still working."
% (left, LIFETIME))
if left < SAFE_MARGIN:
return ("past-the-safe-margin",
"%ds remain, which is inside the %ds margin a refresh needs to "
"cover a slow mint and a retry." % (left, SAFE_MARGIN))
return ("fresh", "%ds remain of the %ds lifetime." % (left, LIFETIME))
def refresh_verdict(interval):
"""Judge a refresh schedule against the fixed lifetime. Pure.
This is the half of the check that finds the bug on a healthy process,
before it has ever fired.
"""
if not interval or interval <= 0:
return ("minted-once-at-startup",
"no refresh interval, so this process mints once and holds. "
"The first 401 arrives %d minutes after start, on everything "
"at once." % (LIFETIME // 60))
if interval >= LIFETIME:
return ("refresh-slower-than-lifetime",
"re-minting every %ds against a %ds lifetime is not a refresh, "
"it is a race. Some days the token is replaced first and some "
"days it is not." % (interval, LIFETIME))
if interval > LIFETIME - SAFE_MARGIN:
return ("refresh-without-margin",
"re-minting every %ds leaves only %ds of margin, which one "
"slow mint or one retry uses up." % (interval, LIFETIME - interval))
return ("refresh-healthy",
"re-minting every %ds leaves %ds of margin."
% (interval, LIFETIME - interval))
def cliff_at(minted_at):
"""The epoch second at which 401s begin, or None. Pure."""
if minted_at is None:
return None
return int(minted_at) + LIFETIME
def reconcile(header_expiry, record_expiry):
"""Compare GitHub's expiry against your own record of the mint. Pure.
A disagreement is not a rounding problem. It means the process is holding a
different token from the one it wrote down, which is a caching or sharing
bug wearing an expiry costume.
"""
if header_expiry is None:
return ("no-header",
"the response carried no expiry, so GitHub's view is "
"unavailable and only your record is in play.")
if record_expiry is None:
return ("header-only",
"there is no recorded mint time to check GitHub's expiry "
"against. Record one; it costs nothing and it is the only way "
"to notice a stale token.")
gap = int(abs(header_expiry - record_expiry))
if gap <= RECONCILE_TOLERANCE:
return ("record-agrees",
"GitHub's expiry and your recorded mint time are %ds apart." % gap)
return ("record-disagrees",
"GitHub's expiry and your recorded mint time are %ds apart, so "
"this process is not holding the token it recorded. Look for a "
"cached token or two workers sharing one variable." % gap)
def interpret(status, message, left):
"""Map the live response to a cause, using the remaining life. Pure.
401 Bad credentials is the same sentence for an expired token, a revoked
one and a truncated one, so the burn-down is what separates them. With no
record to lean on, the honest answer is that they cannot be separated.
"""
if status == 200:
return ("token-live",
"the token answered the installation route, so it is valid "
"right now.")
text = str(message or "").lower()
if status == 401 and "bad credentials" in text:
if left is not None and left <= 0:
return ("expired-as-predicted",
"the token is past its hour and GitHub refused it, which "
"is exactly the arithmetic above.")
if left is not None and left > DANGER_BAND:
return ("not-an-expiry-problem",
"%ds of the lifetime remain and GitHub still refused the "
"token, so it was revoked, truncated or never valid. That "
"is a different investigation." % left)
return ("expired-or-revoked-cannot-tell",
"GitHub refused the token and there is no reliable record of "
"when it was minted, so expiry and revocation look identical "
"from here.")
if status == 403 and "not accessible by integration" in text:
return ("wrong-credential-class",
"this route accepted the credential and refused the action, "
"which means what is being held is not an installation access "
"token at all.")
if status == 404:
return ("route-not-answered",
"a 404 on the installation route usually means the credential "
"is not an installation access token.")
return ("unrelated",
"the response does not look like an expiry, so this failure has "
"another cause.")
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--refresh-interval", type=int, default=0,
help="the seconds between re-mints in your code; 0 means "
"the token is minted once at startup")
ap.add_argument("--minted-at", default=None,
help="when this token was minted, as an epoch second or an "
"ISO-8601 timestamp. Defaults to GITHUB_TOKEN_MINTED_AT")
args = ap.parse_args()
token = os.environ.get("GITHUB_INSTALLATION_TOKEN")
if not token:
log.error("set GITHUB_INSTALLATION_TOKEN to the installation access "
"token the process is holding")
return 2
minted_at = parse_moment(args.minted_at
or os.environ.get("GITHUB_TOKEN_MINTED_AT"))
now = time.time()
r = requests.get(API + "/installation/repositories", timeout=30,
params={"per_page": 1}, headers={
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
try:
body = r.json()
except ValueError:
body = None
message = body.get("message") if isinstance(body, dict) else None
header_expiry = parse_expiry_header(
r.headers.get("github-authentication-token-expiration"))
log.info("GET /installation/repositories returned %d", r.status_code)
left, source = remaining(minted_at, header_expiry, now)
if minted_at is not None:
log.info("minted %ds ago", int(now - minted_at))
if left is not None:
log.info("%ds left, according to the %s", left, source)
state, detail = classify(left)
log.info("%s: %s", state, detail)
plan_state, plan_detail = refresh_verdict(args.refresh_interval)
log.info("%s: %s", plan_state, plan_detail)
record_expiry = None if minted_at is None else minted_at + LIFETIME
match_state, match_detail = reconcile(header_expiry, record_expiry)
log.info("%s: %s", match_state, match_detail)
live_state, live_detail = interpret(r.status_code, message, left)
log.info("%s: %s", live_state, live_detail)
if plan_state != "refresh-healthy" or state in ("expired",
"inside-the-danger-band"):
log.info("repair: re-mint every %ds, and re-mint again on any 401. A "
"timer alone still fails on the day something stalls.",
RECOMMENDED_INTERVAL)
print(json.dumps({"seconds_left": left, "source": source,
"cliff_at": cliff_at(minted_at),
"refresh_interval": args.refresh_interval,
"state": state, "refresh_state": plan_state,
"reconcile_state": match_state,
"live_state": live_state}, indent=2))
return 0 if state == "fresh" and plan_state == "refresh-healthy" else 1
if __name__ == "__main__":
sys.exit(main())
/**
* Say how much of its hour a GitHub App installation token has left.
*
* Read only. One request, GET /installation/repositories, which is the route
* an installation access token can answer and almost nothing else can.
*
* Minting is a write, so this script never does it. The mint moment comes from
* your own record of it, or the expiry comes from the header GitHub attaches
* to a response for a credential that has one. The report says which source it
* used.
*
* The token is read from the environment and never printed.
*/
const API = 'https://api.github.com';
const UA = 'github-installation-token-age/1.0';
/** Fixed, from the moment of minting, and not extended by use. */
export const LIFETIME = 3600;
/** Re-mint with this much of the hour still unspent. */
export const SAFE_MARGIN = 600;
export const RECOMMENDED_INTERVAL = LIFETIME - SAFE_MARGIN;
/** Under this, a long batch will cross the line while it is still working. */
export const DANGER_BAND = 300;
/** Two records of the same token should agree to about this. */
export const RECONCILE_TOLERANCE = 60;
/** Parse an epoch or an ISO-8601 timestamp into epoch seconds. Pure. */
export function parseMoment(value) {
const text = String(value ?? '').trim();
if (!text) return null;
if (/^\d{9,11}$/.test(text)) return Number(text);
const ms = Date.parse(text);
return Number.isNaN(ms) ? null : ms / 1000;
}
/**
* Parse the expiry GitHub puts on a response. Pure.
* It is not ISO-8601: a space-separated date and time followed by a zone name.
*/
export function parseExpiryHeader(value) {
let text = String(value ?? '').trim();
if (!text) return null;
if (text.endsWith(' UTC')) {
text = `${text.slice(0, -4).trim().replace(' ', 'T')}+00:00`;
}
return parseMoment(text);
}
/**
* Seconds of life left, and where the number came from. Pure.
* GitHub's expiry wins when both exist: yours is a record of a mint, GitHub's
* is a statement about the credential in your hand.
*/
export function remaining(mintedAt, expiresAt, now) {
if (expiresAt !== null && expiresAt !== undefined) {
return [Math.trunc(expiresAt - now), 'github'];
}
if (mintedAt !== null && mintedAt !== undefined) {
return [Math.trunc(mintedAt + LIFETIME - now), 'record'];
}
return [null, 'nothing'];
}
/** Turn a remaining life into a band. Pure. */
export function classify(left) {
if (left === null || left === undefined) {
return ['no-record',
'there is no mint time recorded and no expiry on the response, so ' +
'nothing can be said about how much of the hour is left. Record the ' +
'moment you mint, next to the token.'];
}
if (left <= 0) {
return ['expired',
`this token ran out ${-left}s ago. Every call made with it returns 401 ` +
'Bad credentials, all at once, which is why a restart appears to fix it.'];
}
if (left < DANGER_BAND) {
return ['inside-the-danger-band',
`${left}s remain of the ${LIFETIME}s lifetime. A batch that runs longer ` +
'than that will cross the line while it is still working.'];
}
if (left < SAFE_MARGIN) {
return ['past-the-safe-margin',
`${left}s remain, which is inside the ${SAFE_MARGIN}s margin a refresh ` +
'needs to cover a slow mint and a retry.'];
}
return ['fresh', `${left}s remain of the ${LIFETIME}s lifetime.`];
}
/** Judge a refresh schedule against the fixed lifetime. Pure. */
export function refreshVerdict(interval) {
if (!interval || interval <= 0) {
return ['minted-once-at-startup',
'no refresh interval, so this process mints once and holds. The first ' +
`401 arrives ${LIFETIME / 60} minutes after start, on everything at once.`];
}
if (interval >= LIFETIME) {
return ['refresh-slower-than-lifetime',
`re-minting every ${interval}s against a ${LIFETIME}s lifetime is not a ` +
'refresh, it is a race. Some days the token is replaced first and some ' +
'days it is not.'];
}
if (interval > LIFETIME - SAFE_MARGIN) {
return ['refresh-without-margin',
`re-minting every ${interval}s leaves only ${LIFETIME - interval}s of ` +
'margin, which one slow mint or one retry uses up.'];
}
return ['refresh-healthy',
`re-minting every ${interval}s leaves ${LIFETIME - interval}s of margin.`];
}
/** The epoch second at which 401s begin, or null. Pure. */
export function cliffAt(mintedAt) {
if (mintedAt === null || mintedAt === undefined) return null;
return Math.trunc(mintedAt) + LIFETIME;
}
/** Compare GitHub's expiry against your own record of the mint. Pure. */
export function reconcile(headerExpiry, recordExpiry) {
if (headerExpiry === null || headerExpiry === undefined) {
return ['no-header',
'the response carried no expiry, so GitHub\'s view is unavailable and ' +
'only your record is in play.'];
}
if (recordExpiry === null || recordExpiry === undefined) {
return ['header-only',
'there is no recorded mint time to check GitHub\'s expiry against. ' +
'Record one; it costs nothing and it is the only way to notice a stale ' +
'token.'];
}
const gap = Math.trunc(Math.abs(headerExpiry - recordExpiry));
if (gap <= RECONCILE_TOLERANCE) {
return ['record-agrees',
`GitHub's expiry and your recorded mint time are ${gap}s apart.`];
}
return ['record-disagrees',
`GitHub's expiry and your recorded mint time are ${gap}s apart, so this ` +
'process is not holding the token it recorded. Look for a cached token or ' +
'two workers sharing one variable.'];
}
/** Map the live response to a cause, using the remaining life. Pure. */
export function interpret(status, message, left) {
if (status === 200) {
return ['token-live',
'the token answered the installation route, so it is valid right now.'];
}
const text = String(message ?? '').toLowerCase();
if (status === 401 && text.includes('bad credentials')) {
if (left !== null && left !== undefined && left <= 0) {
return ['expired-as-predicted',
'the token is past its hour and GitHub refused it, which is exactly ' +
'the arithmetic above.'];
}
if (left !== null && left !== undefined && left > DANGER_BAND) {
return ['not-an-expiry-problem',
`${left}s of the lifetime remain and GitHub still refused the token, ` +
'so it was revoked, truncated or never valid. That is a different ' +
'investigation.'];
}
return ['expired-or-revoked-cannot-tell',
'GitHub refused the token and there is no reliable record of when it ' +
'was minted, so expiry and revocation look identical from here.'];
}
if (status === 403 && text.includes('not accessible by integration')) {
return ['wrong-credential-class',
'this route accepted the credential and refused the action, which means ' +
'what is being held is not an installation access token at all.'];
}
if (status === 404) {
return ['route-not-answered',
'a 404 on the installation route usually means the credential is not an ' +
'installation access token.'];
}
return ['unrelated',
'the response does not look like an expiry, so this failure has another cause.'];
}
function flag(name, fallback) {
const at = process.argv.indexOf(name);
if (at === -1 || at === process.argv.length - 1) return fallback;
return process.argv[at + 1];
}
async function main() {
const token = process.env.GITHUB_INSTALLATION_TOKEN;
if (!token) {
console.error('set GITHUB_INSTALLATION_TOKEN to the installation access ' +
'token the process is holding');
process.exitCode = 2;
return;
}
const interval = Number(flag('--refresh-interval', 0)) || 0;
const mintedAt = parseMoment(flag('--minted-at', null)
?? process.env.GITHUB_TOKEN_MINTED_AT);
const now = Date.now() / 1000;
const res = await fetch(`${API}/installation/repositories?per_page=1`, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
let body = null;
try { body = await res.json(); } catch { body = null; }
const message = body && typeof body === 'object' ? body.message : null;
const headerExpiry = parseExpiryHeader(
res.headers.get('github-authentication-token-expiration'));
console.log(`GET /installation/repositories returned ${res.status}`);
const [left, source] = remaining(mintedAt, headerExpiry, now);
if (mintedAt !== null) console.log(`minted ${Math.trunc(now - mintedAt)}s ago`);
if (left !== null) console.log(`${left}s left, according to the ${source}`);
const [state, detail] = classify(left);
console.log(`${state}: ${detail}`);
const [planState, planDetail] = refreshVerdict(interval);
console.log(`${planState}: ${planDetail}`);
const recordExpiry = mintedAt === null ? null : mintedAt + LIFETIME;
const [matchState, matchDetail] = reconcile(headerExpiry, recordExpiry);
console.log(`${matchState}: ${matchDetail}`);
const [liveState, liveDetail] = interpret(res.status, message, left);
console.log(`${liveState}: ${liveDetail}`);
if (planState !== 'refresh-healthy' || state === 'expired'
|| state === 'inside-the-danger-band') {
console.log(`repair: re-mint every ${RECOMMENDED_INTERVAL}s, and re-mint ` +
'again on any 401. A timer alone still fails on the day something stalls.');
}
console.log(JSON.stringify({
seconds_left: left,
source,
cliff_at: cliffAt(mintedAt),
refresh_interval: interval,
state,
refresh_state: planState,
reconcile_state: matchState,
live_state: liveState,
}, null, 2));
process.exitCode = (state === 'fresh' && planState === 'refresh-healthy') ? 0 : 1;
}
// Only run when invoked directly, so importing this module from the test file
// does not execute main() and set an exit code that fails a passing suite.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Every case is a pair of timestamps and an integer, so the whole hour can be replayed in a line: a token minted an hour and a minute ago, a refresh timer set to exactly the lifetime, a recorded mint time that disagrees with GitHub's expiry by twenty minutes. The one fake token in here is the string tok, which is short enough and silly enough that nobody will mistake it for a credential. The rule worth pinning is the last one: a 401 with plenty of life left is explicitly not this note's problem, and the script says so rather than claiming the win.
from github_installation_token_age import (
DANGER_BAND, LIFETIME, cliff_at, classify, interpret, parse_expiry_header,
parse_moment, reconcile, refresh_verdict, remaining,
)
NOW = 1_772_000_000.0
# Obviously not a credential.
FAKE = "tok"
def test_a_recorded_mint_time_parses_in_either_shape():
assert parse_moment("1772000000") == NOW
assert parse_moment("2026-02-25T08:33:20Z") == parse_moment("1772008400")
assert parse_moment("") is None
assert parse_moment(None) is None
assert parse_moment("some time last tuesday") is None
def test_the_expiry_github_states_is_not_iso_8601():
assert parse_expiry_header("2026-02-25 08:33:20 UTC") == parse_moment("1772008400")
assert parse_expiry_header("") is None
assert parse_expiry_header(None) is None
def test_the_remaining_life_names_the_source_it_came_from():
left, source = remaining(NOW - 600, None, NOW)
assert (left, source) == (3000, "record")
left, source = remaining(NOW - 600, NOW + 120, NOW)
assert (left, source) == (120, "github")
assert remaining(None, None, NOW) == (None, "nothing")
def test_an_hour_old_token_is_the_headline_finding():
state, detail = classify(-60)
assert state == "expired"
assert "60s ago" in detail
assert "all at once" in detail
def test_the_bands_below_an_hour_are_named_separately():
assert classify(3000)[0] == "fresh"
assert classify(599)[0] == "past-the-safe-margin"
assert classify(DANGER_BAND - 1)[0] == "inside-the-danger-band"
assert classify(0)[0] == "expired"
def test_nothing_recorded_is_a_state_and_not_a_guess():
state, detail = classify(None)
assert state == "no-record"
assert "Record the moment you mint" in detail
def test_minting_once_at_startup_is_found_before_it_fires():
state, detail = refresh_verdict(0)
assert state == "minted-once-at-startup"
assert "60 minutes after start" in detail
assert refresh_verdict(None)[0] == "minted-once-at-startup"
def test_an_hourly_timer_against_an_hourly_token_is_a_race():
state, detail = refresh_verdict(LIFETIME)
assert state == "refresh-slower-than-lifetime"
assert "it is a race" in detail
assert refresh_verdict(7200)[0] == "refresh-slower-than-lifetime"
def test_a_refresh_with_no_room_for_a_retry_is_still_flagged():
assert refresh_verdict(3400)[0] == "refresh-without-margin"
def test_fifty_minutes_is_the_schedule_that_passes():
state, detail = refresh_verdict(3000)
assert state == "refresh-healthy"
assert "600s of margin" in detail
def test_the_cliff_is_an_hour_after_the_mint():
assert cliff_at(NOW) == int(NOW) + LIFETIME
assert cliff_at(None) is None
def test_two_records_of_different_tokens_are_caught():
state, detail = reconcile(NOW + 240, NOW + 1440)
assert state == "record-disagrees"
assert "1200s apart" in detail
assert reconcile(NOW + 240, NOW + 250)[0] == "record-agrees"
assert reconcile(None, NOW + 240)[0] == "no-header"
assert reconcile(NOW + 240, None)[0] == "header-only"
def test_a_401_at_the_end_of_the_hour_is_the_expiry():
state, detail = interpret(401, "Bad credentials", -30)
assert state == "expired-as-predicted"
assert "the arithmetic above" in detail
def test_a_401_with_most_of_the_hour_left_is_explicitly_not_this_problem():
state, detail = interpret(401, "Bad credentials", 2400)
assert state == "not-an-expiry-problem"
assert "revoked, truncated or never valid" in detail
def test_a_401_with_no_record_refuses_to_choose():
assert interpret(401, "Bad credentials", None)[0] == "expired-or-revoked-cannot-tell"
def test_the_other_responses_point_somewhere_else():
assert interpret(200, None, 3000)[0] == "token-live"
assert interpret(403, "Resource not accessible by integration",
3000)[0] == "wrong-credential-class"
assert interpret(404, "Not Found", 3000)[0] == "route-not-answered"
assert interpret(500, "Server Error", 3000)[0] == "unrelated"
def test_the_fixture_token_is_obviously_not_a_credential():
assert len(FAKE) < 20
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
DANGER_BAND, LIFETIME, classify, cliffAt, interpret, parseExpiryHeader,
parseMoment, reconcile, refreshVerdict, remaining,
} from './github-installation-token-age.mjs';
const NOW = 1772000000;
/** Obviously not a credential. */
const FAKE = 'tok';
test('a recorded mint time parses in either shape', () => {
assert.equal(parseMoment('1772000000'), NOW);
assert.equal(parseMoment('2026-02-25T08:33:20Z'), parseMoment('1772008400'));
assert.equal(parseMoment(''), null);
assert.equal(parseMoment(null), null);
assert.equal(parseMoment('some time last tuesday'), null);
});
test('the expiry github states is not iso 8601', () => {
assert.equal(parseExpiryHeader('2026-02-25 08:33:20 UTC'), parseMoment('1772008400'));
assert.equal(parseExpiryHeader(''), null);
assert.equal(parseExpiryHeader(null), null);
});
test('the remaining life names the source it came from', () => {
assert.deepEqual(remaining(NOW - 600, null, NOW), [3000, 'record']);
assert.deepEqual(remaining(NOW - 600, NOW + 120, NOW), [120, 'github']);
assert.deepEqual(remaining(null, null, NOW), [null, 'nothing']);
});
test('an hour old token is the headline finding', () => {
const [state, detail] = classify(-60);
assert.equal(state, 'expired');
assert.match(detail, /60s ago/);
assert.match(detail, /all at once/);
});
test('the bands below an hour are named separately', () => {
assert.equal(classify(3000)[0], 'fresh');
assert.equal(classify(599)[0], 'past-the-safe-margin');
assert.equal(classify(DANGER_BAND - 1)[0], 'inside-the-danger-band');
assert.equal(classify(0)[0], 'expired');
});
test('nothing recorded is a state and not a guess', () => {
const [state, detail] = classify(null);
assert.equal(state, 'no-record');
assert.match(detail, /Record the moment you mint/);
});
test('minting once at startup is found before it fires', () => {
const [state, detail] = refreshVerdict(0);
assert.equal(state, 'minted-once-at-startup');
assert.match(detail, /60 minutes after start/);
assert.equal(refreshVerdict(null)[0], 'minted-once-at-startup');
});
test('an hourly timer against an hourly token is a race', () => {
const [state, detail] = refreshVerdict(LIFETIME);
assert.equal(state, 'refresh-slower-than-lifetime');
assert.match(detail, /it is a race/);
assert.equal(refreshVerdict(7200)[0], 'refresh-slower-than-lifetime');
});
test('a refresh with no room for a retry is still flagged', () => {
assert.equal(refreshVerdict(3400)[0], 'refresh-without-margin');
});
test('fifty minutes is the schedule that passes', () => {
const [state, detail] = refreshVerdict(3000);
assert.equal(state, 'refresh-healthy');
assert.match(detail, /600s of margin/);
});
test('the cliff is an hour after the mint', () => {
assert.equal(cliffAt(NOW), NOW + LIFETIME);
assert.equal(cliffAt(null), null);
});
test('two records of different tokens are caught', () => {
const [state, detail] = reconcile(NOW + 240, NOW + 1440);
assert.equal(state, 'record-disagrees');
assert.match(detail, /1200s apart/);
assert.equal(reconcile(NOW + 240, NOW + 250)[0], 'record-agrees');
assert.equal(reconcile(null, NOW + 240)[0], 'no-header');
assert.equal(reconcile(NOW + 240, null)[0], 'header-only');
});
test('a 401 at the end of the hour is the expiry', () => {
const [state, detail] = interpret(401, 'Bad credentials', -30);
assert.equal(state, 'expired-as-predicted');
assert.match(detail, /the arithmetic above/);
});
test('a 401 with most of the hour left is explicitly not this problem', () => {
const [state, detail] = interpret(401, 'Bad credentials', 2400);
assert.equal(state, 'not-an-expiry-problem');
assert.match(detail, /revoked, truncated or never valid/);
});
test('a 401 with no record refuses to choose', () => {
assert.equal(interpret(401, 'Bad credentials', null)[0],
'expired-or-revoked-cannot-tell');
});
test('the other responses point somewhere else', () => {
assert.equal(interpret(200, null, 3000)[0], 'token-live');
assert.equal(interpret(403, 'Resource not accessible by integration', 3000)[0],
'wrong-credential-class');
assert.equal(interpret(404, 'Not Found', 3000)[0], 'route-not-answered');
assert.equal(interpret(500, 'Server Error', 3000)[0], 'unrelated');
});
test('the fixture token is obviously not a credential', () => {
assert.ok(FAKE.length < 20);
});
FAQ
Can I make the installation token last longer than an hour?
No, and there is no parameter for it. The hour is the point: an installation access token carries the App's permissions across a whole account, so a leaked one is worth having, and a short fixed lifetime bounds how long it is worth anything. What you can do is stop treating it as a startup constant. Minting is one request and it is fast, so a token per unit of work, or a token on a fifty-minute timer, costs almost nothing and removes the failure entirely.
Why does restarting fix it if the token is the problem?
Because a restart mints a new token, which is exactly the repair, applied by accident. That coincidence is what makes this bug expensive: the fix that works is the one that also resets the heap, the connection pool and every other suspect, so the evidence points at all of them equally. The tell is the timing. If the failure lands at close to sixty minutes of process life every time, and lands on every call at once rather than on a growing fraction of them, it is not a leak.
Should I re-mint on a timer or when I get a 401?
Both, because they cover different failures. A timer handles the ordinary case and keeps the credential fresh even when the process is idle. Re-minting on a 401 handles everything the timer cannot model: a container paused by the scheduler, a clock stepped by an hour, a single batch that took twice as long as any batch before it. A timer alone fails on the unusual day, and a 401 handler alone means every unusual day starts with a burst of failed requests.
How do I get expires_at without minting a token?
You do not, and that is why this script does not pretend to. The mint response is where expires_at comes from and minting is a write, which nothing in this section does. What is available read-only is the expiry GitHub attaches to a response for a credential that has one, and your own record of when you minted. The script uses whichever exists, says which one it used, and reports the disagreement when both exist and differ, because that disagreement means you are holding a token you did not record.
Does an installation token expire early when the App's permissions change?
It can stop working before its hour is up for reasons that are not expiry, which is exactly why the script separates them. Suspending the installation, uninstalling the App, or revoking the token all end it immediately, and all of them produce the same 401 Bad credentials as the hour running out. The remaining life is what tells them apart: a 401 with forty minutes still on the clock is not an expiry, and the script says so rather than recommending a shorter refresh interval that would not have helped.
Related field notes
- A token that expires in days with nobody watching
- 401 Bad credentials on every endpoint
- Endpoints that refuse an installation token
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Authenticating as a GitHub App installation — GitHub Docs
- App installations — GitHub REST API
- Best practices for creating a GitHub App — GitHub Docs
- Authenticating as a GitHub App — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.