Diagnostic GitHub API
the webhook exists but somebody switched it off
The hook is right there in the repository's settings, pointed at the right URL, subscribed to the right events, with a secret set and a green tick beside it from the last time anyone looked. The delivery log is not full of failures. It is empty, and it has been empty for five weeks. Nothing is broken because nothing is running: active is false.
Read GET /repos/{owner}/{repo}/hooks and look at the active boolean on each hook. false is the finding, and it is the whole finding — an inactive hook is not attempted, so it generates no delivery record, no failure and no error anywhere.
The part worth a script is what comes next, because a hook reaches active: false by three routes with three different repairs. It was created that way and has never delivered anything; somebody toggled it off during an incident and never toggled it back; or GitHub switched it off after a long run of failed deliveries. last_response.code separates the third from the other two, and created_at against updated_at separates the first from the second. Re-enabling a hook that GitHub disabled, without fixing the receiver first, just gets it disabled again.
The problem in plain words
This one hides because every instinct points somewhere else. An event that never arrives sends you to your receiver: the routing, the ingress, the signature check, the queue. All of them are fine. Then it sends you to GitHub's delivery log, which is where an experienced person goes second, and the log is empty — so the conclusion is that GitHub did not send the event, and the search moves to whether the event fires at all. Nobody looks at a boolean that has been false since March, because the settings page shows the hook and a hook you can see feels like a hook that is running.
The toggling itself is usually legitimate and usually forgotten. Somebody switches a hook off at two in the morning because the receiver is melting under a migration and the retries are making it worse. That is exactly the right call. The incident ends, the receiver is fixed, the deploy goes out, and the one action that has no ticket attached is the one that gets left: turning it back on. The integration stays dark, and because it produces silence rather than errors, the silence is the last thing anyone questions.
The third route is worse to get wrong. GitHub can disable a hook itself after a sustained run of failures, which means an inactive hook is sometimes the aftermath of a completely different problem rather than a problem of its own. Flipping it back on without fixing whatever was returning 500 puts you back where you started in a day, having spent the retention window you needed to replay the lost events.
Why it happens
An inactive hook is not attempted, so nothing records it. This is the difference between this note and a hook whose deliveries are failing. A failing hook writes a record for every attempt, with a status code and a duration you can read. An inactive one writes nothing, so the two symptoms that look identical from your receiver — no request arrived — are opposites in the delivery log: a wall of 5xx or a blank page.
The flag is per hook, not per event and not per repository. Turning one hook off does not affect an organization hook pointed at the same URL, which is how an integration can go half dark: the org-level copy keeps delivering for some repositories while the repo-level hook for the one that matters is off. Read every scope that can reach the repository, not just the one you were told about.
updated_at dates the change, but not precisely. It moves whenever any part of the hook config is edited, so it is evidence rather than proof: a hook whose updated_at equals its created_at has never been edited at all, which makes active: false an original choice. A hook edited later might have been toggled then, or might have had its URL changed then and been toggled at some other moment the API does not record.
last_response is the tell for an automatic disable. It carries the code, status and message from the most recent attempt, and it survives the hook being switched off. An inactive hook whose last recorded response was a 500 or a timeout was almost certainly disabled rather than toggled, and that changes the order of the repair. Fix the receiver, then re-enable.
Re-enabling is a write, so this script does not do it. It prints the request. That is not squeamishness: turning a hook back on is exactly the action you want a human to take deliberately, after deciding the endpoint can survive the replay that follows. The script's job is to tell you which of the three situations you are in, so the decision is an informed one.
The fix, as a flow
One GET carries the finding and it is a boolean, which is why the script spends its effort somewhere else. Reading the flag is a line; knowing which of the three ways a hook arrived at false is the part that decides what you do next. Two timestamps separate a hook that was created switched off from one that somebody toggled later, and the last recorded response separates both from a hook GitHub disabled after a run of failures. That third case is not this note's problem at all, and re-enabling it without fixing the receiver spends the retention window you needed for the replay.
How to fix it
Read the flag before you read anything else
GET /repos/{owner}/{repo}/hooks returns every hook the token can see, each with id, active, events, config, created_at, updated_at and last_response. One request answers the question. Do this before instrumenting the receiver, because an inactive hook makes every other measurement meaningless.
Do not trust a truthy test on the boolean
The field is a real JSON boolean, but it does not always arrive as one. Config files, form posts, Terraform state and hand-rolled clients all have their own idea of how to spell false, and "false" is a non-empty string that a truthy test reads as on. Parse it to three states — on, off, and unreadable — and report the third rather than guessing at it.
Ask how it got there
Compare created_at with updated_at, and read last_response.code. A failing last response on an inactive hook means GitHub probably disabled it, an updated_at later than created_at means it was edited at some point after creation, and the two timestamps being equal means it has been off since the day it was made and has never delivered anything.
Corroborate with an empty delivery log
GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries on an inactive hook returns whatever was delivered before it was switched off, and nothing since. That gives you the date the silence started from the other direction, and it tells you how much of the gap still falls inside the retention window and can therefore be replayed.
Fix the receiver first if the log says so
Re-enable last, not first. If the last response was a failure, the hook was switched off for a reason that has not gone away, and switching it on again buys you a day. When you do re-enable, expect a burst: everything queued behind a redelivery run arrives at once, which is a second reason to be sure the endpoint is healthy before the flag flips.
How to check it worked
Once the flag is back on, the same read reports active and the delivery log starts filling again within a minute of the next matching event.
GITHUB_TOKEN=$GH_READONLY python3 github_hook_active_audit.py --repo acme-corp/api
# 3 hook(s) on acme-corp/api, 1 inactive
# inactive-after-failures: hook 512334455 is switched off, and its last
# recorded response was 502. GitHub disables a hook after a sustained run of
# failures, so this is an aftermath rather than a cause.
# repair: fix the receiver for the 502 first, then re-enable with
# gh api --method PATCH /repos/acme-corp/api/hooks/512334455 -F active=true
# after the receiver is fixed and an owner re-enables it
# active: hook 512334455 is switched onThe full code
One GET carries the finding and one optional GET dates it. Everything that decides anything is pure: a three-state read of a boolean that arrives as a boolean, a number or a string depending on what wrote it; a comparison of two timestamps with enough tolerance that a hook written and read in the same second is not called edited; the last recorded response code, which can be absent, null or a string; and a verdict that separates the three ways a hook ends up switched off, because they do not share a repair. Re-enabling is a write, so the script prints the request rather than making it.
"""Say whether a GitHub webhook is switched off, and which of three ways it happened.
Read only. Every call is a GET. Re-enabling a hook is a write and this script
does not do it: it prints the request for you to run once you have decided the
endpoint can survive being switched back on.
An inactive hook is not attempted, so it produces no delivery record, no
failure and no error. That is what makes it hard to find - the delivery log is
empty rather than full of 5xx, so the evidence looks like the absence of
events rather than the absence of a hook.
Three routes lead to active: false, and they do not share a repair:
created inactive never delivered anything, updated_at == created_at
toggled off later somebody switched it off, updated_at > created_at
disabled by GitHub a sustained run of failures, last_response is 4xx/5xx
Environment:
GITHUB_TOKEN a read-only token that can see the repository's hooks
"""
import argparse
import json
import logging
import os
import sys
from datetime import datetime, timezone
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_hook_active_audit")
API = "https://api.github.com"
UA = "github-hook-active-audit/1.0"
# Spellings of the boolean seen in the wild. GitHub returns a real JSON boolean,
# but the value reaches this script through whatever wrote the hook: a form
# post, a config file, a Terraform state, a client that stringifies everything.
TRUTHY = ("true", "1", "yes", "on")
FALSY = ("false", "0", "no", "off")
# States that mean a hook is delivering nothing at all.
OFF_STATES = ("inactive-after-failures", "inactive-toggled",
"inactive-since-creation", "inactive-undated")
def active_state(hook):
"""Three-state read of the active flag: on, off or unknown. Pure.
Deliberately not a boolean. A truthy test on this field reads the string
"false" as on, and an absent field as off, and both of those are wrong in
the direction that matters: one hides a dead hook, the other invents one.
"""
if not isinstance(hook, dict) or "active" not in hook:
return "unknown"
raw = hook["active"]
if isinstance(raw, bool):
return "on" if raw else "off"
if isinstance(raw, (int, float)):
return "on" if raw else "off"
text = str(raw).strip().lower()
if text in TRUTHY:
return "on"
if text in FALSY:
return "off"
return "unknown"
def last_code(hook):
"""The status code of the most recent delivery attempt, or None. Pure.
last_response survives the hook being switched off, which is the only
reason this script can tell a disable from a toggle.
"""
if not isinstance(hook, dict):
return None
resp = hook.get("last_response")
if not isinstance(resp, dict):
return None
code = resp.get("code")
if code is None or code == "":
return None
try:
return int(code)
except (TypeError, ValueError):
return None
def failed_last(hook):
"""Whether the most recent recorded response was a failure. Pure."""
code = last_code(hook)
return code is not None and code >= 400
def parsed_time(text):
"""An ISO 8601 timestamp as an aware datetime, or None. Pure."""
raw = str(text or "").strip()
if not raw or raw.lower() in ("null", "none"):
return None
if raw.endswith("Z"):
raw = raw[:-1] + "+00:00"
try:
when = datetime.fromisoformat(raw)
except ValueError:
return None
return when if when.tzinfo else when.replace(tzinfo=timezone.utc)
def days_since(text, now):
"""Whole days between a timestamp and now, or None. Pure."""
when = parsed_time(text)
if when is None or now is None:
return None
return (now - when).days
def edited_after_creation(hook, tolerance_seconds=90):
"""True, False or None - was this hook changed after it was made? Pure.
None when either timestamp is missing or unparseable, because "we cannot
tell" is a real answer here and reporting it as "never edited" would put
the reader in the wrong one of three repairs. The tolerance exists because
a hook created and configured in one API call comes back with two
timestamps a second or two apart.
"""
if not isinstance(hook, dict):
return None
created = parsed_time(hook.get("created_at"))
updated = parsed_time(hook.get("updated_at"))
if created is None or updated is None:
return None
return (updated - created).total_seconds() > tolerance_seconds
def newest_delivery(deliveries):
"""The most recent delivered_at across delivery records, or None. Pure."""
best, best_at = None, None
for row in deliveries or []:
if not isinstance(row, dict):
continue
when = parsed_time(row.get("delivered_at"))
if when is None:
continue
if best_at is None or when > best_at:
best, best_at = str(row.get("delivered_at")), when
return best
def silent_days(deliveries, now):
"""Days since the last delivery, or None when there has never been one. Pure."""
return days_since(newest_delivery(deliveries), now)
def classify(hook, deliveries=None, now=None):
"""Sort one hook into a state and a sentence. Pure.
deliveries is optional and only ever corroborates. The finding lives in the
hook record; the log dates it and says how much is still replayable.
"""
ident = "hook %s" % (hook.get("id", "?") if isinstance(hook, dict) else "?")
state = active_state(hook)
if state == "unknown":
return ("unknown",
"%s does not report a readable active flag. Read it in the "
"repository's settings before trusting anything else here."
% ident)
if state == "off":
if failed_last(hook):
return ("inactive-after-failures",
"%s is switched off, and its last recorded response was "
"%d. GitHub disables a hook after a sustained run of "
"failures, so this is an aftermath rather than a cause."
% (ident, last_code(hook)))
edited = edited_after_creation(hook)
if edited is True:
age = days_since(hook.get("updated_at"), now)
return ("inactive-toggled",
"%s is switched off and was last edited %s%s. It delivered "
"before that and has delivered nothing since."
% (ident, hook.get("updated_at", "at an unrecorded time"),
", %d day(s) ago" % age if age is not None else ""))
if edited is False:
return ("inactive-since-creation",
"%s is switched off and has never been edited, so it was "
"created inactive and has never delivered anything."
% ident)
return ("inactive-undated",
"%s is switched off. Its timestamps are missing, so which of "
"the three ways it got there cannot be told from here." % ident)
quiet = silent_days(deliveries, now)
if deliveries is not None and newest_delivery(deliveries) is None:
return ("active-but-silent",
"%s is switched on and the delivery log is empty. The hook is "
"not the problem: either nothing it subscribes to has "
"happened, or it subscribes to the wrong events." % ident)
if quiet is not None and quiet >= 30:
return ("active-but-quiet",
"%s is switched on and its last delivery was %d day(s) ago."
% (ident, quiet))
return ("active", "%s is switched on." % ident)
def repair(state, hook, repo="OWNER/REPO"):
"""The request or the decision a reader has to make. Pure.
Every branch that ends in a config change prints the change. Turning a hook
back on is a deliberate act with a burst of traffic behind it, and this
script is read only in any case.
"""
hook_id = hook.get("id", "HOOK_ID") if isinstance(hook, dict) else "HOOK_ID"
enable = ("gh api --method PATCH /repos/%s/hooks/%s -F active=true"
% (repo, hook_id))
if state == "inactive-after-failures":
return ("fix the receiver for the recorded response code first, then "
"re-enable with %s. Re-enabling before the receiver is fixed "
"gets the hook disabled again and spends the retention window "
"you need for the replay." % enable)
if state == "inactive-toggled":
return ("confirm the endpoint is healthy and can take a burst, then "
"re-enable with %s." % enable)
if state == "inactive-since-creation":
return ("this hook has never delivered anything. Either it was made "
"inactive by mistake, in which case %s, or it was superseded "
"by another hook and should be deleted." % enable)
if state == "inactive-undated":
return ("read the delivery log for the date the silence started, then "
"decide. When you re-enable: %s." % enable)
if state in ("active-but-silent", "active-but-quiet"):
return ("nothing here. The hook is on, so look at its events array "
"and at whether anything it subscribes to has happened.")
if state == "unknown":
return "read the active flag in the repository's settings by hand."
return "nothing. This hook is on."
def summarize(hooks):
"""Counts across every hook read. Pure."""
rows = [h for h in (hooks or []) if isinstance(h, dict)]
off = [h for h in rows if active_state(h) == "off"]
return {"total": len(rows), "inactive": len(off),
"active": len([h for h in rows if active_state(h) == "on"]),
"inactive_ids": [h.get("id") for h in off]}
def get(session, path):
"""One GET. Returns (status, json-or-None)."""
url = API + path if path.startswith("/") else path
r = session.get(url, timeout=30)
try:
body = r.json()
except ValueError:
body = None
return r.status_code, body
def list_hooks(session, scope):
"""Hooks for a repo (owner/name) or an org (@org). Read only."""
path = ("/orgs/%s/hooks?per_page=100" % scope[1:] if scope.startswith("@")
else "/repos/%s/hooks?per_page=100" % scope)
status, body = get(session, path)
if status != 200 or not isinstance(body, list):
log.error("GET %s returned %d; a token that cannot read hooks reports "
"no hooks rather than an error you would notice", path, status)
return []
return body
def list_deliveries(session, scope, hook_id, limit=30):
"""Recent delivery records for one hook. Read only, corroboration only."""
base = ("/orgs/%s/hooks/%s/deliveries" % (scope[1:], hook_id)
if scope.startswith("@")
else "/repos/%s/hooks/%s/deliveries" % (scope, hook_id))
status, body = get(session, "%s?per_page=%d" % (base, limit))
if status != 200 or not isinstance(body, list):
return None
return body
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", action="append", default=[],
help="owner/name; repeatable")
ap.add_argument("--org", action="append", default=[],
help="organization login; repeatable")
ap.add_argument("--no-deliveries", action="store_true",
help="skip the corroborating read of the delivery log")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN to a read-only token that can see the "
"repository's hooks")
return 2
scopes = list(args.repo) + ["@" + o for o in args.org]
if not scopes:
log.error("pass at least one --repo owner/name or --org login")
return 2
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
now = datetime.now(timezone.utc)
findings = []
for scope in scopes:
label = scope[1:] if scope.startswith("@") else scope
hooks = list_hooks(session, scope)
stats = summarize(hooks)
log.info("%d hook(s) on %s, %d inactive", stats["total"], label,
stats["inactive"])
for hook in hooks:
deliveries = None
if not args.no_deliveries:
deliveries = list_deliveries(session, scope, hook.get("id"))
state, detail = classify(hook, deliveries, now)
findings.append({"scope": label, "hook_id": hook.get("id"),
"state": state, "detail": detail,
"last_delivery": newest_delivery(deliveries)})
if state != "active":
log.info("%s: %s", state, detail)
log.info("repair: %s", repair(state, hook, label))
if stats["inactive"] == 0:
log.info("active: no hook on %s is switched off", label)
print(json.dumps({"scopes": scopes, "findings": findings},
indent=2, default=str))
return 1 if any(f["state"] in OFF_STATES for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Say whether a GitHub webhook is switched off, and which of three ways it happened.
*
* Read only. Every call is a GET. Re-enabling a hook is a write and is not done
* here: the script prints the request for you to run once you have decided the
* endpoint can survive being switched back on.
*
* Environment:
* GITHUB_TOKEN a read-only token that can see the repository's hooks
*
* Usage:
* node github-hook-active-audit.mjs acme-corp/api @acme-corp
*/
const API = 'https://api.github.com';
const UA = 'github-hook-active-audit/1.0';
const TRUTHY = ['true', '1', 'yes', 'on'];
const FALSY = ['false', '0', 'no', 'off'];
/** States that mean a hook is delivering nothing at all. */
export const OFF_STATES = [
'inactive-after-failures', 'inactive-toggled',
'inactive-since-creation', 'inactive-undated',
];
/**
* Three-state read of the active flag: on, off or unknown. Pure.
* A truthy test reads the string 'false' as on and an absent field as off.
*/
export function activeState(hook) {
if (!hook || typeof hook !== 'object' || !('active' in hook)) return 'unknown';
const raw = hook.active;
if (typeof raw === 'boolean') return raw ? 'on' : 'off';
if (typeof raw === 'number') return raw ? 'on' : 'off';
const text = String(raw ?? '').trim().toLowerCase();
if (TRUTHY.includes(text)) return 'on';
if (FALSY.includes(text)) return 'off';
return 'unknown';
}
/** The status code of the most recent delivery attempt, or null. Pure. */
export function lastCode(hook) {
if (!hook || typeof hook !== 'object') return null;
const resp = hook.last_response;
if (!resp || typeof resp !== 'object') return null;
const code = resp.code;
if (code === null || code === undefined || code === '') return null;
const n = Number(code);
return Number.isFinite(n) ? Math.trunc(n) : null;
}
/** Whether the most recent recorded response was a failure. Pure. */
export function failedLast(hook) {
const code = lastCode(hook);
return code !== null && code >= 400;
}
/** An ISO 8601 timestamp as epoch milliseconds, or null. Pure. */
export function parsedTime(text) {
const raw = String(text ?? '').trim();
if (!raw || ['null', 'none'].includes(raw.toLowerCase())) return null;
const ms = Date.parse(raw);
return Number.isNaN(ms) ? null : ms;
}
/** Whole days between a timestamp and now, or null. Pure. */
export function daysSince(text, nowMs) {
const when = parsedTime(text);
if (when === null || nowMs === null || nowMs === undefined) return null;
return Math.floor((nowMs - when) / 86400000);
}
/** true, false or null - was this hook changed after it was made? Pure. */
export function editedAfterCreation(hook, toleranceSeconds = 90) {
if (!hook || typeof hook !== 'object') return null;
const created = parsedTime(hook.created_at);
const updated = parsedTime(hook.updated_at);
if (created === null || updated === null) return null;
return (updated - created) / 1000 > toleranceSeconds;
}
/** The most recent delivered_at across delivery records, or null. Pure. */
export function newestDelivery(deliveries) {
let best = null;
let bestAt = null;
for (const row of deliveries ?? []) {
if (!row || typeof row !== 'object') continue;
const when = parsedTime(row.delivered_at);
if (when === null) continue;
if (bestAt === null || when > bestAt) {
best = String(row.delivered_at);
bestAt = when;
}
}
return best;
}
/** Days since the last delivery, or null when there has never been one. Pure. */
export function silentDays(deliveries, nowMs) {
return daysSince(newestDelivery(deliveries), nowMs);
}
/** Sort one hook into a state and a sentence. Pure. */
export function classify(hook, deliveries = null, nowMs = null) {
const ident = `hook ${(hook && typeof hook === 'object' ? hook.id : null) ?? '?'}`;
const state = activeState(hook);
if (state === 'unknown') {
return ['unknown',
`${ident} does not report a readable active flag. Read it in the ` +
'repository\'s settings before trusting anything else here.'];
}
if (state === 'off') {
if (failedLast(hook)) {
return ['inactive-after-failures',
`${ident} is switched off, and its last recorded response was ` +
`${lastCode(hook)}. GitHub disables a hook after a sustained run of ` +
'failures, so this is an aftermath rather than a cause.'];
}
const edited = editedAfterCreation(hook);
if (edited === true) {
const age = daysSince(hook.updated_at, nowMs);
return ['inactive-toggled',
`${ident} is switched off and was last edited ` +
`${hook.updated_at ?? 'at an unrecorded time'}` +
`${age !== null ? `, ${age} day(s) ago` : ''}. It delivered before ` +
'that and has delivered nothing since.'];
}
if (edited === false) {
return ['inactive-since-creation',
`${ident} is switched off and has never been edited, so it was ` +
'created inactive and has never delivered anything.'];
}
return ['inactive-undated',
`${ident} is switched off. Its timestamps are missing, so which of the ` +
'three ways it got there cannot be told from here.'];
}
const quiet = silentDays(deliveries, nowMs);
if (deliveries !== null && newestDelivery(deliveries) === null) {
return ['active-but-silent',
`${ident} is switched on and the delivery log is empty. The hook is not ` +
'the problem: either nothing it subscribes to has happened, or it ' +
'subscribes to the wrong events.'];
}
if (quiet !== null && quiet >= 30) {
return ['active-but-quiet',
`${ident} is switched on and its last delivery was ${quiet} day(s) ago.`];
}
return ['active', `${ident} is switched on.`];
}
/** The request or the decision a reader has to make. Pure. */
export function repair(state, hook, repo = 'OWNER/REPO') {
const hookId = (hook && typeof hook === 'object' ? hook.id : null) ?? 'HOOK_ID';
const enable = `gh api --method PATCH /repos/${repo}/hooks/${hookId} -F active=true`;
if (state === 'inactive-after-failures') {
return 'fix the receiver for the recorded response code first, then ' +
`re-enable with ${enable}. Re-enabling before the receiver is fixed gets ` +
'the hook disabled again and spends the retention window you need for ' +
'the replay.';
}
if (state === 'inactive-toggled') {
return 'confirm the endpoint is healthy and can take a burst, then ' +
`re-enable with ${enable}.`;
}
if (state === 'inactive-since-creation') {
return 'this hook has never delivered anything. Either it was made ' +
`inactive by mistake, in which case ${enable}, or it was superseded by ` +
'another hook and should be deleted.';
}
if (state === 'inactive-undated') {
return 'read the delivery log for the date the silence started, then ' +
`decide. When you re-enable: ${enable}.`;
}
if (state === 'active-but-silent' || state === 'active-but-quiet') {
return 'nothing here. The hook is on, so look at its events array and at ' +
'whether anything it subscribes to has happened.';
}
if (state === 'unknown') {
return 'read the active flag in the repository\'s settings by hand.';
}
return 'nothing. This hook is on.';
}
/** Counts across every hook read. Pure. */
export function summarize(hooks) {
const rows = (hooks ?? []).filter((h) => h && typeof h === 'object');
const off = rows.filter((h) => activeState(h) === 'off');
return {
total: rows.length,
inactive: off.length,
active: rows.filter((h) => activeState(h) === 'on').length,
inactive_ids: off.map((h) => h.id),
};
}
async function get(token, path) {
const res = await fetch(path.startsWith('/') ? API + path : path, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
let body = null;
try { body = await res.json(); } catch { body = null; }
return { status: res.status, body };
}
async function listHooks(token, scope) {
const path = scope.startsWith('@')
? `/orgs/${scope.slice(1)}/hooks?per_page=100`
: `/repos/${scope}/hooks?per_page=100`;
const { status, body } = await get(token, path);
if (status !== 200 || !Array.isArray(body)) {
console.error(`GET ${path} returned ${status}; a token that cannot read ` +
'hooks reports no hooks rather than an error you would notice');
return [];
}
return body;
}
async function listDeliveries(token, scope, hookId, limit = 30) {
const base = scope.startsWith('@')
? `/orgs/${scope.slice(1)}/hooks/${hookId}/deliveries`
: `/repos/${scope}/hooks/${hookId}/deliveries`;
const { status, body } = await get(token, `${base}?per_page=${limit}`);
if (status !== 200 || !Array.isArray(body)) return null;
return body;
}
async function main() {
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN to a read-only token that can see the ' +
"repository's hooks");
process.exitCode = 2;
return;
}
const scopes = process.argv.slice(2);
if (scopes.length === 0) {
console.error('pass at least one owner/name, or @org for an organization');
process.exitCode = 2;
return;
}
const now = Date.now();
const findings = [];
for (const scope of scopes) {
const label = scope.startsWith('@') ? scope.slice(1) : scope;
const hooks = await listHooks(token, scope);
const stats = summarize(hooks);
console.log(`${stats.total} hook(s) on ${label}, ${stats.inactive} inactive`);
for (const hook of hooks) {
const deliveries = await listDeliveries(token, scope, hook.id);
const [state, detail] = classify(hook, deliveries, now);
findings.push({
scope: label,
hook_id: hook.id,
state,
detail,
last_delivery: newestDelivery(deliveries),
});
if (state !== 'active') {
console.log(`${state}: ${detail}`);
console.log(`repair: ${repair(state, hook, label)}`);
}
}
if (stats.inactive === 0) {
console.log(`active: no hook on ${label} is switched off`);
}
}
console.log(JSON.stringify({ scopes, findings }, null, 2));
process.exitCode = findings.some((f) => OFF_STATES.includes(f.state)) ? 1 : 0;
}
// Only run when invoked directly, so importing this module from the test file
// does not execute main(), fail on the missing token and set an exit code that
// fails the suite even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The cases worth pinning are the ones where a shortcut would lie. A truthy test on active calls the string "false" an on hook and an absent field an off one, so both are tested explicitly and the absent case is unknown rather than either. After that: the three routes to off producing three different states and three different repairs, an inactive hook with a failing last response never being reported as a simple toggle, and an on hook with an empty log being sent to a different note instead of being called a finding here.
from datetime import datetime, timezone
from github_hook_active_audit import (
active_state, classify, days_since, edited_after_creation, failed_last,
last_code, newest_delivery, repair, silent_days, summarize,
)
NOW = datetime(2026, 8, 30, 12, 0, 0, tzinfo=timezone.utc)
FRESH = {"id": 1, "active": True, "created_at": "2026-01-04T10:00:00Z",
"updated_at": "2026-01-04T10:00:01Z"}
TOGGLED = {"id": 2, "active": False, "created_at": "2026-01-04T10:00:00Z",
"updated_at": "2026-07-26T02:11:00Z"}
BORN_OFF = {"id": 3, "active": False, "created_at": "2026-01-04T10:00:00Z",
"updated_at": "2026-01-04T10:00:02Z"}
DISABLED = {"id": 4, "active": False, "created_at": "2026-01-04T10:00:00Z",
"updated_at": "2026-07-26T02:11:00Z",
"last_response": {"code": 502, "status": "bad gateway"}}
def test_a_truthy_test_would_get_the_string_false_wrong():
assert active_state({"active": "false"}) == "off"
assert active_state({"active": "0"}) == "off"
assert active_state({"active": 0}) == "off"
assert active_state({"active": "true"}) == "on"
assert active_state({"active": 1}) == "on"
def test_an_absent_flag_is_unknown_and_not_off():
assert active_state({"id": 1}) == "unknown"
assert active_state({"active": None}) == "unknown"
assert active_state({"active": "maybe"}) == "unknown"
assert active_state(None) == "unknown"
def test_the_last_response_code_survives_every_shape_it_arrives_in():
assert last_code(DISABLED) == 502
assert last_code({"last_response": {"code": "500"}}) == 500
assert last_code({"last_response": {"code": None}}) is None
assert last_code({"last_response": {}}) is None
assert last_code({"id": 1}) is None
assert failed_last(DISABLED)
assert not failed_last({"last_response": {"code": 200}})
def test_a_hook_configured_in_one_call_is_not_called_edited():
assert edited_after_creation(BORN_OFF) is False
assert edited_after_creation(TOGGLED) is True
assert edited_after_creation({"created_at": "2026-01-04T10:00:00Z"}) is None
def test_the_three_routes_to_off_are_three_different_states():
assert classify(DISABLED, None, NOW)[0] == "inactive-after-failures"
assert classify(TOGGLED, None, NOW)[0] == "inactive-toggled"
assert classify(BORN_OFF, None, NOW)[0] == "inactive-since-creation"
def test_a_disabled_hook_is_never_reported_as_a_plain_toggle():
state, detail = classify(DISABLED, None, NOW)
assert state == "inactive-after-failures"
assert "502" in detail
assert "aftermath" in detail
def test_an_off_hook_with_no_timestamps_says_so_rather_than_guessing():
state, detail = classify({"id": 9, "active": False}, None, NOW)
assert state == "inactive-undated"
assert "cannot be told from here" in detail
def test_an_on_hook_with_an_empty_log_is_sent_to_a_different_question():
state, detail = classify(FRESH, [], NOW)
assert state == "active-but-silent"
assert "not the problem" in detail
assert "events array" in repair(state, FRESH)
def test_an_on_hook_with_a_recent_delivery_is_simply_active():
log = [{"delivered_at": "2026-08-30T09:00:00Z", "status": "OK"}]
assert classify(FRESH, log, NOW)[0] == "active"
def test_the_delivery_log_is_read_for_its_newest_row_not_its_first():
log = [{"delivered_at": "2026-08-01T09:00:00Z"},
{"delivered_at": "2026-08-29T09:00:00Z"},
{"delivered_at": "not a date"},
"junk"]
assert newest_delivery(log) == "2026-08-29T09:00:00Z"
assert silent_days(log, NOW) == 1
assert newest_delivery([]) is None
assert silent_days([], NOW) is None
def test_the_repair_for_a_disabled_hook_puts_the_receiver_first():
text = repair("inactive-after-failures", DISABLED, "acme-corp/api")
assert text.index("fix the receiver") < text.index("re-enable")
assert "/repos/acme-corp/api/hooks/4" in text
def test_the_summary_counts_the_hooks_that_are_off():
stats = summarize([FRESH, TOGGLED, DISABLED, {"id": 5}])
assert stats["total"] == 4
assert stats["inactive"] == 2
assert stats["active"] == 1
assert stats["inactive_ids"] == [2, 4]
def test_days_since_refuses_to_invent_an_age():
assert days_since("2026-08-27T12:00:00Z", NOW) == 3
assert days_since("", NOW) is None
assert days_since("null", NOW) is None
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
activeState, classify, daysSince, editedAfterCreation, failedLast,
lastCode, newestDelivery, repair, silentDays, summarize,
} from './github-hook-active-audit.mjs';
const NOW = Date.parse('2026-08-30T12:00:00Z');
const FRESH = {
id: 1, active: true,
created_at: '2026-01-04T10:00:00Z', updated_at: '2026-01-04T10:00:01Z',
};
const TOGGLED = {
id: 2, active: false,
created_at: '2026-01-04T10:00:00Z', updated_at: '2026-07-26T02:11:00Z',
};
const BORN_OFF = {
id: 3, active: false,
created_at: '2026-01-04T10:00:00Z', updated_at: '2026-01-04T10:00:02Z',
};
const DISABLED = {
id: 4, active: false,
created_at: '2026-01-04T10:00:00Z', updated_at: '2026-07-26T02:11:00Z',
last_response: { code: 502, status: 'bad gateway' },
};
test('a truthy test would get the string false wrong', () => {
assert.equal(activeState({ active: 'false' }), 'off');
assert.equal(activeState({ active: '0' }), 'off');
assert.equal(activeState({ active: 0 }), 'off');
assert.equal(activeState({ active: 'true' }), 'on');
assert.equal(activeState({ active: 1 }), 'on');
});
test('an absent flag is unknown and not off', () => {
assert.equal(activeState({ id: 1 }), 'unknown');
assert.equal(activeState({ active: null }), 'unknown');
assert.equal(activeState({ active: 'maybe' }), 'unknown');
assert.equal(activeState(null), 'unknown');
});
test('the last response code survives every shape it arrives in', () => {
assert.equal(lastCode(DISABLED), 502);
assert.equal(lastCode({ last_response: { code: '500' } }), 500);
assert.equal(lastCode({ last_response: { code: null } }), null);
assert.equal(lastCode({ last_response: {} }), null);
assert.equal(lastCode({ id: 1 }), null);
assert.ok(failedLast(DISABLED));
assert.ok(!failedLast({ last_response: { code: 200 } }));
});
test('a hook configured in one call is not called edited', () => {
assert.equal(editedAfterCreation(BORN_OFF), false);
assert.equal(editedAfterCreation(TOGGLED), true);
assert.equal(editedAfterCreation({ created_at: '2026-01-04T10:00:00Z' }), null);
});
test('the three routes to off are three different states', () => {
assert.equal(classify(DISABLED, null, NOW)[0], 'inactive-after-failures');
assert.equal(classify(TOGGLED, null, NOW)[0], 'inactive-toggled');
assert.equal(classify(BORN_OFF, null, NOW)[0], 'inactive-since-creation');
});
test('a disabled hook is never reported as a plain toggle', () => {
const [state, detail] = classify(DISABLED, null, NOW);
assert.equal(state, 'inactive-after-failures');
assert.match(detail, /502/);
assert.match(detail, /aftermath/);
});
test('an off hook with no timestamps says so rather than guessing', () => {
const [state, detail] = classify({ id: 9, active: false }, null, NOW);
assert.equal(state, 'inactive-undated');
assert.match(detail, /cannot be told from here/);
});
test('an on hook with an empty log is sent to a different question', () => {
const [state, detail] = classify(FRESH, [], NOW);
assert.equal(state, 'active-but-silent');
assert.match(detail, /not the problem/);
assert.match(repair(state, FRESH), /events array/);
});
test('an on hook with a recent delivery is simply active', () => {
const log = [{ delivered_at: '2026-08-30T09:00:00Z', status: 'OK' }];
assert.equal(classify(FRESH, log, NOW)[0], 'active');
});
test('the delivery log is read for its newest row not its first', () => {
const log = [
{ delivered_at: '2026-08-01T09:00:00Z' },
{ delivered_at: '2026-08-29T09:00:00Z' },
{ delivered_at: 'not a date' },
'junk',
];
assert.equal(newestDelivery(log), '2026-08-29T09:00:00Z');
assert.equal(silentDays(log, NOW), 1);
assert.equal(newestDelivery([]), null);
assert.equal(silentDays([], NOW), null);
});
test('the repair for a disabled hook puts the receiver first', () => {
const text = repair('inactive-after-failures', DISABLED, 'acme-corp/api');
assert.ok(text.indexOf('fix the receiver') < text.indexOf('re-enable'));
assert.match(text, /\/repos\/acme-corp\/api\/hooks\/4/);
});
test('the summary counts the hooks that are off', () => {
const stats = summarize([FRESH, TOGGLED, DISABLED, { id: 5 }]);
assert.equal(stats.total, 4);
assert.equal(stats.inactive, 2);
assert.equal(stats.active, 1);
assert.deepEqual(stats.inactive_ids, [2, 4]);
});
test('daysSince refuses to invent an age', () => {
assert.equal(daysSince('2026-08-27T12:00:00Z', NOW), 3);
assert.equal(daysSince('', NOW), null);
assert.equal(daysSince('null', NOW), null);
});
FAQ
How is this different from deliveries that are failing?
They are opposites in the one place that matters. A failing hook is attempted every time, so GitHub writes a delivery record for each attempt with a status code, a duration and a response body you can read. An inactive hook is not attempted at all, so there is nothing to read: the log stops dead at the moment it was switched off. From the receiver's side both look like a request that never arrived, which is why the delivery log is the first thing to check and the shape of what you find there tells you which note you are in.
Does GitHub really disable webhooks by itself?
Yes, after a sustained run of failed deliveries. That is why the script checks last_response before it checks the timestamps: an inactive hook whose most recent recorded response was a 502 or a timeout was almost certainly switched off for you rather than by anyone, and it is a symptom of a receiver problem rather than a problem of its own. Turning it back on without fixing the receiver gets it disabled again, and burns the retention window you would have used to replay the events you missed.
Why not just re-enable it in the script?
Two reasons, and only one of them is that this section never writes. The other is that re-enabling releases a burst: anything you replay from the retention window arrives at once, on top of live traffic, against an endpoint that has not handled a request in weeks. That is a decision to take deliberately, with somebody watching. The script prints the exact request so the decision is one keystroke away, and so it is still a decision.
Can I tell who switched it off, and when?
Not from the hook record. It carries created_at and updated_at, and updated_at moves for any edit to the hook, so it dates the most recent change of any kind rather than the toggle specifically. There is no actor field. If you need the name, the organization audit log has the event; the API-visible evidence here is limited to the fact that something changed after creation and the date the deliveries stopped, which the script reads from the other end of the delivery log.
The hook is on and still nothing arrives. What now?
Then this is not your note, and the script says so rather than shrugging. An active hook with an empty delivery log means either that nothing it subscribes to has actually happened, or that it subscribes to events your handler is not waiting for, which is a different read of a different field. Check the events array against the events your code handles before you touch anything else, and confirm the hook you are looking at is the one whose URL your receiver serves.
Related field notes
- Deliveries failing where nobody reads the log
- The hook is not subscribed to your event
- Certificate verification switched off on the hook
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Repository webhooks — GitHub REST API
- Organization webhooks — GitHub REST API
- Troubleshooting webhooks — GitHub Docs
- Handling failed webhook deliveries — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.