Diagnostic GitHub API
webhook deliveries are failing and nobody reads the log
Someone opens a pull request and the bot that should comment on it says nothing. You grep your receiver's logs for the last hour and find no request at all, which points the finger at GitHub. It is not GitHub. The delivery happened, your server answered 502, and GitHub wrote that down in a log you have never opened.
Read GET /repos/{owner}/{repo}/hooks and look at last_response on each hook: code, status and message for the most recent attempt. One request, immediate verdict.
Then read GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries?per_page=100, where every attempt carries status, status_code, event, duration, delivered_at, guid and redelivery. Group the failures by status code, because the repair for a 401 and the repair for a timeout have nothing in common. Anything you lost can be replayed with the redelivery endpoint, but only inside the retention window.
The problem in plain words
Every other integration failure leaves a trace on both sides. This one leaves a trace on GitHub's side only. Your receiver's access log records requests it handled; a request that arrived and blew up in a middlebox, or that arrived and returned 500 from a framework error page, may never reach the code that does your logging. From where you are standing, the events simply stopped.
So the discovery path is always the same and always late. A user notices that something downstream of an event did not happen. Someone re-runs it by hand. Weeks later a second report arrives, and only then does anyone open the hook and find several hundred failed deliveries stretching back to a deploy nobody connected to webhooks. By that point the oldest of those deliveries has aged out of the log and cannot be replayed at all.
Why it happens
The delivery record belongs to GitHub, not to you. GitHub stores each attempt with the response it received, including attempts that your application never saw. A reverse proxy returning 413 on a large push payload, a WAF returning 403, a platform returning 502 while a container restarts: all three are invisible in your logs and all three are one field in the delivery record.
A failing hook is not a disabled hook. GitHub keeps trying, so nothing escalates on its own. The hook stays active, the delivery log fills with red, and no state change ever fires an alert. The only way this becomes visible is if something goes and looks.
The status code is the whole diagnosis, and it gets thrown away. A run of 401 or 403 means your own server rejected GitHub, which is what a mismatched webhook secret looks like from the outside. A run of 5xx means the payload arrived and your handler raised. A timeout means the handler is doing its work synchronously and ran past ten seconds. A missing status code means nothing answered at all: DNS, TLS or a closed port. Reporting these as one number called "failures" is how the repair gets guessed at.
The log has a horizon. Deliveries are retained for a limited window, and the redelivery endpoint can only replay what is still in it. Every day this goes unnoticed converts recoverable events into permanently lost ones.
The fix, as a flow
The script buckets deliveries by status code before it judges anything, because the code is the entire diagnosis: a timeout and a 502 and a silent connection failure are three different repairs wearing one number called failures.
How to fix it
Read last_response first, because it costs one request
GET /repos/{owner}/{repo}/hooks returns every hook with a last_response object. A code outside the 2xx range on any hook is enough to know something is wrong before you page a single delivery, and a code of null means the hook has never delivered anything at all — a different problem with a different cause.
Page the delivery log and bucket by what actually happened
GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries?per_page=100, following rel="next" in the Link header. Sort each record into a bucket by status_code and status: rejected (401, 403), server error (5xx), timed out, unreachable (no code at all), other client error (400, 404, 413). The dominant bucket is the finding.
Read the window, not just the count
Keep the earliest and latest delivered_at for both successes and failures. If the newest delivery succeeded and every failure is older, this is already fixed and what remains is a backfill. If failures continue up to the present, it is live. Those two situations produce the same failure count and want opposite responses.
Fix the receiver for the bucket you found
A 5xx run wants the exception traced. A timeout run wants the handler to answer immediately and do its work on a queue. A 401 or 403 run wants the signing secret compared against the one GitHub is signing with — and note that the API will never confirm that comparison for you; the delivery log is the only place a wrong secret becomes visible.
Replay what is still in the window
Once the endpoint answers 2xx, replay each failed delivery with POST /repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}/attempts. The script prints that call for every failure it found rather than making it. Replays arrive with redelivery: true and the same guid, so a receiver that keys on the guid will treat them correctly.
How to check it worked
Re-run the script after the receiver change. Every hook should report clean, or recovered with a backfill count that goes to zero once you have replayed it.
python3 github_hook_delivery_audit.py --repo acme/api
# 2 hook(s), 0 failing, 0 delivery(ies) needing a replayThe full code
Two GETs per hook and nothing else. The bucketing and the verdict are pure functions because the status code is the entire diagnosis here, and a classifier that collapses a timeout into a 5xx sends you to read a stack trace that does not exist. The redelivery endpoint is printed with the exact delivery id, never called.
"""Report GitHub webhooks whose deliveries are failing, and say how they fail.
Read only. Every request is a GET, so a token with read access to the repository
and its hooks is enough. The redelivery call is printed for a human to run, never
made here: this script holds a credential that can reach your repositories.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_hook_delivery_audit")
API = "https://api.github.com"
UA = "github-hook-delivery-audit/1.0"
# Failure buckets, most diagnostic first. Ties in the dominant-bucket scan are
# broken by this order, so a hook failing equally often on two causes reports the
# one that names a specific repair rather than the one that says "other".
FAILURE_ORDER = ("rejected", "server-error", "timeout", "unreachable",
"client-error", "unknown")
def bucket(delivery):
"""Sort one delivery record into a bucket. Pure.
status_code is the diagnosis. A record with no code at all never reached a
server, which is a network problem; a record with 401 or 403 reached one that
refused it, which is usually a signature the receiver would not accept. These
want opposite repairs and are routinely reported as one number.
"""
status = str(delivery.get("status") or "").strip().lower()
raw = delivery.get("status_code")
try:
code = int(raw)
except (TypeError, ValueError):
code = 0
if 200 <= code < 300:
return "ok"
if "tim" in status:
return "timeout"
if not code:
return "unreachable"
if code in (401, 403):
return "rejected"
if 400 <= code < 500:
return "client-error"
if 500 <= code < 600:
return "server-error"
return "unknown"
def triage(hook):
"""Read the hook's last_response, which is the one-request version of this.
Pure. code is null on a hook that has never delivered anything, which is not
a failure and must not be reported as one: it means the hook is new, is
inactive, or is subscribed to events that have not happened.
"""
last = hook.get("last_response") or {}
code = last.get("code")
if code is None:
return ("never", "no delivery attempt recorded yet")
try:
code = int(code)
except (TypeError, ValueError):
return ("unknown", "unreadable last_response code %r" % (last.get("code"),))
if 200 <= code < 300:
return ("ok", "last attempt returned %d" % code)
message = str(last.get("message") or "").strip()
return ("failing", "last attempt returned %d%s"
% (code, ": " + message if message else ""))
def summarize(deliveries):
"""Count deliveries by bucket and keep the ends of the window. Pure.
delivered_at is ISO 8601 in UTC on every record, so string comparison orders
them correctly and nothing needs parsing to find the first and last of each.
"""
out = {"total": 0, "ok": 0, "failed": 0, "redeliveries": 0, "counts": {},
"guids": {}, "last_ok": None, "first_failed": None, "last_failed": None}
for d in deliveries or []:
kind = bucket(d)
when = str(d.get("delivered_at") or "")
out["total"] += 1
if d.get("redelivery"):
out["redeliveries"] += 1
if kind == "ok":
out["ok"] += 1
if when and (out["last_ok"] is None or when > out["last_ok"]):
out["last_ok"] = when
continue
out["failed"] += 1
out["counts"][kind] = out["counts"].get(kind, 0) + 1
ids = out["guids"].setdefault(kind, [])
if len(ids) < 5 and d.get("id") is not None:
ids.append(d.get("id"))
if when:
if out["first_failed"] is None or when < out["first_failed"]:
out["first_failed"] = when
if out["last_failed"] is None or when > out["last_failed"]:
out["last_failed"] = when
return out
def verdict(summary):
"""Classify one hook from its delivery summary. Pure.
Returns (state, detail). "recovered" exists because a fixed hook and a
broken one produce the same failure count, and the difference between them
is whether anything has succeeded since.
"""
total = int(summary.get("total") or 0)
if not total:
return ("empty",
"no deliveries in the retained window. Either nothing this hook "
"subscribes to has happened, or the hook is not active.")
failed = int(summary.get("failed") or 0)
if not failed:
return ("clean", "%d delivery(ies), all accepted" % total)
last_ok = summary.get("last_ok")
last_failed = summary.get("last_failed")
if last_ok and last_failed and last_ok > last_failed:
return ("recovered",
"%d of %d failed, but the most recent delivery succeeded. The "
"receiver is working; %d event(s) are still waiting on a replay."
% (failed, total, failed))
counts = summary.get("counts") or {}
worst = None
for kind in FAILURE_ORDER:
n = counts.get(kind, 0)
if n and (worst is None or n > counts[worst]):
worst = kind
n = counts.get(worst, 0)
if worst == "rejected":
return (worst,
"%d of %d came back 401 or 403. Your own server refused GitHub. "
"This is the only shape a mismatched webhook secret takes from "
"outside: the API will not compare secrets for you." % (n, total))
if worst == "server-error":
return (worst,
"%d of %d returned 5xx. The payload arrived and the handler "
"raised, so the trace is in your application, not in the "
"network." % (n, total))
if worst == "timeout":
return (worst,
"%d of %d timed out. GitHub allows a receiver 10 seconds; a "
"handler doing its real work synchronously runs past that as "
"soon as the payload grows." % (n, total))
if worst == "unreachable":
return (worst,
"%d of %d recorded no status code at all, so nothing answered: "
"DNS, TLS, a closed port, or an allow-list that no longer "
"matches GitHub's hook ranges." % (n, total))
return (worst or "unknown",
"%d of %d failed with a 4xx that is not an auth error, which is "
"usually a route that moved (404) or a body the handler would not "
"parse (400)." % (n, total))
def next_link(response):
"""The rel=next URL from the Link header, or None."""
for part in (response.headers.get("Link") or "").split(","):
chunk = part.strip()
if chunk.startswith("<") and chunk.endswith('rel="next"'):
return chunk[1:chunk.index(">")]
return None
def get(session, url, **params):
r = session.get(url, params=params, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, expired or "
"malformed")
if r.status_code in (403, 404):
raise SystemExit("%d from %s: reading hooks needs admin:repo_hook (or "
"the fine-grained Webhooks: Read permission). GitHub "
"returns 404 rather than 403 when a token cannot see a "
"resource at all." % (r.status_code, url))
r.raise_for_status()
return r
def page(session, url, key=None, limit=1000, **params):
"""Follow Link rel=next until the limit. Returns a flat list."""
out = []
while url and len(out) < limit:
r = get(session, url, **params)
body = r.json()
out.extend(body if isinstance(body, list) else body.get(key) or [])
url, params = next_link(r), {}
return out[:limit]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--hook", type=int, default=None,
help="only this hook id (default: every hook on the repo)")
ap.add_argument("--max-deliveries", type=int, default=300,
help="stop paging each hook's delivery log after this many")
args = ap.parse_args()
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
owner, _, name = args.repo.partition("/")
if not (owner and name):
log.error("--repo takes owner/name, for example acme/api")
return 2
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
base = "%s/repos/%s/%s/hooks" % (API, owner, name)
hooks = page(session, base, per_page=100)
if args.hook:
hooks = [h for h in hooks if h.get("id") == args.hook]
if not hooks:
log.info("no webhooks on %s that this token can see", args.repo)
return 0
failing = replayable = 0
for hook in hooks:
hid = hook.get("id")
url = (hook.get("config") or {}).get("url", "?")
state, detail = triage(hook)
log.info("hook %s %s last_response: %s (%s)", hid, url, state, detail)
deliveries = page(session, "%s/%s/deliveries" % (base, hid),
limit=args.max_deliveries, per_page=100)
summary = summarize(deliveries)
state, detail = verdict(summary)
line = " %-12s %s" % (state, detail)
if state in ("clean", "empty"):
log.info(line)
continue
log.warning(line)
log.warning(" failures from %s to %s, %d redelivery(ies) already in "
"the log", summary["first_failed"], summary["last_failed"],
summary["redeliveries"])
if state != "recovered":
failing += 1
replayable += summary["failed"]
for kind, ids in sorted(summary["guids"].items()):
for did in ids:
log.warning(" repair: POST %s/%s/deliveries/%s/attempts "
"(%s)", base, hid, did, kind)
log.info("%d hook(s), %d failing, %d delivery(ies) needing a replay",
len(hooks), failing, replayable)
return 1 if failing else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report GitHub webhooks whose deliveries are failing, and say how they fail.
*
* Read only. Every request is a GET. The redelivery call is printed for a human
* to run, never made here.
*/
const API = 'https://api.github.com';
const UA = 'github-hook-delivery-audit/1.0';
// Failure buckets, most diagnostic first. Ties are broken by this order.
const FAILURE_ORDER = ['rejected', 'server-error', 'timeout', 'unreachable',
'client-error', 'unknown'];
/**
* Sort one delivery record into a bucket. Pure. A record with no status code
* never reached a server; one with 401 or 403 reached one that refused it.
*/
export function bucket(delivery) {
const status = String(delivery.status ?? '').trim().toLowerCase();
const code = Number.parseInt(delivery.status_code, 10) || 0;
if (code >= 200 && code < 300) return 'ok';
if (status.includes('tim')) return 'timeout';
if (!code) return 'unreachable';
if (code === 401 || code === 403) return 'rejected';
if (code >= 400 && code < 500) return 'client-error';
if (code >= 500 && code < 600) return 'server-error';
return 'unknown';
}
/**
* Read the hook's last_response: the one-request version of this whole check.
* A null code means the hook has never delivered anything, which is not a
* failure.
*/
export function triage(hook) {
const last = hook.last_response ?? {};
if (last.code === null || last.code === undefined) {
return ['never', 'no delivery attempt recorded yet'];
}
const code = Number.parseInt(last.code, 10);
if (!Number.isFinite(code)) {
return ['unknown', `unreadable last_response code ${JSON.stringify(last.code)}`];
}
if (code >= 200 && code < 300) return ['ok', `last attempt returned ${code}`];
const message = String(last.message ?? '').trim();
return ['failing',
`last attempt returned ${code}${message ? `: ${message}` : ''}`];
}
/** Count deliveries by bucket and keep the ends of the window. Pure. */
export function summarize(deliveries) {
const out = {
total: 0, ok: 0, failed: 0, redeliveries: 0, counts: {}, guids: {},
last_ok: null, first_failed: null, last_failed: null,
};
for (const d of deliveries ?? []) {
const kind = bucket(d);
const when = String(d.delivered_at ?? '');
out.total += 1;
if (d.redelivery) out.redeliveries += 1;
if (kind === 'ok') {
out.ok += 1;
if (when && (out.last_ok === null || when > out.last_ok)) out.last_ok = when;
continue;
}
out.failed += 1;
out.counts[kind] = (out.counts[kind] ?? 0) + 1;
const ids = (out.guids[kind] ??= []);
if (ids.length < 5 && d.id !== undefined && d.id !== null) ids.push(d.id);
if (when) {
if (out.first_failed === null || when < out.first_failed) out.first_failed = when;
if (out.last_failed === null || when > out.last_failed) out.last_failed = when;
}
}
return out;
}
/** Classify one hook from its delivery summary. Pure. Returns [state, detail]. */
export function verdict(summary) {
const total = summary.total ?? 0;
if (!total) {
return ['empty',
'no deliveries in the retained window. Either nothing this hook ' +
'subscribes to has happened, or the hook is not active.'];
}
const failed = summary.failed ?? 0;
if (!failed) return ['clean', `${total} delivery(ies), all accepted`];
if (summary.last_ok && summary.last_failed && summary.last_ok > summary.last_failed) {
return ['recovered',
`${failed} of ${total} failed, but the most recent delivery succeeded. ` +
`The receiver is working; ${failed} event(s) are still waiting on a replay.`];
}
const counts = summary.counts ?? {};
let worst = null;
for (const kind of FAILURE_ORDER) {
const n = counts[kind] ?? 0;
if (n && (worst === null || n > counts[worst])) worst = kind;
}
const n = counts[worst] ?? 0;
if (worst === 'rejected') {
return [worst,
`${n} of ${total} came back 401 or 403. Your own server refused GitHub. ` +
'This is the only shape a mismatched webhook secret takes from outside: ' +
'the API will not compare secrets for you.'];
}
if (worst === 'server-error') {
return [worst,
`${n} of ${total} returned 5xx. The payload arrived and the handler ` +
'raised, so the trace is in your application, not in the network.'];
}
if (worst === 'timeout') {
return [worst,
`${n} of ${total} timed out. GitHub allows a receiver 10 seconds; a ` +
'handler doing its real work synchronously runs past that as soon as the ' +
'payload grows.'];
}
if (worst === 'unreachable') {
return [worst,
`${n} of ${total} recorded no status code at all, so nothing answered: ` +
"DNS, TLS, a closed port, or an allow-list that no longer matches GitHub's " +
'hook ranges.'];
}
return [worst ?? 'unknown',
`${n} of ${total} failed with a 4xx that is not an auth error, which is ` +
'usually a route that moved (404) or a body the handler would not parse (400).'];
}
function nextLink(res) {
for (const part of (res.headers.get('link') ?? '').split(',')) {
const chunk = part.trim();
if (chunk.startsWith('<') && chunk.endsWith('rel="next"')) {
return chunk.slice(1, chunk.indexOf('>'));
}
}
return null;
}
async function get(token, url) {
const res = await fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
if (res.status === 401) {
throw new Error('401 from GitHub: GITHUB_TOKEN is missing, expired or malformed');
}
if (res.status === 403 || res.status === 404) {
throw new Error(`${res.status} from ${url}: reading hooks needs ` +
'admin:repo_hook (or the fine-grained Webhooks: Read permission). GitHub ' +
'returns 404 rather than 403 when a token cannot see a resource at all.');
}
if (!res.ok) throw new Error(`${res.status} from ${url}`);
return res;
}
async function page(token, url, limit = 1000) {
const out = [];
let next = url;
while (next && out.length < limit) {
const res = await get(token, next);
out.push(...(await res.json()));
next = nextLink(res);
}
return out.slice(0, limit);
}
async function main() {
const repo = process.argv[2];
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN (a read-only token is enough)');
process.exitCode = 2;
return;
}
if (!repo || !repo.includes('/')) {
console.error('usage: node github-hook-delivery-audit.mjs owner/name');
process.exitCode = 2;
return;
}
const base = `${API}/repos/${repo}/hooks`;
const hooks = await page(token, `${base}?per_page=100`);
if (hooks.length === 0) {
console.log(`no webhooks on ${repo} that this token can see`);
return;
}
let failing = 0;
let replayable = 0;
for (const hook of hooks) {
const url = hook.config?.url ?? '?';
const [tstate, tdetail] = triage(hook);
console.log(`hook ${hook.id} ${url} last_response: ${tstate} (${tdetail})`);
const deliveries = await page(token, `${base}/${hook.id}/deliveries?per_page=100`, 300);
const summary = summarize(deliveries);
const [state, detail] = verdict(summary);
const line = ` ${state.padEnd(12)} ${detail}`;
if (state === 'clean' || state === 'empty') { console.log(line); continue; }
console.warn(line);
console.warn(` failures from ${summary.first_failed} to ${summary.last_failed}, ` +
`${summary.redeliveries} redelivery(ies) already in the log`);
if (state !== 'recovered') failing += 1;
replayable += summary.failed;
for (const [kind, ids] of Object.entries(summary.guids).sort()) {
for (const id of ids) {
console.warn(` repair: POST ${base}/${hook.id}/deliveries/${id}/attempts (${kind})`);
}
}
}
console.log(`${hooks.length} hook(s), ${failing} failing, ` +
`${replayable} delivery(ies) needing a replay`);
process.exitCode = failing ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing token, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The cases worth pinning are the ones that look alike from a distance. A delivery with no status code is not a 5xx. A hook whose failures all predate its last success is not broken. And last_response.code of null means nothing has ever been delivered, which a naive numeric comparison reads as a failure and reports as an outage.
from github_hook_delivery_audit import bucket, summarize, triage, verdict
def delivery(code, status="failure", when="2026-08-01T10:00:00Z", did=1, redelivery=False):
return {"id": did, "status": status, "status_code": code,
"delivered_at": when, "redelivery": redelivery}
def test_two_hundred_is_the_only_success():
assert bucket(delivery(200, "OK")) == "ok"
assert bucket(delivery(204, "OK")) == "ok"
def test_no_status_code_is_unreachable_not_a_server_error():
# Nothing answered, so there is no stack trace to go and read.
assert bucket(delivery(0)) == "unreachable"
assert bucket({"status": "failure"}) == "unreachable"
def test_a_timeout_is_its_own_bucket_whatever_the_code_says():
assert bucket({"status": "timed out", "status_code": 0}) == "timeout"
def test_auth_failures_are_separated_from_other_client_errors():
assert bucket(delivery(401)) == "rejected"
assert bucket(delivery(403)) == "rejected"
assert bucket(delivery(404)) == "client-error"
assert bucket(delivery(502)) == "server-error"
def test_triage_treats_a_null_code_as_never_delivered():
state, detail = triage({"last_response": {"code": None, "status": "unused"}})
assert state == "never"
assert "no delivery" in detail
def test_triage_reads_the_failing_code_and_message():
state, detail = triage({"last_response": {"code": 502, "message": "Bad Gateway"}})
assert state == "failing"
assert "502" in detail and "Bad Gateway" in detail
def test_summarize_keeps_both_ends_of_the_window():
s = summarize([
delivery(200, "OK", "2026-08-01T10:00:00Z"),
delivery(500, when="2026-08-02T10:00:00Z", did=2),
delivery(500, when="2026-08-03T10:00:00Z", did=3, redelivery=True),
])
assert s["total"] == 3 and s["ok"] == 1 and s["failed"] == 2
assert s["first_failed"] == "2026-08-02T10:00:00Z"
assert s["last_failed"] == "2026-08-03T10:00:00Z"
assert s["last_ok"] == "2026-08-01T10:00:00Z"
assert s["redeliveries"] == 1
assert s["guids"]["server-error"] == [2, 3]
def test_an_empty_log_is_not_a_healthy_hook():
state, _ = verdict(summarize([]))
assert state == "empty"
def test_failures_older_than_the_last_success_are_already_fixed():
s = summarize([delivery(500, when="2026-08-01T10:00:00Z"),
delivery(200, "OK", "2026-08-02T10:00:00Z", did=2)])
state, detail = verdict(s)
assert state == "recovered"
assert "replay" in detail
def test_the_dominant_bucket_names_the_repair():
s = summarize([delivery(500), delivery(500, did=2), delivery(404, did=3)])
state, detail = verdict(s)
assert state == "server-error"
assert "handler" in detail
def test_a_run_of_401s_points_at_the_secret_without_claiming_to_read_it():
s = summarize([delivery(401), delivery(401, did=2)])
state, detail = verdict(s)
assert state == "rejected"
assert "will not compare secrets" in detail
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
bucket, summarize, triage, verdict,
} from './github-hook-delivery-audit.mjs';
const delivery = (code, status = 'failure', when = '2026-08-01T10:00:00Z',
id = 1, redelivery = false) =>
({ id, status, status_code: code, delivered_at: when, redelivery });
test('two hundred is the only success', () => {
assert.equal(bucket(delivery(200, 'OK')), 'ok');
assert.equal(bucket(delivery(204, 'OK')), 'ok');
});
test('no status code is unreachable, not a server error', () => {
assert.equal(bucket(delivery(0)), 'unreachable');
assert.equal(bucket({ status: 'failure' }), 'unreachable');
});
test('a timeout is its own bucket whatever the code says', () => {
assert.equal(bucket({ status: 'timed out', status_code: 0 }), 'timeout');
});
test('auth failures are separated from other client errors', () => {
assert.equal(bucket(delivery(401)), 'rejected');
assert.equal(bucket(delivery(403)), 'rejected');
assert.equal(bucket(delivery(404)), 'client-error');
assert.equal(bucket(delivery(502)), 'server-error');
});
test('triage treats a null code as never delivered', () => {
const [state, detail] = triage({ last_response: { code: null, status: 'unused' } });
assert.equal(state, 'never');
assert.match(detail, /no delivery/);
});
test('triage reads the failing code and message', () => {
const [state, detail] = triage({ last_response: { code: 502, message: 'Bad Gateway' } });
assert.equal(state, 'failing');
assert.match(detail, /502: Bad Gateway/);
});
test('summarize keeps both ends of the window', () => {
const s = summarize([
delivery(200, 'OK', '2026-08-01T10:00:00Z'),
delivery(500, 'failure', '2026-08-02T10:00:00Z', 2),
delivery(500, 'failure', '2026-08-03T10:00:00Z', 3, true),
]);
assert.equal(s.total, 3);
assert.equal(s.failed, 2);
assert.equal(s.first_failed, '2026-08-02T10:00:00Z');
assert.equal(s.last_failed, '2026-08-03T10:00:00Z');
assert.equal(s.last_ok, '2026-08-01T10:00:00Z');
assert.equal(s.redeliveries, 1);
assert.deepEqual(s.guids['server-error'], [2, 3]);
});
test('an empty log is not a healthy hook', () => {
assert.equal(verdict(summarize([]))[0], 'empty');
});
test('failures older than the last success are already fixed', () => {
const s = summarize([
delivery(500, 'failure', '2026-08-01T10:00:00Z'),
delivery(200, 'OK', '2026-08-02T10:00:00Z', 2),
]);
const [state, detail] = verdict(s);
assert.equal(state, 'recovered');
assert.match(detail, /replay/);
});
test('the dominant bucket names the repair', () => {
const s = summarize([delivery(500), delivery(500, 'failure', '2026-08-01T10:00:00Z', 2),
delivery(404, 'failure', '2026-08-01T10:00:00Z', 3)]);
const [state, detail] = verdict(s);
assert.equal(state, 'server-error');
assert.match(detail, /handler/);
});
test('a run of 401s points at the secret without claiming to read it', () => {
const s = summarize([delivery(401), delivery(401, 'failure', '2026-08-01T10:00:00Z', 2)]);
const [state, detail] = verdict(s);
assert.equal(state, 'rejected');
assert.match(detail, /will not compare secrets/);
});
FAQ
How long does GitHub keep webhook deliveries?
Long enough to diagnose a problem you notice quickly and not long enough to diagnose one you notice late. The delivery log is a bounded window, and the redelivery endpoint can only replay what is still in it, so the practical answer is that every day a failure goes unnoticed converts recoverable events into lost ones. Read last_response on a schedule rather than reading the log after a complaint.
Why does my receiver have no record of the request?
Because it never reached your application code. A reverse proxy rejecting a large push payload, a WAF returning 403, or a platform returning 502 during a restart all answer GitHub without your handler running. GitHub records what it received; your log records what your handler saw, and those are different sets.
Does the script redeliver the failed events for me?
No. This section is read only, so it prints POST /repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}/attempts with the exact delivery id for each failure and leaves the decision to you. Replaying into a receiver that is still broken just refills the log.
A redelivery arrived and my handler ran twice. Is that expected?
Yes. A replay carries redelivery: true and the same guid as the original attempt, so it is the same event, not a new one. If running it twice caused a visible side effect, the handler is not idempotent, and GitHub's at-least-once delivery would have found that eventually anyway.
Every delivery returns 401. Can the script tell me whether my secret is wrong?
No, and nothing can. The API returns the secret masked as ******** when it is set, so a wrong secret is indistinguishable from a right one at the configuration level. A sustained run of 401 or 403 in the delivery log is the only evidence that exists, which is exactly why the script reports that bucket separately instead of counting it as a generic failure.
Related field notes
- A webhook with no secret sends no signature
- The hook is not subscribed to your event
- The same URL registered twice
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Handling failed webhook deliveries — GitHub Docs
- Repository webhooks — GitHub REST API
- Troubleshooting webhooks — GitHub Docs
- Best practices for using webhooks — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.