Diagnostic GitHub API
the hook is not subscribed to the event you are waiting for
The handler was written, reviewed, unit tested against a saved payload, and deployed. In production it has never executed once. There is no error anywhere because there is no failure anywhere: the hook was created years ago with push and pull_request, and the event your handler waits for has never been sent to it.
Read GET /repos/{owner}/{repo}/hooks and diff the hook's events array against the set of events your handlers actually implement. An event in your code but not in that array can never arrive, and produces no error of any kind.
Then read GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries?per_page=100 and collect the distinct event values that really appeared in the window. That splits "not subscribed" from "subscribed but nothing has happened", which look the same from your handler's point of view and want completely different responses.
The problem in plain words
Most integration bugs announce themselves. This one is defined by the absence of an event, and absence has no timestamp, no status code and no log line. The handler is not broken; it is simply never invoked, which is indistinguishable from a quiet week.
The usual sequence: a hook is created early on with the two or three events the first feature needed. A year later someone adds a release handler, tests it locally by replaying a payload file, and ships. It works perfectly in every environment where it is exercised by hand, and never fires in the one where it matters. The bug is not in the code that was reviewed; it is in a configuration object nobody opened.
Why it happens
An unsubscribed event is not refused, it is not generated. GitHub delivers only what the hook's events array lists. There is no rejected delivery, no entry in the log, nothing to alert on. The only artefact is a gap, and gaps are not monitored.
Event names and action names get confused. The event is pull_request; opened, closed and synchronize are actions inside its payload. Subscribing to pull_request.opened is not a thing you can do, and a handler registered under that string will not match the header X-GitHub-Event: pull_request either. The same trap catches pull-request with a hyphen, which is how the resource is spelled in URLs and not how the event is spelled anywhere.
Silence is ambiguous without the delivery log. A subscribed event that has not occurred looks exactly like an unsubscribed one. Collecting the distinct event values from recent deliveries turns that into two separate findings: one is a configuration change, the other is patience or a wrong repository.
The wildcard hides the problem and creates another. events: ["*"] means nothing is ever missing, at the cost of receiving every event type GitHub has now and every one it adds later. It converts a subscription bug into a volume and cost problem, which is why the script reports it as its own state rather than as success.
The fix, as a flow
The script compares three lists rather than two: what your receiver implements, what the hook subscribes to, and what actually arrived. The third one is what separates never subscribed from a quiet week.
How to fix it
Write down the events your handlers actually implement
Take the list from the router in your receiver — the switch on X-GitHub-Event — not from memory or from a document. That list is the contract; everything else in this check is a comparison against it. The script takes it as a repeated --handles argument.
Read the hook's events array
GET /repos/{owner}/{repo}/hooks. Compare canonically: lowercase, and treat a hyphen as an underscore, so pull-request in your list is recognised as a spelling of pull_request rather than reported as a missing subscription. Strip anything after a dot, because that is an action name and never a subscription.
Collect the events that were really delivered
GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries?per_page=100 and take the distinct event values. Subscribed and seen is healthy. Subscribed and never seen inside a busy window is worth a look: it is often a hook on a fork, or on the wrong repository in an org that has three with similar names.
Report the traffic no handler implements
The comparison runs both ways. Events arriving that nothing handles are volume you pay to receive, verify and discard — and on a monorepo the push payloads doing that are not small. Those are candidates for removal from the array, not additions to your code.
Add the missing events explicitly, not with a wildcard
Update the hook's events to the exact GitHub names. Resist ["*"]: it subscribes you to every event type that exists now and every one added in future, and it makes this class of bug undetectable by turning every question about coverage into "yes".
How to check it worked
Re-run with the same --handles list. Every handler should map to a subscribed event, and the report should show no unhandled traffic worth removing.
python3 github_hook_event_coverage.py --repo acme/api --handles push --handles pull_request --handles release
# 1 hook(s), 0 handler(s) with no subscription, 0 unhandled event(s) arrivingThe full code
One list request plus one page of deliveries per hook. The whole check is a set comparison, so it is one pure function with the network on either side of it — and it is the normalisation inside that function that does the real work, because pull-request, pull_request.opened and Pull_Request are all the same subscription and none of them is spelled the way GitHub spells it.
"""Compare the events a webhook is subscribed to against the ones you handle.
Read only. Two GETs per hook: the hook list, and one page of its delivery log to
see which events really arrived. An unsubscribed event produces no failure and no
delivery record, so the only way to find one is to compare two lists.
"""
import argparse
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_hook_event_coverage")
API = "https://api.github.com"
UA = "github-hook-event-coverage/1.0"
def normalize(name):
"""Canonical form of an event name. Pure.
Three spellings reach this function and none of them is always right. GitHub
names events with underscores (pull_request), URLs use hyphens, and a handler
is often registered under an action (pull_request.opened) which is a field
inside the payload rather than something a hook can subscribe to.
"""
base = str(name or "").strip().lower().replace("-", "_")
if "." in base:
base = base.split(".", 1)[0]
return base
def coverage(handled, subscribed, seen=()):
"""Compare handlers, subscriptions and observed traffic. Pure.
Returns a list of rows, one per event on either side, each with a state:
missing subscribed nowhere, so the handler can never run
delivered subscribed and seen in the delivery window
quiet subscribed but not seen, which may just mean nothing happened
wildcard the hook subscribes to everything, including future events
unhandled arriving or subscribed with no handler behind it
"""
subs = {}
wildcard = False
for raw in subscribed or []:
if str(raw).strip() == "*":
wildcard = True
continue
subs[normalize(raw)] = str(raw)
seen_events = {}
for raw in seen or []:
key = normalize(raw)
seen_events[key] = seen_events.get(key, 0) + 1
rows = []
claimed = set()
for raw in handled or []:
key = normalize(raw)
claimed.add(key)
note = ""
if str(raw) != key:
note = "your handler is registered as %r; GitHub spells this %r" % (
str(raw), key)
if wildcard:
state = "wildcard"
elif key not in subs:
state = "missing"
elif key in seen_events:
state = "delivered"
else:
state = "quiet"
rows.append({"event": key, "handler": str(raw), "state": state,
"seen": seen_events.get(key, 0), "note": note})
for key in sorted(set(subs) | set(seen_events)):
if key in claimed:
continue
rows.append({"event": key, "handler": None, "state": "unhandled",
"seen": seen_events.get(key, 0),
"note": "subscribed" if key in subs else "arriving without a subscription"})
return rows
def next_link(response):
"""The rel=next URL from the Link header, or None."""
for part in (response.headers.get("Link") or "").split(","):
chunk = part.strip()
if chunk.startswith("<") and chunk.endswith('rel="next"'):
return chunk[1:chunk.index(">")]
return None
def get(session, url, **params):
r = session.get(url, params=params, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from GitHub: GITHUB_TOKEN is missing, expired or "
"malformed")
if r.status_code in (403, 404):
raise SystemExit("%d from %s: reading hooks needs admin:repo_hook, and "
"GitHub answers 404 rather than 403 when the token "
"cannot see the resource at all" % (r.status_code, url))
r.raise_for_status()
return r
def page(session, url, limit=500, **params):
out = []
while url and len(out) < limit:
r = get(session, url, **params)
out.extend(r.json())
url, params = next_link(r), {}
return out[:limit]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--handles", action="append", default=[],
help="an event your receiver implements; repeat per event")
ap.add_argument("--max-deliveries", type=int, default=200,
help="deliveries to read per hook when collecting the "
"events that really arrived")
args = ap.parse_args()
if not args.handles:
log.error("pass --handles once per event your receiver implements, "
"taken from its switch on X-GitHub-Event")
return 2
token = os.environ.get("GITHUB_TOKEN")
if not token:
log.error("set GITHUB_TOKEN (a read-only token is enough)")
return 2
owner, _, name = args.repo.partition("/")
if not (owner and name):
log.error("--repo takes owner/name, for example acme/api")
return 2
session = requests.Session()
session.headers.update({
"Authorization": "Bearer " + token,
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": UA,
})
base = "%s/repos/%s/%s/hooks" % (API, owner, name)
hooks = page(session, base, per_page=100)
if not hooks:
log.info("no webhooks on %s that this token can see", args.repo)
return 0
missing = unhandled = 0
for hook in hooks:
hid = hook.get("id")
url = (hook.get("config") or {}).get("url", "?")
subscribed = hook.get("events") or []
deliveries = page(session, "%s/%s/deliveries" % (base, hid),
limit=args.max_deliveries, per_page=100)
seen = [d.get("event") for d in deliveries]
log.info("hook %s %s subscribes to %d event(s), %d delivery(ies) read",
hid, url, len(subscribed), len(deliveries))
for row in coverage(args.handles, subscribed, seen):
line = " %-10s %s%s" % (row["state"], row["event"],
" " + row["note"] if row["note"] else "")
if row["state"] in ("delivered", "quiet"):
log.info(line)
continue
log.warning(line)
if row["state"] == "missing":
missing += 1
log.warning(" repair: add %r to this hook's events array; "
"until then the handler cannot run and nothing will "
"report an error", row["event"])
elif row["state"] == "unhandled":
unhandled += 1
log.warning(" %d delivery(ies) of an event nothing handles: "
"volume you receive, verify and discard",
row["seen"])
elif row["state"] == "wildcard":
log.warning(" the hook subscribes to *, so this arrives "
"along with every event type GitHub adds in future")
log.info("%d hook(s), %d handler(s) with no subscription, %d unhandled "
"event(s) arriving", len(hooks), missing, unhandled)
return 1 if missing else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Compare the events a webhook is subscribed to against the ones you handle.
*
* Read only. Two GETs per hook: the hook list, and one page of its delivery log.
* An unsubscribed event produces no failure and no delivery record, so the only
* way to find one is to compare two lists.
*/
const API = 'https://api.github.com';
const UA = 'github-hook-event-coverage/1.0';
/**
* Canonical form of an event name. Pure. GitHub names events with underscores,
* URLs use hyphens, and handlers are often registered under an action
* (pull_request.opened), which is a payload field and not a subscription.
*/
export function normalize(name) {
const base = String(name ?? '').trim().toLowerCase().replaceAll('-', '_');
return base.includes('.') ? base.slice(0, base.indexOf('.')) : base;
}
/**
* Compare handlers, subscriptions and observed traffic. Pure. States:
* missing, delivered, quiet, wildcard, unhandled.
*/
export function coverage(handled, subscribed, seen = []) {
const subs = new Map();
let wildcard = false;
for (const raw of subscribed ?? []) {
if (String(raw).trim() === '*') { wildcard = true; continue; }
subs.set(normalize(raw), String(raw));
}
const seenEvents = new Map();
for (const raw of seen ?? []) {
const key = normalize(raw);
seenEvents.set(key, (seenEvents.get(key) ?? 0) + 1);
}
const rows = [];
const claimed = new Set();
for (const raw of handled ?? []) {
const key = normalize(raw);
claimed.add(key);
const note = String(raw) !== key
? `your handler is registered as '${raw}'; GitHub spells this '${key}'`
: '';
let state;
if (wildcard) state = 'wildcard';
else if (!subs.has(key)) state = 'missing';
else if (seenEvents.has(key)) state = 'delivered';
else state = 'quiet';
rows.push({ event: key, handler: String(raw), state,
seen: seenEvents.get(key) ?? 0, note });
}
for (const key of [...new Set([...subs.keys(), ...seenEvents.keys()])].sort()) {
if (claimed.has(key)) continue;
rows.push({ event: key, handler: null, state: 'unhandled',
seen: seenEvents.get(key) ?? 0,
note: subs.has(key) ? 'subscribed' : 'arriving without a subscription' });
}
return rows;
}
function nextLink(res) {
for (const part of (res.headers.get('link') ?? '').split(',')) {
const chunk = part.trim();
if (chunk.startsWith('<') && chunk.endsWith('rel="next"')) {
return chunk.slice(1, chunk.indexOf('>'));
}
}
return null;
}
async function get(token, url) {
const res = await fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': UA,
},
});
if (res.status === 401) {
throw new Error('401 from GitHub: GITHUB_TOKEN is missing, expired or malformed');
}
if (res.status === 403 || res.status === 404) {
throw new Error(`${res.status} from ${url}: reading hooks needs ` +
'admin:repo_hook, and GitHub answers 404 rather than 403 when the token ' +
'cannot see the resource at all');
}
if (!res.ok) throw new Error(`${res.status} from ${url}`);
return res;
}
async function page(token, url, limit = 500) {
const out = [];
let next = url;
while (next && out.length < limit) {
const res = await get(token, next);
out.push(...(await res.json()));
next = nextLink(res);
}
return out.slice(0, limit);
}
async function main() {
const [repo, ...handles] = process.argv.slice(2);
const token = process.env.GITHUB_TOKEN;
if (!token) {
console.error('set GITHUB_TOKEN (a read-only token is enough)');
process.exitCode = 2;
return;
}
if (!repo || !repo.includes('/') || handles.length === 0) {
console.error('usage: node github-hook-event-coverage.mjs owner/name push pull_request');
process.exitCode = 2;
return;
}
const base = `${API}/repos/${repo}/hooks`;
const hooks = await page(token, `${base}?per_page=100`);
if (hooks.length === 0) {
console.log(`no webhooks on ${repo} that this token can see`);
return;
}
let missing = 0;
let unhandled = 0;
for (const hook of hooks) {
const url = hook.config?.url ?? '?';
const subscribed = hook.events ?? [];
const deliveries = await page(token,
`${base}/${hook.id}/deliveries?per_page=100`, 200);
const seen = deliveries.map((d) => d.event);
console.log(`hook ${hook.id} ${url} subscribes to ${subscribed.length} ` +
`event(s), ${deliveries.length} delivery(ies) read`);
for (const row of coverage(handles, subscribed, seen)) {
const line = ` ${row.state.padEnd(10)} ${row.event}` +
(row.note ? ` ${row.note}` : '');
if (row.state === 'delivered' || row.state === 'quiet') {
console.log(line);
continue;
}
console.warn(line);
if (row.state === 'missing') {
missing += 1;
console.warn(` repair: add '${row.event}' to this hook's events ` +
'array; until then the handler cannot run and nothing will report an error');
} else if (row.state === 'unhandled') {
unhandled += 1;
console.warn(` ${row.seen} delivery(ies) of an event nothing ` +
'handles: volume you receive, verify and discard');
} else if (row.state === 'wildcard') {
console.warn(' the hook subscribes to *, so this arrives along with ' +
'every event type GitHub adds in future');
}
}
}
console.log(`${hooks.length} hook(s), ${missing} handler(s) with no ` +
`subscription, ${unhandled} unhandled event(s) arriving`);
process.exitCode = missing ? 1 : 0;
}
// Only run when invoked directly, so importing this module from the test file
// does not run main(), fail on the missing token, and fail the test file with it.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Two cases carry the note. A handler for an event the hook does not carry has to come out as missing even though nothing anywhere has failed, and a handler registered as pull_request.opened has to be recognised as the pull_request subscription it already has rather than reported as a second missing event that no one can add.
from github_hook_event_coverage import coverage, normalize
def rows_by_event(rows):
return {r["event"]: r for r in rows}
def test_normalize_accepts_the_three_spellings_people_use():
assert normalize("pull_request") == "pull_request"
assert normalize("pull-request") == "pull_request"
assert normalize("Pull_Request.opened") == "pull_request"
assert normalize(None) == ""
def test_an_unsubscribed_handler_is_the_finding():
rows = rows_by_event(coverage(["release"], ["push", "pull_request"], ["push"]))
assert rows["release"]["state"] == "missing"
def test_an_action_suffix_matches_the_event_it_belongs_to():
# pull_request.opened is not something a hook can subscribe to, and treating
# it as a separate event invents a repair that cannot be carried out.
rows = rows_by_event(coverage(["pull_request.opened"], ["pull_request"],
["pull_request"]))
assert rows["pull_request"]["state"] == "delivered"
assert "GitHub spells this" in rows["pull_request"]["note"]
def test_subscribed_but_unseen_is_not_the_same_as_unsubscribed():
rows = rows_by_event(coverage(["release"], ["release", "push"], ["push"]))
assert rows["release"]["state"] == "quiet"
def test_a_wildcard_is_reported_rather_than_counted_as_success():
rows = rows_by_event(coverage(["release"], ["*"], ["push"]))
assert rows["release"]["state"] == "wildcard"
def test_traffic_nothing_handles_is_reported_too():
rows = rows_by_event(coverage(["push"], ["push", "status"],
["push", "status", "status"]))
assert rows["status"]["state"] == "unhandled"
assert rows["status"]["seen"] == 2
assert rows["push"]["state"] == "delivered"
def test_an_event_arriving_without_a_subscription_is_still_surfaced():
rows = rows_by_event(coverage(["push"], ["push"], ["push", "ping"]))
assert rows["ping"]["state"] == "unhandled"
assert "without a subscription" in rows["ping"]["note"]
def test_case_and_hyphens_do_not_create_phantom_findings():
rows = coverage(["Pull-Request"], ["pull_request"], ["pull_request"])
assert [r["state"] for r in rows] == ["delivered"]
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { coverage, normalize } from './github-hook-event-coverage.mjs';
const byEvent = (rows) => Object.fromEntries(rows.map((r) => [r.event, r]));
test('normalize accepts the three spellings people use', () => {
assert.equal(normalize('pull_request'), 'pull_request');
assert.equal(normalize('pull-request'), 'pull_request');
assert.equal(normalize('Pull_Request.opened'), 'pull_request');
assert.equal(normalize(null), '');
});
test('an unsubscribed handler is the finding', () => {
const rows = byEvent(coverage(['release'], ['push', 'pull_request'], ['push']));
assert.equal(rows.release.state, 'missing');
});
test('an action suffix matches the event it belongs to', () => {
const rows = byEvent(coverage(['pull_request.opened'], ['pull_request'],
['pull_request']));
assert.equal(rows.pull_request.state, 'delivered');
assert.match(rows.pull_request.note, /GitHub spells this/);
});
test('subscribed but unseen is not the same as unsubscribed', () => {
const rows = byEvent(coverage(['release'], ['release', 'push'], ['push']));
assert.equal(rows.release.state, 'quiet');
});
test('a wildcard is reported rather than counted as success', () => {
const rows = byEvent(coverage(['release'], ['*'], ['push']));
assert.equal(rows.release.state, 'wildcard');
});
test('traffic nothing handles is reported too', () => {
const rows = byEvent(coverage(['push'], ['push', 'status'],
['push', 'status', 'status']));
assert.equal(rows.status.state, 'unhandled');
assert.equal(rows.status.seen, 2);
assert.equal(rows.push.state, 'delivered');
});
test('an event arriving without a subscription is still surfaced', () => {
const rows = byEvent(coverage(['push'], ['push'], ['push', 'ping']));
assert.equal(rows.ping.state, 'unhandled');
assert.match(rows.ping.note, /without a subscription/);
});
test('case and hyphens do not create phantom findings', () => {
const rows = coverage(['Pull-Request'], ['pull_request'], ['pull_request']);
assert.deepEqual(rows.map((r) => r.state), ['delivered']);
});
FAQ
Why is there no error when an event is not subscribed?
Because nothing happens. GitHub generates a delivery only for events in the hook's events array; an unsubscribed event is not refused, it is never created. There is no failed delivery, no status code and nothing to alert on, which is why this has to be found by comparing two lists rather than by watching for errors.
Can I subscribe to pull_request.opened?
No. pull_request is the event; opened is the action field inside its payload. Hooks subscribe to events, and your receiver branches on the action after it has already been handed the delivery. The script normalises an action suffix back to its event so that a handler named this way is not reported as an unfixable missing subscription.
The event is in the array but I have never received it. What now?
That is the quiet state, and it has two ordinary explanations: the event genuinely has not occurred in the retained window, or the hook is on a different repository than you think, most often a fork or a similarly named repo in the same org. Check the hook's own repository before changing any configuration.
Should I just subscribe to everything with a wildcard?
It removes this bug and adds two others. You receive every event type GitHub has and every one it ships later, so a monorepo's push payloads dominate your receiver's time and the amount of repository data leaving GitHub grows for no benefit. The script reports a wildcard as its own state rather than as coverage.
Does this work for organization hooks and GitHub App events too?
The same comparison applies, with a different source list. Org hooks have their own events array on GET /orgs/{org}/hooks. A GitHub App's subscriptions are set on the App itself rather than per installation, so they are not in this endpoint at all and are read with the App's own credentials.
Related field notes
- The same URL registered twice
- Deliveries failing where nobody reads the log
- A webhook with no secret sends no signature
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Webhook events and payloads — GitHub Docs
- Troubleshooting webhooks — GitHub Docs
- Repository webhooks — GitHub REST API
- Creating webhooks — GitHub Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.