Diagnostic Stripe
events still show pending_webhooks hours after they fired
Most payments are processed and some are not. There is no pattern anyone can see: the same customer, the same product, the same code path, and one order exists while the next does not. The endpoint is enabled, the Dashboard shows no banner, and your logs show the handler running successfully for every request it received.
pending_webhooks on the event object is the number of destinations that have not yet returned a 2xx for it. Page GET /v1/events?created[lt]=<now-3600> and flag every event where that count is above zero. An hour after creation, first delivery and the early retries are long finished, so a nonzero value is a delivery that failed rather than one still in flight.
Then group the stuck events by type. That tally is the finding: one type dominating means a single handler branch is failing, while an even spread means the endpoint as a whole is timing out.
The problem in plain words
This is the partial failure between the two states people know how to look for. The endpoint has not been disabled, so nothing draws attention to it. It has not stopped delivering either, so the graph of received webhooks looks approximately normal. Some fraction of events is failing, being retried, failing again, and eventually being dropped, and the fraction is small enough that it reads as flakiness rather than as a bug.
The confusing part is that your own logs support the wrong conclusion. Every request you received you handled correctly, so the handler looks innocent. The requests that failed are not in your logs at all — they timed out before the response, or the load balancer answered before your code did, or your framework issued a redirect. Stripe records those as failures and you record them as nothing.
Why it happens
The handler does its work before it answers. A route that writes to the database, calls a fulfilment API and sends an email before returning 200 is fine at four events an hour and fails at four hundred. Stripe's timeout does not care that the work eventually succeeded; a slow 200 is a failed delivery, and the retry runs the same expensive work again.
Redirects count as failures. A 301 from an https canonicalisation rule, a trailing-slash redirect, or an auth middleware that bounces unauthenticated requests to a login page all return 3xx. Stripe treats a redirect on a webhook request as a failed delivery and does not follow it, so the endpoint looks reachable in a browser and unreachable to Stripe.
One bad branch fails only its own events. If invoice.payment_failed throws on a null customer, only that type accumulates. Everything else delivers perfectly, which is exactly why the failure looks random from the outside: it is not random at all, it is one type, and nobody has grouped it.
Retries hide the start and the scale. Stripe retries with exponential backoff for up to three days, so an event created this morning may still be pending this evening and succeed tomorrow. The count you get is a snapshot of a moving backlog, which is why the age filter matters more than the raw number.
The fix, as a flow
The script asks the API for events older than an hour, then counts the ones still waiting on a 2xx and tallies them by type, because the shape of the backlog is what names the cause.
How to fix it
Page the events with an age filter, not the whole window
GET /v1/events?limit=100&created[lt]=<now minus one hour>. Excluding the last hour at the API rather than in your own loop is what keeps the count meaningful: an event created two minutes ago with pending_webhooks of 1 is a delivery in progress, not a problem.
Count the stuck ones and their share of the sample
The absolute count tells you how much replaying there is to do. The share of the sample tells you what kind of failure it is, which is the more useful of the two.
Group by event type before touching any code
If four fifths of the stuck events are one type, you have found the branch. If they are spread across everything the endpoint receives, the handler is not the problem: the response time is, or the route is answering with a redirect.
Answer first, work second
Return 200 as soon as the signature verifies, then process from a queue. This is the only structural fix; every other change just moves the timeout around. It also makes the retry harmless, because a duplicate delivery enqueues a job your processed-event table will drop.
Replay what was lost, oldest first
GET /v1/events?delivery_success=false paginated, pushed through your own handler in chronological order and guarded by your processed-event table. Do the oldest first: those are the ones closest to leaving the 30 day retention window.
How to check it worked
Re-run the script. Nothing older than an hour should still have deliveries outstanding.
python3 stripe_pending_webhooks.py
# clear 412 event(s) older than the grace period, all deliveredThe full code
One paginated GET and no writes — a restricted key with read access to Events is enough. The classification is a pure function of four numbers: how many events were old enough to judge, how many of those are still outstanding, and which single type accounts for the most of them. That last pair is what separates a broken branch from a slow endpoint, and it should be a visible rule rather than a judgement call made while reading output.
"""Report Stripe events whose deliveries are still outstanding hours after they fired.
Read only. One paginated GET, no writes: give this a RESTRICTED key with read
access to Events. The repair is printed, never performed, because this script
holds a credential to a live payments account.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stripe_pending_webhooks")
API = "https://api.stripe.com/v1"
# An event younger than this is a delivery in progress, not a failure.
GRACE_SECONDS = 3600
# A single type holding this share of the stuck events is a handler branch.
CONCENTRATED = 0.8
# Stuck events at this share of the whole sample is an endpoint level failure.
WIDESPREAD = 0.5
def verdict(sampled, stuck, top_type, top_count):
"""Classify a window of events. Pure, so the rules can be tested offline.
`sampled` is how many events were old enough to judge, `stuck` how many of
those still have pending_webhooks above zero, and `top_type`/`top_count` the
most common type among the stuck ones. Returns (state, detail).
"""
if sampled <= 0:
return ("empty",
"no events older than the grace period in the retained window: "
"nothing here can be judged yet")
if stuck <= 0:
return ("clear",
"%d event(s) older than the grace period, all delivered" % sampled)
share = top_count / float(stuck)
if share >= CONCENTRATED:
return ("one-branch",
"%d of %d stuck event(s) are %s. That is one handler branch "
"failing, not the endpoint." % (top_count, stuck, top_type))
if stuck / float(sampled) >= WIDESPREAD:
return ("endpoint-wide",
"%d of %d sampled event(s) never got a 2xx, across %s and other "
"types. The route is timing out or answering with a redirect."
% (stuck, sampled, top_type))
return ("intermittent",
"%d of %d sampled event(s) stuck, spread across types. This is the "
"handler running out of time under load rather than one bad branch."
% (stuck, sampled))
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from Stripe: the key is wrong, or is for the other mode")
if r.status_code == 403:
raise SystemExit("403 from Stripe: the restricted key lacks read access to "
+ path)
r.raise_for_status()
return r.json()
def scan(session, cutoff, limit):
"""Count events older than `cutoff` and tally the stuck ones by type."""
sampled = 0
stuck = 0
by_type = {}
params = {"limit": 100, "created[lt]": cutoff}
while True:
page = get(session, "/events", **params)
data = page.get("data", [])
for ev in data:
sampled += 1
if (ev.get("pending_webhooks") or 0) > 0:
stuck += 1
t = ev.get("type", "unknown")
by_type[t] = by_type.get(t, 0) + 1
if not data or not page.get("has_more") or sampled >= limit:
break
params["starting_after"] = data[-1]["id"]
return sampled, stuck, by_type
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--max-events", type=int, default=1000,
help="stop after sampling this many events")
ap.add_argument("--grace", type=int, default=GRACE_SECONDS,
help="ignore events younger than this many seconds")
args = ap.parse_args()
key = os.environ.get("STRIPE_API_KEY")
if not key:
log.error("set STRIPE_API_KEY (use a restricted, read-only key)")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + key})
cutoff = int(time.time()) - args.grace
sampled, stuck, by_type = scan(s, cutoff, args.max_events)
# Sorted by count then name so a tie reports the same type on every run.
ranked = sorted(by_type.items(), key=lambda kv: (-kv[1], kv[0]))
top_type, top_count = ranked[0] if ranked else ("none", 0)
state, detail = verdict(sampled, stuck, top_type, top_count)
if state in ("empty", "clear"):
log.info("%-13s %s", state, detail)
return 0
log.warning("%-13s %s", state, detail)
for t, n in ranked[:8]:
log.warning(" %5d %s", n, t)
log.warning(" repair: return 200 as soon as the signature verifies and move "
"the work to a queue. A slow 200 is a failed delivery.")
log.warning(" then replay: GET %s/events?delivery_success=false paginated "
"oldest first, guarded by your processed event table", API)
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Report Stripe events whose deliveries are still outstanding hours after they fired.
*
* Read only. One paginated GET, no writes: give this a RESTRICTED key with read
* access to Events. The repair is printed, never performed.
*/
const API = 'https://api.stripe.com/v1';
const GRACE_SECONDS = 3600;
const CONCENTRATED = 0.8;
const WIDESPREAD = 0.5;
/**
* Classify a window of events. Pure, so the rules can be tested offline.
*/
export function verdict(sampled, stuck, topType, topCount) {
if (sampled <= 0) {
return ['empty',
'no events older than the grace period in the retained window: nothing ' +
'here can be judged yet'];
}
if (stuck <= 0) {
return ['clear', `${sampled} event(s) older than the grace period, all delivered`];
}
const share = topCount / stuck;
if (share >= CONCENTRATED) {
return ['one-branch',
`${topCount} of ${stuck} stuck event(s) are ${topType}. That is one ` +
'handler branch failing, not the endpoint.'];
}
if (stuck / sampled >= WIDESPREAD) {
return ['endpoint-wide',
`${stuck} of ${sampled} sampled event(s) never got a 2xx, across ` +
`${topType} and other types. The route is timing out or answering with a ` +
'redirect.'];
}
return ['intermittent',
`${stuck} of ${sampled} sampled event(s) stuck, spread across types. This is ` +
'the handler running out of time under load rather than one bad branch.'];
}
async function get(key, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401) {
throw new Error('401 from Stripe: the key is wrong, or is for the other mode');
}
if (res.status === 403) {
throw new Error(`403 from Stripe: the restricted key lacks read access to ${path}`);
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
export async function scan(key, cutoff, limit = 1000) {
let sampled = 0;
let stuck = 0;
const byType = new Map();
const params = { limit: 100, 'created[lt]': cutoff };
for (;;) {
const page = await get(key, '/events', params);
const data = page.data ?? [];
for (const ev of data) {
sampled += 1;
if ((ev.pending_webhooks ?? 0) > 0) {
stuck += 1;
const t = ev.type ?? 'unknown';
byType.set(t, (byType.get(t) ?? 0) + 1);
}
}
if (data.length === 0 || !page.has_more || sampled >= limit) break;
params.starting_after = data[data.length - 1].id;
}
return { sampled, stuck, byType };
}
async function main() {
const key = process.env.STRIPE_API_KEY;
if (!key) {
console.error('set STRIPE_API_KEY (use a restricted, read-only key)');
process.exitCode = 2;
return;
}
const cutoff = Math.floor(Date.now() / 1000) - GRACE_SECONDS;
const { sampled, stuck, byType } = await scan(key, cutoff);
// Sorted by count then name so a tie reports the same type on every run.
const ranked = [...byType.entries()].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
const [topType, topCount] = ranked[0] ?? ['none', 0];
const [state, detail] = verdict(sampled, stuck, topType, topCount);
if (state === 'empty' || state === 'clear') {
console.log(`${state.padEnd(13)} ${detail}`);
return;
}
console.warn(`${state.padEnd(13)} ${detail}`);
for (const [t, n] of ranked.slice(0, 8)) {
console.warn(` ${String(n).padStart(5)} ${t}`);
}
console.warn(' repair: return 200 as soon as the signature verifies and move ' +
'the work to a queue. A slow 200 is a failed delivery.');
console.warn(` then replay: GET ${API}/events?delivery_success=false paginated ` +
'oldest first, guarded by your processed event table');
process.exitCode = 1;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The shape of the failure is the finding, so the tests pin the two thresholds that decide it. Eighty percent of the stuck events being one type is a branch; below that it is the endpoint, and which side of the line a given tally falls on has to be a rule rather than an impression. The empty sample is worth a test of its own, because dividing by a stuck count of zero is exactly how a check like this ends up crashing on the healthy accounts.
from stripe_pending_webhooks import verdict
def test_an_empty_sample_reports_nothing_rather_than_dividing_by_zero():
state, _ = verdict(0, 0, "none", 0)
assert state == "empty"
def test_everything_delivered_is_clear():
state, detail = verdict(412, 0, "none", 0)
assert state == "clear"
assert "412" in detail
def test_one_type_dominating_names_the_branch():
state, detail = verdict(500, 40, "invoice.payment_failed", 36)
assert state == "one-branch"
assert "invoice.payment_failed" in detail
def test_the_concentration_threshold_is_inclusive():
# 80 of 100 is a branch; 79 of 100 is not.
assert verdict(1000, 100, "charge.refunded", 80)[0] == "one-branch"
assert verdict(1000, 100, "charge.refunded", 79)[0] != "one-branch"
def test_a_majority_stuck_across_types_is_the_endpoint():
state, detail = verdict(100, 60, "payment_intent.succeeded", 20)
assert state == "endpoint-wide"
assert "redirect" in detail
def test_a_thin_spread_is_load_not_a_bad_branch():
state, detail = verdict(1000, 40, "payment_intent.succeeded", 12)
assert state == "intermittent"
assert "under load" in detail
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { verdict } from './stripe-pending-webhooks.mjs';
test('an empty sample reports nothing rather than dividing by zero', () => {
assert.equal(verdict(0, 0, 'none', 0)[0], 'empty');
});
test('everything delivered is clear', () => {
const [state, detail] = verdict(412, 0, 'none', 0);
assert.equal(state, 'clear');
assert.match(detail, /412/);
});
test('one type dominating names the branch', () => {
const [state, detail] = verdict(500, 40, 'invoice.payment_failed', 36);
assert.equal(state, 'one-branch');
assert.match(detail, /invoice\.payment_failed/);
});
test('the concentration threshold is inclusive', () => {
assert.equal(verdict(1000, 100, 'charge.refunded', 80)[0], 'one-branch');
assert.notEqual(verdict(1000, 100, 'charge.refunded', 79)[0], 'one-branch');
});
test('a majority stuck across types is the endpoint', () => {
const [state, detail] = verdict(100, 60, 'payment_intent.succeeded', 20);
assert.equal(state, 'endpoint-wide');
assert.match(detail, /redirect/);
});
test('a thin spread is load not a bad branch', () => {
const [state, detail] = verdict(1000, 40, 'payment_intent.succeeded', 12);
assert.equal(state, 'intermittent');
assert.match(detail, /under load/);
});
FAQ
What exactly does pending_webhooks count?
The number of destinations that have not yet returned a 2xx for that event. It starts at the number of endpoints subscribed to the type and drops as each one acknowledges, so a value above zero long after the event was created means at least one destination is still failing.
Why ignore events from the last hour?
Because first delivery and the first retries happen inside it. An event created minutes ago with a pending delivery is normal traffic; counting it produces a number that looks alarming on a completely healthy account.
Is a 3xx really a failure?
Yes. Stripe does not follow redirects on webhook requests, so an https canonicalisation rule, a trailing-slash rule, or an auth middleware bouncing to a login page all read as failed deliveries. The URL will look perfectly fine in a browser.
The endpoint is enabled, so how bad is this?
It is the window before it is not. Sustained failures end with Stripe disabling the endpoint after about three days of retries, and once disabled it stops retrying the queued events. Fixing it now costs a replay; fixing it later costs a replay plus a re-enable plus whatever left the retention window.
Can I tell which endpoint is failing when there are several?
Yes, from the event object: it reports the destinations that have not acknowledged it, so you can attribute the stuck count per endpoint rather than guessing. With a single endpoint the type tally is the more useful cut.
Related field notes
- Undelivered events aging out of the 30 day window
- A webhook endpoint sits disabled after days of retries
- Two endpoints on one URL handle every event twice
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- The event object — Stripe API reference
- List all events — Stripe API reference
- Process undelivered events — Stripe Docs
- Receive Stripe events in your webhook endpoint — Stripe Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.