Diagnostic Stripe

a live webhook endpoint points at a dev tunnel or http

It worked for exactly one afternoon. The endpoint was registered during a development session, against a tunnel that happened to be running at the time; the tunnel was closed at the end of the day and the hostname stopped resolving. Every delivery attempt since then has failed before it reached any code you wrote.

Read-only key Python and Node.js Tests included

Woman standing and holding smartphones — Photo by Clay Banks on Unsplash

The short answer

Read GET /v1/webhook_endpoints and judge the url of every entry with livemode true. Stripe delivers from its own network over HTTPS, so the host has to be a publicly resolvable name with a valid certificate. A tunnel hostname, a localhost, or an RFC1918 address cannot be any of those things.

An http:// URL in a live configuration is the same failure written differently: deliveries go over TLS 1.2 or 1.3, and there is nothing at the other end of a plaintext port to negotiate one.

The problem in plain words

The distinctive part of this failure is that it produces no partial success. A handler with a bug fulfils some orders and not others; an endpoint pointed at a hostname that does not resolve fulfils none, ever, from the first event onwards. Stripe reports it as a connection error rather than an application error, which means it never appears in your logs, your error tracker, or your uptime monitor — the request does not reach any of them.

It also has an expiry date. Stripe retries a failing destination for up to three days in live mode and then disables it, so a tunnel URL left in the live configuration turns into a disabled endpoint about seventy-two hours after the last payment that anybody was watching.

Nothing reaches your code, so nothing appears in your logs, your error tracker or your uptime monitor.

Why it happens

A tunnel is the correct tool for local development and a terrible one to leave behind. ngrok, loca.lt, trycloudflare.com and friends exist precisely so Stripe can reach a laptop, and they work perfectly for that. The hostname is allocated per session on the free tiers, so it is valid for one sitting and dead afterwards. What survives is the endpoint object holding it.

Test-mode and live-mode endpoints are separate objects, and people copy between them. The URL that is right in test mode is very often the one that gets pasted into the live configuration during a rushed launch, because the two forms look identical and the mode is a toggle somewhere else on the screen.

Nothing validates reachability at registration time. Creating an endpoint records a string. Stripe does not have to resolve or connect to the host to accept it, so a typo, a staging hostname, or an internal address is stored without complaint and fails silently later, one event at a time.

The host and the handshake fail identically from the outside. An expired certificate, a self-signed one, or a server that will only speak TLS 1.0 produces the same result as an unresolvable name: nothing arrives. The API tells you about the host-shaped problems, because the URL is a field you can read. The handshake-shaped ones need a TLS test against the host itself.

The fix, as a flow

The script parses the hostname out of every endpoint URL before judging it, because the mode decides the verdict: the same tunnel name is a working day in test mode and an outage in live.

Judged on the parsed hostname, not the string. A suffix match keeps localhost-tools.example.com out of the findings.

How to fix it

List every endpoint and read url alongside livemode

The mode matters more than usual here. A tunnel hostname on a test-mode endpoint is somebody working, exactly as intended. The same string on a live-mode endpoint is an outage.

Judge the host, not the URL string

Parse out the hostname and decide on that. https://myapp.eu.ngrok.io/stripe/webhook is HTTPS, has a certificate, and looks entirely respectable as a string. It is still a tunnel. The reverse case matters too: a substring match on localhost flags a perfectly good https://localhost-tools.example.com.

Separate cannot resolve from cannot handshake

This is where the read-only check stops. If the host is public and the scheme is right, the remaining failure modes are certificate and protocol ones, and you find those with a TLS server test against the hostname rather than through the Stripe API.

Repoint the endpoint rather than recreating it

Updating the endpoint's url keeps the signing secret, so nothing has to be redeployed. Deleting and recreating gives you a new whsec_ and an unplanned config change on the same afternoon you are already fixing an outage. If the endpoint is purely a development leftover, remove it instead of repointing it.

Use the CLI listener for local work, not a registered tunnel

stripe listen --forward-to localhost:4242/stripe/webhook creates an ephemeral destination and prints its own signing secret. Nothing persists in the account afterwards, which is the whole point: there is no object left behind to be found in live mode six months later.

How to check it worked

Re-run the script. Every live endpoint should report a public HTTPS host.

python3 stripe_webhook_url_check.py
# ok        https://example.com/stripe/webhook  public https host
# 2 endpoint(s), 0 unreachable

The full code

One GET and no writes — a restricted key with read access to Webhook Endpoints is the whole requirement. The classifier takes the URL and the mode and nothing else, so the interesting rule is visible: the same tunnel hostname is a finding in live mode and ordinary in test mode, and that distinction is the one a grep for ngrok gets wrong.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Read only, it never writes. One of 114 Stripe fixes, free and open source.

stripe_webhook_url_check.py

"""Report Stripe webhook endpoints Stripe cannot reach: tunnels, localhost, http.

Read only. One GET, no writes: give this a RESTRICTED key with read access to
Webhook Endpoints. The repair is printed, never performed, because this script
holds a credential to a live payments account.
"""
import argparse
import logging
import os
import sys
from urllib.parse import urlparse

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stripe_webhook_url_check")

API = "https://api.stripe.com/v1"

# Hostname suffixes handed out by development tunnels. Matched as suffixes, not
# substrings: https://localhost-tools.example.com is a real production host.
TUNNELS = (".ngrok.io", ".ngrok-free.app", ".ngrok.app", ".ngrok.dev",
           ".loca.lt", ".trycloudflare.com", ".serveo.net")

LOOPBACK = {"localhost", "127.0.0.1", "0.0.0.0", "::1", "[::1]"}


def split_url(url):
    """(scheme, host) lowercased, or ('', '') when the URL is not usable."""
    try:
        parts = urlparse(url or "")
    except ValueError:
        return ("", "")
    if not parts.scheme or not parts.hostname:
        return ("", "")
    return (parts.scheme.lower(), parts.hostname.lower())


def unroutable_ip(host):
    """True for loopback and RFC1918 literals, which Stripe can never reach."""
    octets = host.split(".")
    if len(octets) != 4 or not all(o.isdigit() for o in octets):
        return False
    a, b = int(octets[0]), int(octets[1])
    return a == 10 or a == 127 or (a == 172 and 16 <= b <= 31) or (a == 192 and b == 168)


def verdict(url, livemode):
    """Classify one endpoint URL. Pure, so the rules can be tested offline.

    Returns (state, detail). The mode is part of the input on purpose: a tunnel
    hostname is how local development works and is only a fault in live mode.
    """
    scheme, host = split_url(url)
    if not host:
        return ("unparseable", "%r is not a URL with a scheme and a host" % (url,))

    if host in LOOPBACK or unroutable_ip(host):
        kind = ("unroutable",
                "%s is not reachable from outside your network, so no event has "
                "ever arrived and none will." % host)
    elif any(host == t.lstrip(".") or host.endswith(t) for t in TUNNELS):
        kind = ("tunnel",
                "%s is a development tunnel host. It resolves only while the "
                "tunnel process is running and the name changes when it restarts."
                % host)
    elif scheme != "https":
        kind = ("plaintext",
                "the scheme is %s. Stripe delivers over HTTPS with TLS 1.2 or "
                "1.3, and there is nothing to negotiate on a plaintext port."
                % scheme)
    else:
        return ("ok", "public https host")

    if not livemode:
        return ("dev",
                "test mode: %s Expected while developing. The risk is this URL "
                "being copied into the live endpoint." % kind[1])
    return kind


def get(session, path, **params):
    r = session.get(API + path, params=params, timeout=30)
    if r.status_code == 401:
        raise SystemExit("401 from Stripe: the key is wrong, or is for the other mode")
    r.raise_for_status()
    return r.json()


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--include-test-mode", action="store_true",
                    help="report test-mode endpoints too, as information")
    args = ap.parse_args()

    key = os.environ.get("STRIPE_API_KEY")
    if not key:
        log.error("set STRIPE_API_KEY (use a restricted, read-only key)")
        return 2

    s = requests.Session()
    s.headers.update({"Authorization": "Bearer " + key})

    endpoints = get(s, "/webhook_endpoints", limit=100).get("data", [])
    if not endpoints:
        log.info("no webhook endpoints configured for this key's mode")
        return 0

    bad = 0
    for ep in endpoints:
        state, detail = verdict(ep.get("url"), bool(ep.get("livemode")))
        line = "%-11s %s  %s" % (state, ep.get("url", "?"), detail)
        if state == "ok":
            log.info(line)
            continue
        if state == "dev":
            if args.include_test_mode:
                log.info(line)
            continue
        bad += 1
        log.warning(line)
        log.warning("  repair: update %s/webhook_endpoints/%s with "
                    "url=https://<your-domain>/stripe/webhook, which keeps the "
                    "signing secret", API, ep["id"])
        log.warning("  or remove the endpoint if it is a development leftover, "
                    "and use: stripe listen --forward-to localhost:4242/webhook")

    log.info("%d endpoint(s), %d unreachable", len(endpoints), bad)
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())

stripe-webhook-url-check.mjs

/**
 * Report Stripe webhook endpoints Stripe cannot reach: tunnels, localhost, http.
 *
 * Read only. One GET, no writes: give this a RESTRICTED key with read access to
 * Webhook Endpoints. The repair is printed, never performed.
 */
const API = 'https://api.stripe.com/v1';

// Hostname suffixes handed out by development tunnels. Matched as suffixes, not
// substrings: https://localhost-tools.example.com is a real production host.
const TUNNELS = ['.ngrok.io', '.ngrok-free.app', '.ngrok.app', '.ngrok.dev',
  '.loca.lt', '.trycloudflare.com', '.serveo.net'];

const LOOPBACK = new Set(['localhost', '127.0.0.1', '0.0.0.0', '::1', '[::1]']);

export function splitUrl(url) {
  try {
    const parsed = new URL(url ?? '');
    const host = parsed.hostname.toLowerCase().replace(/^\[|\]$/g, '');
    if (!host) return ['', ''];
    return [parsed.protocol.replace(':', '').toLowerCase(), host];
  } catch {
    return ['', ''];
  }
}

/** True for loopback and RFC1918 literals, which Stripe can never reach. */
export function unroutableIp(host) {
  const octets = host.split('.');
  if (octets.length !== 4 || !octets.every((o) => /^\d+$/.test(o))) return false;
  const [a, b] = [Number(octets[0]), Number(octets[1])];
  return a === 10 || a === 127 || (a === 172 && b >= 16 && b <= 31)
    || (a === 192 && b === 168);
}

/**
 * Classify one endpoint URL. Pure, so the rules can be tested offline.
 * The mode is part of the input on purpose: a tunnel hostname is how local
 * development works and is only a fault in live mode.
 */
export function verdict(url, livemode) {
  const [scheme, host] = splitUrl(url);
  if (!host) return ['unparseable', `${JSON.stringify(url)} is not a URL with a scheme and a host`];

  let kind;
  if (LOOPBACK.has(host) || unroutableIp(host)) {
    kind = ['unroutable',
      `${host} is not reachable from outside your network, so no event has ever ` +
      'arrived and none will.'];
  } else if (TUNNELS.some((t) => host === t.slice(1) || host.endsWith(t))) {
    kind = ['tunnel',
      `${host} is a development tunnel host. It resolves only while the tunnel ` +
      'process is running and the name changes when it restarts.'];
  } else if (scheme !== 'https') {
    kind = ['plaintext',
      `the scheme is ${scheme}. Stripe delivers over HTTPS with TLS 1.2 or 1.3, ` +
      'and there is nothing to negotiate on a plaintext port.'];
  } else {
    return ['ok', 'public https host'];
  }

  if (!livemode) {
    return ['dev',
      `test mode: ${kind[1]} Expected while developing. The risk is this URL ` +
      'being copied into the live endpoint.'];
  }
  return kind;
}

async function get(key, path, params = {}) {
  const url = new URL(API + path);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
  if (res.status === 401) {
    throw new Error('401 from Stripe: the key is wrong, or is for the other mode');
  }
  if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
  return res.json();
}

async function main() {
  const key = process.env.STRIPE_API_KEY;
  if (!key) {
    console.error('set STRIPE_API_KEY (use a restricted, read-only key)');
    process.exitCode = 2;
    return;
  }

  const { data: endpoints = [] } = await get(key, '/webhook_endpoints', { limit: 100 });
  if (endpoints.length === 0) {
    console.log("no webhook endpoints configured for this key's mode");
    return;
  }

  const includeTest = process.argv.includes('--include-test-mode');
  let bad = 0;
  for (const ep of endpoints) {
    const [state, detail] = verdict(ep.url, Boolean(ep.livemode));
    const line = `${state.padEnd(11)} ${ep.url ?? '?'}  ${detail}`;
    if (state === 'ok') { console.log(line); continue; }
    if (state === 'dev') { if (includeTest) console.log(line); continue; }
    bad += 1;
    console.warn(line);
    console.warn(`  repair: update ${API}/webhook_endpoints/${ep.id} with ` +
                 'url=https://<your-domain>/stripe/webhook, which keeps the signing secret');
    console.warn('  or remove the endpoint if it is a development leftover, and use: ' +
                 'stripe listen --forward-to localhost:4242/webhook');
  }

  console.log(`${endpoints.length} endpoint(s), ${bad} unreachable`);
  process.exitCode = bad ? 1 : 0;
}

// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}

Add a test

Two cases carry the note. The first is the same tunnel URL in both modes: in test mode it is somebody working, in live mode it is an outage, and a check that cannot tell them apart is either noisy or useless. The second is suffix matching — localhost-tools.example.com is a production hostname that any substring check flags, and a false positive on a working endpoint is how a check gets ignored.

test_stripe_webhook_url_check.py

from stripe_webhook_url_check import verdict


def test_a_public_https_url_is_fine():
    state, _ = verdict("https://example.com/stripe/webhook", True)
    assert state == "ok"


def test_a_tunnel_host_in_live_mode_is_flagged():
    state, detail = verdict("https://a1b2.eu.ngrok.io/stripe/webhook", True)
    assert state == "tunnel"
    assert "ngrok.io" in detail


def test_the_same_tunnel_host_in_test_mode_is_not_a_fault():
    # The whole point of a tunnel is local development. Only live mode matters.
    state, _ = verdict("https://a1b2.eu.ngrok.io/stripe/webhook", False)
    assert state == "dev"


def test_a_private_address_is_unroutable():
    assert verdict("https://10.4.1.9/stripe/webhook", True)[0] == "unroutable"
    assert verdict("http://localhost:4242/webhook", True)[0] == "unroutable"


def test_plain_http_on_a_public_host_is_flagged():
    state, detail = verdict("http://example.com/stripe/webhook", True)
    assert state == "plaintext"
    assert "TLS 1.2" in detail


def test_a_hostname_containing_localhost_is_not_flagged():
    # Suffix matching, not substring: this is a real production host.
    assert verdict("https://localhost-tools.example.com/hook", True)[0] == "ok"


def test_a_missing_url_is_reported_not_passed():
    assert verdict(None, True)[0] == "unparseable"
    assert verdict("example.com/webhook", True)[0] == "unparseable"

stripe-webhook-url-check.test.mjs

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { verdict } from './stripe-webhook-url-check.mjs';

test('a public https url is fine', () => {
  assert.equal(verdict('https://example.com/stripe/webhook', true)[0], 'ok');
});

test('a tunnel host in live mode is flagged', () => {
  const [state, detail] = verdict('https://a1b2.eu.ngrok.io/stripe/webhook', true);
  assert.equal(state, 'tunnel');
  assert.match(detail, /ngrok\.io/);
});

test('the same tunnel host in test mode is not a fault', () => {
  assert.equal(verdict('https://a1b2.eu.ngrok.io/stripe/webhook', false)[0], 'dev');
});

test('a private address is unroutable', () => {
  assert.equal(verdict('https://10.4.1.9/stripe/webhook', true)[0], 'unroutable');
  assert.equal(verdict('http://localhost:4242/webhook', true)[0], 'unroutable');
});

test('plain http on a public host is flagged', () => {
  const [state, detail] = verdict('http://example.com/stripe/webhook', true);
  assert.equal(state, 'plaintext');
  assert.match(detail, /TLS 1\.2/);
});

test('a hostname containing localhost is not flagged', () => {
  assert.equal(verdict('https://localhost-tools.example.com/hook', true)[0], 'ok');
});

test('a missing url is reported, not passed', () => {
  assert.equal(verdict(null, true)[0], 'unparseable');
  assert.equal(verdict('example.com/webhook', true)[0], 'unparseable');
});

FAQ

Can a Stripe webhook endpoint use http instead of https?

Not usefully in live mode. Stripe delivers over HTTPS and supports TLS 1.2 and 1.3, so a plaintext URL has nothing to negotiate with. An http:// URL sitting in a live configuration is almost always a test-mode value that was copied across.

Why did my ngrok endpoint work once and then never again?

Because the hostname was allocated for that tunnel session. On the free tiers a new hostname is issued each time the tunnel starts, so the URL stored on the endpoint object stops resolving the moment you close the terminal. The endpoint keeps trying to reach a name that no longer exists.

Can Stripe deliver to localhost or a private IP if I open a firewall port?

No. Stripe delivers from its own network, so the host has to resolve publicly. 127.0.0.1 and RFC1918 addresses mean something different on every network, including Stripe's. For local work use stripe listen, which forwards from the CLI process rather than registering an endpoint.

If I fix the URL, do the missed events get delivered?

No. Fixing the URL only affects future deliveries. Events are retained for 30 days and can be fetched with GET /v1/events?delivery_success=false and replayed through your own handler, but nothing is resent automatically.

Does changing the url change the signing secret?

No, as long as you update the existing endpoint. The secret belongs to the endpoint object, so a URL change keeps it and needs no redeploy. Deleting and recreating issues a new whsec_ that every environment then has to be given.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.