allanninal.dev/build

Part 2 of 7 · Dunning recovery series ~8 min read

How a failed charge gets caught

Everything downstream depends on hearing about the failed charge promptly and exactly once. This post is about the front door: a Stripe-style webhook lands on a Lambda Function URL, the system verifies it really came from the processor, throws away duplicates and replays, and records that this subscription has entered dunning — without trusting anything the request claims about money.

Key takeaways

  • The processor fires a signed webhook the instant a renewal charge fails; a Lambda Function URL catches it.
  • The signature is verified against the webhook secret before anything is trusted — an unsigned request does nothing.
  • Events are de-duplicated by id, because processors retry their own webhooks and replays are normal.
  • The handler trusts no amount in the request; it records ids and lets the retrier read real figures back from the processor.
  • It returns 200 in milliseconds and hands the work to SQS, so the processor never sees a slow endpoint.

The front door is a webhook, not a poll

The system finds out about a failed charge the same way everything else in a Stripe-style processor finds out: a webhook. When a subscription’s renewal charge is declined, the processor immediately fires an event — invoice.payment_failed, or your processor’s equivalent — to an HTTPS endpoint you registered. The alternative, polling the processor every few minutes asking “anything failed yet?”, would be slower, cost more, and miss the exact moment. So the front door is a single endpoint that listens.

That endpoint is a Lambda Function URL — a plain HTTPS address that invokes dunr-webhook directly. No API Gateway sits in front of it; for one webhook receiver, the Function URL does the whole job and costs nothing per request. The processor is configured to send the four events that matter: a charge failed, a charge succeeded, a payment method was attached or updated, and a subscription was cancelled in the billing tool. Everything the system does begins here.

Three checks before anything is trusted

A public URL that puts customers into a payment-recovery flow is exactly the kind of thing you don’t want fired by a random request. So dunr-webhook does three things, in order, before it believes a word of the payload.

  • Verify the signature. The processor signs every webhook with a shared secret and puts the signature in a header. The handler reads the webhook signing secret from Secrets Manager (dunr/stripe/webhook-secret), recomputes the HMAC over the raw request body and the timestamp, and compares. If it doesn’t match — or the timestamp is older than five minutes, which blocks replay of a captured request — the handler returns 400 and stops. An attacker who doesn’t have the secret cannot make this system do anything.
  • De-duplicate by event id. Processors guarantee at-least-once webhook delivery, which means they will happily send the same event twice if they don’t get a fast 200. Every event carries a unique id. The handler does a conditional write of that id to a small dunr-dedupe table (with a 30-day TTL); if the id is already there, this is a duplicate and the handler returns 200 immediately without doing anything again. This is the first line of defence against acting on the same failure twice.
  • Trust ids, not amounts. The handler reads only the identifiers from the payload — the subscription id, the invoice id, the customer id, and the failure reason code. It deliberately does not trust any amount or currency in the request body as gospel; when the retrier later attempts the charge, it reads the live invoice from the processor using its API key, so the figure charged is always the processor’s real number, not whatever arrived in a webhook.

Entering dunning, then handing off

Once an event is verified, fresh, and parsed, the handler does the smallest possible amount of work and gets out of the way.

How a failed-charge webhook is caught, verified, and handed off A left-to-right flow. On the far left, outside the AWS account, a box labelled "Payment processor" fires a signed webhook for a failed charge. An arrow crosses into a dotted AWS account container. Inside, the first box is a Lambda Function URL feeding the dunr-webhook Lambda. From dunr-webhook, three checks are drawn as a vertical stack of small decision boxes: first, verify the HMAC signature against the webhook secret read from Secrets Manager, with a branch that returns HTTP 400 and stops if it fails; second, de-duplicate by event id using a conditional write to the dunr-dedupe table with a thirty-day time-to-live, with a branch that returns HTTP 200 and stops if the id was already seen; third, parse only the identifiers — subscription, invoice, customer, and failure code — trusting no amount from the body. After the three checks pass, dunr-webhook writes the subscription into the dunr-dunning table in a retrying state, sends a message to the dunr-events SQS queue, and returns HTTP 200 in milliseconds. The SQS queue has a dead-letter queue attached for messages that fail processing twice. An arrow leaves the queue to the right toward the retry engine covered in the next post. A note at the bottom reads: verify, de-duplicate, then hand off fast — the processor never waits on a slow endpoint. Payment processor signed webhook POST AWS account · eu-west-2 dunr-webhook Lambda Function URL 1. Verify HMAC else 400 & stop 2. De-dupe by id dunr-dedupe, TTL 30d 3. Parse ids only no amount trusted Secrets Manager webhook secret dunr-dunning state: retrying dunr-events SQS + DLQ → retry engine (Part 3) Verify, de-duplicate, then hand off fast — the processor never waits on a slow endpoint.
Fig 2. The failed-charge front door. A signed webhook hits the Function URL; dunr-webhook verifies the signature, drops duplicates by id, parses only the identifiers, marks the subscription retrying, queues the work, and returns 200 in milliseconds.

It writes one row to the dunr-dunning table — keyed on the subscription id, marked retrying, recording the invoice id, the customer id, the failure code, and a timestamp — then drops a single message on the dunr-events SQS queue and returns 200. The decision-making (how to retry, when, what to say) all happens downstream, off the back of the queue. The handler itself is deliberately dumb: catch, verify, record, hand off.

Returning 200 fast matters more than it looks. If the endpoint is slow, the processor treats the webhook as failed and retries it — which is why de-duplication exists — and a backlog of slow responses can get your endpoint disabled by the processor. By doing nothing heavy in the handler and pushing real work onto SQS, the front door responds in milliseconds however busy the retry engine is. The SQS queue also has a dead-letter queue: if a message fails to process twice, it lands there with an alarm, so a single malformed event never silently blocks the pipeline or gets lost.

The other three events

The same front door handles the good news too. A charge.succeeded for a subscription that’s in dunning means a retry (or a fresh attempt after a card update) worked — the handler routes it to the recovery path that cancels remaining retries and restores access. A payment_method.updated means the customer fixed their card; the handler nudges the retrier to try again now rather than waiting for the next scheduled slot. And a subscription.cancelled — which only ever comes from a human acting in your billing tool — tells the system to stand down: cancel the schedule, stop the emails, and mark the row closed. The system never originates a cancellation; it only reacts to one.

Design rules that shaped the front door

  • Verify before you trust. No signature, or a stale timestamp, means a 400 and nothing happens.
  • Assume duplicates. At-least-once delivery is normal; de-duplicate by event id on a conditional write.
  • Trust ids, not amounts. The real figure is read live from the processor at retry time, never from the webhook body.
  • Answer fast, work later. Return 200 in milliseconds; push the real work onto SQS with a dead-letter queue.
  • One door, four events. Failed, succeeded, card updated, cancelled — all arrive here and route from one place.

Next: how the retry engine takes that queued event and lays out a humane backoff schedule — the next business day, then every few business days, skipping weekends — and guarantees a charge is never attempted twice.

All posts