allanninal.dev/build

Part 1 of 7 · Dunning recovery series ~11 min read

A dunning recovery system on AWS for a few dollars a month

A subscription business loses more revenue to failed payments than to customers who actually decide to leave. A card expires, a charge is declined, the renewal silently fails — and weeks later someone notices the account is still active but hasn’t paid since March. Recovering those payments by hand is fiddly and easy to drop: you have to retry the charge at sensible intervals, email the customer without nagging, and pause access only once you’ve genuinely given up — never double-charging and never cutting someone off by mistake. This post walks through the design of a small system that catches every failed charge, runs a smart retry schedule, sends a branded dunning email at each step, and pauses or restores access on its own — while never moving money without a guard.

Key takeaways

  • One trigger starts everything: a failed-charge webhook from your payment processor (Stripe-style).
  • The retry schedule is plain Python — a backoff over several days that skips weekends, four retries in about ten days.
  • A branded dunning email goes out at every attempt, each with a one-tap link to update the card.
  • Access pauses only after the final failed retry, and restores the instant a retry succeeds or the card is updated.
  • Designed on AWS for about $1.90/month at small-business volume. It never double-charges and never silently cancels.

The whole system on one page

Before any code, here’s the shape of what we’re designing. There are three things outside AWS the system talks to, and three pieces inside that do the work.

System architecture: three things outside, three pieces inside AWS At the top, three external boxes in a row. Far left, "Payment processor" — a Stripe-style billing provider that fires webhooks when a subscription charge fails, succeeds, or a card is updated, and that the system calls back to retry a charge. Centre, "Your app" — the product the customer pays for, which checks entitlement on every request and shows a paused screen with an update-card link when access is paused. Far right, "Customers" — where branded dunning emails land at each retry attempt, each carrying a one-tap link to update the card. Each connects by an arrow to the AWS account container below. The payment processor sends failed-charge webhooks in and receives retry charge calls back. Your app calls the entitlement check. Customers receive dunning emails and click the update-card link. Inside the AWS account are three components in a row. On the left, the Failed-charge intake — a webhook Lambda behind a Function URL that verifies the processor signature, de-duplicates the event, and puts the subscription into dunning. In the middle, the Retry engine — decides a backoff schedule in plain Python, writes one-off EventBridge Scheduler rules, retries the charge at each step, and advances the state. On the right, Access and mail — a gatekeeper that pauses and restores access and a mailer that sends a branded dunning email at each attempt. Internal arrows flow left to right, with the retry engine feeding both the gatekeeper and the mailer. A note at the bottom reads: it never double-charges — every retry is idempotent — and it never silently cancels; the worst it does on its own is pause, which is reversible. Payment processor webhooks out, retries in Your app checks entitlement Customers where dunning mail lands failed charge paused? dunning email AWS account · eu-west-2 Failed-charge intake verify, de-dupe, enter dunning Retry engine backoff schedule, retry the charge Access & mail pause / restore, branded dunning email event state Never double-charges — every retry is idempotent — and never silently cancels; the worst it does is pause.
Fig 1. Three things outside, three pieces inside AWS. A failed-charge webhook flows in from the processor. The retry engine lays out a backoff schedule and retries the charge at each step. The access-and-mail piece sends a branded dunning email and pauses or restores access — on its own, but reversibly.

What you set up once (the outside)

  • The payment processor. You already use a Stripe-style processor to take subscription payments. Two things connect it to this system. First, a webhook endpoint — you point the processor at a Lambda Function URL and subscribe to the events that matter: a charge failed, a charge succeeded, a payment method was updated, a subscription changed. Second, a secret API key (in Secrets Manager) the system uses to call back and retry a charge. You do not store card numbers anywhere; the processor holds the card, and the system only ever asks it to try the saved card again.
  • Your app’s entitlement check. Somewhere your product already decides “is this customer allowed in?” — on login, on an API call, on a page load. You change that one check to ask the gatekeeper (Part 5) instead of reading a flag directly. That single change is what lets the system pause and restore access without touching the rest of your app. While a subscription is healthy the answer is always granted; the moment dunning gives up it becomes paused, and the customer sees an update-card screen instead of the product.
  • The email brand. One sender identity verified in SES (with DKIM and SPF), your logo, and a short HTML template in S3. The dunning emails go out from billing@your-company.com looking like you, not like a generic decline notice. You set the tone once — the copy at each attempt is written from that template — and you decide how many retries to run before access pauses.

What runs on every failed charge (the inside)

  • The failed-charge intake. The processor fires a webhook the instant a renewal charge fails. dunr-webhook, behind a Lambda Function URL, verifies the signature against the webhook secret so it knows the event is genuine, throws the event away if it has seen its id before (processors retry their own webhooks, so duplicates are normal), and writes the subscription into the dunr-dunning table in a retrying state. It does not trust any amount the request claims; it records the subscription and invoice ids and lets the retry engine read the real figures back from the processor. Then it drops a message on an SQS queue and returns 200 fast, so the processor never sees a slow endpoint. Part 2 is all about this front door.
  • The retry engine. A scheduler Lambda reads the queue and lays out the retry schedule in plain Python: the next business day, then roughly every three business days, four retries over about ten days, skipping weekends. Each step becomes a one-off EventBridge Scheduler rule that fires dunr-retrier at the exact minute. The retrier asks the processor to charge the saved card again — guarded by an idempotency key tied to the invoice and attempt number, so a duplicated trigger can never charge twice. A success cancels every remaining retry and marks the subscription recovered; a failure advances the attempt and, if it was the last one, hands off to the gatekeeper to pause access. Parts 3 covers the schedule.
  • Access and mail. Two small pieces close the loop. dunr-mailer sends a branded dunning email at each attempt (Part 4) — one Bedrock Haiku call writes the copy in your voice, gentle on the first nudge and clear about the consequence by the last, each with a one-tap update-card link. dunr-gatekeeper is the entitlement check your app calls (Part 5); it pauses access only after the final failed retry and restores it the instant a retry succeeds or the customer updates their card. Every attempt, email, and state change is written to the dunr-log audit table.

The five states every subscription moves through

A subscription in this system is always in exactly one state, and the transitions are deterministic — no model decides them.

  • Active. Paying normally. The system isn’t doing anything; the gatekeeper answers granted.
  • Retrying. A charge has failed and the backoff schedule is running. Access is still granted — you don’t lock someone out the second a card blips. Each retry sends a dunning email.
  • Paused. Every retry failed. Access is now paused; the customer sees the update-card screen. This is reversible, and it is the strongest action the system takes on its own.
  • Recovered. A retry succeeded, or the customer updated their card and the next charge cleared. Remaining retries are cancelled, access is restored in the same second, and a short “you’re all set” receipt goes out.
  • Cancelled. Reached only by a human, in your billing tool. The system never cancels a subscription on its own; pausing is as far as it goes.

In plain words

A customer on your £29/month plan renews on a Tuesday and the charge declines — expired card. Within a second the processor fires a charge.failed webhook; dunr-webhook verifies it and puts the subscription into retrying. The scheduler lays out four retries: Wednesday, the following Monday, Thursday, and the Monday after — all weekdays. Wednesday’s email goes out: “We couldn’t process your payment — no need to worry, we’ll try again, or you can update your card here.” The Wednesday and Monday retries both fail; the customer is busy and ignores the emails. Thursday’s email is firmer: “Your access will pause on Monday if we can’t take payment.” On Saturday the customer finally taps the link and updates their card. The gatekeeper sees the new card, the next retry clears on the spot, the two remaining scheduled retries are cancelled, access stays on, and a receipt confirms it. The customer never lost access, you never lost the £29, and nobody on your team touched it.

The cost of running this is about $1.90 a month at small-business volume. The cost of not running it is the renewals that fail and never get chased — the quiet, compounding leak that subscription businesses call involuntary churn.

Design rules that shaped every decision

  • One trigger, verified. Everything starts from a signed failed-charge webhook; an unsigned or replayed request does nothing.
  • The schedule is plain Python. A backoff over business days — no model decides when to retry.
  • Every retry is idempotent. An idempotency key on the invoice-and-attempt means a duplicate trigger can never double-charge.
  • Pause, never cancel. The strongest action the system takes on its own is a reversible pause; cancelling is always a human decision.
  • Restore is instant. A successful retry or an updated card flips access back in the same second — no overnight batch.
  • Everything is logged. Every attempt, email, and state change writes to an audit table you can read a year later.

Why this shape

Most small subscription businesses handle failed payments one of three ways: the processor’s built-in dunning runs with default settings nobody tuned, a founder manually emails people when they notice, or nothing happens and the customer keeps their access for free until someone audits the books. The default-settings approach often retries on weekends and sends generic decline emails that look like phishing. The manual approach doesn’t scale past a few dozen customers. And doing nothing is how a subscription business slowly gives its product away.

The setup above keeps the processor as the source of truth for whether a charge succeeded — the system never guesses — and adds a small layer that reacts to failures with a sensible, branded, fully logged recovery flow. Retries happen on weekdays at humane intervals. Emails look like you and escalate gently. Access pauses only when recovery has genuinely failed, and comes back the instant the customer fixes their card. And because the schedule and the pause are plain deterministic Python guarded by an idempotency key, the two things you most fear — charging someone twice and cutting someone off by mistake — simply can’t happen.

The next four posts walk through each piece in turn: how a failed charge gets caught, how a retry schedule gets decided, how a dunning email gets sent, and how access gets paused and restored. One diagram per post. A cost breakdown and a final engineering reference at the end.

All posts