allanninal.dev/build

Part 3 of 7 · Dunning recovery series ~7 min read

How a retry schedule gets decided

A good dunning schedule is mostly restraint. Retry too fast and you burn attempts while the card is still frozen; retry too slow and the customer has moved on. This post is about how the system lays out a sensible backoff — the next business day, then every few business days, skipping weekends — turns each step into a one-off scheduled job, and makes absolutely sure a single charge is never attempted twice.

Key takeaways

  • The schedule is plain Python: a first retry the next business day, then roughly every three business days — four retries over about ten days.
  • Weekends are skipped, because a card frozen on Friday rarely clears on Saturday and a dunning email lands better midweek.
  • Each step is a one-off EventBridge Scheduler rule that fires dunr-retrier at the exact minute, then deletes itself.
  • An idempotency key on the invoice-and-attempt means a duplicated trigger can never charge a customer twice.
  • A success cancels every remaining retry; the final failure hands off to pause access. No model decides any of it.

A good schedule is mostly restraint

The temptation with a failed payment is to hammer the card — retry every hour until it works. That’s the wrong instinct. Most renewal failures are an expired card, a temporary hold, or an insufficient-funds decline that resolves on payday, not in the next sixty minutes. Retrying too fast burns your handful of attempts while the card is still frozen, and each failed attempt can ding your account’s risk profile with the processor. Retrying too slowly is worse in the other direction — wait two weeks between attempts and the customer has forgotten they ever used you.

So the schedule is built around a simple, opinionated backoff that a person would recognise as reasonable: try again tomorrow, then give it a few days, then a few more, then one last try before pausing. Four retries spread over about ten days. Crucially, the schedule is plain Python doing date arithmetic — there is no model involved in deciding when to charge someone, because that is a decision that touches money and must be predictable.

The backoff, in business days

When dunr-scheduler reads a failed-charge message off the queue, it computes the retry dates from the failure date, skipping Saturdays and Sundays:

  • Retry 1 — next business day. Gives a temporary hold or a same-day top-up time to clear, without waiting long.
  • Retry 2 — three business days later. Catches the customer who needed a few days, and lands after the first dunning email has had time to be seen.
  • Retry 3 — three more business days. The midpoint nudge, with a firmer email.
  • Retry 4 — three more business days (final). The last automatic attempt. Its email warns that access will pause if this one fails.

Why skip weekends? Two reasons, both practical. A card that was declined for insufficient funds on a Friday is no more likely to clear on a Saturday — payday and bank processing run on weekdays. And a dunning email that arrives on a Sunday gets buried by Monday morning; a Tuesday email gets read. So a failure on a Tuesday produces retries on Wednesday, the following Monday, Thursday, and the Monday after — every one a weekday. The number of retries and the gaps are config, not code: you can run three retries over a week or five over three weeks by changing values, with no redeploy of logic.

The retry backoff across business days, with weekends skipped A horizontal timeline running left to right across about two weeks. A baseline runs along the middle with day labels: Tuesday (the original failure), Wednesday, Thursday, Friday, then a shaded weekend block for Saturday and Sunday, then Monday, Tuesday, Wednesday, Thursday, Friday, a second shaded weekend block, and a final Monday. Above the line, four retry markers are placed only on weekdays. Retry 1 sits on the first Wednesday, labelled "next business day". Retry 2 sits on the following Monday, labelled "+3 business days". Retry 3 sits on the Thursday after, labelled "+3 business days". Retry 4 sits on the final Monday, labelled "+3 business days, final". Each retry marker has a small envelope icon above it indicating a branded dunning email goes out at that attempt. The two weekend blocks are shaded grey and labelled "skipped". Below the line, a callout under Retry 1 reads "success here cancels retries 2-4 and restores access"; a callout under Retry 4 reads "final failure pauses access". A note at the bottom reads: each retry is a one-off EventBridge Scheduler rule fired at the exact minute, guarded by an idempotency key so it can never charge twice. skipped skipped Tue (fail) Wed Thu Fri Mon Tue Wed Thu Mon ✉ Retry 1 +1 day ✉ Retry 2 +3 days ✉ Retry 3 +3 days ✉ Retry 4 final success here cancels retries 2–4, restores access final failure pauses access Each retry is a one-off EventBridge Scheduler rule, fired at the exact minute and guarded by an idempotency key.
Fig 3. The backoff in business days. Four retries over about ten days, weekends skipped, an email at each. A success at any step cancels the rest; the final failure is what pauses access.

One job per retry, with EventBridge Scheduler

Each retry date becomes a one-off EventBridge Scheduler rule. The scheduler creates a rule with an at(2026-06-24T09:00:00) expression that targets dunr-retrier with a payload of the subscription id and the attempt number. EventBridge Scheduler fires it once, at that minute, and — with --action-after-completion DELETE — deletes the rule afterward, so there is no accumulating litter of stale schedules. This is the right tool here precisely because the work is sparse and far apart: there is no Lambda sitting awake waiting, no cron table to scan, nothing running between retries. Ten days can pass between a customer’s first and last retry and the system uses no compute at all in between.

The schedule names encode what they are — dunr-retry-{subscription}-{attempt} — which makes the next part easy: when a retry succeeds, the retrier can cancel the remaining rules by name in one pass, so retries 2 through 4 simply never fire once retry 1 has recovered the payment.

The one guarantee that matters: never twice

EventBridge Scheduler, like most of AWS, is at-least-once: in rare cases it can fire the same schedule twice. A webhook can be replayed. A retry could overlap with a customer manually updating their card and triggering a fresh attempt. None of these may ever result in charging a customer twice. So every charge the retrier makes carries an idempotency key — a deterministic string built from the invoice id and the attempt number, for example dunr-inv_8842-a3. The processor treats two requests with the same idempotency key as the same request: the second one returns the result of the first without taking any money. The key is deterministic, not random, precisely so that a duplicate trigger reconstructs the same key and is harmlessly collapsed.

This is the load-bearing safety property of the whole system. Combined with the de-duplication at the front door (Part 2), it means there is no path — duplicate webhook, double-fired schedule, overlapping manual update — that charges a customer’s card more than once for one attempt.

Design rules that shaped the schedule

  • Restraint over speed. Four retries over about ten days, not a barrage in the first hour.
  • Business days only. Skip weekends — the card won’t clear and the email won’t get read.
  • Plain Python, no model. A decision that moves money is deterministic and predictable.
  • One job per retry. One-off EventBridge schedules that self-delete — no always-on compute between attempts.
  • Idempotency, always. A deterministic key per invoice-and-attempt makes a double-charge impossible.
  • Config, not code. The count and gaps are values you can change without redeploying logic.

Next: the message that goes out at each of those four attempts — a branded dunning email in your voice, with a one-tap link to update the card.

All posts