allanninal.dev/build

Part 7 of 7 · Dunning recovery series ~10 min read

Engineering reference: the dunning recovery architecture

Same system, drawn for engineers. Region, service names, resource identifiers, the Bedrock model id, the Lambda inventory, IAM scopes, the SES configuration set, the EventBridge Scheduler rules, the DynamoDB schemas, and the idempotency design that stops a double-charge. Read it alongside the previous six posts — this one is the build sheet.

Key takeaways

  • One region throughout: eu-west-2. One AWS account dedicated to the system keeps the IAM blast radius small.
  • Seven Lambdas, three DynamoDB tables, one S3 bucket, one SQS queue with a DLQ, one SES configuration set.
  • EventBridge Scheduler drives one-off retries (self-deleting) plus a daily sweep and a monthly report.
  • The double-charge guard is a deterministic idempotency key on the invoice-and-attempt, passed to the processor.
  • Bedrock Haiku 4.5 has exactly two callsites — the mailer and the monthly summary — never the money path.

Region and account shape

Default region: eu-west-2 (London). SES outbound, Bedrock cross-Region inference, EventBridge Scheduler, and Lambda Function URLs are all in good shape there, and it keeps customer data in-region for a UK/EU subscriber base. A second region for resilience isn’t worth the setup at this volume: the failure mode is a retry firing an hour late, not a regional outage, and because every charge is idempotent there is no in-flight payment to protect across a failover. One AWS account dedicated to dunning recovery (separate from your other workloads) keeps the IAM blast radius small and lets a single AWS Budgets alarm cover the whole system.

Topology

AWS topology of the dunning recovery system A three-band topology inside one AWS account in eu-west-2. The top band, ingress, shows the payment processor outside the account firing a signed webhook into dunr-webhook behind a Lambda Function URL; dunr-webhook verifies the signature against Secrets Manager, de-duplicates via the dunr-dedupe table, writes the dunr-dunning table, and sends to the dunr-events SQS queue, which has a dead-letter queue. The middle band, the retry engine, shows dunr-scheduler consuming the SQS queue, computing a business-day backoff, and creating one-off EventBridge Scheduler rules named dunr-retry-subscription-attempt; those rules fire dunr-retrier, which calls the payment processor with an idempotency key built from the invoice and attempt number, updates dunr-dunning, and on success cancels the remaining scheduler rules. The bottom band, egress, shows dunr-mailer sending branded email through SES configuration set dunr-mail using a template from the dunr-assets S3 bucket and one Bedrock Haiku 4.5 call, and dunr-gatekeeper behind a Lambda Function URL answering the app's entitlement check by reading dunr-dunning. Bounce and complaint events flow from SES through SNS to a suppression list. EventBridge Scheduler also drives dunr-sweep daily and dunr-summary monthly. A note at the bottom reads: every charge is idempotent and the system never cancels — the strongest state it sets on its own is paused. Ingress Payment processor signed webhook (outside account) dunr-webhook Function URL, verify, de-dupe, enter dunning dunr-events SQS + DLQ dunr-dedupe event ids TTL 30d Retry engine dunr-scheduler SQS consumer, business-day backoff, creates schedules EventBridge Scheduler dunr-retry-{sub}-{n} at(...) one-off, self-deletes dunr-retrier charge w/ idempotency key inv+attempt, advance or pause dunr- dunning state Egress dunr-mailer template (dunr-assets) + Bedrock copy, SES dunr-mail SES + SNS feedback branded email out, bounce/complaint → suppression list dunr-gatekeeper Function URL, reads dunr-dunning, granted / paused Your app entitlement check Every charge is idempotent and the system never cancels — the strongest state it sets on its own is paused.
Fig 7. AWS topology in three bands: ingress (the verified webhook into a queue), the retry engine (the scheduler creating one-off rules that fire the retrier against shared state), and egress (the mailer and the gatekeeper). Every Lambda is event- or schedule-driven; nothing is synchronous-chained.

Lambda functions

All Lambdas use the arm64 architecture, the smallest memory size that meets latency targets (typically 256 MB), Python 3.14 runtime, and CloudWatch Logs at 7-day retention. Each function has its own least-privilege IAM role. None run inside a VPC.

  • dunr-webhook — Lambda Function URL, public with AuthType: NONE; security comes from HMAC signature verification, not network auth. Verifies the processor signature against dunr/stripe/webhook-secret, rejects stale timestamps (replay window 5 min), de-duplicates by event id with a conditional PutItem on dunr-dedupe, writes dunr-dunning, and sends to dunr-events. Routes the four event types (failed / succeeded / payment-method-updated / subscription-cancelled). Returns 200 in milliseconds. Memory: 256 MB. Timeout: 10 s.
  • dunr-scheduler — SQS trigger on dunr-events. Computes the business-day backoff in plain Python (next business day, then +3 business days ×3, weekends skipped; count and gaps from config), creates one-off EventBridge Scheduler rules dunr-retry-{sub}-{n} with at(...) expressions and --action-after-completion DELETE, and triggers the first dunning email. Memory: 256 MB. Timeout: 30 s. No Bedrock.
  • dunr-retrier — EventBridge Scheduler target, one invocation per retry. Reads the live invoice from the processor (API key in dunr/stripe/api-key) and attempts the charge with a deterministic idempotency key dunr-{invoice}-a{n}. On success: marks dunr-dunning recovered, deletes remaining dunr-retry-{sub}-* rules, restores access, fires a receipt email. On failure: advances the attempt or, if final, sets paused. Memory: 256 MB. Timeout: 30 s. No Bedrock.
  • dunr-mailer — invoked per attempt by scheduler/retrier. Reads the template, logo, and tone brief from s3://dunr-assets/, makes one Bedrock Haiku 4.5 call for subject and body copy (falls back to a plain template on error), builds the signed single-use update-card link (HMAC from dunr/links/secret), and sends via SES with configuration set dunr-mail. Memory: 512 MB. Timeout: 30 s.
  • dunr-gatekeeper — Lambda Function URL, public with AuthType: NONE but called with a short app-issued bearer token. Single keyed read of dunr-dunning; returns granted for active/retrying/recovered and paused for paused. Also exposes the internal pause/restore writes used by the retrier. Single-digit-ms responses; the app caches for 1–2 min. Memory: 256 MB. Timeout: 5 s.
  • dunr-feedback — SNS trigger from the SES configuration set. Writes hard-bounce and complaint addresses to the suppression list so future sends skip them. Memory: 256 MB. Timeout: 10 s.
  • dunr-sweep — EventBridge Scheduler target, daily at 9am local. Reconciles state: catches a retry whose schedule never fired (re-creates it), re-pings paused subscriptions that have a stale card, and closes rows past a configurable abandonment horizon. No Bedrock; a plain pass. Memory: 256 MB.
  • dunr-summary — EventBridge Scheduler target, monthly on the first Monday at 9am. Reads the past month of dunr-log; one Bedrock Haiku 4.5 call writes a short narrative (charges recovered, value saved, recovery rate by failure code); emails it via SES and writes it to s3://dunr-assets/reports/. Memory: 512 MB.

Storage

  • DynamoDB · dunr-dunning — one row per subscription in or recently out of dunning. PK subscription_id; attributes: customer_id, status (active/retrying/paused/recovered/cancelled), attempt_no, max_attempts, next_retry_at, invoice_id, failure_code, access (granted/paused), schedule_names, entered_dunning_at. On-demand.
  • DynamoDB · dunr-log — one row per attempt or state change. PK subscription_id; SK ts (ISO-8601 + ULID suffix). Attributes: event (charge_failed/retry_attempted/retry_succeeded/email_sent/access_paused/access_restored/card_updated/cancelled), attempt_no, result, failure_code, before, after. On-demand. No TTL — this is the long-term audit trail.
  • DynamoDB · dunr-dedupe — one row per processed webhook event id, for idempotency at the front door. PK event_id; attribute seen_at. TTL 30 days. On-demand.
  • S3 · dunr-assets — email HTML template, logo, tone brief, and the monthly reports under reports/. Versioning enabled so a bad template edit rolls back in one click. No public access.

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0 via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Two callsites only: dunr-mailer for the dunning-email copy, and dunr-summary for the monthly narrative. No model is ever on the schedule, charge, pause, or restore path.
  • Embeddings / Knowledge Base. Not used. There is no corpus to retrieve over — the state is a handful of structured rows.

EventBridge Scheduler config

  • One-off retry rules — created by dunr-scheduler, named dunr-retry-{sub}-{n}, at(YYYY-MM-DDTHH:MM:SS) expressions in the configured TZ, target dunr-retrier, with --action-after-completion DELETE so each self-cleans. Deleted en masse by name prefix when a retry succeeds.
  • dunr-daily-sweep — cron(0 9 * * ? *) in TZ. Target: dunr-sweep.
  • dunr-monthly-summary — cron(0 9 ? * 2#1 *) (first Monday at 9am) in TZ. Target: dunr-summary.

SES (outbound only)

  • Verify a sender identity at billing@your-company.com with DKIM and SPF on the parent domain; out of sandbox by request. No inbound rule set — this system only sends.
  • Configuration set dunr-mail with event publishing to an SNS topic for Bounce and Complaint; dunr-feedback consumes it and maintains the suppression list. Sends check the suppression list first.
  • One template per escalation level lives in s3://dunr-assets/; the rendered subject/body come from Bedrock at send time, with the template as the deterministic fallback.

IAM (least privilege per Lambda)

Each Lambda has its own role scoped to exact ARNs. Sketch:

  • dunr-webhook role: secretsmanager:GetSecretValue on the webhook secret; dynamodb:PutItem on dunr-dedupe and dunr-dunning; sqs:SendMessage on dunr-events. No processor API key — it never charges.
  • dunr-scheduler role: sqs:ReceiveMessage/DeleteMessage on dunr-events; scheduler:CreateSchedule/DeleteSchedule on the dunr-retry-* name prefix; dynamodb:UpdateItem on dunr-dunning; lambda:InvokeFunction on dunr-mailer. No bedrock:*.
  • dunr-retrier role: secretsmanager:GetSecretValue on the processor API key; dynamodb:UpdateItem on dunr-dunning and PutItem on dunr-log; scheduler:DeleteSchedule on the dunr-retry-* prefix; lambda:InvokeFunction on dunr-mailer. The only role that can move money.
  • dunr-mailer role: s3:GetObject on dunr-assets; bedrock:InvokeModel on the Haiku ARN; ses:SendRawEmail from the verified sender; secretsmanager:GetSecretValue on the link-signing secret.
  • dunr-gatekeeper role: dynamodb:GetItem on dunr-dunning, and UpdateItem scoped to the access attribute. No secrets, no Bedrock, no charging.

The idempotency design (why it can’t double-charge)

Two independent guards make a double-charge impossible. At the front door, dunr-webhook de-duplicates webhook events by id on a conditional write, so a replayed or re-delivered event is a no-op. At the charge, dunr-retrier passes the processor a deterministic idempotency key — dunr-{invoice}-a{n}, built from the invoice id and attempt number, never random. If EventBridge Scheduler fires the same retry twice, or a manual card update overlaps a scheduled retry, both requests reconstruct the same key, and the processor collapses the second into the first without taking money. The key is keyed on the attempt, not the clock, precisely so duplicates converge.

Observability and cost gates

  • CloudWatch Logs: all Lambdas, 7-day retention, structured JSON. Metric filter on "error", "throttle", "signature_fail".
  • SQS DLQ: dunr-events has a dead-letter queue; a message that fails twice lands there with an alarm, so a malformed event never silently blocks the pipeline.
  • Alarms: retrier failures > 0/hour; DLQ depth > 0; webhook signature failures > 5/hour (the secret may have rotated); SES complaint rate over threshold.
  • AWS Budgets: $10/month threshold, alarm at 80% and 100% to an SNS topic dunr-cost-alarm.

Deploy

GitHub Actions with OIDC into a deploy role (no long-lived keys) and AWS SAM. The opinionated bits: turn on S3 versioning for dunr-assets so a bad template edit rolls back in one click; give dunr-scheduler an SQS source with a DLQ so a single bad event never wedges the pipeline; and scope the processor API key to the retrier role alone, so it is the one and only function that can charge a card. Total deployable surface: seven Lambdas, three DynamoDB tables, one S3 bucket, one SQS queue with a DLQ, one SES configuration set, two recurring EventBridge schedules (plus the one-off retry rules created at runtime), and one Budgets alarm.

That’s the full system. Six narrative posts and this engineering reference. If you want to talk about adapting it for your business, see Work with me.

All posts