allanninal.dev/build

Part 7 of 7 · Chargeback responder series ~12 min read

Engineering reference: the chargeback responder architecture

Same system, drawn for engineers. Region, service names, resource identifiers, the Bedrock model id, the Lambda inventory, IAM scopes, the SES configuration, the two Function URLs, the EventBridge dispute rule and the one-off Scheduler deadline watch, and the DynamoDB schemas for the disputes, the evidence index, and the deadlines. Read it alongside the previous six posts — this one is the build sheet.

Region and account shape

Default region: eu-west-2 (London). Lambda Function URLs, EventBridge Scheduler, SES outbound, DynamoDB, S3, and Bedrock cross-Region inference are all in good shape there, and it keeps the data close to a UK or EU customer base. A second region for multi-region resilience isn’t worth the extra work at SMB volume — the failure mode for a chargeback responder is a single dispute taking an hour longer to file, and the deadline guard already gives a day of slack, so a regional blip never causes a missed deadline. One AWS account dedicated to the responder keeps the IAM blast radius small and lets a single AWS Budgets alarm cover the whole system.

Topology

AWS topology of the chargeback responder A topology diagram with three regions stacked vertically inside one AWS account boundary, eu-west-2. Top region: ingress. The payment processor posts a signed dispute webhook to cbr-webhook, a Lambda Function URL that verifies the HMAC signature, de-duplicates on the dispute id with a conditional write to the cbr-disputes table, records the evidence deadline in cbr-deadlines, arms a one-off EventBridge Scheduler job timed before due_by targeting cbr-deadline-guard, and emits a cbr.dispute.received event onto the EventBridge default bus, which enqueues a job on the SQS work queue with its dead-letter queue. Middle region: event-driven processing. cbr-gather is triggered by the SQS queue; it reads the dispute, pulls the order from the commerce platform, proof of delivery from the carrier API, the customer messages from the support inbox, and the agreed policy from the policy store, writes each piece to the S3 evidence bucket cbr-evidence under a per-dispute prefix, indexes each in the cbr-evidence DynamoDB table marking found or missing, and enqueues assembly. cbr-assemble reads the index and the reason-code playbook from S3 and makes one Bedrock Claude Haiku 4.5 call via Global cross-Region inference to produce a structured rebuttal with cited evidence and a winnability verdict, renders a packet PDF to cbr-evidence, and either marks the case ready to file or recommends accepting the loss. Bottom region: filing and audit. cbr-file submits the packet through the processor API using the key from Secrets Manager, with an idempotent conditional transition to the filed state. cbr-deadline-guard is the Scheduler target that fires before due_by and files the assembled packet if the case is winnable and neither filed nor marked accept-loss. cbr-ack is a second Lambda Function URL handling the owner’s signed File, Hold, and Accept-loss buttons from the SES email. A closed dispute webhook returns through cbr-webhook and is recorded as won or lost. CloudWatch Logs collects from every Lambda at 7-day retention; the SQS dead-letter queue catches anything that fails twice; an AWS Budgets alarm at $25 posts to an SNS topic. A note at the bottom reads: the deadline guard guarantees a winnable case is filed before the cutoff, and nothing in the packet is ever fabricated. Payment processor · webhook + API Ingress Lambda · cbr-webhook Function URL verify HMAC, de-dupe → cbr-disputes + cbr-deadlines arm Scheduler job EventBridge bus cbr.dispute.received cbr.dispute.closed rule → SQS work SQS · cbr-work one msg per dispute DLQ after 2 fails target: cbr-gather Event-driven processing Lambda · cbr-gather pulls order, delivery, comms, agreed policy → S3 cbr-evidence + cbr-evidence index found/missing per piece Lambda · cbr-assemble reads index + playbook one Bedrock call, cited rebuttal + verdict → packet PDF in S3 ready / accept-loss Bedrock Claude Haiku 4.5 Global cross-Region inference profile assemble only Filing & audit Lambda · cbr-file submits via processor API key from Secrets Manager idempotent: ready→filed always before due_by Lambda · cbr-deadline-guard Scheduler target fires before due_by files if winnable & not filed/accept-loss Lambda · cbr-ack Function URL signed File/Hold/Accept from SES owner email closed → won/lost The deadline guard files a winnable case before the cutoff — and nothing in the packet is ever fabricated.
Fig 7. AWS topology, in three regions of the diagram: ingress (the verified webhook arming the deadline watch), event-driven processing (gather then assemble, with Bedrock used only to assemble), and filing and audit (the filer, the deadline guard backstop, and the owner’s buttons). Every Lambda is event- or schedule-driven; nothing is synchronous-chained.

Lambda functions

All Lambdas use the arm64 architecture, the smallest memory size that meets latency targets (typically 256 MB), Python 3.14 runtime, and CloudWatch Logs at 7-day retention. Each function has its own least-privilege IAM role. None run inside a VPC.

  • cbr-webhook — Lambda Function URL, AuthType: NONE; authenticated by the processor’s signature. Verifies the HMAC over the raw body using the signing secret from Secrets Manager and checks the timestamp window. On charge.dispute.created: de-duplicates with a conditional PutItem on cbr-disputes, records due_by and the reason code in cbr-deadlines, creates a one-off EventBridge Scheduler rule targeting cbr-deadline-guard, emits cbr.dispute.received, and returns 200. On charge.dispute.closed: records won/lost on the case and deletes any remaining Scheduler rule. Memory: 256 MB. Timeout: 15 s.
  • cbr-gather — SQS trigger on cbr-work. Reads the dispute, then pulls the four sources targeted by the reason code: the order from the commerce platform, proof of delivery from the carrier API (key in Secrets Manager), the customer’s messages from the support inbox/chat, and the accepted policy snapshot. Writes each raw piece to s3://cbr-evidence/<dispute_id>/ and an index row to the cbr-evidence table with a found/missing flag; a not-yet-available delivery confirmation is recorded for re-gather, not treated as an error. Enqueues assembly. Memory: 512 MB. Timeout: 60 s. No Bedrock calls.
  • cbr-assemble — triggered after gather. Reads the evidence index and the reason-code playbook from s3://cbr-evidence/playbook.txt, pulls the cited pieces from S3, and makes one Bedrock Haiku 4.5 call (anthropic.claude-haiku-4-5-20251001-v1:0 via global.anthropic.claude-haiku-4-5-20251001-v1:0) returning structured JSON: cover narrative, cited rebuttal points, exhibit list, winnability verdict. Drops any claim whose cited evidence id is absent or marked missing, renders the packet PDF (and the field-and-attachment form) to S3, and sets the case ready or recommend_accept_loss. Emails the owner via SES. In auto-file mode, a strong case is handed straight to cbr-file. Memory: 1024 MB. Timeout: 60 s.
  • cbr-file — submits the assembled evidence through the processor’s evidence API using the key from Secrets Manager. Idempotent: a conditional update moves the case from ready to filed only once, so the owner’s tap and the guard can never double-file. Writes the processor’s submission id and timestamp to cbr-disputes and deletes the Scheduler rule. Memory: 256 MB. Timeout: 30 s.
  • cbr-deadline-guard — EventBridge Scheduler target, one per dispute, fired a safe margin (default 24 h) before due_by. Reads case state: if filed or accept_loss, does nothing; if weak and auto-file off, raises a last-call alert only; otherwise triggers a re-gather of any pending piece and calls cbr-file on the assembled packet. Memory: 256 MB. Timeout: 60 s.
  • cbr-ack — Lambda Function URL, AuthType: NONE; verifies a signed, single-use token (HMAC over (dispute_id, action, nonce, expiry) keyed from Secrets Manager) so a forwarded email link can’t be replayed. On File: calls cbr-file. On Hold: sets held without disarming the guard. On Accept loss: sets accept_loss and deletes the Scheduler rule. Always writes the action to cbr-disputes. Memory: 256 MB. Timeout: 15 s.
  • cbr-summary — EventBridge Scheduler target, monthly on the first Monday at 9am. Reads the past month’s disputes and outcomes; calls Bedrock Haiku 4.5 to write a one-paragraph narrative (disputes received, value defended, win rate by reason code, recommended-accept-loss count); emails it via SES to the owner. Memory: 512 MB.

Storage

  • DynamoDB · cbr-disputes — one row per dispute. PK dispute_id; attributes: processor, charge_id, reason_code, amount, currency, customer, state (received/gathering/assembled/ready/held/filed/accept_loss/won/lost), winnability, submission_id, created_at, filed_at. On-demand.
  • DynamoDB · cbr-evidence — the evidence index, one row per gathered piece. PK dispute_id; SK evidence_type (order/delivery/comms/policy); attributes: s3_key, source, gathered_at, status (found/missing/pending), note. On-demand.
  • DynamoDB · cbr-deadlines — one row per dispute deadline. PK dispute_id; attributes: due_by, safety_cutoff, schedule_arn, filed (bool). On-demand. The authoritative record the guard checks against.
  • S3 · cbr-evidence — one prefix per dispute holding the raw gathered evidence (order JSON, carrier tracking, message exports, policy snapshot) and the compiled packet PDF, plus the editable playbook.txt at the bucket root. Versioning enabled. Lifecycle to Glacier at 90 days; expiry at 7 years for the audit trail.

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0 via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Two callsites: cbr-assemble for the rebuttal, and cbr-summary for the monthly narrative. Neither cbr-webhook, cbr-gather, cbr-file, nor cbr-deadline-guard ever calls Bedrock — every money or deadline decision is deterministic.
  • Grounding. The assemble prompt carries only the gathered evidence with its index ids; the renderer drops any claim citing an absent or missing id. The model cannot introduce evidence, only arrange what was gathered.
  • Heavier model. anthropic.claude-sonnet-4-6-... is configured but off by default; worth switching cbr-assemble to Sonnet only for high-value disputes where a more careful narrative is justified, via a per-dispute amount threshold.
  • Embeddings. Not used. There’s no corpus to search — each dispute’s evidence is gathered fresh. No Knowledge Base, no S3 Vectors.

EventBridge and Scheduler config

  • Dispute rule — an EventBridge rule on the default bus matching cbr.dispute.received, target the cbr-work SQS queue (which triggers cbr-gather).
  • Deadline watch — one-off Scheduler rules created by cbr-webhook, one per dispute, with an at(YYYY-MM-DDTHH:MM:SS) expression set to due_by minus the safety margin, target cbr-deadline-guard, and --action-after-completion DELETE so the rule self-cleans. Deleted early by cbr-file or cbr-ack when the case is filed or accept-loss.
  • cbr-monthly-summary — cron(0 9 ? * 2#1 *) (first Monday at 9am) in TZ. Target: cbr-summary Lambda.

SES and the two Function URLs

  • SES outbound only. Verify a sender identity at disputes@your-company.com with DKIM and SPF on the parent domain; out of sandbox by request. Used for the per-dispute owner alert and the monthly summary. There is no SES inbound — disputes arrive by webhook, not email.
  • Webhook Function URL — cbr-webhook, public, signature-checked inside the function. This is the address registered in the processor’s webhook settings.
  • Ack Function URL — cbr-ack, public, signed single-use token in each button link. The owner’s email renders File, Hold, and Accept-loss as links to this URL.

IAM (least privilege per Lambda)

Each Lambda has its own role with policies scoped to exact ARNs. Sketch:

  • cbr-webhook role: dynamodb:PutItem/UpdateItem on cbr-disputes and cbr-deadlines; scheduler:CreateSchedule/DeleteSchedule; events:PutEvents on the default bus; secretsmanager:GetSecretValue on the webhook signing secret. No bedrock:*.
  • cbr-gather role: sqs:ReceiveMessage/DeleteMessage on cbr-work; s3:PutObject on cbr-evidence; dynamodb:PutItem on the cbr-evidence table; secretsmanager:GetSecretValue on the carrier key; outbound network to the carrier, commerce, and inbox hosts. No bedrock:*.
  • cbr-assemble role: s3:GetObject/PutObject on cbr-evidence; dynamodb:Query on the evidence index; bedrock:InvokeModel on the Haiku ARN; ses:SendEmail from the verified sender; secretsmanager:GetSecretValue on the ack-link signing secret.
  • cbr-file role: dynamodb:UpdateItem on cbr-disputes and cbr-deadlines; scheduler:DeleteSchedule; s3:GetObject on the packet; secretsmanager:GetSecretValue on the processor API key.
  • cbr-deadline-guard role: the cbr-file permissions plus dynamodb:GetItem on the deadlines and disputes tables, and permission to re-invoke cbr-gather.
  • cbr-ack role: dynamodb:UpdateItem on cbr-disputes; scheduler:DeleteSchedule; permission to invoke cbr-file; secretsmanager:GetSecretValue on the ack-link signing secret.

Idempotency and the deadline invariant

Two invariants hold the system together. First, a dispute is filed at most once: filing is a conditional state transition from ready to filed, so the owner’s tap, an auto-file, and the deadline guard all race safely — whoever wins moves the state, the others find it already filed and stop. Second, a winnable dispute is never filed late: the one-off Scheduler rule is armed before any slow work runs and is only deleted when the case is genuinely filed or deliberately dropped, so there is no path where a case is forgotten. The guard fires on a timer that AWS owns, independent of whether any earlier step succeeded.

Observability and cost gates

  • CloudWatch Logs: all Lambdas, 7-day retention, structured JSON. Subscription filter on "error" + "throttle" + "timeout" to a metric for alerting.
  • SQS DLQ: cbr-work has a dead-letter queue; a gather that fails twice lands there with an alarm, so a flaky carrier API never silently blocks a case from being assembled in time.
  • Alarms: any case within 12 h of due_by still in ready (the guard should have it, but page anyway); DLQ depth > 0; cbr-file failures > 0; cbr-webhook signature-verification failures > 5/hour.
  • X-Ray: off by default. Not worth the cost at SMB volume.
  • AWS Budgets: $25/month threshold, alarm at 80% and 100%, posts to SNS topic cbr-cost-alarm subscribed to the owner’s email.

Config and secrets

The processor API key and webhook signing secret live in Secrets Manager under cbr/processor/*; the carrier API key under cbr/carrier/key; the ack-link signing secret under cbr/links/secret. The mode (auto-file vs owner-approval), the safety-margin hours, the high-value Sonnet threshold, and the owner’s email live in Parameter Store under /cbr/config/. The reason-code playbook is plain text in S3 so it can be tuned without a deploy. Lambdas fetch config on cold start and cache for the lifetime of the execution environment.

Deploy

GitHub Actions with OIDC into a deploy role (no long-lived keys) and AWS SAM. The opinionated bits: turn on S3 versioning for cbr-evidence so a packet is always reproducible and a bad playbook edit rolls back in one click; give cbr-gather an SQS source with a DLQ so one unreachable carrier never wedges the pipeline; and keep the deadline guard’s schedule creation in cbr-webhook so the safety net is armed at the earliest possible moment. Total deployable surface: seven Lambdas, three DynamoDB tables, one S3 bucket, one SQS queue with a DLQ, one EventBridge rule on the default bus (plus the per-dispute Scheduler rules), two Function URLs, an SES sender identity, and one Budgets alarm.

That’s the full system. Six narrative posts and this engineering reference. If you want to talk about adapting it for your business, see Work with me.

All posts