Engineering reference: the upsell recommender architecture

The architecture, for engineers

This is the same system as Part 1 with the friendly labels removed and the real resources named. Everything is in one region, eu-west-2 (London), in one account. There is no API Gateway, no NAT Gateway, and nothing always-on; the only inbound surfaces are two Lambda Function URLs, the outbound nudge goes through SNS, and work is buffered on a single SQS queue.

Lambda functions

Six functions, all Python 3.14 on arm64, all with CloudWatch Logs at 7-day retention. Each does one job and hands off; the SQS queue (upsr-jobs, with upsr-jobs-dlq as its dead-letter queue after five attempts) decouples the public webhook from the slower model call.

• upsr-order — a public surface. Backs the order Function URL: verifies the store’s signature, de-duplicates against upsr-orders with a conditional write on the order id, checks upsr-optout and the frequency cap on upsr-customers, gathers the basket and history, and enqueues a suggest job. Nothing slow happens here — no catalogue join, no model call.
• upsr-picker — SQS-triggered on suggest jobs. Makes the single Bedrock call to propose a ranked shortlist, applies the deterministic catalogue filters (in stock, complementary, price-appropriate, not already owned) against upsr-catalogue, and writes the top survivor with a send_at to upsr-offers — or writes a no-fit record and stops. The only function with bedrock:InvokeModel.
• upsr-sender — scheduled. Queries upsr-offers by the state-and-send-at index for offers whose delay has elapsed and are still chosen, re-checks opt-out and the cap, builds the signed one-tap link, sends one nudge via SNS, stamps the frequency cap on upsr-customers, and flips the offer to sent with a conditional update so it is never sent twice.
• upsr-addlink — a public surface. Backs the add-link Function URL: validates the signed token, records the tap against the offer in upsr-offers (marking it taken if inside the attribution window, or out-of-window if not), and issues an HTTP 302 to the store’s add-to-cart URL for the chosen SKU.
• upsr-sweep — scheduled. Queries upsr-offers for sent offers past their attribution window with no tap, marks each ignored, and emails the shop owner a rolled-up take-up report via SES.
• upsr-catalog-sync — scheduled. Pulls the product list — stock, prices, categories and complementary tags — and upserts rows into upsr-catalogue, so the picker’s filters always read current stock.

Idempotency and exactly-once

Three points in the flow must not double-act, and each is pinned by a conditional write rather than by hope. First, one offer per order: upsr-order writes upsr-orders with a condition that the order id doesn’t already exist, so a store that fires the webhook three times for one checkout creates exactly one job. Second, one nudge per offer: upsr-sender flips the offer from chosen to sent with a condition on the current state, so two overlapping scheduler runs can never both send — the loser’s conditional update simply fails. Third, one terminal state per offer: both upsr-addlink and upsr-sweep only transition an offer that is still sent, so a tap that races the sweep resolves to a single outcome, and a re-run of the sweep skips anything already closed. The offers table’s state field is the single source of truth for where each offer is, and every transition is a guarded write.

Data stores, schedules, and messaging

• DynamoDB (all on-demand). upsr-orders — PK order_id; one item per order with its dedup marker, basket snapshot, and gate outcome. upsr-offers — PK offer_id, holding the order, customer, chosen SKU, line, send_at, and state (chosen / sent / taken / out-of-window / ignored / no-fit); a GSI on state + send_at drives the sender and sweep queries, and the attribution window is a TTL attribute. upsr-catalogue — PK sku, the product mirror with stock, price, category and complementary tags. upsr-customers — PK customer_id, the history plus the frequency-cap fields (last_nudged_at, nudge_count). upsr-optout — PK customer_id, the suppression list checked before every send.
• Function URLs. Two — upsr-order for the inbound webhook and upsr-addlink for the one-tap redirect, both AuthType NONE at the edge because authenticity is enforced in-function: the webhook by the store’s shared-secret signature, the add link by the signed token it carries. No API Gateway.
• SNS and SES. SNS sends the outbound nudge (or SES does, if you send the offer by email); SES sends the internal take-up report to the shop owner from a verified domain with DKIM.
• EventBridge Scheduler. Three rules — upsr-sender at rate(5 minutes) to release due offers, upsr-sweep at rate(1 hour) to close windows, and upsr-catalog-sync at rate(15 minutes) to refresh stock and prices.
• Secrets Manager. Two secrets — the webhook signing secret and the store API key — fetched at call time, never in env vars or the catalogue sheet. The add-link token signing key lives here too, alongside the webhook secret.
• Bedrock. Model id anthropic.claude-haiku-4-5 via the Global cross-Region inference profile, invoked only by upsr-picker.

IAM scope, observability, and region

Each function gets its own execution role scoped to exactly what it touches, no wildcards. upsr-order can read upsr-customers and upsr-optout, conditionally write upsr-orders, read the signing secret, and send to upsr-jobs — it cannot call Bedrock, SNS, or the catalogue. upsr-picker is the only role with bedrock:InvokeModel, scoped to the one Haiku profile; it reads upsr-catalogue and writes upsr-offers, and cannot send anything. upsr-sender can read upsr-offers, upsr-optout and upsr-customers, publish to SNS, and update the offer and cap — but cannot call Bedrock. upsr-addlink can validate the token, update upsr-offers, and redirect, nothing more. The scheduled functions hold the narrow catalogue, offers and SES permissions they need and, apart from upsr-addlink, no inbound surface at all. Everything runs in eu-west-2; the only cross-Region path is Bedrock’s Global inference profile, which routes the model call for capacity and is not a data store. CloudWatch Logs at 7-day retention carry every function’s trace, and an AWS Budgets alarm watches the monthly spend — with Bedrock and SMS the lines most likely to move, it’s the cheapest early warning that volume (or a runaway loop) is running hot.

That’s the whole system: an order comes in, a model proposes and the catalogue rules dispose, one tasteful nudge goes out with a one-tap link, and every offer — taken, ignored, or never sent — ends up counted. No always-on compute, one small model call per order, and a hard filter that means a bad suggestion can never reach a customer. It sells the add-on that actually fits, and stays quiet when none does.

All posts