Skip to content
allanninal.dev/build
Collaborate

Part 7 of 7 · Product recall tracer series ~7 min read

Engineering reference: the product recall tracer architecture

The same system with the service names filled in: what reads a delivery, what reads a notice, what runs the trace, and where the register and the evidence sit.

Key takeaways

  • Single region, single account. Every resource is regional; nothing is global except the IAM roles.
  • 5 Lambda functions, each with its own execution role. No shared role, no wildcards on resources.
  • 3 DynamoDB tables, each keyed so the concurrency story is a condition expression rather than a lock.
  • One Bedrock model, called once, with a JSON schema it must fill or leave null.
  • Nothing always-on: no instance, no container, no provisioned capacity.

The system, by service name

The product recall tracer drawn with AWS service namesThree boxes across the top sit outside the AWS account. Delivery capture, photos from a phone. Recall notices, from an agency feed or a mailbox. Stock and sales, CSV exports arriving nightly. Each connects to the AWS account container below. Inside are three components. S3 with EventBridge handling uploads, feeds and the nightly load. Five Lambda functions covering goods-in, notice, trace, match and evidence. And three DynamoDB tables holding batches, notices and traces. A note says us-east-1, one account, deliveries arrive all day, notices are rare and urgent, and the sales load is nightly and never on the critical path of a trace.AWS ACCOUNTDelivery capturephotos from aphoneRecall noticesagency feed ormailboxStock and salesCSV exports,nightlyS3 + EventBridgeuploads, feeds andthe nightly loadLambda x5goods-in, notice, trace,match, evidenceDynamoDB x3batches, notices,tracesingroundsoutus-east-1. One account. Deliveries arrive all day; notices are rare and urgent; the sales load is nightly and never on the critical path of a trace.
Fig 1. The same shape as Part 1 with the service names filled in. Nothing here is new; it is the same three groups, named.
  • Compute
  • Storage
  • Database
  • Management
  • Front-end & mobile

Region and account

  • Region: us-east-1. Chosen because SES inbound receipt rules exist in only a subset of regions and this one has the widest Bedrock model availability. If your data has to stay elsewhere, check both constraints before moving: inbound SES is the binding one.
  • Account: one. This is a small system, and a separate account per environment costs more in wiring than it saves. A dev and a prod stack in the same account, with distinct resource prefixes, is the right size here.
  • Everything is regional. The only global resources are the IAM roles and policies. There is no CloudFront, no global table and no cross-region replication, because nothing here has a latency or durability requirement that would justify them.

Lambda inventory

FunctionTriggerDoesTimeout / memory
rt-goods-inS3 put, delivery prefixReads a note or label photo into batch rows; opens a balance60s / 1024MB
rt-noticeSES inbound or feed pollReads a notice into products, batch codes, dates and hazard60s / 1024MB
rt-traceAPI, on demandRuns a stored scope against batches, stock and sales120s / 1024MB
rt-matchStep after traceResolves traced sales to nameable customers; counts the rest120s / 1024MB
rt-evidenceStep after matchFreezes inputs, queries, results and negatives into one object300s / 2048MB

Splitting this into separate functions is not about modularity. It is that only one of them needs Bedrock permissions and only one is reachable from the public internet, and neither of those is true if it is one handler behind a router.

IAM, scoped

RoleAllowedOn
rt-goods-in-rolebedrock:InvokeModel, s3:GetObject, dynamodb:PutItemOne model id; the deliveries prefix; batches
rt-notice-rolebedrock:InvokeModel, s3:GetObject, dynamodb:PutItemOne model id; the notices prefix; notices
rt-trace-roledynamodb:Query, s3:GetObject, dynamodb:PutItembatches and notices read; traces write; the sales prefix
rt-evidence-roledynamodb:Query, s3:PutObject, ses:SendEmailAll three tables read; the evidence prefix write-once; one verified identity

No role has a Resource: “*” on anything that writes, and every GetSecretValue grant names a single secret arn. That is why there is more than one secret rather than one JSON blob with everything in it.

DynamoDB schemas

Table: batches

PK   site#product_code S
SK   batch_code        S   verbatim, exactly as printed
     best_before       S   separate field, never derived from the batch
     supplier_id       S   one step back
     received_on       S   ISO date
     qty_received      N
     qty_remaining     N   drawn down; a batch is a balance, not an event
     confidence        S   scanned | read | assumed
     evidence_key      S   the photograph the code came from

batch_code is stored verbatim and never normalised at ingest. Two real
batches merged by a tidy-up are unrecoverable; a fuzzy match at trace
time is reversible and a human can see what it did.

Table: notices

PK   notice_id         S
SK   ’#claim’          S   one item per notice
     source            S   agency | supplier | internal
     received_at       S
     action            S   withdrawal | recall
     hazard            S   free text, from the notice
     product_codes     L
     batch_codes       L   what the notice actually named
     best_befores      L   often named instead of a batch
     window_from       S   null means explicitly open
     window_to         S
     widened_reason    S   set only when the scope is broader than the codes

widened_reason is null on a normal trace. A non-null value is the record
that somebody deliberately searched wider than the notice specified.

Table: traces

PK   notice_id         S
SK   trace_id          S   one item per run; runs are never overwritten
     ran_at            S
     qty_received      N
     qty_on_hand       N   observed
     qty_sold_est      N   inferred
     qty_sold_low      N   the range, where FIFO leaves one
     qty_sold_high     N
     basis             S   scanned | picked | fifo
     matched_customers N
     unmatched_units   N   reported as prominently as the matched count
     negatives         L   batch codes checked and not held
     evidence_key      S

qty_on_hand and qty_sold_est are deliberately separate fields with
separate names. Summing them into one ’affected’ number is the error the
whole table exists to prevent.

Inbound and outbound

  • Deliveries arrive as photographs from whatever phone is at the back door. There is no app to install and no terminal to buy, because the step has to survive a busy morning or it will not happen at all.
  • Notices arrive by mailbox or feed and both land in the same function. A supplier email and an agency alert are the same document for our purposes.
  • Stock and sales are parsed, not read by a model. They are fixed-header CSV and they are the largest files in the system.
  • A trace runs on demand and is never scheduled. It is caused by a notice, it is frozen when it finishes, and a follow-up question is answered from the frozen result rather than by running it again.

The model call

  • One call per delivery, one per notice. Nothing per unit, per sale or per customer. The bill does not move when trade does.
  • A mid-tier model for deliveries. A delivery note is short and structured and a case label is mostly character recognition; paying for a frontier model here buys nothing.
  • A capable model for notices, because there are a handful a year and a batch code read wrongly is the one error nothing downstream can catch.
  • Codes come back as strings, never as numbers. 0041 parsed as an integer is 41, and it will match nothing for the rest of the system’s life.
  • No model touches the trace. The forward trace is a query and an assumption, both of which have to be re-runnable and explainable years later.

Things worth knowing before you build it

  • Store the batch code verbatim. Normalising at ingest can merge two real batches, and that is not recoverable; compare fuzzily at trace time where a person can see the match.
  • Keep best-before in its own field. Roughly half of published notices name a date rather than a lot code, and deriving one from the other loses both.
  • Never sum on-hand and sold into a single affected figure. One is counted and the other is inferred, and the sum inherits the weaker claim without saying so.
  • Report the unmatched customer count everywhere the matched count appears. The ratio between them is what decides whether a public notice is required.
  • Freeze the trace. Re-running against live stock a week later produces a different answer to the same question, which is exactly what an audit trail exists to prevent.
  • Do not build an identity graph to raise the match rate. It is a large new data protection surface, and a probabilistic identification is a poor basis for a safety notice.

That is the whole system. Seven posts, one diagram at a time, and nothing in it that needs a server.

All posts