Skip to content

Part 7 of 7 · Cold chain monitor series ~7 min read

Engineering reference: the cold chain monitor architecture

The first six posts are for the person deciding whether to build this. This one is for the person building it. Same system, no analogies: the services by name, the functions, the two tables, the fail-closed sweep, and the append-only record.

Key takeaways

  • Single region, single account. Every resource is regional; nothing is global except the IAM roles.
  • 3 Lambda functions, each with its own execution role. No shared role, no wildcards on resources.
  • 2 DynamoDB tables, each keyed so the concurrency story is a condition expression rather than a lock.
  • One Bedrock model, called once, with a JSON schema it must fill or leave null.
  • Nothing always-on: no instance, no container, no provisioned capacity.

The system, by service name

The cold chain monitor drawn with AWS service namesThree boxes across the top outside the AWS account. Sensors, reporting over MQTT or HTTP every five minutes. The on-call rota, three people deep. And the Inspection export, produced as one document. Inside the account, three groups. IoT Core handling ingest and EventBridge running a five-minute sweep. Three Lambda functions named ingest, judge and escalate. And two DynamoDB tables named readings and events. A note gives the region as us-east-1, one account, and states that the system fails closed: a missed sweep is a breach, not an absence.AWS ACCOUNTSensorsMQTT or HTTP,every 5 minutesOn-call rotathree people deepInspection exportone documentIoT Core + EventBridgeingest,5-minute sweepLambda x3ingest, judge, escalateDynamoDB x2readings, eventsingroundsoutus-east-1. One account. Fail closed: a missed sweep is a breach, not an absence.
Fig 1. The same shape as Part 1 with the service names filled in. Nothing here is new; it is the same three groups, named.
  • Compute
  • Database
  • Networking
  • People

Region and account

  • Region: us-east-1. Chosen because SES inbound receipt rules exist in only a subset of regions and this one has the widest Bedrock model availability. If your data has to stay elsewhere, check both constraints before moving: inbound SES is the binding one.
  • Account: one. This is a small system, and a separate account per environment costs more in wiring than it saves. A dev and a prod stack in the same account, with distinct resource prefixes, is the right size here.
  • Everything is regional. The only global resources are the IAM roles and policies. There is no CloudFront, no global table and no cross-region replication, because nothing here has a latency or durability requirement that would justify them.

Lambda inventory

FunctionTriggerDoesTimeout / memory
cc-ingestIoT ruleAppends the reading with both timestamps; flags implausible values without discarding them5s / 512 MB
cc-judgeEventBridge, every 5 minutesChecks every sensor for a due reading; runs the consecutive and cumulative rules; opens breaches60s / 1024 MB
cc-escalateEventBridge, every minute while a breach is openRepeats and escalates until acknowledged; never stops on its own15s / 512 MB

Splitting this into separate functions is not about modularity. It is that only one of them needs Bedrock permissions and only one is reachable from the public internet, and neither of those is true if it is one handler behind a router.

IAM, scoped

RoleAllowedOn
cc-ingest-roledynamodb:PutItemReadings only; no update, no delete
cc-judge-roledynamodb:Query, dynamodb:PutItemReadings; appends to events
cc-escalate-roledynamodb:Query, dynamodb:PutItem, sns:Publish, ses:SendEmailEvents; one topic; one verified identity

No role has a Resource: “*” on anything that writes, and every GetSecretValue grant names a single secret arn. That is why there is more than one secret rather than one JSON blob with everything in it.

DynamoDB schemas

Table: readings

PK   sensor_id         S
SK   taken_at          S   the sensor’s own clock
     received_at       S   when it reached us; often differs
     celsius           N   null on a gap record
     kind              S   reading | gap | implausible | correction
     battery_pct       N
     rssi              N
     note              S   set on correction records only, with a name

No update path exists in any role. A correction is a new item of kind
`correction` referencing the original, and both appear on every chart.

Table: events

PK   unit_id           S   walkin_2
SK   opened_at         S
     kind              S   excursion | breach | gap_breach | sensor_stuck
                           | drift_suspected | calibration_due
     closed_at         S
     peak_celsius      N
     minutes_out       N   consecutive
     cumulative_today  N   minutes, for the review rule
     acknowledged_by   S   a named person, and when
     decision          S   dispose | release | testing
     decided_by        S   a named person
     reason            S   free text, in their words

A release is recorded as carefully as a disposal. The asymmetry in how
carefully those two get written down is where problems hide.

Inbound and outbound

  • Sensors report every five minutes over IoT Core. Devices that buffer during signal loss send batches, which is why both timestamps are stored.
  • The judge sweeps every sensor every five minutes rather than reacting to arrivals, because the whole point is to notice what did not arrive.
  • Unit rules are configuration — range, consecutive limit, cumulative limit — owned by whoever is responsible for the stock, not by the code.
  • Escalation runs until acknowledged. There is no maximum number of attempts and no automatic close.

The model call

  • There is no model in this system. Everything here is comparisons and durations, which is the correct level of sophistication for a safety record.
  • The tempting use is predicting failures from the temperature curve. Battery level, door-open frequency and compressor cycle time already say it more directly.
  • The wrong use is any judgement about whether stock is safe. Part 5 is entirely about why that decision needs a person’s name on it.
  • Summarising a breach for the record is defensible, and the summary sits alongside the readings rather than replacing them.
  • The cost page assumes none, which is why reading writes are the whole bill.

Things worth knowing before you build it

  • Fail closed on missing data. The default behaviour of most alerting is to stay quiet when nothing arrives, which makes total failure look like perfect health.
  • Alarm on duration, never on a single reading. Two hundred door-opening alarms a month is how the one that matters gets silenced.
  • Check for zero variance. A stuck sensor passes every range check and produces the most reassuring chart in the building.
  • Give the readings table no update or delete path in any role. An editable temperature record is not a record.
  • Record releases as carefully as disposals, with a name and a reason. Nothing visible happens on a release, which is exactly why it gets written down badly.

That is the whole system. Seven posts, one diagram at a time, and nothing in it that needs a server.

All posts