Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Cold chain monitor series ~6 min read

A cold chain monitor on AWS for a few dollars a month

A freezer alarm goes off at ten past three in the morning. Somebody silences it from their phone because the last four were door-open spikes during the evening restock. This one was the compressor. By seven the contents are at minus four and the question is what to do with eleven thousand pounds of stock and no record of exactly what happened when.

Young woman focused on a computer and documents at an office desk
Photo by cornerstone accounting on Unsplash

Key takeaways

  • An excursion is normal; a breach is an excursion with duration. Alarm on the second.
  • Missing data is a breach. Silence must never read as compliance.
  • A sensor that has failed often looks better than one that is working.
  • The system never decides whether stock is safe. It produces the evidence.
  • Designed on AWS for about $3 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: temperature readings recorded, judged and escalatedThree boxes across the top sit outside the AWS account. On the left, Sensors in each unit, reporting or not. In the middle, The unit's rules: its range, and how long out of it is allowed. On the right, Whoever is on call, and whoever signs. Each connects by an arrow to the AWS account container below. Readings, or silence, flow down into the account. Limits per unit feed in. An alarm that does not stop goes back out. Inside the AWS account are three components in a row. On the left, the Recorder, capturing readings and the gaps between them. In the middle, the Judge, deciding excursion or breach. On the right, the Escalator, continuing until somebody acknowledges. A note at the bottom says the recorder's second job, noticing gaps, is the one that catches dead sensors.AWS ACCOUNTSensorsin each unit,reporting or notThe unit's rulesrange, and how longout of it is allowedWhoever is on calland whoever signsRecorderreadings, andthe gaps betweenJudgeexcursion, orbreach?Escalatoruntil somebodyacknowledgesreadings, orsilencelimits per unitan alarm that doesnot stopThe recorder's second job -- noticing gaps -- is the one that catches dead sensors.
Fig 1. Three things outside the account, three pieces inside it. The escalator on the right does not give up, which is a deliberate departure from how most alerting works.
  • Compute
  • Database
  • Management
  • Analytics
  • People

The rule that shapes everything

Fail closed. A sensor that has stopped reporting is treated exactly as if it were reporting a breach, because from a food safety point of view those two situations are identical: nobody knows what the temperature is.

This sounds obvious and is the opposite of how most monitoring is built. The default behaviour of almost every alerting system is to alarm on bad values and stay silent on no values, which means the most complete failure produces the calmest dashboard.

What runs (the inside)

  • The recorder. Takes readings at a fixed interval and treats a missed interval as an event in its own right. Part 2.
  • The judge. Distinguishes a door-open excursion from a genuine breach, using duration rather than a single reading. Part 3.
  • The escalator. Keeps going until a person acknowledges, and catches the sensor that is lying. Parts 4 and 5.

One night, end to end

One night of temperature monitoring from a door-open spike to a real breachA horizontal row of five boxes joined by arrows. Eighteen forty, door open: two degrees above range for four minutes. Excursion: logged, no alarm. Two ten, rising and not returning. Two twenty-two, breach: twelve minutes out of range. Escalated until acknowledged. A note says the first two boxes are why the fourth one gets taken seriously.ONE NIGHT, END TO END18:40 door open+2C for 4 minExcursionlogged, no alarm02:10 risingand not returning02:22 breach12 min out of rangeEscalateduntil acknowledgedThe first two boxes are why the fourth one gets taken seriously.
Fig 2. The same night as one line. Not alarming at half past six is what makes the alarm at twenty past two credible.
  • Compute
  • Machine learning
  • Management
  • People

In plain words

At twenty to seven in the evening somebody opens the walk-in to restock. The temperature rises two degrees over four minutes and comes back down. That is an excursion, it is entirely normal, and it is recorded and not alarmed on.

At ten past two in the morning the temperature starts rising with nobody near it. Twelve minutes later it is still out of range and still climbing, which crosses the breach rule for this unit: out of range for more than ten consecutive minutes.

The alarm goes to the person on call. It repeats. If it is not acknowledged in fifteen minutes it goes to a second person, and then to a third, and it does not stop. When somebody does acknowledge, they get the last six hours of readings on a chart, not just the current number, because what has been happening matters more than what is happening.

Design rules that shaped every decision

  • Fail closed. Missing data is a breach.
  • Alarm on duration out of range, never on a single reading.
  • Every alarm carries the recent history, not just the current value.
  • Escalate to a second and third person. An unacknowledged alarm is not handled.
  • Records are append-only and timestamped at the sensor, for inspection.
  • The system never says whether stock is safe. A person decides and signs.

Why this shape

Temperature monitoring is a solved problem in the sense that the sensors are cheap and the arithmetic is trivial, and it fails constantly in practice for reasons that are entirely about human behaviour: too many false alarms, so alarms get silenced; a sensor dies and nothing notices; the record turns out to be a spreadsheet somebody filled in from memory.

So the design spends its effort on being believed. Not alarming on door openings, alarming loudly on silence, escalating rather than giving up, and keeping a record that cannot be quietly adjusted.

The next four posts walk through each piece: how a reading becomes a record, how an excursion is told from a breach, how a broken sensor is caught, and who decides what happens to the stock. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts