Skip to content
allanninal.dev/build
Collaborate

Part 2 of 7 · Shelf price checker series ~5 min read

Reading a shelf from a photograph

Nobody is going to key four thousand labels into a spreadsheet, and the stores that try get through one aisle. A phone photograph of a shelf section holds about seven labels, and a model can read all seven in a few seconds, provided the photograph is good enough and the model is not told what it ought to find.

Key takeaways

  • Photograph by bay, so every label arrives with a position as well as a price.
  • Send the retake verdict back to the phone while the person is still in the aisle.
  • Read every field as printed. Units and prices are parsed in code, not by the model.
  • Read blind. The model never sees the till price or the last walk.
  • An unreadable label is a result, not a gap. It goes on the list as unreadable.

From a photo to label records

A shelf photo read into label recordsA vertical chain inside an AWS account container, entered from a box on the left labelled A shelf photo, carrying its store, bay and time. Good enough to read, meaning sharp, lit and square to the shelf, with a side exit reading no: Retake now, while they are still in the bay. Find the labels, meaning every shelf-edge label in the frame. Read each one, as printed, field by field. Every field legible, covering price, unit price, promotion and barcode, with a side exit reading no: Partial record, holding what it could read with the rest null. The final step is Store the record, with the photo and its place in the bay. A note says the verdict comes back in seconds for a reason, because a photo rejected an hour later means walking back to the bay, and nobody does.AWS ACCOUNTA shelf photostore, bayand timeGood enough to read?sharp, lit, andsquare to the shelfRetake nowwhile they arestill in the baynoFind the labelsevery shelf-edgelabel in the frameRead each oneas printed, fieldby fieldEvery field legible?price, unit price,promotion, barcodePartial recordwhat it could read,the rest nullnoStore the recordwith the photo andits place in the bayThe verdict comes back in seconds for a reason. A photo rejected an hour later means walking back to the bay, and nobody does.
Fig 1. Five steps and two branches. The first branch goes back to the phone; the second is kept, because a partial read is still evidence of what the shelf said.
  • Database
  • Machine learning
  • Management
  • Analytics
  • Front-end & mobile

Bay first, then photos

Every store already numbers its aisles and bays for planograms and replenishment, and the walk page asks for that number before it opens the camera. That one field does more work than anything the model reads. It turns a photograph into a place, which is what lets part three tell a correct label from a correct label sitting under the wrong product.

The page itself is deliberately plain: a bay number, a camera button, a strip of thumbnails and a Done button when the bay is finished. It is a web page rather than an app, because installing anything on a shared store phone is a conversation with somebody who is not in the building. A photo uploaded twice, which a weak signal at the back of a store makes common, is recognised by its hash and never read twice.

The retake has to happen in the aisle

Shelf-edge strips are glossy, chilled cabinets have glass doors that reflect everything, and a photo taken at arm’s length on a busy morning is often soft. On top of the 583 photos kept from the first full walk of the largest store, 37 were rejected and retaken, almost all of them in the chilled and frozen runs.

So the verdict goes back to the phone within a few seconds, as a red border on the thumbnail. The person is still in the bay, the fix is to step to one side and shoot again, and it costs ten seconds. The same verdict delivered at the end of the walk costs a second walk, which in practice means the bay is simply not checked.

What came back from the first walk

Labels found in one walk, grouped by how completely they were readThree vertical bars showing the labels found in one full walk of the largest store, grouped by how completely each was read. Read in full accounts for three thousand eight hundred and seventy-four labels. Read in part accounts for three hundred and fifty-one. Unreadable accounts for ninety-one. A note says the middle bar is mostly promotion labels, whose end dates are printed small, and the third is mostly labels torn, faded or half covered by stock.02000400060008000~3874Read in full~351Read in part~91UnreadableLabels found in one full walk of the largest storeThe middle bar is mostly promotion labels, whose end dates are printed small. The third is mostly labels torn, faded or half covered by stock.
Fig 2. The same 4,316 labels, grouped by how completely they were read. Nearly nine in ten needed nothing from a person.

Read as printed, parse in code

The model returns what the label says, in the characters it says it: £1.25, 65p, 2 for £3, £2.50 per kg, 44.6p/100g. It does not convert pence to pounds, per 100 grams to per kilogram, or a multibuy into a price per item. Every one of those conversions happens afterwards in ordinary code, where a slipped decimal point is a bug that fails a test rather than a plausible number in a report.

This matters more here than in most extraction jobs, because the mistakes are small. A model asked to normalise units will usually turn 44.6p per 100 grams into £4.46 per kilogram, which is right, and occasionally into £44.60, which is not. Both look like prices.

Why the read is blind

It would be easy to hand the model the till price file and ask it to check the labels. It would also destroy the point of the exercise. A model told that a product costs £1.85, looking at a slightly blurred label, will read £1.85, and a mismatch that exists will be read out of existence.

So the read knows the store, the bay and nothing else. Its only job is to report what a customer standing in front of that shelf would see. Deciding whether that is right is the next step, and it is done by a comparison that cannot be persuaded.

What a label record holds

  • Where. Store, bay, photo, and position in the frame.
  • When. The time the photo was taken, from the upload, not from the read.
  • Price as printed. The characters on the label, never a converted value.
  • Unit price as printed. The figure and the unit, kept separately.
  • Promotion. The mechanic wording and end date, where printed.
  • Identifiers. Barcode digits or shelf code where legible, and the description.
  • Legibility. Which fields were read, and which came back null.

Unreadable is an answer

Ninety-one labels could not be read at all. The temptation is to drop them, because there is nothing to compare. They go on the manager’s list instead, as unreadable, because a label a camera cannot read at arm’s length is usually a label a customer cannot read either, and a price has to be clearly legible to count as displayed.

The next post is where the reading stops and the judgement starts: which product a label belongs to, and which price it should be compared with.

All posts