allanninal.dev/build

Part 2 of 7 · Product image cleaner series ~7 min read

How a raw photo gets ingested

Before anything can be cleaned up, the photo has to arrive and be turned into something predictable to work on. This post is about that first step alone: the lanes that get a snapshot into S3, the S3 upload event that kicks off the work, and the unglamorous normalising — rotate, convert, downscale, record — that every later stage quietly depends on.

Key takeaways

  • Three lanes get a photo in — a watched Drive folder, an email-a-snapshot address, and a direct upload — all landing in one S3 inbox/ prefix.
  • The pipeline is started by the S3 upload event itself, routed through EventBridge to a queue. Nothing polls and nothing waits.
  • Ingest fixes what phones get wrong: EXIF rotation, HEIC conversion, and a cap on the longest edge so big snaps don’t cost extra.
  • A job row is opened in DynamoDB the moment a file is accepted, so every photo has a status from its first second.
  • Non-images and oddities are rejected at the door — the heavy cutout step never runs on a PDF or a screenshot.

Three ways in, one place they land

The point of the ingest step is to make everything that follows simple, and the way it does that is to collapse every possible way a photo might arrive into one fact: an object now exists under the inbox/ prefix of the images bucket. How it got there doesn’t matter to anything downstream.

There are three lanes, chosen so the owner can use whichever is least friction in the moment. A watched folder — a Google Drive folder that a small scheduled function mirrors into inbox/ every few minutes — suits a batch of photos taken at a desk. An email lane — forward or send a photo to a dedicated address, and SES writes the attachment straight to inbox/ — suits a single shot taken on the shop floor, sent before you’ve put the phone down. And a direct upload straight into the prefix covers anyone wiring this into their own tooling. Three doors, one room.

The upload event, not a poll

What turns a dropped file into work is the S3 upload event. The images bucket has event notifications switched on; when an object is created under inbox/, S3 emits an Object Created event to EventBridge. A single rule matches that prefix and forwards the event to an SQS queue, and the queue triggers the ingest Lambda. There is no schedule checking the folder, no “is anything new?” loop, and therefore no cost or delay while nothing is happening. A photo that lands at 2am is picked up at 2am; a quiet week costs nothing because nothing fires.

The queue between the event and the function is doing real work, not decoration. If someone uploads forty photos from a shoot at once, forty events arrive together; the queue absorbs the burst and lets the functions work through it at their own pace, and a file that fails to process is retried and, after a few attempts, parked in a dead-letter queue rather than lost or retried forever. The upload event is the spark; the queue is what keeps a busy afternoon from becoming a thundering herd.

Ingest: three lanes into one S3 prefix, an upload event, and normalisation On the left, three external lanes stacked vertically: a watched Drive folder mirrored by a scheduled sync function; an email address whose attachments SES writes out; and a direct upload. All three arrows converge on a single S3 bucket box labelled inbox slash, inside the AWS account container. From that bucket, an arrow labelled Object Created event goes to an EventBridge box; from EventBridge an arrow goes to an SQS queue with a dead-letter queue; from the queue an arrow goes to the Ingest Lambda. The Ingest Lambda has three outputs: it writes a normalised working image to a work slash prefix in S3; it opens a job row in a DynamoDB table picr-jobs; and it enqueues the next stage, cutout. A small note shows that non-image files are rejected at the door and never reach cutout. A bottom note reads: the upload event starts everything — nothing polls the folder. Watched folder Drive, synced Email a snapshot SES inbound Direct upload into the prefix AWS account inbox/ S3, versioned EventBridge prefix rule event picr-work SQS + DLQ picr-ingest validate & normalise work/ clean working image picr-jobs DynamoDB job row enqueue cutout next stage The upload event starts everything — nothing polls the folder, and a non-image is rejected before cutout runs.
Fig 2. Three lanes converge on one inbox/ prefix. The S3 upload event flows through EventBridge to a queue and triggers picr-ingest, which writes a clean working image, opens a job row, and enqueues the cutout step.

What normalising actually does

Phone photos are messy in predictable ways, and ingest exists to make them boring before any expensive step sees them. It does four things. It reads the EXIF orientation tag and physically rotates the pixels the right way up, because a photo that merely says it’s rotated will be cut out sideways. It converts HEIC — the format iPhones default to — into a format the rest of the pipeline reads without special handling. It caps the longest edge, downscaling, say, a 4,032-pixel snap to around 2,000 pixels, which is more than enough for every output variant and roughly a quarter of the work for the container Lambda to chew through. And it strips the rest of the EXIF block, so the customer’s GPS location doesn’t quietly travel out with a published image.

The output is a single clean working image written to work/<job_id>/source.jpg — correctly oriented, a sensible size, no surprises — and a job row in DynamoDB keyed by the job id, opened with status ingested. From this point every photo in the system has an identity and a status, which is what makes the review lane and the daily sweep possible later.

What ingest refuses

The cheapest place to stop a mistake is at the door, before the one expensive step runs. Ingest validates that the object really is a raster image it can open — a JPEG, PNG, HEIC, or WebP — and quietly rejects everything else: a PDF someone dragged in by accident, a screenshot, a zero-byte file from a failed upload, a video. A rejected file gets its job row marked rejected with a reason and is moved out of inbox/ so it isn’t picked up twice; the owner sees it on the next notification rather than wondering why nothing happened. Nothing that isn’t a photo of a product ever reaches the container Lambda, which is exactly where you don’t want to be spending money on a misfire.

Design rules for the way in

  • One landing place. Every lane ends as an object in inbox/; nothing downstream knows or cares which lane it was.
  • The event is the trigger. S3 → EventBridge → queue → function; no polling, no idle cost.
  • A queue absorbs the burst. Forty photos at once are smoothed out, retried on failure, and dead-lettered if they keep failing.
  • Normalise early. Rotate, convert, downscale, and strip EXIF before any expensive step looks at the file.
  • Reject at the door. Non-images never reach the container Lambda; the costly step only ever sees a real photo.
All posts