How a background gets removed

Why a container Lambda, and not a zip

Every other function in this system is a small zip package: a few kilobytes of Python plus Pillow. Background removal can’t be, and the reason is size. A zip-deployed Lambda is capped at 250 MB unzipped, and a background-removal model doesn’t fit — the U^2-Net weights alone are tens of megabytes, the inference runtime that loads them is much larger, and together with their dependencies they blow past the limit comfortably. A container-image Lambda raises that ceiling to 10 GB, which is room to spare. So this one function is built as an image: the model weights are baked into the image at build time, so they’re already on disk when the function starts rather than downloaded on every call, and the whole thing is pushed to a private registry and deployed from there.

It runs on arm64, which is cheaper per unit of compute than x86, and with more memory than the rest — around 6 GB. The memory figure isn’t about holding the image in RAM; it’s that on Lambda, CPU is allocated in proportion to memory. Asking for 6 GB gets you several vCPUs, and since this is the one genuinely CPU-bound step in the pipeline, the extra cores cut the wall-clock time of each cutout, which is what you actually pay for. A bigger, faster function that runs for twenty seconds can cost less than a small, starved one that grinds for two minutes.

What the model actually produces

The function reads the clean working image, hands it to the segmentation model, and gets back an alpha mask — a greyscale image the same size as the photo, where each pixel says how much of the product is there: white for “definitely product,” black for “definitely background,” and the soft greys in between for the fuzzy edges, the wisps of a candle wick, the slight blur where a glass curves away. The default engine is rembg driving U^2-Net, a model built for exactly this — salient-object detection, the “what’s the subject of this photo” problem — and an alpha-matting pass cleans up those soft edges so the cutout doesn’t end up with a hard, scissored outline.

The mask is then used to composite: the product’s pixels are kept, everything the mask calls background is made transparent, and the result is written as a 32-bit PNG — work/<job_id>/cutout.png — with a real alpha channel. That transparent PNG is the only thing the next stage needs; the kitchen counter, the toaster, and the windowsill are gone. Crucially, the model’s job ends here. It decides which pixels are product; it does not crop, resize, choose a channel, or know what a watermark is. One job, cleanly bounded.

Knowing when it went wrong

A segmentation model is confident even when it’s wrong, so the function doesn’t ask it “are you sure?” — it measures the result instead, on cheap, deterministic signals that don’t need a second model. The main one is coverage: what fraction of the frame the cutout keeps. A normal product photo, after trimming, leaves something between roughly 8% and 92% of the pixels as product. Near zero means the model decided almost everything was background — it ate the product. Near 100% means it kept almost everything — it failed to find a subject at all, usually a busy or low-contrast photo. Both fall outside the safe band.

The function also looks at how fragmented the mask is — a clean cutout is one connected blob, whereas a result speckled into many disconnected islands is the signature of the see-through-vase failure, where the model carves holes through the middle of a transparent object. When coverage is outside the band, or the mask is badly fragmented, the job is marked needs review, the variant step is not enqueued, and the owner gets a notification with the original photo and the attempted cutout side by side. The channel variants are only ever built from a cutout that passed. That single rule — never build from a cutout you don’t trust — is what keeps a bad mask from becoming three published images with a hole in them.

The hosted alternative

The default is the local model, because it has no per-call fee and no third party in the loop — once the image is built, a cutout costs only the Lambda time it runs for. But the function is written so the engine is swappable: point it at a hosted background-removal API instead, with the API key held in Secrets Manager and fetched at call time, and everything around it — the normalisation, the confidence gate, the review lane — stays exactly the same. You’d reach for the hosted option if you wanted a particular vendor’s quality on difficult subjects like hair or jewellery and were happy to pay per image for it. The architecture doesn’t care which engine produced the mask; it only cares that a mask came back and passed the gate.

All posts