allanninal.dev/build

Part 2 of 7 · Lead enricher series ~8 min read

How a raw lead gets normalized

A raw lead arrives looking like a person typed it in a hurry, because one did. The name is “jane DOE”, the phone is “(0) 7700 900123”, the email has a trailing space and a capital letter, and the country is blank. None of that is wrong, exactly — it’s just not in a shape anything downstream can match or trust. This post is about the first stage: turning the messy, partial fields of a raw lead into clean, canonical ones, deterministically where it can and with a small model call only where fuzzy judgement actually helps.

Key takeaways

  • Normalising turns the messy, partial fields of a raw lead into canonical ones: clean name, one email form, an E.164 phone, a resolved country.
  • Most of it is plain deterministic Python — case-fixing, trimming, lower-casing, phone parsing — with predictable, testable results.
  • One small Bedrock Haiku 4.5 call handles only the fuzzy bits: splitting an awkward full name, or inferring a country from thin clues.
  • The original raw payload is kept untouched in S3; normalisation writes a new clean copy, so nothing is ever destroyed in place.
  • If a field can’t be normalised confidently, it’s left blank and flagged — the enricher never guesses a value it can’t support.

Why normalise first

Everything downstream depends on clean fields. The enrichment lookup needs a real email domain, not Jane.Doe@Acme-Widgets.CO.UK  with a stray capital and a trailing space. The dedupe stage matches on normalised keys — if one copy of Jane has +447700900123 and another has 07700 900123, they won’t line up and she’ll land twice. The junk filter needs a parsed email to check the domain against a blocklist. So normalisation runs first, and its single job is to take whatever shape the lead arrived in and produce canonical fields the rest of the pipeline can rely on.

The work splits cleanly into two kinds. Most of it has a single right answer and no judgement involved — trim the whitespace, lowercase the email, parse the phone. That’s plain Python, fully deterministic, and easy to test. A smaller part is genuinely fuzzy — is “Jan Van Der Berg” a first name of “Jan” and a surname of “Van Der Berg”, or something else? — and that’s where one small model call earns its place. The design keeps the two strictly separate, and never lets the model touch a field the rules can already settle.

The deterministic pass

The first pass is all rules, applied field by field:

  • Email. Trim, lowercase, strip a display name if the value arrived as "Jane Doe" <jane@acme.co.uk>, and validate the basic shape. The domain is pulled out and kept separately — it’s the key the enrichment stage will use. A plus-tag like jane+newsletter@acme.co.uk is preserved in the stored email but the bare form is also recorded for matching.
  • Phone. Parsed against the resolved country into E.164 (+447700900123), the single canonical form the dedupe index keys on. A number that can’t be parsed to a valid form is left blank and flagged, rather than stored half-formatted — a half-parsed number is worse than none, because it matches nothing and looks deceptively complete.
  • Name. Whitespace collapsed, obvious all-caps or all-lowercase fixed to title case with a sensible particle list (so “McDonald” and “van der Berg” survive), and surrounding junk like trailing commas removed.
  • Country. Resolved from explicit fields first (a country dropdown, a postal code pattern), then from the phone’s dialling code, then from the email’s country-code top-level domain as a weak last hint. Each source carries a confidence, and the strongest wins.

By the end of this pass, most leads are fully normalised with no model call at all. The clean fields are written as a new record in S3 alongside — never overwriting — the raw original.

The normalise stage: a deterministic pass, then a model call only for what’s left fuzzy On the left, a box labelled Raw lead listing messy fields: name jane DOE, email with a capital and a trailing space, phone 07700 900123, country blank. An arrow leads into a dotted AWS account container. Inside, the ler-normalize Lambda contains two stacked stages. The top stage, Deterministic pass, is plain Python: trim and lowercase the email, parse the phone to E.164, fix name casing, resolve the country from explicit fields then dialling code. A decision diamond below it asks: anything still ambiguous? A No arrow goes straight to the output. A Yes arrow goes to the second stage, a Bedrock Haiku 4.5 call that handles only the fuzzy bits — split an awkward full name, infer a country from thin clues — whose output is then bounded by rules. Both paths converge on a box labelled Clean record written to S3, listing Jane Doe, the lowercased email, the plus-format phone, country United Kingdom. A separate arrow shows the original raw payload kept untouched in S3. A note reads: a field that can’t be normalised confidently is left blank and flagged, never guessed. Raw lead name: jane DOE email: Jane.Doe@…  phone: 07700 900123 country: (blank) ler-normalize — AWS Deterministic pass (Python) trim & lowercase email · parse phone → E.164 fix name casing · resolve country anything still fuzzy? yes Bedrock Haiku 4.5 split name, infer country — bounded by rules no Clean record → S3 Jane Doe · +447700900123 · UK into the clean record A field that can’t be normalised confidently is left blank and flagged — never guessed. The raw payload stays untouched in S3; normalisation only ever writes a new copy.
Fig 2. Normalisation is a deterministic pass first, with one small Bedrock call reserved for the genuinely fuzzy fields. The raw payload is never modified; a clean copy is written alongside it.

Where the model earns its place

Some fields resist rules. A single “full name” box that contains “Dr. Maria Garcia-Lopez” needs to become a title, a first name, and a compound surname — and the split isn’t something a regex gets right across cultures. A lead with no country field, a generic .com email, and a phone number written as “+1 ext. 204” needs a judgement call to land on the United States. These are the cases the deterministic pass flags as ambiguous and hands to one Bedrock Haiku 4.5 call.

The call is deliberately narrow. It’s given only the specific ambiguous field and asked for a structured answer — first name, last name, and country code — not invited to rewrite the whole record. Crucially, its output is then bounded by the same rules: a country it returns must be a real ISO country, a name split must reconstruct to the original characters, and anything outside those bounds is rejected and the field left blank and flagged. The model proposes; the rules dispose. That keeps a confident-sounding but wrong guess from quietly becoming a stored fact.

Because the model only fires when the deterministic pass couldn’t settle a field, most leads never reach it — which is exactly why the Bedrock line on the cost breakdown stays small. A clean lead from a UK form with separate name fields and a valid phone is fully normalised by Python alone.

What normalisation deliberately doesn’t do

It doesn’t enrich — it won’t go and find the company behind the domain, that’s the next stage. It doesn’t dedupe — it doesn’t care yet whether this Jane already exists. And it doesn’t judge whether the lead is junk; a gibberish name is normalised just like a real one and left for the junk filter to catch with full context. Each stage does one job, so each one is simple to reason about and to test. Normalisation’s contract is narrow and total: take messy fields in, hand canonical fields out, and never invent a value you can’t defend.

Why this shape

  • Deterministic first, model last. Rules settle the fields that have a right answer; the model only sees what’s genuinely ambiguous.
  • The model is fenced in. Its output must pass the same rules as everything else — a real country, a reconstructable name — or it’s rejected.
  • Blank beats wrong. A field that can’t be normalised confidently is left empty and flagged, not filled with a plausible guess.
  • The raw lead is immutable. Normalisation writes a new clean copy to S3; the original payload is always there to re-run or audit against.
  • Canonical forms are chosen for matching. E.164 phones and lowercased emails exist so the dedupe stage downstream actually lines duplicates up.
All posts