allanninal.dev/build

Part 4 of 7 · Lead enricher series ~7 min read

How a duplicate gets caught

The same person reaches you more than once. They fill in the form on Monday, reply to an ad on Thursday, and get imported from a list at the weekend — three enquiries, one human. If each one lands in the CRM as a fresh contact, your sales pipeline quietly doubles and nobody trusts the numbers. This post is about the dedupe stage: how the enricher checks a clean, enriched lead against the people already in your CRM, catches the duplicate even when the details don’t line up exactly, and links rather than blindly merges.

Key takeaways

  • A DynamoDB identity index mirrors the people already in your CRM on normalised keys: lowercased email, E.164 phone, and a name-plus-company fingerprint.
  • An exact-key check catches the obvious repeats instantly and for almost nothing — most duplicates are caught here.
  • A fuzzy pass compares normalised name, company, and domain for near-matches that don’t share an exact key.
  • Only genuinely ambiguous near-matches go to Bedrock Haiku 4.5 for a same-person / different-person judgement.
  • A confident match links the lead to the existing record; a confident non-match is written as new. It links, it never blindly merges.

One person, many enquiries

The same human reaches you through more than one door. Jane fills in the website form on Monday, replies to an email campaign on Thursday, and turns up in a list import at the weekend. That’s three enquiries and one person. If each one lands in the CRM as a fresh contact, the pipeline count doubles, the same person gets three follow-up emails, and every report built on contact counts is quietly wrong. The dedupe stage exists to make sure a lead that’s really an existing person gets attached to that person, not added as a stranger.

The hard part is that the three enquiries rarely look identical. The form gave a work email; the campaign reply came from a personal one. One has a phone, one doesn’t. The name is “Jane Doe” once and “Jane M. Doe” another time. Exact string matching catches the easy cases and misses the ones that matter most. So dedupe runs in layers: cheap and certain first, fuzzy where it has to be, and a model only for the genuinely undecidable.

The identity index

Dedupe doesn’t query the CRM on every lead — that would be slow, rate-limited, and expensive. Instead it reads a small DynamoDB identity index that mirrors the CRM. Each person in the CRM has entries under a few normalised keys:

  • Email key — the lowercased, plus-stripped email address.
  • Phone key — the E.164 phone number.
  • Name-and-company fingerprint — a normalised, lower-cased combination of the person’s name and their company, used when neither email nor phone lines up.

The index is seeded once from a CRM export and kept current: every time the push stage writes or links a record, it updates the index too. Because the keys are exactly the canonical forms the normalise stage produces, a new lead’s keys can be looked up directly — no re-cleaning, no guessing.

The dedupe stage: exact-key check, then fuzzy pass, then a model judgement only for the ambiguous A clean enriched lead enters a dotted AWS account container holding the ler-dedupe Lambda. First, an Exact-key check looks the lead’s email, phone, and name-plus-company keys up against a DynamoDB identity index that mirrors the CRM. A decision diamond asks: exact match on any key? A Yes arrow goes to an outcome box, Link to existing record. A No arrow goes to a Fuzzy pass that compares normalised name, company, and domain and produces a similarity score. A second decision diamond asks: confident? Three branches leave it: clearly the same goes to Link to existing record; clearly different goes to Write as new; ambiguous goes to a Bedrock Haiku 4.5 box that makes a same-person or different-person judgement, whose answer then routes to either Link or Write as new. A note reads: it links to a record, it never blindly merges — and it never overwrites a field a human already entered. Enriched lead ler-dedupe — AWS Exact-key check Identity index DynamoDB exact match? yes Link to existing no Fuzzy pass name, company, domain → score confident ? same Link to existing diff. Write as new ambiguous Bedrock Haiku 4.5 same person or not? routes to link / new It links to a record — it never blindly merges, and never overwrites a field a human already entered.
Fig 4. Dedupe runs cheap and certain first: an exact-key check against the identity index, then a fuzzy pass, with Bedrock judging only the genuinely ambiguous near-matches.

Exact first, fuzzy second, model last

The first check is exact and instant. The lead’s email key, phone key, and name-and-company fingerprint are looked up directly in the identity index. A hit on any of them is a confident duplicate — the same email is the same person — and the lead is linked to the existing record with no further work. This catches the large majority of real duplicates, costs a couple of DynamoDB reads, and involves no model at all.

When no exact key matches, the fuzzy pass runs. It pulls the small set of candidate records that share something — the same company, a similar surname, the same email domain — and scores each against the lead on normalised name similarity, company match, and domain. A high score with corroborating signals (same company, same domain, near-identical name) is treated as the same person; a low score is a clean stranger. Both of those are decided by the deterministic score against tuned thresholds — still no model.

Only the middle band — a near-match the score can’t confidently call — is handed to Bedrock Haiku 4.5. “J. Doe at Acme” with a personal email versus “Jane Doe at Acme Widgets” with a work email: plausibly the same person, plausibly two. The model is given just the two candidate records and asked for a same-person / different-person verdict with a brief reason. Its answer routes the lead to link or write-as-new, and that judgement is logged. Because the model only sees the ambiguous slice, it’s a small fraction of leads — which keeps both the cost and the risk down.

A confident match doesn’t mean overwriting the existing record with the new one. The enricher links: it attaches the new enquiry to the existing person and fills only genuinely empty fields with the new lead’s sourced data. If the CRM record already has a phone number and the new lead has a different one, the enricher does not replace it — it records the new number as additional information for a human to reconcile, and leaves the existing value alone. This is the same restraint that runs through the whole system: the enricher completes and connects, it never destroys what a person already entered.

The choice to mirror the CRM in a cheap index, rather than hammer the CRM’s own API on every lead, is what makes dedupe fast and affordable. The index can be wrong at the edges — a record added directly in the CRM a minute ago might not be mirrored yet — so the push stage does one final exact check against the CRM itself before creating a contact, as a backstop. Two layers, cheap one first, authoritative one last.

Why this shape

  • Cheap and certain first. Exact-key lookups catch most duplicates for two DynamoDB reads and no model call.
  • The model only sees the hard slice. Confident same and confident different are decided deterministically; only the ambiguous band reaches Bedrock.
  • An index, not the live CRM. Dedupe reads a cheap DynamoDB mirror; the push stage does one authoritative CRM check as a backstop.
  • Link, never blindly merge. A match attaches the enquiry to the existing person and fills only empty fields — it never overwrites a human’s value.
  • Every verdict is logged. Whether matched by key, by score, or by model, the decision and its reason are recorded for audit.
All posts