allanninal.dev/build

Part 1 of 7 · Lead enricher series ~9 min read

A lead enricher on AWS for a few dollars a month

A small business collects more leads than anyone has time to tidy. The form submission with the name in all caps and a phone number missing its country code. The webhook lead from a free email address that turns out to be a competitor poking around. The same person who fills in the contact form twice in a week and lands in the CRM as two half-complete contacts. The lead with a company name but no idea what the company actually does. Cleaning each one by hand is dull, and it’s the first thing that slips when the week gets busy — so the CRM slowly fills with duplicates, junk, and half-records. This post walks through the design of a small enricher that catches every raw lead, cleans it, completes it from public data, checks it against what’s already in the CRM, and drops the obvious junk — before anything is written.

Key takeaways

  • Three lanes bring leads in: a form/webhook lane, an emailed-enquiry lane, and a manual upload — all landing as raw, messy records.
  • Every lead runs the same five-stage pipeline: normalise, enrich, dedupe, junk-filter, then push to the CRM.
  • It cleans and completes leads — it does not score them and it does not route them. That’s a different job, deliberately left out.
  • Clean, enriched records go to the CRM; obvious junk is logged and dropped before anything is written.
  • Designed on AWS for about $2.50/month at typical small-business volume. It never overwrites a field a human already entered.

The whole system on one page

Before any code, here’s the shape of what we’re designing.

System architecture: three lead sources, a five-stage pipeline inside AWS, the CRM on the far side At the top, three external boxes in a row labelled Inbound leads: a form and webhook lane, an emailed-enquiry lane, and a manual upload lane. Each feeds an arrow down into a dotted container labelled AWS account. Inside the AWS account, a left-to-right pipeline of five boxes: Normalise (fix name casing, phone and email format, country), Enrich (company from the email domain, plus size, industry, location), Dedupe (check against the CRM identity index), Junk filter (drop obvious spam), and Push to CRM. Below the pipeline sit two supporting boxes: an Identity index (DynamoDB) that the Dedupe stage reads and the Push stage updates, and a Junk log that the Junk filter and the early spam pre-screen write dropped leads to. On the far right, outside the container, a box labelled Your CRM receives clean enriched records from the Push stage. A note at the bottom reads: it cleans and completes — it never scores, never routes, and never overwrites a field a human already entered. Form & webhook the busy lane Emailed enquiry SES inbound Manual upload a dropped CSV Inbound leads — raw & messy AWS account — eu-west-2 Normalise case, phone, email, country Enrich company, size, industry, place Dedupe match against the CRM Junk filter drop obvious spam Push write clean record Identity index DynamoDB — mirrors CRM Junk log dropped, not lost reads / updates drops Your CRM It cleans and completes — never scores, never routes, never overwrites a human’s field.
Fig 1. Three lanes in, one pipeline, the CRM on the far side. Every lead is normalised, enriched, deduped against the CRM, and junk-filtered before a clean record is written. Junk is logged and dropped; nothing reaches the CRM unchecked.

What you set up once

  • The lead lanes. Most leads arrive through the form/webhook lane: your website form, an ad platform, or a partner posts a JSON payload, and a small intake function drops the raw body straight into S3 and onto a queue. A second email lane catches enquiries sent to an address like hello@your-company.com — SES writes the raw message to S3 and the same pipeline picks it up. A third manual lane lets someone drop a CSV of leads (from an event, say) into a folder. Whatever the source, the lead enters as a raw, messy record — nothing is assumed clean.
  • The enrichment source. One public firmographic data provider, reached over its API, with the key held in Secrets Manager. This is what turns an email domain into a company name, a rough headcount, an industry, and a location. It’s also the one line on the bill that grows with volume, so the design caches aggressively and never calls it for a lead that’s already been flagged as junk.
  • The identity index. A small DynamoDB table that mirrors the people already in your CRM on a few normalised keys — lowercased email, the phone number in international format, and a name-plus-company fingerprint. It’s seeded once from a CRM export and kept current as new records are written. The dedupe stage reads it; the push stage updates it. The CRM stays the source of truth; this is just a fast, cheap mirror to check against.

What runs on every lead

  • Normalise. The first stage turns the raw fields into canonical ones: the name to sensible casing, the email to a single lowercase form, the phone to E.164, and the country resolved from whatever clues are present. It’s mostly plain Python, with one small Bedrock Haiku 4.5 call reserved for the genuinely fuzzy parts — splitting an awkward full name, or inferring a country when only a dialling code is present. Part 2 walks through it.
  • Enrich. The clean email domain becomes the key for a firmographic lookup: company name, size band, industry, and headquarters location, attached to the record with a note of where each field came from. Cached results mean the same domain is never looked up twice in a hurry. Part 3 covers it.
  • Dedupe. The enriched lead is checked against the identity index — first for an exact match on a normalised key, then a fuzzy pass for near-matches, with Bedrock judging only the genuinely ambiguous cases. A confident match links the lead to the existing person; a confident non-match is written as new. Part 4 explains the matching.
  • Junk filter. Before anything is written, the lead faces a layered junk check: cheap deterministic rules first (disposable domains, gibberish, role addresses), a bounded model judgement only for the borderline ones. Obvious junk is logged and dropped. Part 5 is about deciding what isn’t worth keeping.
  • Push. A clean, enriched, deduped, non-junk lead is written to the CRM — created as a new contact, or attached to the existing one if dedupe found a match. It never overwrites a field a human already entered; it only fills genuine gaps. The identity index is updated so the next duplicate is caught.

In plain words

A lead arrives from your website form: name “jane DOE”, email “Jane.Doe@acme-widgets.co.uk ” (capitalised, trailing space), phone “07700 900123”, company blank, message “interested in a quote.” The enricher normalises it to “Jane Doe”, jane.doe@acme-widgets.co.uk, and +447700900123, country United Kingdom. It takes the domain acme-widgets.co.uk, looks it up, and fills in: Acme Widgets Ltd, 50–200 staff, industrial supplies, Birmingham. It checks the identity index, finds that Jane already enquired three weeks ago, and links this enquiry to her existing record rather than creating a second “Jane Doe.” It runs the junk checks — real domain, real name, plausible message — and writes the completed record. A rep opens the CRM to a single, full contact with a company profile attached, not a bare name and a guess.

The cost of running this is about $2.50 a month at small-business volume. The cost of not running it is a CRM that fills with duplicates nobody trusts, junk that wastes a rep’s morning, and half-records that make every report a little bit wrong.

Design rules that shaped every decision

  • Clean and complete, full stop. It does not score leads and it does not route them — those are someone else’s job, deliberately kept out.
  • Deterministic by default. The model is called only for fuzzy judgement, and its output is always bounded by plain rules afterwards.
  • Never overwrite a human’s field. The enricher fills genuine gaps and links duplicates; it never replaces a value somebody already entered.
  • Junk is dropped, never silently lost. Every dropped lead is written to a log with the reason, so you can audit and reverse a bad call.
  • Check before you spend. The cheap junk rules run before the paid enrichment lookup, so junk never costs you an API call.
  • One region, no always-on compute. Everything is event-driven and serverless, so it costs almost nothing when it’s quiet.

Why this shape

Most small teams handle raw leads in one of three ways: they let the form write straight into the CRM and tidy up later (which never happens), they paste leads in by hand (slow, and inconsistent between people), or they buy a heavy all-in-one platform that scores and routes and emails and does ten other things they don’t need. The first fills the CRM with junk and duplicates. The second doesn’t scale past a few leads a day. The third is expensive and couples cleaning — a job with a clear right answer — to scoring and routing, jobs that depend on your sales process and change all the time.

The setup above does exactly one thing well: it makes sure that whatever lands in your CRM is clean, complete, deduped, and real. It deliberately stops there. Scoring a lead and routing it to the right rep is a separate concern with its own logic; bolting it on here would mean redeploying the cleaner every time the sales playbook changes. By keeping the enricher to cleaning and completing, the rules that matter — what counts as junk, which fields to fill — stay stable, and the expensive, changeable parts live somewhere else.

The next four posts walk through each stage in turn: how a raw lead gets normalised, how it gets enriched from public data, how a duplicate gets caught against the CRM, and how junk gets filtered out. One diagram per post. A cost breakdown and a full engineering reference at the end.

All posts