allanninal.dev/build

Part 7 of 7 · Lead enricher series ~9 min read

Engineering reference: the lead enricher architecture

This is the enricher with the friendly labels stripped off. Same system, drawn for someone who has to build or review it: the Lambdas and what triggers each one, the EventBridge schedules, the DynamoDB identity index and its exact key schema, the S3 buckets, the SES setup, the IAM scopes that keep each function to its own job, the Bedrock model id, and the single region it all runs in. If the earlier posts are the “what” and the “why”, this one is the “how, exactly.”

Key takeaways

  • Seven Lambdas (Python 3.14, arm64): one per lead lane and one per pipeline stage, chained over the EventBridge default bus.
  • Two DynamoDB tables on-demand: ler-identity (the dedupe index) and ler-enrich-cache (per-domain firmographics with a TTL).
  • Three S3 buckets: raw payloads (versioned), clean enriched records, and the junk log. SES handles the inbound email lane and the outbound digest.
  • EventBridge Scheduler runs three jobs: a retry sweep, an overnight backfill + identity re-sync, and the daily junk digest.
  • One Bedrock model (global.anthropic.claude-haiku-4-5-20251001-v1:0), one region (eu-west-2), least-privilege IAM per function, no API Gateway and no always-on compute.

The whole architecture, drawn for engineers

Same system as the earlier posts, with the friendly labels off. Three ingress lanes, a five-stage event-driven chain, two outputs, and three scheduled jobs — all in one region.

Engineering topology of the lead enricher: ingress, event-driven pipeline, outputs, and schedules in one region A topology diagram inside one dotted AWS account boundary labelled eu-west-2, in three horizontal bands. Top band, ingress: three boxes. A Lambda Function URL feeding the ler-intake Lambda for the form and webhook lane; an SES inbound rule set writing raw mail to the S3 bucket ler-raw and triggering ler-intake; and an S3 manual-upload path into ler-raw that also triggers ler-intake. ler-intake runs a cheap spam pre-screen and writes a message to the SQS queue ler-intake-queue. Middle band, the event-driven pipeline: ler-normalize consumes ler-intake-queue, calls Bedrock Haiku 4.5 for fuzzy fields, and emits an le.normalized event on the EventBridge default bus. An EventBridge rule routes that to ler-enrich, which reads its key from Secrets Manager, reads and writes the DynamoDB table ler-enrich-cache, calls the firmographic API, and emits le.enriched. That routes to ler-dedupe, which reads and writes the DynamoDB table ler-identity and calls Bedrock for ambiguous matches, then emits le.deduped. That routes to ler-filter, which calls Bedrock for borderline junk and emits either le.clean or le.junk. Bottom band, outputs and control: le.clean routes to ler-push, which does a final CRM check, writes the contact to the external CRM, writes the clean record to the S3 bucket ler-records, and updates ler-identity; le.junk routes to a writer that appends to the S3 bucket ler-junk. A dead-letter queue ler-dlq catches failed events. EventBridge Scheduler drives three jobs: ler-retry every fifteen minutes drains the DLQ and retries pending enrichments; ler-backfill nightly re-enriches pending leads and re-syncs ler-identity from the CRM; ler-digest daily emails the junk digest via SES outbound. CloudWatch Logs collects from every Lambda at 7-day retention, and an AWS Budgets alarm watches the monthly cost. A note reads: the enricher only creates or links records and drops junk — it never overwrites a human’s field, and every decision is logged. AWS account — eu-west-2 Ingress Function URL form / webhook SES inbound email lane → ler-raw S3 ler-raw manual upload ler-intake spam pre-screen → ler-intake-queue Event-driven pipeline (EventBridge default bus) ler-normalize SQS-triggered ler-enrich le.normalized ler-dedupe le.enriched ler-filter le.deduped Bedrock Haiku 4.5 fuzzy calls ler-enrich-cache DynamoDB · TTL ler-identity DynamoDB · dedupe Secrets Manager enrichment key Outputs & control ler-push le.clean Your CRM create / link S3 ler-records clean record S3 ler-junk le.junk ler-dlq SQS dead-letter AWS Budgets cost alarm EventBridge Scheduler ler-retry — 15 min, drains DLQ ler-backfill — nightly re-sync ler-digest — daily junk email It only creates or links records and drops junk — never overwrites a human’s field. Every decision is logged.
Fig 7. The full topology: three ingress lanes, a five-stage chain over the EventBridge default bus, two outputs, a dead-letter queue, and three scheduled jobs — all in eu-west-2.

Lambdas

All Python 3.14 on arm64, 256–512 MB, with a dead-letter queue (ler-dlq) on every asynchronous path.

  • ler-intake — the front door. Triggered three ways: a Lambda Function URL (IAM-auth or a shared secret) for the form/webhook lane, an S3 PUT on ler-raw for the manual-upload and SES-inbound lanes. Writes the raw payload to ler-raw (versioned), runs the cheap spam pre-screen (honeypot, disposable domain, timing), drops obvious bot junk straight to ler-junk, and enqueues survivors to the SQS queue ler-intake-queue. Memory: 256 MB.
  • ler-normalize — consumes ler-intake-queue (batch size 1). Deterministic field cleanup (email, phone → E.164, name casing, country); calls Bedrock Haiku 4.5 only for flagged-ambiguous fields, bounding the result against ISO/reconstruction rules. Writes the clean record and emits le.normalized to the EventBridge default bus. Memory: 512 MB.
  • ler-enrich — EventBridge rule on le.normalized. Skips personal/free domains; reads/writes ler-enrich-cache; on a miss, fetches the firmographic key from Secrets Manager and calls the provider, writing the result back to the cache with a TTL. Tags each field with source + date. Emits le.enriched (or marks enrichment_pending on provider failure and still emits). Memory: 256 MB. Timeout: 30 s.
  • ler-dedupe — EventBridge rule on le.enriched. Exact-key lookup against ler-identity; deterministic fuzzy scoring for near-matches; Bedrock Haiku 4.5 only for the ambiguous band. Emits le.deduped carrying the verdict (new or link:<crm_id>). Memory: 256 MB.
  • ler-filter — EventBridge rule on le.deduped. Weighs deterministic junk signals; Bedrock Haiku 4.5 only for borderline cases. Emits le.clean or le.junk. Memory: 256 MB.
  • ler-push — EventBridge rule on le.clean. Does one final authoritative duplicate check against the live CRM (backstop to the index), then creates or links the contact via the CRM API (key in Secrets Manager), filling only empty fields — never overwriting. Writes the clean record to ler-records and updates ler-identity. Memory: 256 MB. Timeout: 30 s.
  • ler-junk-writer — EventBridge rule on le.junk. Appends the dropped record and its reason to ler-junk. Memory: 128 MB.

EventBridge Scheduler

  • ler-retry — every 15 minutes. Re-drives messages from ler-dlq and retries any lead marked enrichment_pending from a transient provider failure.
  • ler-backfill — nightly (03:00). Re-enriches still-pending leads once the provider has recovered, and re-syncs ler-identity from a fresh CRM export so the index doesn’t drift from the source of truth.
  • ler-digest — daily (08:00). Summarises the previous day’s ler-junk drops and emails the digest via SES so a human watches the boundary.

DynamoDB

  • ler-identity (on-demand) — the dedupe/identity index. Partition key: id_key (string) — a typed, normalised key such as EMAIL#jane.doe@acme-widgets.co.uk, PHONE#+447700900123, or FP#janedoe|acmewidgets. Attributes: crm_id, person_name, company, source, updated_at. A global secondary index on crm_id lets the backfill rewrite every key for a person in one pass. Multiple id_key rows can point at the same crm_id — that’s how one person is reachable by email, phone, or fingerprint.
  • ler-enrich-cache (on-demand) — per-domain firmographic cache. Partition key: domain (string). Attributes: profile (JSON), fetched_at, ttl (epoch seconds; DynamoDB TTL enabled, ~90 days). A miss or an expired item triggers a fresh provider call.

S3 & SES

  • S3 ler-raw — raw inbound payloads (form JSON, SES MIME, uploaded CSVs). Versioning on; the immutable original behind every lead.
  • S3 ler-records — the clean, enriched, deduped record written for each lead that reached the CRM.
  • S3 ler-junk — the junk log: every dropped lead with its reason. The replay source if the filter ever drops something real.
  • SES inbound — rule set for hello@your-company.com, action S3 PUT to ler-raw, which triggers ler-intake.
  • SES outbound — the daily junk digest and ad-hoc ops alerts (provider down, DLQ backing up).

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0, invoked via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Three callsites: ler-normalize (fuzzy field resolution), ler-dedupe (ambiguous same-person judgement), and ler-filter (borderline keep-or-drop). Each call is narrow, structured, and bounded by deterministic rules afterwards.
  • Heavier model. anthropic.claude-sonnet-4-6-... is configured but off by default — available behind a per-callsite flag if a particular dedupe or junk decision ever needs more judgement than Haiku gives. The match and filter logic never depends on a model; the model only ever breaks a tie.

IAM (least privilege per function)

  • ler-intake role: s3:PutObject/GetObject on ler-raw and ler-junk; sqs:SendMessage on ler-intake-queue. No Bedrock, no DynamoDB.
  • ler-normalize role: sqs:ReceiveMessage/DeleteMessage on ler-intake-queue; s3:GetObject/PutObject on ler-raw/ler-records; bedrock:InvokeModel on the Haiku ARN; events:PutEvents.
  • ler-enrich role: dynamodb:GetItem/PutItem on ler-enrich-cache; secretsmanager:GetSecretValue on the enrichment-key secret only; events:PutEvents. No write access to ler-identity.
  • ler-dedupe role: dynamodb:Query/GetItem on ler-identity (read); bedrock:InvokeModel on the Haiku ARN; events:PutEvents. No CRM credentials.
  • ler-filter role: bedrock:InvokeModel on the Haiku ARN; events:PutEvents. No data-store writes.
  • ler-push role: secretsmanager:GetSecretValue on the CRM-key secret; dynamodb:PutItem/UpdateItem on ler-identity; s3:PutObject on ler-records. The only role that can write the identity index or talk to the CRM.

Region & operations

  • Region. Everything in eu-west-2 (London). The Bedrock Global cross-Region inference profile routes the model call for capacity, but no data store, queue, or bucket leaves the region.
  • CloudWatch Logs. Every Lambda logs at 7-day retention — enough to debug a bad day, short enough to stay free-tier-adjacent.
  • DLQ. ler-dlq (SQS) catches any event that fails twice; ler-retry re-drives it and alerts via SES if it backs up.
  • AWS Budgets. A monthly cost alarm (first two budgets free) on the account, posting to SNS, so a runaway loop or a provider price change is caught the same day.
  • No API Gateway, no NAT Gateway, no always-on compute. Ingress is a Lambda Function URL plus SES and S3 events; everything else is event- or schedule-driven.
All posts