A similarity score answers the wrong question. Two records can be ninety per cent similar and obviously different people, and eighty per cent similar and certainly the same person. What matters is what agrees, how unlikely that agreement is by chance, and what disagrees.
Key takeaways
Agreement on a rare value is strong evidence; on a common one it is nearly none.
Three signals argue against: different first names, conflicting strong identifiers, and explicit separation.
The score is a sum of evidence, and the report shows the terms rather than the total.
Automatic merging needs a conclusive identifier and zero evidence against.
A pair marked not-a-duplicate is remembered forever and never proposed again.
Rarity, not agreement
Two records with the surname Smith agreeing tells you almost nothing; two with an unusual surname agreeing is meaningful. The same applies to first names, postcodes, and company names. Weighting agreement by how rare the value is in your own database is the single change that most improves scoring, and it costs one count query per value.
Agreement
Weight if rare
Weight if common
Email, exact
Conclusive
Conclusive — email is always rare
Phone, normalised
Strong
Weak if it is the company’s own number
Surname
Moderate
Nearly none
Postcode
Moderate
Weak in a dense postcode
Company name
Strong
Weak for a common trading name
The second column is the interesting one. A phone number is normally a strong signal and becomes worthless when four hundred records share it, and the same computation that flagged that as an oversized block also tells the scorer to discount it.
Evidence against
Fig 1. How a pair is scored. The against-branches are not negative weights but hard blocks, because some evidence is conclusive in the other direction.
Database
App integration
Machine learning
Security & identity
Analytics
People
Households
The most common false positive in any consumer database. Two people at the same address with the same surname, the same landline, and often the same email domain are frequently a couple, and every similarity measure scores them very highly.
Different first names is the signal that separates them, and it needs to be strong enough to override a high agreement score rather than being one term among many. A pair agreeing on address, surname and phone but disagreeing on first name is a household until proven otherwise, and proposing it as a merge wastes somebody’s attention every single time.
Remembering a no
The most important small feature in the whole system. A pair a person has looked at and declared separate must never be proposed again, ever, and that record has to survive re-imports, re-runs and schema changes.
Without it, the same six hundred rejected pairs reappear in every run and the review becomes something people stop doing within two rounds. The not-a-duplicate list is keyed on both record identifiers and is the one piece of state that is never expired.
The automatic band
Fig 2. The only case that merges without a person. All three conditions must hold, which keeps the band narrow enough that automatic merging is uncontroversial.
App integration
Machine learning
Management
People
The third condition is the one that gets left out and matters most. Two records with the same email address where one has free-text notes somebody typed are not a safe automatic merge, because the merge rules have to decide what happens to those notes and that decision benefits from a person seeing it.