How a pair gets scored
A similarity score answers the wrong question. Two records can be ninety per cent similar and obviously different people, and eighty per cent similar and certainly the same person. What matters is what agrees, how unlikely that agreement is by chance, and what disagrees.
Key takeaways
- Agreement on a rare value is strong evidence; on a common one it is nearly none.
- Three signals argue against: different first names, conflicting strong identifiers, and explicit separation.
- The score is a sum of evidence, and the report shows the terms rather than the total.
- Automatic merging needs a conclusive identifier and zero evidence against.
- A pair marked not-a-duplicate is remembered forever and never proposed again.
Rarity, not agreement
Two records with the surname Smith agreeing tells you almost nothing; two with an unusual surname agreeing is meaningful. The same applies to first names, postcodes, and company names. Weighting agreement by how rare the value is in your own database is the single change that most improves scoring, and it costs one count query per value.
| Agreement | Weight if rare | Weight if common |
|---|---|---|
| Email, exact | Conclusive | Conclusive — email is always rare |
| Phone, normalised | Strong | Weak if it is the company’s own number |
| Surname | Moderate | Nearly none |
| Postcode | Moderate | Weak in a dense postcode |
| Company name | Strong | Weak for a common trading name |
The second column is the interesting one. A phone number is normally a strong signal and becomes worthless when four hundred records share it, and the same computation that flagged that as an oversized block also tells the scorer to discount it.
Evidence against
- Database
- App integration
- Machine learning
- Security & identity
- Analytics
- People
Households
The most common false positive in any consumer database. Two people at the same address with the same surname, the same landline, and often the same email domain are frequently a couple, and every similarity measure scores them very highly.
Different first names is the signal that separates them, and it needs to be strong enough to override a high agreement score rather than being one term among many. A pair agreeing on address, surname and phone but disagreeing on first name is a household until proven otherwise, and proposing it as a merge wastes somebody’s attention every single time.
Remembering a no
The most important small feature in the whole system. A pair a person has looked at and declared separate must never be proposed again, ever, and that record has to survive re-imports, re-runs and schema changes.
Without it, the same six hundred rejected pairs reappear in every run and the review becomes something people stop doing within two rounds. The not-a-duplicate list is keyed on both record identifiers and is the one piece of state that is never expired.
The automatic band
- App integration
- Machine learning
- Management
- People
The third condition is the one that gets left out and matters most. Two records with the same email address where one has free-text notes somebody typed are not a safe automatic merge, because the merge rules have to decide what happens to those notes and that decision benefits from a person seeing it.
Next: what a proposal actually shows.
All posts