Home / Work / Entity resolution at billion scale
Entity resolution at billion scale
Finding the records that describe the same company or person across databases with no shared ID.
- Problem
- Duplicate and unlinked records across sources
- Scale
- 1.25 billion candidate pairs
- Result
- About 40,000 comparisons, run in seconds; 98% accuracy in evaluation
The problem
When two databases share no identifier, every record in one is a possible match for every record in the other. At this size, comparing all pairs is not slow, it is out of reach, and the records disagree on spelling, format and completeness.
What I did
- Data quality before matching. Each source is profiled and cleaned first, so that matching works on comparable records.
- The comparison bottleneck treated first. 1.25 billion candidate pairs were brought down to about 40,000 comparisons before any detailed matching.
- Human review for borderline cases. Clear matches and clear non-matches are decided automatically. Uncertain cases go to a person.
Result
1.25 billion candidate pairs reduced to about 40,000 comparisons, run in seconds, with 98% accuracy in evaluation.
Why it matters
Linked records are what make counts, search, CRM and AI training data trustworthy, and what make a GDPR erasure request answerable. You can’t delete what you can’t find explains why.
Facing a similar problem? Email me.