Home / Matching demo
Entity matching, hands on
Entity resolution decides which records, across systems with no shared ID, describe the same thing. Two problems decide whether it can work at all, before any matching: the quality of the data and the number of comparisons. Walk through both below.
Before matching anything, measure the data. These are 10 Amazon and 13 Google records from the public Amazon-Google benchmark. Pick a check to see what it flags.
Amazon records
Google records
Brute force compares every record with every other. Here that is every Amazon record against every Google record: one square per comparison.
Now grow the data
Ten times more records means a hundred times more comparisons. That is the bottleneck to treat before any matching.
The same bottleneck in one of my production projects, step by step. The figures come from that project; the reduction method itself is not shown.
- Data quality controls
- Candidate reductionmethod not shown
- Detailed matching
- Human review of borderline cases
Request: erase every record of Nora Keller, born 1984-03-12.
Fictional records: the example from my LinkedIn post.
Data quality comes first
Records of the same product disagree before a single pair is compared. In these 23 benchmark records, the manufacturer is empty in 9 of the 13 Google records, the same companies are written differently in the two sources, and titles carry tags and symbols. Measuring this first tells you what matching will have to cope with, and what to clean.
Brute force does not scale
Comparing every record with every other grows with the square of the data: ten times more records, a hundred times more comparisons. The original Amazon-Google benchmark crosses 1,363 Amazon records with 3,226 Google records: 4,397,038 possible pairs for 1,300 true matches. One of my production projects started from 1.25 billion candidate pairs and was brought down to about 40,000 comparisons, run in seconds. Treating that bottleneck is where much of the engineering lies: the reduction step tab shows the result, not the method.
Why it is a privacy problem
A GDPR erasure request is an entity resolution problem in disguise. Before you can delete, export or exclude one person’s data, you have to find every record of them across systems that share no ID. Delete only the exact matches and the rest stays in your systems, and in whatever you train next. Read the full article.
About the data
The products are 10 Amazon and 13 Google records from the Amazon-Google benchmark, copied as published in the matchbench repository. The benchmark comes from the Database Group of Leipzig University (Köpcke, Thor and Rahm, VLDB 2010), which shares it under a Creative Commons licence. The patient records are fictional. Everything in the demo is computed in your browser, including the speed used for the time estimates.
A matching problem of your own? Email me.