Fuzzy Matching Definition
Fuzzy matching is a technique for finding values or records that are similar but not exactly the same, such as "Jon Smith" and "John Smith" or "12 Main St." and "12 Main Street." Instead of giving a simple yes-or-no answer, it measures how close two values are and returns a similarity score, so near-matches caused by typos, abbreviations, or formatting differences can still be found.
How does fuzzy matching work?
Before comparing, values are usually cleaned up by removing punctuation, making everything lowercase, and expanding common abbreviations. The values are then compared using one or more methods:
- Character differences: counting how many letters must be added, removed, or changed to turn one value into the other. "Smith" and "Smyth" differ by one letter, so they score as very similar.
- Sound-alike comparison: comparing how words are pronounced rather than spelled, so "Meyer" and "Maier" can be recognized as the same name.
- Word-based comparison: comparing the words in each value regardless of order, so "Smith, John" and "John Smith" are treated as a match.
- Partial overlap: comparing short fragments of text, which helps with long values like product descriptions or company names.
The result is a score, often expressed as a percentage. Values that score above a chosen threshold are treated as matches or flagged for review.
Where is fuzzy matching used?
- Finding duplicates: spotting the same customer, supplier, or product entered more than once, as part of data deduplication.
- Combining data from different sources: linking records from two systems that spell names or addresses differently, for example after a company merger.
- Search: returning useful results even when a user misspells a search term.
- Checking incoming data: comparing new records against existing ones before they are added, so duplicates are caught early.
How is it related to probabilistic matching?
Fuzzy matching compares one value at a time, such as two names or two addresses. Probabilistic matching builds on it by combining the similarity scores of several fields, like name, address, and phone number, into an overall likelihood that two records describe the same entity. In practice, fuzzy matching is often one of the building blocks inside a larger match and merge process.
What are its limitations?
Fuzzy matching can produce false matches, especially with short values: "Jon" and "Joan" look similar but are different names, and two products with nearly identical names may be different models. It can also miss matches where the same thing is described in completely different words, such as a company's legal name and its brand name. Comparing every record with every other record is slow on large datasets, so systems usually narrow the comparison first, for example by only comparing records from the same postal code. Thresholds need testing and adjustment to balance missed matches against incorrect ones.
