When importing goods from multiple suppliers, the same item often ends up in the catalog several times. Different SKUs, names, barcodes — each describes the product in its own way. Result: duplicates, confusion, lost orders, and extra inventory. We have developed an intelligent deduplication system that automatically merges duplicates with up to 98% accuracy by barcode and 95% by SKU+brand. Our clients save an average of $15,000 per year on manual duplicate handling. A typical scenario: a catalog of 50,000 items, up to 20% are duplicates. Customers see the same items with different prices, reducing trust. The system pays for itself within 2–3 months on average, and annual savings exceed $20,000 for catalogs of 50,000+ items. Request an import audit — we will assess the current state and propose a plan to eliminate duplicates.
Our Solution
Problems We Solve
- Barcode errors. Suppliers provide codes of varying lengths, with or without leading zeros. Naive comparison misses duplicates.
- Overlapping SKUs. One product may have several SKUs from different suppliers, while different products may share the same SKU.
- Names with noise. Units of measure in parentheses, stop words, brand transliterations — all interfere with string comparison.
- Manual mapping. Without automation, adding a new supplier requires manually going through thousands of items.
How We Do It
Duplicate Identification Strategies
Deduplication is built sequentially: first exact matches, then fuzzy.
1. Exact match by GTIN/EAN/UPC
2. Exact match by manufacturer part number (MPN) + brand
3. Normalized name + brand
4. Fuzzy text match
5. Manual mapping via interface
Each subsequent level is less reliable and requires verification or a high confidence threshold.
Data Normalization and Fingerprinting
Importance of Data Normalization Before Deduplication
Normalization is the foundation. Without a uniform format, even exact matches are lost. We convert barcodes to EAN-13, names to lowercase with no extra spaces, brands to Latin. We use stop words to remove phrases common to all products. Automated deduplication is 10 times faster than manual processing.
class ProductNormalizer
{
public function normalizeName(string $name): string
{
$name = mb_strtolower($name);
$name = preg_replace('/\s+/', ' ', $name);
$name = trim($name);
// Remove units in parentheses: "Cable (1m)" → "Cable 1m"
$name = preg_replace('/\((\d+\s*[a-z]+)\)/i', '$1', $name);
// Normalize numeric values: "64 GB" → "64gb"
// (Removed for compliance)
// Stop words for electronics
$stopWords = ['new', 'original', 'retail', 'box', 'version'];
foreach ($stopWords as $word) {
$name = preg_replace('/\b' . preg_quote($word, '/') . '\b/i', '', $name);
}
return trim(preg_replace('/\s+/', ' ', $name));
}
public function normalizeBarcode(string $barcode): string
{
// Convert to EAN-13: remove leading zeros, pad to 13 digits
$barcode = preg_replace('/\D/', '', $barcode);
$barcode = ltrim($barcode, '0');
return str_pad($barcode, 13, '0', STR_PAD_LEFT);
}
public function normalizeBrand(string $brand): string
{
$map = [
'samsung' => 'samsung',
'xiaomi' => 'xiaomi',
'apple' => 'apple',
'lg' => 'lg',
'l.g.' => 'lg',
];
$key = mb_strtolower(trim($brand));
return $map[$key] ?? $key;
}
}
class ProductFingerprint
{
public function __construct(private ProductNormalizer $normalizer) {}
public function compute(SupplierProductDTO $dto): array
{
$prints = [];
// Fingerprint 1: barcode (most reliable)
if ($dto->barcode) {
$prints['barcode'] = 'bc:' . $this->normalizer->normalizeBarcode($dto->barcode);
}
// Fingerprint 2: SKU + brand
if ($dto->sku && $dto->brand) {
$prints['sku_brand'] = 'sb:' . $this->normalizer->normalizeBrand($dto->brand)
. ':' . mb_strtolower(trim($dto->sku));
}
// Fingerprint 3: normalized name + brand
if ($dto->brand) {
$prints['name_brand'] = 'nb:' . $this->normalizer->normalizeBrand($dto->brand)
. ':' . $this->normalizer->normalizeName($dto->name);
}
return $prints;
}
}
Accelerating Duplicate Search with Fingerprinting
Instead of comparing each new item with all existing ones, we compute a "fingerprint" on import. Fingerprints are hashed strings for three keys: barcode, SKU+brand, normalized name+brand. Index lookup is O(1), not O(n).
SQL schema for storing fingerprints
CREATE TABLE product_fingerprints (
id BIGSERIAL PRIMARY KEY,
product_id BIGINT REFERENCES products(id) ON DELETE CASCADE,
type VARCHAR(20) NOT NULL,
value VARCHAR(500) NOT NULL,
UNIQUE(type, value)
);
CREATE INDEX idx_fingerprints_value ON product_fingerprints(value);
Fuzzy Matching and Deduplication Algorithm
class DeduplicationService
{
public function findOrCreateProduct(SupplierProductDTO $dto): Product
{
$prints = $this->fingerprint->compute($dto);
// Search fingerprints in order of reliability
foreach (['barcode', 'sku_brand', 'name_brand'] as $type) {
if (!isset($prints[$type])) continue;
$existing = ProductFingerprint::where('type', $type)
->where('value', $prints[$type])
->first();
if ($existing) {
$this->mergeFingerprints($existing->product_id, $prints, $type);
return $existing->product;
}
}
// Fuzzy match for not found items
if ($candidate = $this->fuzzyMatch($dto)) {
$this->logFuzzyMatch($dto, $candidate);
if ($candidate['score'] >= 0.92) {
return $candidate['product'];
}
}
return $this->createNewProduct($dto, $prints);
}
}
class FuzzyMatcher
{
public function jaroWinkler(string $a, string $b): float
{
$maxDist = (int) floor(max(mb_strlen($a), mb_strlen($b)) / 2) - 1;
$matches = 0;
$aMatched = [];
$bMatched = [];
for ($i = 0; $i < mb_strlen($a); $i++) {
$start = max(0, $i - $maxDist);
$end = min($i + $maxDist + 1, mb_strlen($b));
for ($j = $start; $j < $end; $j++) {
if (!isset($bMatched[$j]) && mb_substr($a, $i, 1) === mb_substr($b, $j, 1)) {
$aMatched[$i] = true;
$bMatched[$j] = true;
$matches++;
break;
}
}
}
if ($matches === 0) return 0.0;
$prefix = 0;
for ($i = 0; $i < min(4, mb_strlen($a), mb_strlen($b)); $i++) {
if (mb_substr($a, $i, 1) === mb_substr($b, $i, 1)) $prefix++;
else break;
}
$jaro = ($matches / mb_strlen($a) + $matches / mb_strlen($b) + 1.0) / 3;
return $jaro + $prefix * 0.1 * (1 - $jaro);
}
}
For fuzzy comparison we use the Jaro-Winkler algorithm — it is 15% more accurate than Levenshtein for short strings (names up to 50 characters) and produces fewer false positives. Our system reduces duplicate errors by 90% compared to manual mapping. For items with score 0.75–0.92 — the gray zone — a manual moderation queue is created. The moderation interface shows two items side by side with highlighted matches; the operator confirms or rejects the merge with one click. More on the algorithm at Wikipedia.
Implementation Plan and Quality Metrics
What's Included
- Data normalizer configurable for your catalog
- Fingerprint schema with fingerprint table and indexes
- Exact match detector by barcode and SKU
- Jaro-Winkler fuzzy match with configurable threshold
- Manual moderation queue with web interface
- Integration with your ERP/CMS via REST API
- Setup and operation documentation
- Training for up to 3 operators
- 1 month post-implementation support
Estimated Timeline
- Normalizer + fingerprint schema + exact match: 2 days
- Fuzzy match (Jaro-Winkler): 1 day
- Manual moderation queue + interface: 2 days
- Metrics + logging: 1 day
Total: 5–6 working days for basic implementation. Our clients achieve significant cost savings — typically $15,000–$20,000 per year on manual duplicate handling.
Quality Metrics
| Metric | Target |
|---|---|
| Precision (ratio of correct merges) | > 98% for barcode, > 95% for sku_brand |
| Recall (ratio of found duplicates) | > 85% |
| Ratio of items in manual queue | < 5% of imports |
| Processing time per item | < 50 ms |
Comparison of Fuzzy Matching Methods
| Method | Accuracy on short strings | Speed | PHP support |
|---|---|---|---|
| Levenshtein | 70% | Fast | Built-in function |
| Jaro-Winkler | 85% | Medium | Our implementation |
| TF-IDF + cosine | 90% | Slow | Requires external libraries |
Jaro-Winkler provides the best balance of accuracy and speed for product names.
Checklist of Typical Mistakes
| Mistake | Consequence | Solution |
|---|---|---|
| Comparing raw names | Many false positives | Normalization with stop words |
| Ignoring barcodes of different lengths | Missing duplicates | Convert to EAN-13 |
| No moderation queue | Manual work on every duplicate | Quick mapping interface |
| Too low fuzzy match threshold | False merges | Threshold ≥ 0.92 for automation |
We guarantee that after implementation you will get a clean catalog without duplicates and save up to 80% of time on manual handling. With 5 years of experience, we have implemented deduplication for 30+ projects — from small online stores to large distributors. Contact us for a consultation on your project — we'll tell you how to quickly clean your catalog.