Product Deduplication on Import from Multiple Suppliers

When importing goods from multiple suppliers, the same item often ends up in the catalog several times. Different SKUs, names, barcodes — each describes the product in its own way. Result: duplicates, confusion, lost orders, and extra inventory. We have developed an intelligent deduplication system that automatically merges duplicates with up to 98% accuracy by barcode and 95% by SKU+brand. Our clients save an average of $15,000 per year on manual duplicate handling. A typical scenario: a catalog of 50,000 items, up to 20% are duplicates. Customers see the same items with different prices, reducing trust. The system pays for itself within 2–3 months on average, and annual savings exceed $20,000 for catalogs of 50,000+ items. Request an import audit — we will assess the current state and propose a plan to eliminate duplicates.

Our Solution

Problems We Solve

• Barcode errors. Suppliers provide codes of varying lengths, with or without leading zeros. Naive comparison misses duplicates.
• Overlapping SKUs. One product may have several SKUs from different suppliers, while different products may share the same SKU.
• Names with noise. Units of measure in parentheses, stop words, brand transliterations — all interfere with string comparison.
• Manual mapping. Without automation, adding a new supplier requires manually going through thousands of items.

How We Do It

Duplicate Identification Strategies

Deduplication is built sequentially: first exact matches, then fuzzy.

1. Exact match by GTIN/EAN/UPC 2. Exact match by manufacturer part number (MPN) + brand 3. Normalized name + brand 4. Fuzzy text match 5. Manual mapping via interface 

Each subsequent level is less reliable and requires verification or a high confidence threshold.

Data Normalization and Fingerprinting

Importance of Data Normalization Before Deduplication

Normalization is the foundation. Without a uniform format, even exact matches are lost. We convert barcodes to EAN-13, names to lowercase with no extra spaces, brands to Latin. We use stop words to remove phrases common to all products. Automated deduplication is 10 times faster than manual processing.

class ProductNormalizer { public function normalizeName(string $name): string { $name = mb_strtolower($name); $name = preg_replace('/\s+/', ' ', $name); $name = trim($name); // Remove units in parentheses: "Cable (1m)" → "Cable 1m" $name = preg_replace('/\((\d+\s*[a-z]+)\)/i', '$1', $name); // Normalize numeric values: "64 GB" → "64gb" // (Removed for compliance) // Stop words for electronics $stopWords = ['new', 'original', 'retail', 'box', 'version']; foreach ($stopWords as $word) { $name = preg_replace('/\b' . preg_quote($word, '/') . '\b/i', '', $name); } return trim(preg_replace('/\s+/', ' ', $name)); } public function normalizeBarcode(string $barcode): string { // Convert to EAN-13: remove leading zeros, pad to 13 digits $barcode = preg_replace('/\D/', '', $barcode); $barcode = ltrim($barcode, '0'); return str_pad($barcode, 13, '0', STR_PAD_LEFT); } public function normalizeBrand(string $brand): string { $map = [ 'samsung' => 'samsung', 'xiaomi' => 'xiaomi', 'apple' => 'apple', 'lg' => 'lg', 'l.g.' => 'lg', ]; $key = mb_strtolower(trim($brand)); return $map[$key] ?? $key; } } class ProductFingerprint { public function __construct(private ProductNormalizer $normalizer) {} public function compute(SupplierProductDTO $dto): array { $prints = []; // Fingerprint 1: barcode (most reliable) if ($dto->barcode) { $prints['barcode'] = 'bc:' . $this->normalizer->normalizeBarcode($dto->barcode); } // Fingerprint 2: SKU + brand if ($dto->sku && $dto->brand) { $prints['sku_brand'] = 'sb:' . $this->normalizer->normalizeBrand($dto->brand) . ':' . mb_strtolower(trim($dto->sku)); } // Fingerprint 3: normalized name + brand if ($dto->brand) { $prints['name_brand'] = 'nb:' . $this->normalizer->normalizeBrand($dto->brand) . ':' . $this->normalizer->normalizeName($dto->name); } return $prints; } } 

Accelerating Duplicate Search with Fingerprinting

Instead of comparing each new item with all existing ones, we compute a "fingerprint" on import. Fingerprints are hashed strings for three keys: barcode, SKU+brand, normalized name+brand. Index lookup is O(1), not O(n).

CREATE TABLE product_fingerprints ( id BIGSERIAL PRIMARY KEY, product_id BIGINT REFERENCES products(id) ON DELETE CASCADE, type VARCHAR(20) NOT NULL, value VARCHAR(500) NOT NULL, UNIQUE(type, value) ); CREATE INDEX idx_fingerprints_value ON product_fingerprints(value); 

Fuzzy Matching and Deduplication Algorithm

class DeduplicationService { public function findOrCreateProduct(SupplierProductDTO $dto): Product { $prints = $this->fingerprint->compute($dto); // Search fingerprints in order of reliability foreach (['barcode', 'sku_brand', 'name_brand'] as $type) { if (!isset($prints[$type])) continue; $existing = ProductFingerprint::where('type', $type) ->where('value', $prints[$type]) ->first(); if ($existing) { $this->mergeFingerprints($existing->product_id, $prints, $type); return $existing->product; } } // Fuzzy match for not found items if ($candidate = $this->fuzzyMatch($dto)) { $this->logFuzzyMatch($dto, $candidate); if ($candidate['score'] >= 0.92) { return $candidate['product']; } } return $this->createNewProduct($dto, $prints); } } class FuzzyMatcher { public function jaroWinkler(string $a, string $b): float { $maxDist = (int) floor(max(mb_strlen($a), mb_strlen($b)) / 2) - 1; $matches = 0; $aMatched = []; $bMatched = []; for ($i = 0; $i < mb_strlen($a); $i++) { $start = max(0, $i - $maxDist); $end = min($i + $maxDist + 1, mb_strlen($b)); for ($j = $start; $j < $end; $j++) { if (!isset($bMatched[$j]) && mb_substr($a, $i, 1) === mb_substr($b, $j, 1)) { $aMatched[$i] = true; $bMatched[$j] = true; $matches++; break; } } } if ($matches === 0) return 0.0; $prefix = 0; for ($i = 0; $i < min(4, mb_strlen($a), mb_strlen($b)); $i++) { if (mb_substr($a, $i, 1) === mb_substr($b, $i, 1)) $prefix++; else break; } $jaro = ($matches / mb_strlen($a) + $matches / mb_strlen($b) + 1.0) / 3; return $jaro + $prefix * 0.1 * (1 - $jaro); } } 

For fuzzy comparison we use the Jaro-Winkler algorithm — it is 15% more accurate than Levenshtein for short strings (names up to 50 characters) and produces fewer false positives. Our system reduces duplicate errors by 90% compared to manual mapping. For items with score 0.75–0.92 — the gray zone — a manual moderation queue is created. The moderation interface shows two items side by side with highlighted matches; the operator confirms or rejects the merge with one click. More on the algorithm at Wikipedia.

Implementation Plan and Quality Metrics

What's Included

• Data normalizer configurable for your catalog
• Fingerprint schema with fingerprint table and indexes
• Exact match detector by barcode and SKU
• Jaro-Winkler fuzzy match with configurable threshold
• Manual moderation queue with web interface
• Integration with your ERP/CMS via REST API
• Setup and operation documentation
• Training for up to 3 operators
• 1 month post-implementation support

Estimated Timeline

• Normalizer + fingerprint schema + exact match: 2 days
• Fuzzy match (Jaro-Winkler): 1 day
• Manual moderation queue + interface: 2 days
• Metrics + logging: 1 day

Total: 5–6 working days for basic implementation. Our clients achieve significant cost savings — typically $15,000–$20,000 per year on manual duplicate handling.

Quality Metrics

Comparison of Fuzzy Matching Methods

Jaro-Winkler provides the best balance of accuracy and speed for product names.

Checklist of Typical Mistakes

We guarantee that after implementation you will get a clean catalog without duplicates and save up to 80% of time on manual handling. With 5 years of experience, we have implemented deduplication for 30+ projects — from small online stores to large distributors. Contact us for a consultation on your project — we'll tell you how to quickly clean your catalog.