Scraping multiple sources inevitably leads to duplicates: a product appears on the manufacturer's site, three distributor catalogs, and a marketplace. Naively comparing by URL or name works poorly—we use smarter approaches. Our experience shows that without proper deduplication, the catalog bloats by 20–40%, and page load speed drops due to unnecessary requests. Duplicates in the catalog not only slow down the site but also reduce conversion by 10–15%: a visitor sees two identical products and doubts the store's reliability. Errors in stock, duplicate orders, price confusion—all consequences of uncleaned data. Our deduplication system solves these problems by ensuring a single source of truth. Our certified engineers guarantee at least 95% accuracy on a test sample.
Why Simple Deduplication Doesn't Work?
Problem 1: Different Data Formats
One supplier provides an article as "ART-123", another as "ART123". Direct comparison will miss the duplicate.
Problem 2: Word Order Variation
"iPhone 15 Pro Max 256GB" and "iPhone 15 256GB Pro Max" are the same product, but the string differs.
Problem 3: Typographical Errors
"Samsung Galaxy S24 Ultra" and "Samsung Galaxy S24 Ulta" are nearly identical but don't match character by character.
Levels of Deduplication
Exact Match
By a normalized key: SKU, EAN/GTIN, manufacturer article. The most reliable method, works where a unique identifier exists.
def normalize_sku(raw_sku: str) -> str: # remove spaces, dashes, convert to uppercase return re.sub(r'[\s\-_/]', '', raw_sku).upper() Content Hashing
For content (articles, descriptions)—normalize the text and compute a hash.
def content_hash(text: str) -> str: normalized = ' '.join(text.lower().split()) # remove extra spaces return hashlib.sha256(normalized.encode()).hexdigest() Fuzzy Matching
For products without an explicit SKU—compare titles by Levenshtein distance or Token Sort/Token Set Ratio algorithms.
from rapidfuzz import fuzz, process def find_duplicate(new_title: str, existing_titles: list[str], threshold=85): result = process.extractOne( new_title, existing_titles, scorer=fuzz.token_sort_ratio ) if result and result[1] >= threshold: return result[0] return None token_sort_ratio sorts words before comparison—works well with word order variations in product titles.
Vector Similarity
For texts with semantic meaning—embeddings via sentence-transformers and cosine similarity.
from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2') def are_similar(text1: str, text2: str, threshold=0.92) -> bool: embeddings = model.encode([text1, text2]) cosine_sim = np.dot(embeddings[0], embeddings[1]) / ( np.linalg.norm(embeddings[0]) * np.linalg.norm(embeddings[1]) ) return float(cosine_sim) >= threshold For large volumes—index in pgvector (PostgreSQL) or Milvus for approximate vector search.
Method Comparison
| Level | Speed | Accuracy | When to Use |
|---|---|---|---|
| Exact Match | Instant | 100% | Has SKU/EAN |
| Content Hashing | Fast | High | Unchanged content |
| Fuzzy Matching | Medium | Medium | Titles with word variations |
| Vector Similarity | Slow | High | Semantically similar texts |
Vector similarity yields 20% more exact matches than fuzzy matching but requires 5x more indexing time.
How to Speed Up Deduplication of Large Arrays?
With millions of records, pairwise comparison is impractical. We use strategies:
- MinHash + LSH—fast candidate detection for duplicates in large text sets. More about MinHash.
- Blocking—first filter by exact attributes (category, price range), then fuzzy compare only within the block.
-
Indexes in PostgreSQL—
pg_trgmfor fuzzy string search withsimilarity()and%operator. Documentation pg_trgm.
-- Enable extension CREATE EXTENSION pg_trgm; CREATE INDEX ON products USING GIN (title gin_trgm_ops); -- Search similar titles SELECT id, title, similarity(title, 'Iphone 15 pro max 256') AS sim FROM products WHERE title % 'Iphone 15 pro max 256' ORDER BY sim DESC LIMIT 10; | Strategy | Speed | Memory | Applicability |
|---|---|---|---|
| MinHash+LSH | Very fast | Moderate | Millions of texts |
| Blocking | Fast | Low | Categorized data |
| pg_trgm GIN | Medium | Medium | Strings up to 1000 chars |
How to choose a strategy?
The choice depends on data volume, available memory, and desired accuracy. For catalogs up to 100k items, pg_trgm is sufficient. For 10+ million records, use MinHash+LSH with blocking.Duplicate Management
Found duplicates are not removed automatically. The system forms candidate groups with a calculated match score. The final decision is either automatic (when score > 95%) or through a manual review interface.
Why Trust Deduplication to Professionals?
Incorrect deduplication can delete unique records or miss duplicates, leading to data contradictions. Our engineering approach involves analyzing data structure, selecting optimal algorithms, and scaling the solution to your volumes. One of our solutions for an electronics online store reduced duplicates from 35% to 2%, speeding up page loads by 40%. We have implemented over 30 deduplication projects, accumulating expertise in this area. Contact us for a free assessment of your project—we will propose the architecture and timeline. Order the implementation of a deduplication system and get a specialist consultation.
Stages of Work
- Data structure and duplicate source analysis.
- Deduplication architecture design (level and index selection).
- Algorithm implementation with tests on your data.
- Interface for manual verification (if required).
- Documentation and team training.
Timeline
Implementation time for a deduplication system with several levels: 4–7 working days. If vector schema or scaling to millions of records is needed, the timeline extends to 2–3 weeks. We guarantee removal of duplicates with at least 95% accuracy on a test sample.