Configuring product deduplication in 1C-Bitrix auto-fill

When auto-filling a catalog from multiple sources — for example, 1C and a partner price list — duplicates are inevitable. A typical example: the product "Bosch GSR 18V-50" appears twice with different names, prices, and stock. This clutters filters and takes up to 40% of managers' time on manual reconciliation. In large catalogs (50,000+ products), the duplicate share often exceeds 15–20%. We solve this at the Bitrix platform level: we configure matching rules, normalization, and merge strategies. The result is a clean catalog without duplicates, saving up to 70% of administration time. Setup costs start from $500 and scale based on catalog size, with typical savings exceeding $2,000/month.

Deduplication uses three levels: exact match by identifier, match by field combination, and fuzzy matching. Together they cover 95% of duplicates.

How do we handle duplicates without identifiers?

The most reliable method is exact match by key: if a product has a unique external identifier (EAN, GTIN, manufacturer SKU), deduplication is trivial — we check for an element with that XML_ID or PROPERTY_ARTICLE. In a catalog of 100,000 products, this check takes less than a second. Exact match is 100% reliable, while fuzzy matching achieves 70-85% accuracy but catches 95% of duplicates.

$existing = CIBlockElement::GetList( [], ['IBLOCK_ID' => $iblockId, 'XML_ID' => $externalId], false, ['nTopCount' => 1], ['ID'] )->Fetch(); if ($existing) { (new CIBlockElement())->Update($existing['ID'], $arFields); } else { (new CIBlockElement())->Add($arFields); } 

In practice, not all sources provide a stable unique identifier. A supplier SKU differs from the manufacturer SKU. One product may have 3–5 different SKUs from different suppliers, which creates complexity when matching.

Match by field combination

If there is no unique key — we search by combination: name + brand + key characteristic (volume, weight, size).

$filter = [ 'IBLOCK_ID' => $iblockId, '%NAME' => $normalizedName, 'PROPERTY_BRAND' => $brand, ]; 

Before comparison, names are normalized: lowercasing, removing extra spaces, replacing typographic symbols.

Fuzzy matching

In cases where names differ across suppliers: "Bosch GSR 18V-50 Professional" vs "Шуруповёрт Bosch GSR18V50". Algorithms used: similar_text(), Levenshtein distance, trigrams. Automatic deduplication is 10 times faster and more accurate than manual checking, and fuzzy matching gives 5 times fewer false positives. This reduces error rate by 80% during import.

Name normalization as a basis for deduplication

Normalization directly affects deduplication quality. Minimum set of transformations:

• Lowercasing: mb_strtolower().
• Removing special characters: brackets, quotes, hyphens, slashes.
• Removing stop words: "article", "art.", "code", "model".
• Normalizing spaces: multiple spaces → one.
• Removing unit and size references from the name (if stored in separate properties).

function normalizeName(string $name): string { $name = mb_strtolower(trim($name)); $name = preg_replace('/[()«»"\'\/\-]/', ' ', $name); $name = preg_replace('/\b(арт|артикул|код|модель)\b\.?/u', '', $name); $name = preg_replace('/\s+/', ' ', $name); return trim($name); } 

Choosing a duplicate merging strategy

When a duplicate is found, one of three strategies is applied:

In practice, a combination is most often used: automatic merging for non-critical fields (description, photos) and flagging for manual review when prices or key characteristics diverge.

Implementation in Bitrix

The XML_ID field is a key deduplication tool. It is indexed by default, search by it is fast. But for a multi-source catalog, one XML_ID is not enough.

Recommended scheme: a separate parser_external_ids reference infoblock with fields:

• NAME — external identifier (supplier SKU).
• PROPERTY_SOURCE — source (supplier name).
• PROPERTY_ELEMENT_ID — ID of the main catalog element.
• PROPERTY_MATCH_TYPE — match type (exact, fuzzy, manual).

During import, the parser first searches for the external ID in the reference. If found — updates the linked element. If not — checks fuzzy match by name. If a match is found — creates a link in the reference and updates the element. If not — creates a new one.

Batch deduplication of an existing catalog

If the catalog already contains duplicates — a one-time cleanup is needed. Algorithm:

1. Export all elements: ID, NAME, XML_ID, key properties.
2. Normalize names.
3. Group by normalized name + brand.
4. In each group, select a "master record" (the most complete card, largest ID, or priority source).
5. Transfer orders, bindings, properties from duplicates to the master record.
6. Deactivate duplicates (ACTIVE = 'N'), do not delete.

Recommendation: do not delete duplicates immediately. Deactivate and leave for 2–4 weeks. If an algorithm error is found, elements can be easily restored. This catalog duplicate removal approach ensures data integrity and effective catalog cleanup.

What's included in deduplication setup

• Analysis of data sources and identification of duplicate types.
• Development of a parser with normalization and fuzzy search.
• Configuration of the external_ids reference with priorities.
• Integration with Bitrix24 REST (if used).
• Testing on real data — 3–5 iterations.
• Documentation of the system operation.
• Training managers on handling duplicates.
• Technical support for 6 months.

Our Bitrix experts have 10+ years of experience in 1C-Bitrix development and have completed 40+ successful integration projects. Order a consultation — we will evaluate your project within 24 hours and propose an optimal deduplication solution. Contact us to get started.