Parsing prices from competitors' websites for 1C-Bitrix

Parsing Competitor Prices for 1C-Bitrix

Competitor price monitoring is one of the few automations that delivers a direct, measurable impact on margins. Without it, managers either fail to respond to changes or spend hours gathering data manually. A properly configured price parser closes this gap with accuracy down to a few hours.

How price parsing differs from catalog parsing

The key difference is frequency and volume. A catalog is parsed infrequently (once a week or at launch). Prices need to be collected every 2–6 hours on a live system with thousands of items. This means:

• No time for a headless browser on every product — fast HTTP requests are needed
• Product identification is critical — the same product on different sites has different URLs
• Storing price history is mandatory — the trend matters more than the current value

Identifying competitor products

The hardest technical challenge is linking your own SKU to a competitor's product card. Three approaches:

By SKU/EAN: the most reliable. Search for the manufacturer's article number in the competitor's page text. Works when both sell the same branded products.

By name: fuzzy matching via similar_text() or Levenshtein. Requires manual verification on the first run — many false matches.

By URL from external databases: price aggregators (Yandex Market, Price.ru) often provide direct links to competitors tied to a specific model. Parse the aggregator as an intermediary.

The mapping result is stored in a CompetitorLinks Highload block:

UF_OWN_PRODUCT_ID → ID of our infoblock element
UF_COMPETITOR_URL → URL of the competitor's product card
UF_COMPETITOR_NAME → competitor identifier
UF_LAST_PRICE → last collected price
UF_LAST_CHECK → date of last check

Storing history and analytics

Storing the current price in a Highload block is fine, but history belongs in a dedicated table. Create a custom table competitor_price_history:

CREATE TABLE competitor_price_history (
    id SERIAL PRIMARY KEY,
    own_product_id INT NOT NULL,
    competitor_name VARCHAR(100),
    price DECIMAL(12,2),
    currency CHAR(3),
    in_stock BOOLEAN,
    parsed_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_cph_product ON competitor_price_history(own_product_id, parsed_at DESC);

This enables building price trend charts and finding patterns (e.g., a competitor drops prices every Friday).

Automated response to changes

After collecting data, apply business logic. Response options:

Manager notification — via \Bitrix\Main\Mail\Event::send() when a competitor's price deviates from yours by more than X%.

Automatic price update — via CCatalogProduct::Update() with a new value in b_catalog_price. Requires aligning rules with the business (minimum margin, prohibition on going below cost price).

Repricing flag — add a NEEDS_REPRICING = Y property to the infoblock element; the manager sees the list in the admin panel.

Case study: monitoring 5 competitors in the auto parts niche

Goal: 4,200 SKUs, 5 competitors, updates every 4 hours, automatic notification when the difference exceeds 7%.

Issues encountered:

• Two competitors used Cloudflare — had to add proxy rotation and random delays
• One site rendered prices via JS — a dedicated Puppeteer worker with a pool of 3 instances was set up for it
• SKUs at one competitor were stored in a non-standard data-sku attribute

Result: the system collects ~21,000 price data points per cycle (4,200 × 5) within the 4-hour window. The manager receives a deviation digest in the morning and evening.

Work timeline

Total for 5 competitors: 6–10 working days.