Parsing Data from Yandex.Market for 1C-Bitrix

Filling a 1C-Bitrix catalog with products from Yandex.Market is a typical task for online stores. The problem is that Market does not have a public API for bulk card export. The partner Content API only gives sellers access to their own data. For a third-party integrator, the only path is parsing, but with serious technical limitations: JavaScript rendering, captcha, IP blocking, and fingerprinting. Plus legal risks, as it violates the terms of service. Our certified team (over 7 years in Bitrix development and integrations) has implemented more than 30 such projects—from simple catalogs to large marketplaces. Our guaranteed solution includes proxy rotation and captcha handling. This article describes a proven approach that allows stable data collection and import into information blocks with minimal risks.

Parsed Data

A product card on Yandex.Market contains name, description, characteristics (key-value pairs), dynamic seller prices, images (1–15 photos), reviews, rating, and category. For a 1C-Bitrix catalog, you most often need name, description, characteristics, and images—prices are not worth parsing due to high volatility.

Technical Implementation

Yandex.Market is a single-page application. Data is loaded through internal APIs and rendered on the client. A simple HTTP request returns an empty shell. We use two approaches:

• Headless browser (Puppeteer, Playwright)—slow (3–5 seconds per page), but reliable.
• Intercepting internal APIs—10–20 times better (faster) than headless browser for mass extraction. The combined approach is better than using only one method.

Combination: headless for initial analysis and token retrieval, then direct requests for mass extraction.

Bypassing Yandex.Market Protection

Yandex blocks automated requests using SmartCaptcha, fingerprinting, and rate limiting. For stable parsing, you must use residential proxy rotation, random delays (2–10 sec), User-Agent rotation, and captcha handling. Without proxy rotation, a single IP gets blocked within an hour.

Mapping Data to Information Block

Market's structure does not match the information block. A transformation layer is needed:

Market characteristics are a flat list, while information block properties are typed. A mapping table is needed: "Weight, g" → PROPERTY_WEIGHT (number), "Color" → PROPERTY_COLOR (list).

Why an Intermediate Layer Is Important

The loading process should go through an intermediate storage (a separate table or JSON files). The import script uses the information block API:

CIBlockElement::Add($arFields);
CIBlockElement::SetPropertyValuesEx($elementId, $iblockId, $propertyValues);

Direct import from the parser is risky: if the parser breaks mid-way, partially filled cards remain in the catalog. For catalogs over 5,000 products, use \Bitrix\Iblock\ElementTable::add() — D7 API is faster and supports batch operations.

Step-by-Step Implementation Plan

1. Analyze target pages: determine data structure and tokens.
2. Develop the parser: choose approach (headless + intercept), configure proxies and captcha.
3. Create intermediate storage and mapping script.
4. Import into 1C-Bitrix information blocks with deduplication.
5. Set up incremental updates on a schedule.

Maintaining Freshness

For catalogs up to 1,000 products, a full reimport once a week works (2–4 hours). For 1,000–10,000, incremental crawling daily (4–8 hours). For over 10,000, combine incremental and trigger-based updates.

Deliverables and Scope of Work

When ordering Yandex.Market parsing for 1C-Bitrix, we provide:

• Parser development with proxy rotation, captcha handling, and protection against blocks.
• Creation of a characteristic mapping table.
• Scripts for initial import and incremental updates.
• Documentation, accesses, staff training.
• Support during the launch phase (7-day post-launch support).
• Cost estimate and timeline.

Automated parsing saves an average of $1,000–$3,000 per month compared to manual entry. Parsing violates Yandex's terms of service. In practice, claims are rare, but using descriptions and photos "as is" is risky. We recommend rewriting descriptions and checking image licenses.

How to Order

If you need a stable solution for filling a Bitrix catalog with data from Yandex.Market—contact us. We will evaluate the project, choose a strategy, and propose timelines (from 2 to 6 weeks depending on volume). The cost is calculated individually—prices start from $500 for small catalogs, ranging up to $5,000. We have completed over 30 such integrations. Get a consultation—discuss the details.

Parser Development for 1C-Bitrix: Where to Start?

XMLReader, not SimpleXML — the choice of tool determines the project's fate. SimpleXML loads the entire XML into memory, and with an 800 MB supplier file, PHP will crash with a fatal error on a 512 MB limit. XMLReader processes streamingly, node by node, consuming 20–30 MB — 30 times more efficient. This detail starts any parser development for Bitrix. With over 10 years of Bitrix development and 50+ parser projects delivered, we know the pitfalls. Contact us to start your parser development today.

What Problems Does Parsing Solve?

• Primary catalog filling — 15,000 cards with descriptions, characteristics, photos. Manually, that's three months of content manager work; a parser takes a week with debugging.
• Competitor price monitoring — collecting data from Ozon, Wildberries, competitor sites. A competitor drops the price on a hot item — you find out in two hours, not two weeks.
• Supplier aggregation — five price lists in different formats (CSV with CP1251, XML in CommerceML, Excel with merged cells) become a single catalog with a unified property system.
• Card enrichment — pulling characteristics, instructions, 3D models from manufacturer sites. Without this, a product card is an SEO empty shell.
• Assortment update — products missing from the supplier feed are deactivated via CIBlockElement::Update($ID, ['ACTIVE' => 'N']). New ones are created. The catalog stays synchronized.

What Tools Do We Use in Parser Development?

Static websites — PHP (Goutte, Symfony DomCrawler) or Python (Scrapy, lxml). Speed: 50–100 pages/sec. Sufficient for catalogs without JS rendering.

SPA and dynamic websites — Puppeteer or Playwright. Infinite scroll, AJAX filters, lazy-load images — headless browser handles it all. Speed drops to 1–10 pages/sec, but there is no alternative: data exists only after JavaScript execution.

Supplier files:

• Excel (XLS, XLSX) — PhpSpreadsheet. Beware of merged cells and formulas — they break automatic mapping.
• CSV — fgetcsv() with correct encoding. Suppliers love CP1251, BOM in UTF-8, and semicolons instead of commas. All need detection and handling.
• XML/YML — XMLReader for large files, SimpleXML for feeds up to 50 MB.
• CommerceML — standard exchange format with 1C. We parse import.xml and offers.xml, map to information block structure.

API — Supplier REST endpoints, marketplace APIs (Ozon Seller API, Wildberries API). We work within rate limits, handle pagination.

How Is the Auto-Population Pipeline Structured?

Four stages. Each can break in its own way.

1. Collection. Parser crawls sources via cron schedule. Raw data goes to an intermediate table — not directly into b_iblock_element. Log everything: pages visited, elements parsed, where we got 403 or timeout. Without logs, debugging a parser is like fortune-telling.

2. Normalization. Main work here:

   • Clean HTML tags, extra spaces, Unicode garbage
   • Units: "mm" → "mm", "millimeters" → "mm", "миллиметр" → "mm"
   • Map supplier categories to Bitrix information block sections. One supplier has "Notebooks", another "Notebooks and tablets", third "Laptops" — all into one section
   • Deduplication by SKU, EAN/GTIN. One product from three suppliers should not appear three times

3. Load into Bitrix. Via CIBlockElement::Add() for new elements, CIBlockElement::Update() for existing. Images: download, resize via CFile::ResizeImageGet(), convert to WebP. Properties via CIBlockElement::SetPropertyValuesEx(). SEO meta via \Bitrix\Iblock\InheritedProperty\ElementValues. SEF URLs generated from name transliteration.

4. Update. Key point — not overwrite manual edits by content manager. Update only price, stock, activity. Description and photos manually edited are flagged with UF_MANUAL_EDIT property and skipped during import. Products missing from feed are deactivated, not deleted.

Why Is Competitor Price Monitoring Necessary?

A separate subsystem with its own specifics:

Result — dashboard: your product vs competitors, price history, trends. The manager sees where to raise price without losing position, and where to react.

CSV/XML Import Module: Customization for Your Format

For supplier files — custom module with admin panel:

• Configurable mapping: "column B in file → BRAND property of information block"
• Auto-detect encoding (CP1251, UTF-8, UTF-16) via mb_detect_encoding() with validation
• Download images from URL with queue — to avoid channel saturation
• Incremental update by row hash: row changed — update, no — skip
• Cron schedule, report: created 145, updated 892, errors 3 (with details)

Large files: CSV processed in batches of 1000 rows via fgetcsv() (10 times faster than row-by-row), XML streamed via XMLReader, background execution via Bitrix agent queue — no PHP timeouts.

Legal Aspects to Consider

• robots.txt — respect it. Crawl-delay — comply.
• Request frequency — 1–2 per second, no more. Don't DDoS someone else's site.
• Manufacturer content — use it. Unique author texts — don't copy.
• Personal data — don't collect.

What Is Included in a Turnkey Parser Development?

How We Work and Deadlines

1. Prototype — parser for 1–2 sources in 2–3 days. Assess data quality, pitfalls (Cloudflare protection, captcha, dynamic loading).
2. Development — full pipeline: parser → normalization → import into Bitrix → admin panel for management.
3. Testing — run on full catalog volume, check edge cases (empty fields, malformed HTML, broken images).
4. Launch — configure cron, error monitoring via Telegram bot.
5. Support — competitor changed layout? Update CSS selectors in parser.

Get in touch for a free consultation — we will analyze your data sources and propose the optimal parser architecture. Request a project assessment today and get a fixed deadline. We guarantee stable parser operation and full support throughout the usage period.