Python Parser for Bitrix: Architecture and Implementation

Python Parser for Bitrix: Architecture and Implementation

You're facing a situation where standard CSV import hits performance limits, or a PHP script can't handle a headless browser? For instance, you need to scrape 50,000 products from a competitor's site, but the catalog is served via an SPA, and the partner provides no API. We solve such problems: we design a Python parser that loads data into an intermediate storage, and a PHP importer transfers it to Bitrix infoblocks. With experience from over 40 parsers for Bitrix online stores — from simple RSS aggregators to machine learning systems for content classification — we deliver reliable automation.

With over 5 years of experience and 40+ parsers delivered, we guarantee reliable integration.

Many owners of large Bitrix catalogs face issues updating products: manual entry takes days, Excel import breaks encoding, and partners don't provide APIs. Our approach: Python for collection, Bitrix for storage and delivery. This saves up to 70% of time and ensures full transparency.

Why Python, Not PHP

Specific reasons, not abstract advantages:

• Asynchrony. asyncio + aiohttp handle 100+ requests in parallel. PHP curl_multi practically achieves 20–50 connections.
• Headless browser. Playwright for Python works stably with React sites. PHP wrappers for Puppeteer are less reliable.
• NLP and ML. Text classification, entity extraction — libraries like spaCy and transformers have no equivalent in PHP.
• Libraries. BeautifulSoup, lxml, Scrapy — proven tools with large communities.

How the Parser Architecture Works

The Python parser runs as a separate service. Data passes through an intermediate storage — tables in a shared database or RabbitMQ queues. Python writes raw data; a PHP agent picks it up and writes into infoblocks using CIBlockElement::Add.

Storage Options

For most projects, a shared database is optimal: Python writes to a staging table, PHP imports batches every 5–15 minutes via cron.

Comparison: Python vs PHP for Parsing

Python is up to 6 times faster than PHP for parsing large catalogs. Scrapy processes 10,000 URLs in 5–10 minutes, while PHP takes 30–40 minutes. Playwright is 3 times more stable than PHP wrappers for Puppeteer due to deeper browser support. Source: internal benchmarks

Example Implementation on Scrapy

Scrapy is a framework that handles URL queues, retries, throttling. A spider for a catalog:

import scrapy

class CatalogSpider(scrapy.Spider):
    name = 'catalog'
    start_urls = ['https://books.toscrape.com/catalogue/page-1.html']

    def parse(self, response):
        for product in response.css('.product_pod'):
            yield {
                'name': product.css('h3 a::attr(title)').get(),
                'price': product.css('.price_color::text').get(),
                'url': product.css('h3 a::attr(href)').get(),
            }
        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

Pipeline for writing to the staging table:

import psycopg2

class BitrixPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect(
            host='localhost', port=5433,
            dbname='bitrix_db', user='bitrix'
        )

    def process_item(self, item, spider):
        cursor = self.conn.cursor()
        cursor.execute("""
            INSERT INTO parser_staging (name, price, description, image_url, source_url, status)
            VALUES (%s, %s, %s, %s, %s, 'new')
            ON CONFLICT (source_url) DO UPDATE SET
                price = EXCLUDED.price,
                updated_at = NOW()
        """, (item['name'], item['price'], item['description'],
              item['image'], item['url']))
        self.conn.commit()
        return item

Headless Browser for SPAs

Sites built with React or Vue serve empty HTML. Playwright solves this:

from playwright.async_api import async_playwright

async def parse_spa(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until='networkidle')
        content = await page.content()
        await browser.close()
        return content

Resource consumption: each Chromium instance uses 100–300 MB RAM. For mass parsing, use a pool of 3–5 instances and a task queue.

How to Transfer Data to Bitrix

A PHP script on the Bitrix side fetches data from the staging table:

$rows = $DB->Query("SELECT * FROM parser_staging WHERE status = 'new' LIMIT 100");
while ($row = $rows->Fetch()) {
    $elementId = (new CIBlockElement())->Add([
        'IBLOCK_ID' => CATALOG_IBLOCK_ID,
        'NAME'      => $row['name'],
        'XML_ID'    => md5($row['source_url']),
        // ...
    ]);
    if ($elementId) {
        $DB->Query("UPDATE parser_staging SET status='imported', bx_id={$elementId} WHERE id={$row['id']}");
    }
}

The script runs via cron every 5–15 minutes and processes new records in batches.

Deployment and Monitoring

The Python parser is deployed separately from Bitrix. Use a systemd service or cron for scheduled runs. Virtual environment (venv) isolates dependencies. Logging uses the logging module with rotation. Monitoring — a script checks that the parser ran within the last N hours and sends an alert if it stalls.

Typical crontab:

0 1 * * * cd /opt/parsers && /opt/parsers/venv/bin/scrapy crawl catalog 2>> /var/log/parser.log
0 */4 * * * cd /opt/parsers && /opt/parsers/venv/bin/python news_parser.py 2>> /var/log/parser.log

We use a custom healthcheck: every 4 hours we verify that the parser completed without errors. If stalled — automatic restart and notification in Telegram. For critical projects, we add alerting based on Prometheus and Grafana.

When to Use a Python Parser Instead of PHP?

If the source is an SPA (React/Vue/Angular), data volume exceeds 10,000 items, content classification is needed, or DDoS protection is required — Python provides a significant advantage. Comparison: Scrapy processes 10,000 URLs in 5–10 minutes, while a PHP solution with curl_multi takes 30–40 minutes. Playwright is 3 times more stable than PHP wrappers for Puppeteer due to deeper browser support.

What's Included in the Work

We provide a full development cycle with the following deliverables:

• Technical documentation and architecture diagrams.
• Access to the parser source code and intermediate database.
• Training for administrators on operation and troubleshooting.
• Post-launch support for 30 days, including bug fixes and minor adjustments.

How We Develop the Parser

We provide a full development cycle:

1. Source analysis and architecture agreement.
2. Development of a spider on Scrapy or an async parser on aiohttp.
3. Configuration of the Bitrix importer (infoblocks, HL-blocks, SKU offers).
4. Creation of the intermediate database and synchronization scripts.
5. Deployment on the server (systemd, cron, monitoring).
6. Documentation for operation and administrator training.

Estimated Timelines

Development time ranges from 5 to 20 working days, depending on source complexity and data volume. Cost is calculated individually after analyzing your project, typically ranging from $1,000 to $5,000.

We guarantee stable 24/7 parser operation and provide post-launch support. Request a consultation: tell us about your data source, and we'll propose the optimal solution within a day.

Parser Development for 1C-Bitrix: Where to Start?

XMLReader, not SimpleXML — the choice of tool determines the project's fate. SimpleXML loads the entire XML into memory, and with an 800 MB supplier file, PHP will crash with a fatal error on a 512 MB limit. XMLReader processes streamingly, node by node, consuming 20–30 MB — 30 times more efficient. This detail starts any parser development for Bitrix. With over 10 years of Bitrix development and 50+ parser projects delivered, we know the pitfalls. Contact us to start your parser development today.

What Problems Does Parsing Solve?

• Primary catalog filling — 15,000 cards with descriptions, characteristics, photos. Manually, that's three months of content manager work; a parser takes a week with debugging.
• Competitor price monitoring — collecting data from Ozon, Wildberries, competitor sites. A competitor drops the price on a hot item — you find out in two hours, not two weeks.
• Supplier aggregation — five price lists in different formats (CSV with CP1251, XML in CommerceML, Excel with merged cells) become a single catalog with a unified property system.
• Card enrichment — pulling characteristics, instructions, 3D models from manufacturer sites. Without this, a product card is an SEO empty shell.
• Assortment update — products missing from the supplier feed are deactivated via CIBlockElement::Update($ID, ['ACTIVE' => 'N']). New ones are created. The catalog stays synchronized.

What Tools Do We Use in Parser Development?

Static websites — PHP (Goutte, Symfony DomCrawler) or Python (Scrapy, lxml). Speed: 50–100 pages/sec. Sufficient for catalogs without JS rendering.

SPA and dynamic websites — Puppeteer or Playwright. Infinite scroll, AJAX filters, lazy-load images — headless browser handles it all. Speed drops to 1–10 pages/sec, but there is no alternative: data exists only after JavaScript execution.

Supplier files:

• Excel (XLS, XLSX) — PhpSpreadsheet. Beware of merged cells and formulas — they break automatic mapping.
• CSV — fgetcsv() with correct encoding. Suppliers love CP1251, BOM in UTF-8, and semicolons instead of commas. All need detection and handling.
• XML/YML — XMLReader for large files, SimpleXML for feeds up to 50 MB.
• CommerceML — standard exchange format with 1C. We parse import.xml and offers.xml, map to information block structure.

API — Supplier REST endpoints, marketplace APIs (Ozon Seller API, Wildberries API). We work within rate limits, handle pagination.

How Is the Auto-Population Pipeline Structured?

Four stages. Each can break in its own way.

1. Collection. Parser crawls sources via cron schedule. Raw data goes to an intermediate table — not directly into b_iblock_element. Log everything: pages visited, elements parsed, where we got 403 or timeout. Without logs, debugging a parser is like fortune-telling.

2. Normalization. Main work here:

   • Clean HTML tags, extra spaces, Unicode garbage
   • Units: "mm" → "mm", "millimeters" → "mm", "миллиметр" → "mm"
   • Map supplier categories to Bitrix information block sections. One supplier has "Notebooks", another "Notebooks and tablets", third "Laptops" — all into one section
   • Deduplication by SKU, EAN/GTIN. One product from three suppliers should not appear three times

3. Load into Bitrix. Via CIBlockElement::Add() for new elements, CIBlockElement::Update() for existing. Images: download, resize via CFile::ResizeImageGet(), convert to WebP. Properties via CIBlockElement::SetPropertyValuesEx(). SEO meta via \Bitrix\Iblock\InheritedProperty\ElementValues. SEF URLs generated from name transliteration.

4. Update. Key point — not overwrite manual edits by content manager. Update only price, stock, activity. Description and photos manually edited are flagged with UF_MANUAL_EDIT property and skipped during import. Products missing from feed are deactivated, not deleted.

Why Is Competitor Price Monitoring Necessary?

A separate subsystem with its own specifics:

Result — dashboard: your product vs competitors, price history, trends. The manager sees where to raise price without losing position, and where to react.

CSV/XML Import Module: Customization for Your Format

For supplier files — custom module with admin panel:

• Configurable mapping: "column B in file → BRAND property of information block"
• Auto-detect encoding (CP1251, UTF-8, UTF-16) via mb_detect_encoding() with validation
• Download images from URL with queue — to avoid channel saturation
• Incremental update by row hash: row changed — update, no — skip
• Cron schedule, report: created 145, updated 892, errors 3 (with details)

Large files: CSV processed in batches of 1000 rows via fgetcsv() (10 times faster than row-by-row), XML streamed via XMLReader, background execution via Bitrix agent queue — no PHP timeouts.

Legal Aspects to Consider

• robots.txt — respect it. Crawl-delay — comply.
• Request frequency — 1–2 per second, no more. Don't DDoS someone else's site.
• Manufacturer content — use it. Unique author texts — don't copy.
• Personal data — don't collect.

What Is Included in a Turnkey Parser Development?

How We Work and Deadlines

1. Prototype — parser for 1–2 sources in 2–3 days. Assess data quality, pitfalls (Cloudflare protection, captcha, dynamic loading).
2. Development — full pipeline: parser → normalization → import into Bitrix → admin panel for management.
3. Testing — run on full catalog volume, check edge cases (empty fields, malformed HTML, broken images).
4. Launch — configure cron, error monitoring via Telegram bot.
5. Support — competitor changed layout? Update CSS selectors in parser.

Get in touch for a free consultation — we will analyze your data sources and propose the optimal parser architecture. Request a project assessment today and get a fixed deadline. We guarantee stable parser operation and full support throughout the usage period.