Proxy Rotation Setup for 1C-Bitrix Parser

After 200–300 requests from a single IP, the source starts returning 429 Too Many Requests or a captcha. This is standard protection: Cloudflare, DataDome, PerimeterX — all track request frequency by IP. The only way to bypass rate-limiting in industrial parsing is proxy rotation. Let's break down integrating a proxy pool into a Bitrix parser. We set up such a system turnkey: from pool selection to health-check agent. According to Cloudflare, about 30% of all internet traffic is generated by bots, and protection against them is only getting tougher. This guide covers proxy setup for a robust Bitrix parser.

Types of Proxies and What to Choose

Datacenter proxies are 10x cheaper than residential but twice as likely to be blocked. For most catalog auto-population tasks in Bitrix, a pool of 20–50 datacenter proxies is enough. Residential — if the source actively blocks. Setup cost typically ranges from $200 to $500 for a basic proxy rotation integration, with monthly proxy pool costs starting at $50 for 20 datacenter proxies.

How Proxy Rotation Works

Rotation means changing the IP before each request or after N requests. A parser in Bitrix usually uses \Bitrix\Main\Web\HttpClient or cURL directly. The proxy is set via connection options. The task is to select the next proxy from the pool before each request. We implement a ProxyRotator class with cooldown support for banned proxies.

Pool storage — a table or configuration file:

// /local/php_interface/parser/proxy_pool.php
return [
    ['host' => '185.1.2.3', 'port' => 8080, 'user' => 'u1', 'pass' => 'p1', 'type' => 'http'],
    ['host' => '185.1.2.4', 'port' => 8080, 'user' => 'u2', 'pass' => 'p2', 'type' => 'socks5'],
    // ...
];

Rotator class:

class ProxyRotator
{
    private array $pool;
    private array $failed = [];
    private int $index = 0;

    public function next(): ?array
    {
        $attempts = count($this->pool);
        while ($attempts-- > 0) {
            $proxy = $this->pool[$this->index % count($this->pool)];
            $this->index++;
            $key = $proxy['host'] . ':' . $proxy['port'];
            if (!isset($this->failed[$key]) || $this->failed[$key] < time()) {
                return $proxy;
            }
        }
        return null; // all proxies in cooldown
    }

    public function markFailed(array $proxy, int $cooldownSec = 300): void
    {
        $key = $proxy['host'] . ':' . $proxy['port'];
        $this->failed[$key] = time() + $cooldownSec;
    }
}

Rotation strategies:

For catalog parsing, we recommend sticky per source with rotation every 50–100 requests or upon receiving 429/403.

Why Sticky per Source is Better Than Round-robin

Sticky per source imitates real user behavior: one IP per site. This reduces block frequency by 3–4 times compared to round-robin. In our projects, proxy rotation reduces blocking rates by up to 80% compared to single-IP parsing. However, if the source has multiple domains, sticky per source requires a separate proxy per domain.

Integration with Bitrix HttpClient

$proxy = $rotator->next();
$http = new \Bitrix\Main\Web\HttpClient();
$http->setProxy($proxy['host'], $proxy['port'], $proxy['user'], $proxy['pass']);
$http->setTimeout(15);
$http->setStreamTimeout(30);
$response = $http->get($url);

if ($http->getStatus() === 429 || $http->getStatus() === 403) {
    $rotator->markFailed($proxy, 600);
    // retry with another proxy
}

When using cURL directly — options CURLOPT_PROXY, CURLOPT_PROXYUSERPWD, CURLOPT_PROXYTYPE (CURLPROXY_HTTP or CURLPROXY_SOCKS5).

The principle is universal and can be adapted to any PHP platform, ensuring seamless parser integration with proxy rotation.

Pool Health Monitoring

Proxies die — expiration, IP gets banned, provider disconnects. A regular health-check is needed. An agent runs once an hour, iterates through the pool and checks each proxy with a request to https://httpbin.org/ip. Result — status update in the configuration (active/dead). Dead proxies are automatically excluded from rotation.

Log statistics for each proxy: number of successful requests, number of 429/403, average response time. This allows you to identify "slow" proxies and exclude them before complete death.

What to Do When a Proxy Gets Blocked

If a proxy receives 429 or 403, our ProxyRotator puts it into cooldown for 5–10 minutes. But this is not enough. Additionally, you should implement exponential backoff: after each error, increase the wait time for that IP. Also useful is a blacklist of proxies that have been permanently banned — they need to be removed from the pool and replaced with fresh ones.

Additional Measures

• Delay between requests — 1–5 seconds random pause. Even with proxy rotation, machine-gun frequency looks suspicious.
• User-Agent rotation — pool of 10–20 current UA strings, switched along with the proxy.
• Referer and headers — send Accept-Language, Accept-Encoding, Referer from the previous page. Without them, the request looks like a bot.

Deliverables

1. Selection and purchase of the proxy pool (datacenter or residential).
2. Writing a configuration file or table for storing the proxy list.
3. Implementation of the ProxyRotator class with the chosen rotation strategy.
4. Integration with HttpClient or cURL in the parser.
5. Writing the health-check agent and logging statistics.
6. Documentation of the process for your team.
7. Access to a proxy pool management panel.
8. Training session for your team (1 hour).
9. Ongoing support for 1 month after deployment.

With 5+ years of experience in Bitrix development, 50+ parsing projects, and 5 years on the market, we guarantee stable parser operation even against aggressive anti-bots. Contact us for a preliminary assessment of your task. Order turnkey parsing setup — we will prepare an individual solution.

Parser Development for 1C-Bitrix: Where to Start?

XMLReader, not SimpleXML — the choice of tool determines the project's fate. SimpleXML loads the entire XML into memory, and with an 800 MB supplier file, PHP will crash with a fatal error on a 512 MB limit. XMLReader processes streamingly, node by node, consuming 20–30 MB — 30 times more efficient. This detail starts any parser development for Bitrix. With over 10 years of Bitrix development and 50+ parser projects delivered, we know the pitfalls. Contact us to start your parser development today.

What Problems Does Parsing Solve?

• Primary catalog filling — 15,000 cards with descriptions, characteristics, photos. Manually, that's three months of content manager work; a parser takes a week with debugging.
• Competitor price monitoring — collecting data from Ozon, Wildberries, competitor sites. A competitor drops the price on a hot item — you find out in two hours, not two weeks.
• Supplier aggregation — five price lists in different formats (CSV with CP1251, XML in CommerceML, Excel with merged cells) become a single catalog with a unified property system.
• Card enrichment — pulling characteristics, instructions, 3D models from manufacturer sites. Without this, a product card is an SEO empty shell.
• Assortment update — products missing from the supplier feed are deactivated via CIBlockElement::Update($ID, ['ACTIVE' => 'N']). New ones are created. The catalog stays synchronized.

What Tools Do We Use in Parser Development?

Static websites — PHP (Goutte, Symfony DomCrawler) or Python (Scrapy, lxml). Speed: 50–100 pages/sec. Sufficient for catalogs without JS rendering.

SPA and dynamic websites — Puppeteer or Playwright. Infinite scroll, AJAX filters, lazy-load images — headless browser handles it all. Speed drops to 1–10 pages/sec, but there is no alternative: data exists only after JavaScript execution.

Supplier files:

• Excel (XLS, XLSX) — PhpSpreadsheet. Beware of merged cells and formulas — they break automatic mapping.
• CSV — fgetcsv() with correct encoding. Suppliers love CP1251, BOM in UTF-8, and semicolons instead of commas. All need detection and handling.
• XML/YML — XMLReader for large files, SimpleXML for feeds up to 50 MB.
• CommerceML — standard exchange format with 1C. We parse import.xml and offers.xml, map to information block structure.

API — Supplier REST endpoints, marketplace APIs (Ozon Seller API, Wildberries API). We work within rate limits, handle pagination.

How Is the Auto-Population Pipeline Structured?

Four stages. Each can break in its own way.

1. Collection. Parser crawls sources via cron schedule. Raw data goes to an intermediate table — not directly into b_iblock_element. Log everything: pages visited, elements parsed, where we got 403 or timeout. Without logs, debugging a parser is like fortune-telling.

2. Normalization. Main work here:

   • Clean HTML tags, extra spaces, Unicode garbage
   • Units: "mm" → "mm", "millimeters" → "mm", "миллиметр" → "mm"
   • Map supplier categories to Bitrix information block sections. One supplier has "Notebooks", another "Notebooks and tablets", third "Laptops" — all into one section
   • Deduplication by SKU, EAN/GTIN. One product from three suppliers should not appear three times

3. Load into Bitrix. Via CIBlockElement::Add() for new elements, CIBlockElement::Update() for existing. Images: download, resize via CFile::ResizeImageGet(), convert to WebP. Properties via CIBlockElement::SetPropertyValuesEx(). SEO meta via \Bitrix\Iblock\InheritedProperty\ElementValues. SEF URLs generated from name transliteration.

4. Update. Key point — not overwrite manual edits by content manager. Update only price, stock, activity. Description and photos manually edited are flagged with UF_MANUAL_EDIT property and skipped during import. Products missing from feed are deactivated, not deleted.

Why Is Competitor Price Monitoring Necessary?

A separate subsystem with its own specifics:

Result — dashboard: your product vs competitors, price history, trends. The manager sees where to raise price without losing position, and where to react.

CSV/XML Import Module: Customization for Your Format

For supplier files — custom module with admin panel:

• Configurable mapping: "column B in file → BRAND property of information block"
• Auto-detect encoding (CP1251, UTF-8, UTF-16) via mb_detect_encoding() with validation
• Download images from URL with queue — to avoid channel saturation
• Incremental update by row hash: row changed — update, no — skip
• Cron schedule, report: created 145, updated 892, errors 3 (with details)

Large files: CSV processed in batches of 1000 rows via fgetcsv() (10 times faster than row-by-row), XML streamed via XMLReader, background execution via Bitrix agent queue — no PHP timeouts.

Legal Aspects to Consider

• robots.txt — respect it. Crawl-delay — comply.
• Request frequency — 1–2 per second, no more. Don't DDoS someone else's site.
• Manufacturer content — use it. Unique author texts — don't copy.
• Personal data — don't collect.

What Is Included in a Turnkey Parser Development?

How We Work and Deadlines

1. Prototype — parser for 1–2 sources in 2–3 days. Assess data quality, pitfalls (Cloudflare protection, captcha, dynamic loading).
2. Development — full pipeline: parser → normalization → import into Bitrix → admin panel for management.
3. Testing — run on full catalog volume, check edge cases (empty fields, malformed HTML, broken images).
4. Launch — configure cron, error monitoring via Telegram bot.
5. Support — competitor changed layout? Update CSS selectors in parser.

Get in touch for a free consultation — we will analyze your data sources and propose the optimal parser architecture. Request a project assessment today and get a fixed deadline. We guarantee stable parser operation and full support throughout the usage period.