Proxy Rotation Setup for 1C-Bitrix Parser

After 200–300 requests from a single IP, the source starts returning 429 Too Many Requests or a captcha. This is standard protection: Cloudflare, DataDome, PerimeterX — all track request frequency by IP. The only way to bypass rate-limiting in industrial parsing is proxy rotation. Let's break down integrating a proxy pool into a Bitrix parser. We set up such a system turnkey: from pool selection to health-check agent. According to Cloudflare, about 30% of all internet traffic is generated by bots, and protection against them is only getting tougher. This guide covers proxy setup for a robust Bitrix parser.

Types of Proxies and What to Choose

Datacenter proxies are 10x cheaper than residential but twice as likely to be blocked. For most catalog auto-population tasks in Bitrix, a pool of 20–50 datacenter proxies is enough. Residential — if the source actively blocks. Setup cost typically ranges from $200 to $500 for a basic proxy rotation integration, with monthly proxy pool costs starting at $50 for 20 datacenter proxies.

How Proxy Rotation Works

Rotation means changing the IP before each request or after N requests. A parser in Bitrix usually uses \Bitrix\Main\Web\HttpClient or cURL directly. The proxy is set via connection options. The task is to select the next proxy from the pool before each request. We implement a ProxyRotator class with cooldown support for banned proxies.

Pool storage — a table or configuration file:

// /local/php_interface/parser/proxy_pool.php return [ ['host' => '185.1.2.3', 'port' => 8080, 'user' => 'u1', 'pass' => 'p1', 'type' => 'http'], ['host' => '185.1.2.4', 'port' => 8080, 'user' => 'u2', 'pass' => 'p2', 'type' => 'socks5'], // ... ]; 

Rotator class:

class ProxyRotator { private array $pool; private array $failed = []; private int $index = 0; public function next(): ?array { $attempts = count($this->pool); while ($attempts-- > 0) { $proxy = $this->pool[$this->index % count($this->pool)]; $this->index++; $key = $proxy['host'] . ':' . $proxy['port']; if (!isset($this->failed[$key]) || $this->failed[$key] < time()) { return $proxy; } } return null; // all proxies in cooldown } public function markFailed(array $proxy, int $cooldownSec = 300): void { $key = $proxy['host'] . ':' . $proxy['port']; $this->failed[$key] = time() + $cooldownSec; } } 

Rotation strategies:

For catalog parsing, we recommend sticky per source with rotation every 50–100 requests or upon receiving 429/403.

Why Sticky per Source is Better Than Round-robin

Sticky per source imitates real user behavior: one IP per site. This reduces block frequency by 3–4 times compared to round-robin. In our projects, proxy rotation reduces blocking rates by up to 80% compared to single-IP parsing. However, if the source has multiple domains, sticky per source requires a separate proxy per domain.

Integration with Bitrix HttpClient

$proxy = $rotator->next(); $http = new \Bitrix\Main\Web\HttpClient(); $http->setProxy($proxy['host'], $proxy['port'], $proxy['user'], $proxy['pass']); $http->setTimeout(15); $http->setStreamTimeout(30); $response = $http->get($url); if ($http->getStatus() === 429 || $http->getStatus() === 403) { $rotator->markFailed($proxy, 600); // retry with another proxy } 

When using cURL directly — options CURLOPT_PROXY, CURLOPT_PROXYUSERPWD, CURLOPT_PROXYTYPE (CURLPROXY_HTTP or CURLPROXY_SOCKS5).

The principle is universal and can be adapted to any PHP platform, ensuring seamless parser integration with proxy rotation.

Pool Health Monitoring

Proxies die — expiration, IP gets banned, provider disconnects. A regular health-check is needed. An agent runs once an hour, iterates through the pool and checks each proxy with a request to https://httpbin.org/ip. Result — status update in the configuration (active/dead). Dead proxies are automatically excluded from rotation.

Log statistics for each proxy: number of successful requests, number of 429/403, average response time. This allows you to identify "slow" proxies and exclude them before complete death.

What to Do When a Proxy Gets Blocked

If a proxy receives 429 or 403, our ProxyRotator puts it into cooldown for 5–10 minutes. But this is not enough. Additionally, you should implement exponential backoff: after each error, increase the wait time for that IP. Also useful is a blacklist of proxies that have been permanently banned — they need to be removed from the pool and replaced with fresh ones.

Additional Measures

• Delay between requests — 1–5 seconds random pause. Even with proxy rotation, machine-gun frequency looks suspicious.
• User-Agent rotation — pool of 10–20 current UA strings, switched along with the proxy.
• Referer and headers — send Accept-Language, Accept-Encoding, Referer from the previous page. Without them, the request looks like a bot.

Deliverables

1. Selection and purchase of the proxy pool (datacenter or residential).
2. Writing a configuration file or table for storing the proxy list.
3. Implementation of the ProxyRotator class with the chosen rotation strategy.
4. Integration with HttpClient or cURL in the parser.
5. Writing the health-check agent and logging statistics.
6. Documentation of the process for your team.
7. Access to a proxy pool management panel.
8. Training session for your team (1 hour).
9. Ongoing support for 1 month after deployment.

With 5+ years of experience in Bitrix development, 50+ parsing projects, and 5 years on the market, we guarantee stable parser operation even against aggressive anti-bots. Contact us for a preliminary assessment of your task. Order turnkey parsing setup — we will prepare an individual solution.