How to configure anti-scraping bypass for 1C-Bitrix?
Configuring anti-scraping bypass for 1C-Bitrix becomes necessary when the data source updates its protection — and a parser that worked for months suddenly stops receiving content. Instead of HTML with prices, you get a page with CAPTCHA, a JavaScript challenge, or an empty body. This is the reality of industrial scraping: protection systems evolve, and the parser must adapt. We solve this problem turnkey — from diagnostics to setting up fingerprint rotation and a headless browser. Below we break down the main protection types and technical approaches to handling them.
How to choose the right tool for each protection type?
JavaScript Challenge (Cloudflare, DataDome) – Configuring Bypass
The server returns HTTP 503 with JavaScript code that must execute in a browser to set a cookie like cf_clearance or datadome. Indication: the body contains <noscript> and window._cf_chl_opt or similar obfuscated script.
Our approach: a headless browser (Puppeteer/Playwright) on Node.js runs as a microservice. The PHP parser sends the URL to http://localhost:3000/render?url=..., receives the rendered HTML and cookies, and then uses them in the regular HttpClient. As noted by Puppeteer developers, a headless browser fully emulates real user behavior, allowing you to avoid running every request through the browser — cookies live 15–30 minutes, during which hundreds of normal requests can be made.
Rate Limiting and Browser Fingerprinting
HTTP 429 or 403 after N requests is classic rate limiting. Fingerprinting checks TLS fingerprint (JA3), header order, and JavaScript API availability. A default cURL has a distinct JA3 fingerprint that differs from a browser's.
We use curl-impersonate — a cURL fork that emulates the TLS fingerprint of Chrome or Firefox. In the PHP parser, we configure CURLOPT_SSL_CIPHER_LIST and CURLOPT_SSLVERSION to mimic a real browser. Proxy rotation (SOCKS5, residential) complements the setup.
Honeypot Links
Hidden via CSS links (display:none, visibility:hidden) that only bots click. Clicking one results in an instant IP ban. We check the computed styles of an element before any click: display, visibility, opacity, position outside viewport. If the parser uses DOMDocument, we analyze inline styles and classes.
Headless Browser: Faster and More Reliable than Plain cURL
A standard HTTP request without bypass returns 503 in 0.1 seconds; a headless browser executes JS in 2–3 seconds and returns real HTML. The speed difference of 20–30 times is compensated by stability — once you get good HTML, the parser doesn't waste time on retries. Our headless browser solution is 20 times faster than a pure cURL approach with retries. For large volumes (1000+ pages), the headless browser is used only to obtain cookies, and data is downloaded with a regular HttpClient — this reduces server load by 5 times.
How to handle CAPTCHA without external services?
If the source shows a CAPTCHA, we use a recognition service like 2Captcha or Anti-Captcha — we send the image and receive the answer via API. Recognition of a single CAPTCHA costs a small fee, delay 10–30 seconds. Often CAPTCHA appears as a reaction to rate limiting; reducing frequency and rotating proxies can eliminate the CAPTCHA entirely without external services.
Integration with 2Captcha from a PHP parser:
$taskId = file_get_contents("http://2captcha.com/in.php?key={$apiKey}&method=base64&body=" . base64_encode($captchaImage)); // Wait for solution (polling) $result = file_get_contents("http://2captcha.com/res.php?key={$apiKey}&action=get&id={$taskId}"); Tools for Bypassing Protection
For each protection type we apply a targeted tool combination. Cloudflare requires a headless browser with fingerprint emulation, DataDome similarly. Rate limiting is bypassed with residential proxies and custom headers. Honeypot is excluded by style analysis.
| Protection Type | Complexity | Setup Time | Tool |
|---|---|---|---|
| JavaScript Challenge | Medium | 1 day | Headless browser (Puppeteer/Playwright) |
| Rate Limiting | Low | 0.5 day | Proxies + delays + header rotation |
| Browser Fingerprinting | High | 0.5 day | curl-impersonate + custom headers |
| Honeypot | Low | 0.25 day | Computed styles check |
| CAPTCHA | Medium | 0.5 day | 2Captcha / frequency reduction |
| Work Stage | Duration | Result |
|---|---|---|
| Protection diagnostics | 2–4 hours | Report with protection type and recommendations |
| Headless renderer setup | 4–8 hours | Working service with API |
| Integration with Bitrix parser | 4–6 hours | Stable data collection |
| Testing on real source | 24–48 hours | Confirmed stability |
| Documentation | 2 hours | Maintenance instructions |
More about fingerprint masking
For bypassing Browser Fingerprinting we use curl-impersonate with custom JA3 and HTTP/2 headers. Additionally we configure User-Agent, Accept-Language, and header order according to the target browser. In the headless browser we disable automation flags (webdriver, navigator.webdriver) and emulate mouse/scroll for realism.What's Included in the Work
- Protection diagnostics — we determine the type and version of protection on your source.
- Headless renderer setup — if a JS challenge is required, we deploy a service on Puppeteer/Playwright with masking.
- Integration with the Bitrix parser — we connect cookie/HTML retrieval via HTTP API and configure automatic renewal.
- Testing on the real source — we adjust delays, proxy rotation, and verify stability over 48 hours.
- Documentation — we describe protection behavior, update algorithm, and recommendations for changes.
Typical project cost ranges from $500 to $2000 depending on complexity. Our clients typically save $2,000–$5,000 annually by preventing data access disruptions.
Contact us for a project evaluation — we'll send a test drive within 24 hours. With over 5 years of experience and 100+ scraping projects in 1C-Bitrix, we guarantee quick adaptation to protection changes. Request an estimate right now.