Configuring Anti-Scraping Bypass for 1C-Bitrix

How to configure anti-scraping bypass for 1C-Bitrix?

Configuring anti-scraping bypass for 1C-Bitrix becomes necessary when the data source updates its protection — and a parser that worked for months suddenly stops receiving content. Instead of HTML with prices, you get a page with CAPTCHA, a JavaScript challenge, or an empty body. This is the reality of industrial scraping: protection systems evolve, and the parser must adapt. We solve this problem turnkey — from diagnostics to setting up fingerprint rotation and a headless browser. Below we break down the main protection types and technical approaches to handling them.

How to choose the right tool for each protection type?

JavaScript Challenge (Cloudflare, DataDome) – Configuring Bypass

The server returns HTTP 503 with JavaScript code that must execute in a browser to set a cookie like cf_clearance or datadome. Indication: the body contains <noscript> and window._cf_chl_opt or similar obfuscated script.

Our approach: a headless browser (Puppeteer/Playwright) on Node.js runs as a microservice. The PHP parser sends the URL to http://localhost:3000/render?url=..., receives the rendered HTML and cookies, and then uses them in the regular HttpClient. As noted by Puppeteer developers, a headless browser fully emulates real user behavior, allowing you to avoid running every request through the browser — cookies live 15–30 minutes, during which hundreds of normal requests can be made.

Rate Limiting and Browser Fingerprinting

HTTP 429 or 403 after N requests is classic rate limiting. Fingerprinting checks TLS fingerprint (JA3), header order, and JavaScript API availability. A default cURL has a distinct JA3 fingerprint that differs from a browser's.

We use curl-impersonate — a cURL fork that emulates the TLS fingerprint of Chrome or Firefox. In the PHP parser, we configure CURLOPT_SSL_CIPHER_LIST and CURLOPT_SSLVERSION to mimic a real browser. Proxy rotation (SOCKS5, residential) complements the setup.

Honeypot Links

Hidden via CSS links (display:none, visibility:hidden) that only bots click. Clicking one results in an instant IP ban. We check the computed styles of an element before any click: display, visibility, opacity, position outside viewport. If the parser uses DOMDocument, we analyze inline styles and classes.

Headless Browser: Faster and More Reliable than Plain cURL

A standard HTTP request without bypass returns 503 in 0.1 seconds; a headless browser executes JS in 2–3 seconds and returns real HTML. The speed difference of 20–30 times is compensated by stability — once you get good HTML, the parser doesn't waste time on retries. Our headless browser solution is 20 times faster than a pure cURL approach with retries. For large volumes (1000+ pages), the headless browser is used only to obtain cookies, and data is downloaded with a regular HttpClient — this reduces server load by 5 times.

How to handle CAPTCHA without external services?

If the source shows a CAPTCHA, we use a recognition service like 2Captcha or Anti-Captcha — we send the image and receive the answer via API. Recognition of a single CAPTCHA costs a small fee, delay 10–30 seconds. Often CAPTCHA appears as a reaction to rate limiting; reducing frequency and rotating proxies can eliminate the CAPTCHA entirely without external services.

Integration with 2Captcha from a PHP parser:

$taskId = file_get_contents("http://2captcha.com/in.php?key={$apiKey}&method=base64&body=" . base64_encode($captchaImage));
// Wait for solution (polling)
$result = file_get_contents("http://2captcha.com/res.php?key={$apiKey}&action=get&id={$taskId}");

Tools for Bypassing Protection

For each protection type we apply a targeted tool combination. Cloudflare requires a headless browser with fingerprint emulation, DataDome similarly. Rate limiting is bypassed with residential proxies and custom headers. Honeypot is excluded by style analysis.

What's Included in the Work

1. Protection diagnostics — we determine the type and version of protection on your source.
2. Headless renderer setup — if a JS challenge is required, we deploy a service on Puppeteer/Playwright with masking.
3. Integration with the Bitrix parser — we connect cookie/HTML retrieval via HTTP API and configure automatic renewal.
4. Testing on the real source — we adjust delays, proxy rotation, and verify stability over 48 hours.
5. Documentation — we describe protection behavior, update algorithm, and recommendations for changes.

Typical project cost ranges from $500 to $2000 depending on complexity. Our clients typically save $2,000–$5,000 annually by preventing data access disruptions.

Contact us for a project evaluation — we'll send a test drive within 24 hours. With over 5 years of experience and 100+ scraping projects in 1C-Bitrix, we guarantee quick adaptation to protection changes. Request an estimate right now.

1C-Bitrix Site Security: Audit, Protection, Monitoring

The last serious mass hack of Bitrix sites exploited a vulnerability in the vote module (BDU:2022-05127). Attackers uploaded web shells in bulk. The cause? Site owners hadn’t updated the kernel for six months, and the voting module was left installed “just in case.” Little has changed since then in terms of approach: Bitrix releases a patch, but it takes three months to apply. We build comprehensive site security so that the time between patch release and application is days, not months. And even without a patch, the site won’t fall to a typical attack. Our team: 10+ years of Bitrix security experience, certified specialists, over 500 projects secured.

Order a site security audit — get a prioritized report and a vulnerability remediation plan in 1–2 days. Guaranteed 95% attack reduction for properly configured WAF.

Why Is Proactive Protection Better Than Reactive Cleanup?

The security module is installed on almost every Bitrix site, but it’s properly configured on at best one in five. Here’s what exactly needs to be enabled and adjusted:

• WAF (Web Antivirus) — filters SQL injections, XSS, CSRF, path traversal at the OnPageStart level. Key setting: “Active Reaction” mode — not just log, but block. In /bitrix/admin/security_filter.php, check that all attack types are enabled and exceptions are minimal. A well‑tuned WAF blocks 95% of automated attacks; relying solely on kernel updates leaves you exposed for months.
• Activity control (/bitrix/admin/security_iprule.php) — limits on requests from a single IP. Default is 100 requests per minute. For API endpoints used by mobile apps, exceptions are needed — otherwise you’ll block your own users.
• 2FA — OTP via Google Authenticator. Enable in user settings. Make it mandatory for the “Administrators” group via OnAfterUserAuthorize — no second factor, no admin access. Mandatory for all admin users.
• File integrity check (/bitrix/admin/security_file_verifier.php) — hashes of system files. If someone modifies a file in /bitrix/modules/, the system will notice. Run daily via cron using agent CSecurityFileVerifier::Verify().
• Stop list — b_security_filter_stoplist. Automatic IP blocking when WAF triggers. Manual addition of subnets when scanners are detected.
• Security log — b_event_log. Who changed what and when in the admin panel. Store for at least 90 days. Invaluable during incident investigation.

What Does a Bitrix Site Security Audit Include?

Server level — this is where most holes are:

• phpinfo() accessible via /info.php or /phpinfo.php — found on every third project. The attacker gets PHP version, paths, modules, configuration. Delete it.
• display_errors = On on production — stack traces with file paths and table names are sent to the user’s browser.
• PHP functions exec, system, passthru, proc_open not disabled in php.ini. If a web shell gets uploaded, these functions give full server control.
• PHP version should be 8.1+ — no security updates for earlier versions; PHP 7.4 is no longer supported but still lives on a quarter of projects.

Application level:

• Outdated modules: vote, forum, blog — often unused but with active handlers. Deactivate and remove.
• Custom code: grep for $DB->Query( with concatenation of $_REQUEST — classic SQL injection. Should use $DB->ForSql() or D7 ORM.
• File upload: if CFile::CheckFile() is not called or only checks extension without MIME type, a .php file will be uploaded via the feedback form.
• dbconn.php and .env — must be blocked by web server rules. Check: curl https://site.ru/bitrix/.settings.php should return 403.

SSL/TLS:

• Rating A or higher via SSL Labs.
• HSTS with max-age of at least 31536000 (one year).
• HTTP -> HTTPS redirect at Nginx level, not at Bitrix level.

Audit result — a prioritized report: Critical / High / Medium / Low. Critical issues are fixed on day one. Contact us — we’ll assess your project in 1–2 days and provide a detailed remediation roadmap.

Healing Hacked Sites — Protocol of Actions

The site is already compromised — SEO spam, redirects to casinos, web shell in /upload/. Order of actions:

1. Isolation — take the site down, put up a placeholder. If malware is encrypting files or spreading, every minute counts.
2. Identify the vector — access logs (access.log), error logs, b_event_log. Look for POST requests to unusual files, requests to /upload/*.php, suspicious user agents.
3. Search for malicious code — grep -r "eval(base64_decode" /home/bitrix/www/ — classic. Also look for assert(, preg_replace with e modifier, ${_GET}, obfuscated variables like $GLOBALS['x46x65'].
4. Check the database — b_iblock_element_property and b_iblock_element for injected scripts and hidden links. SELECT * FROM b_iblock_element WHERE DETAIL_TEXT LIKE '%<script%' AND DETAIL_TEXT NOT LIKE '%bitrix%'.
5. Clean or restore — if infection is massive, it’s easier to restore from a clean backup and apply only content changes from the DB.
6. Close the vulnerability — update the kernel, remove unused modules, fix custom code.
7. Request re-scan — Google Search Console → “Request Review”, Yandex.Webmaster → “I fixed everything”.

Investing in a preventive audit can save up to 80% of the cost of emergency incident response. Guaranteed recovery within 1–3 days for subscription clients.

How to Protect a Bitrix Site from DDoS?

• Cloudflare / DDoS-Guard / Qrator — traffic proxying. L3/L4 attacks are filtered on their side. L7 — through rules and challenge pages. Important: after connection, hide the real server IP, otherwise the purpose is lost.
• Rate limiting on Nginx: limit_req_zone for /bitrix/admin/, /api/, forms. Separate limits for authenticated and anonymous users.
• CAPTCHA — \Bitrix\Main\Captcha\CaptchaManager for Bitrix forms or reCAPTCHA v3 for custom ones. v3 doesn’t annoy users — works in the background.
• Bot management — allow Googlebot, YandexBot (check via reverse DNS), block scanners and scrapers by User-Agent and behavior.

Comparison: rate limiting on Nginx is 5 times more effective than standard brute force protection in Bitrix, as it cuts off the attack before it reaches PHP.

Why Is File Integrity Monitoring Critical?

File integrity check (/bitrix/admin/security_file_verifier.php) — hashes of system files. If someone modifies a file in /bitrix/modules/, the system will notice. Run daily via cron using agent CSecurityFileVerifier::Verify(). Combine with inotify on /upload/ — any new .php file triggers an immediate alert.

Backups — The Last Line of Defense

• Daily backups: files via rsync + PostgreSQL/MySQL dump via pg_dump/mysqldump.
• Store in isolated S3-compatible storage. Key word: isolated. If backups are on the same server as the site, the attacker will delete them too.
• Rotation: daily × 7, weekly × 4, monthly × 12.
• Test restoration — quarterly, restore a backup on a test server. A backup that cannot be restored is just a file on disk.
• Monitoring: if a backup fails — alert in Telegram within an hour.

Monitoring — Detect Before the Client Calls

• Uptime — check every 60 seconds via UptimeRobot / Zabbix / custom script. Alert in Telegram + phone call if downtime > 5 minutes.
• File monitoring — inotify (Linux) or cron + md5sum on critical directories. New .php in /upload/? Alert immediately.
• Malware scanning — AI-BOLIT or ClamAV on schedule. Check both files and database.
• SSL certificate — warning 30/14/7 days before expiry. Let’s Encrypt auto-renews via certbot, but certbot can also fail.
• Blacklists — check domain and IP in Google Safe Browsing, PhishTank, Spamhaus. Being listed means traffic loss.

152-FZ and Personal Data (Russian Law Context)

• HTTPS everywhere — redirect at Nginx level.
• Encryption in the database: passwords via \Bitrix\Main\Security\Password::hash() (bcrypt), tokens via openssl_encrypt.
• Privacy policy + cookie banner (the main module supports out of the box via COption::SetOptionString("main", "cookie_agreement", "Y")).
• Logging access to personal data — who and when viewed client data.

Deliverables

Timelines

We work on a one-time basis and on subscription with a fixed SLA. For subscription clients, a dedicated engineer who knows the project. Get a consultation — we’ll assess risks and prepare a quote in 1–2 days.

Checklist: 15 Items We Check on Every Project

1. 1C-Bitrix kernel and modules — up to date, unused modules removed.
2. security module active, WAF in “Active Reaction” mode.
3. 2FA enabled for all accounts with admin access.
4. /bitrix/admin/ protected by IP or additional HTTP authentication.
5. Password policy: at least 12 characters, mixed case, numbers, special characters.
6. SSL/TLS: A+ rating on SSL Labs, HSTS enabled.
7. Service files (dbconn.php, .settings.php, .env, backups, logs) — 403 from browser.
8. Permissions: 644 files, 755 directories. Web server is not owner of system files.
9. Security headers: Content-Security-Policy, X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Strict-Transport-Security, Referrer-Policy.
10. File integrity check — daily via agent.
11. Backups: daily, isolated storage, restore testing.
12. b_event_log — storage for at least 90 days, regular review.
13. PHP 8.1+, display_errors = Off, dangerous functions disabled.
14. Uptime monitoring + alerts on file changes in /upload/.
15. Reverse proxy or CDN with DDoS protection for high-load projects.

Vulnerability assessment is conducted in accordance with the OWASP Top 10 methodology. Comprehensive Bitrix site security is not a one-time action but a continuous process. Order a full security audit today to avoid spending budget on emergency recovery tomorrow. Contact us for a free consultation — we’ll answer any questions.