robots.txt Setup for Bitrix CMS
robots.txt controls which site sections search robots can crawl and which cannot. For typical Bitrix site, standard robots.txt out-of-the-box is usually absent or contains only basic bans. Proper setup saves crawl budget and excludes technical pages from index.
What to Block for Bitrix Site
Technical paths that shouldn't be indexed:
Disallow: /bitrix/
Disallow: /upload/iblock/
Disallow: /auth/
Disallow: /personal/
Disallow: /cart/
Disallow: /order/
Disallow: /search/?
Disallow: /catalog/*?*
Disallow: /?*
/bitrix/ — admin panel, core, cache.
/upload/iblock/ — original images. Enough for search engine to index images via product pages, not direct URLs.
/catalog/*?* — catalog pages with parameters (sorting, pagination). Prevents duplicate indexing.
Management via Administrative Panel
Bitrix has no built-in visual robots.txt editor. File is edited directly via Content → Site Structure → [root] → robots.txt, or via FTP/SSH.
The seo module adds sitemap link automatically when generating sitemap. Check that end of file has line:
Sitemap: https://your-site.ru/sitemap_index.xml
Directives for Different Search Engines
Yandex and Google support User-agent, Disallow, Allow, Crawl-delay and Sitemap directives. Different rules can be set for different robots:
User-agent: Yandex
Disallow: /catalog/*?*
Crawl-delay: 1
User-agent: Googlebot
Disallow: /catalog/*?*
User-agent: *
Disallow: /bitrix/
Disallow: /upload/tmp/
Disallow: /auth/
Crawl-delay: 1 for Yandex — 1 second delay between requests. Use for weak server sites to prevent crawler overloading hosting.
Verification
After changes, check file via webmaster tools:
- Yandex.Webmaster → Tools → Analyze robots.txt
- Google Search Console → Tools → Test file robots.txt
Tools show if specific URL is blocked for specific robot.
Implementation Timeline
Drafting and setting up robots.txt for typical Bitrix site — 30–60 minutes.