Schedule Parser Setup (cron) for 1C-Bitrix: Automation & Reliability

We often see: a Bitrix site with manual price and stock updates — errors, delays, lost sales. A client once came to us with a catalog of 15,000 items: prices were updated once a week manually via Excel. In a month, losses from expired promotions amounted to 8% of turnover. One-time parsing doesn't solve it — you need automatic scheduled execution. Setting up a parser on a schedule (cron) for 1C-Bitrix is a key task for automating data updates. We select the mechanism (Bitrix agents vs system cron), implement parallel run locking, set up logging and alerts. The result: data updates automatically, without human intervention.

How does system cron solve the data update problem?

Agents (b_agent) are Bitrix's built-in mechanism. They only fire when a visitor hits the site and CAgent::CheckAgents() runs. With low traffic, agents run late. The exact launch mode via cron solves the issue but requires system cron configuration. Our experience shows: for parsers that update every 30 minutes, system cron is more reliable.

System cron directly calls PHP-CLI. No dependency on the web server:

*/30 * * * * /usr/bin/php -f /var/www/site/local/scripts/parser_run.php >> /var/log/parser.log 2>&1

For heavy parsers (10,000+ items), system cron is preferable — no PHP-FPM execution time limits. We guarantee stable execution at any time, even when the site is idle.

Why we recommend system cron for heavy parsers?

On a project with 50,000 products updated every 30 minutes, agents struggled: with 200 visitors per day, execution slipped by 10-15 minutes. Switching to system cron eliminated the delays — updates became precise to the second. Support costs dropped by 30% due to automation.

How to protect the parser from parallel runs?

The parser runs for 25 minutes, cron triggers the next after 30 — two instances won't overlap. But what if the script hangs for 35 minutes? You need protection. We use file-based locking or Redis:

$lockFile = '/tmp/parser_' . md5($parserName) . '.lock';
if (file_exists($lockFile) && (time() - filemtime($lockFile)) < 3600) {
    exit('Already running');
}
touch($lockFile);
// ... parser work ...
unlink($lockFile);

For distributed systems (multiple servers) — Redis locking: SET lock:parser NX EX 3600. The cost of a parallel run error: duplicate data and server load. We account for this in every project.

Agents or cron: which to choose? Comparison

Step-by-step cron setup for a parser

1. Determine the launch command — full path to PHP and script. Check version: which php.
2. Set up logging — redirect stdout and stderr to a dated log file.
3. Add the job to crontab — crontab -e, specify schedule and command.
4. Implement locking — lock file or Redis.
5. Test — run the command manually, check logs.
6. Set up alerts — send notifications on errors.

We apply this algorithm in every project — it reduces debugging time by 40%.

Setting up Bitrix agents for parsing

If agents are still used — register via CAgent::AddAgent():

CAgent::AddAgent(
    'MyParserAgent::Run();',  // agent function
    'mymodule',               // module
    'N',                      // not exact
    3600,                     // interval in seconds
    '',                       // first launch date
    'Y',                      // active
    date('d.m.Y H:i:s', time() + 3600)  // next launch
);

The agent must be registered in the module's include.php or in init.php. For production, we recommend combining with system cron via 1C-Bitrix documentation. This gives maximum flexibility and precision.

What does structured logging provide?

A parser without logs is a black box. Minimum scheme:

file_put_contents($logFile, date('[Y-m-d H:i:s] ') . $message . PHP_EOL, FILE_APPEND);

For production — structured logs with levels (info/warning/error) and rotation via logrotate. Alert on error: if the last run's log contains ERROR lines — send an email via \Bitrix\Main\Mail\Event. This cuts response time to failures and halves support costs. According to Bitrix recommendations, such monitoring is mandatory for critical updates.

What's included in the work

• Audit of the current parser or writing a new one
• Selection and setup of the launch mechanism (cron/agents)
• Implementation of parallel run locking
• Configuration of logging and alerts
• Schedule testing and error scenario testing
• Documentation on access and parser operation
• One-week stability guarantee after launch

Timeline

Total: 6–12 hours. Cost is calculated individually — we'll evaluate your project within one business day.

Get a consultation on parser setup — write to us, we'll find the best solution for your budget. Leave a request — we'll tell you how to automate data updates on your project. Contact us to start automation today.

Parser Development for 1C-Bitrix: Where to Start?

XMLReader, not SimpleXML — the choice of tool determines the project's fate. SimpleXML loads the entire XML into memory, and with an 800 MB supplier file, PHP will crash with a fatal error on a 512 MB limit. XMLReader processes streamingly, node by node, consuming 20–30 MB — 30 times more efficient. This detail starts any parser development for Bitrix. With over 10 years of Bitrix development and 50+ parser projects delivered, we know the pitfalls. Contact us to start your parser development today.

What Problems Does Parsing Solve?

• Primary catalog filling — 15,000 cards with descriptions, characteristics, photos. Manually, that's three months of content manager work; a parser takes a week with debugging.
• Competitor price monitoring — collecting data from Ozon, Wildberries, competitor sites. A competitor drops the price on a hot item — you find out in two hours, not two weeks.
• Supplier aggregation — five price lists in different formats (CSV with CP1251, XML in CommerceML, Excel with merged cells) become a single catalog with a unified property system.
• Card enrichment — pulling characteristics, instructions, 3D models from manufacturer sites. Without this, a product card is an SEO empty shell.
• Assortment update — products missing from the supplier feed are deactivated via CIBlockElement::Update($ID, ['ACTIVE' => 'N']). New ones are created. The catalog stays synchronized.

What Tools Do We Use in Parser Development?

Static websites — PHP (Goutte, Symfony DomCrawler) or Python (Scrapy, lxml). Speed: 50–100 pages/sec. Sufficient for catalogs without JS rendering.

SPA and dynamic websites — Puppeteer or Playwright. Infinite scroll, AJAX filters, lazy-load images — headless browser handles it all. Speed drops to 1–10 pages/sec, but there is no alternative: data exists only after JavaScript execution.

Supplier files:

• Excel (XLS, XLSX) — PhpSpreadsheet. Beware of merged cells and formulas — they break automatic mapping.
• CSV — fgetcsv() with correct encoding. Suppliers love CP1251, BOM in UTF-8, and semicolons instead of commas. All need detection and handling.
• XML/YML — XMLReader for large files, SimpleXML for feeds up to 50 MB.
• CommerceML — standard exchange format with 1C. We parse import.xml and offers.xml, map to information block structure.

API — Supplier REST endpoints, marketplace APIs (Ozon Seller API, Wildberries API). We work within rate limits, handle pagination.

How Is the Auto-Population Pipeline Structured?

Four stages. Each can break in its own way.

1. Collection. Parser crawls sources via cron schedule. Raw data goes to an intermediate table — not directly into b_iblock_element. Log everything: pages visited, elements parsed, where we got 403 or timeout. Without logs, debugging a parser is like fortune-telling.

2. Normalization. Main work here:

   • Clean HTML tags, extra spaces, Unicode garbage
   • Units: "mm" → "mm", "millimeters" → "mm", "миллиметр" → "mm"
   • Map supplier categories to Bitrix information block sections. One supplier has "Notebooks", another "Notebooks and tablets", third "Laptops" — all into one section
   • Deduplication by SKU, EAN/GTIN. One product from three suppliers should not appear three times

3. Load into Bitrix. Via CIBlockElement::Add() for new elements, CIBlockElement::Update() for existing. Images: download, resize via CFile::ResizeImageGet(), convert to WebP. Properties via CIBlockElement::SetPropertyValuesEx(). SEO meta via \Bitrix\Iblock\InheritedProperty\ElementValues. SEF URLs generated from name transliteration.

4. Update. Key point — not overwrite manual edits by content manager. Update only price, stock, activity. Description and photos manually edited are flagged with UF_MANUAL_EDIT property and skipped during import. Products missing from feed are deactivated, not deleted.

Why Is Competitor Price Monitoring Necessary?

A separate subsystem with its own specifics:

Result — dashboard: your product vs competitors, price history, trends. The manager sees where to raise price without losing position, and where to react.

CSV/XML Import Module: Customization for Your Format

For supplier files — custom module with admin panel:

• Configurable mapping: "column B in file → BRAND property of information block"
• Auto-detect encoding (CP1251, UTF-8, UTF-16) via mb_detect_encoding() with validation
• Download images from URL with queue — to avoid channel saturation
• Incremental update by row hash: row changed — update, no — skip
• Cron schedule, report: created 145, updated 892, errors 3 (with details)

Large files: CSV processed in batches of 1000 rows via fgetcsv() (10 times faster than row-by-row), XML streamed via XMLReader, background execution via Bitrix agent queue — no PHP timeouts.

Legal Aspects to Consider

• robots.txt — respect it. Crawl-delay — comply.
• Request frequency — 1–2 per second, no more. Don't DDoS someone else's site.
• Manufacturer content — use it. Unique author texts — don't copy.
• Personal data — don't collect.

What Is Included in a Turnkey Parser Development?

How We Work and Deadlines

1. Prototype — parser for 1–2 sources in 2–3 days. Assess data quality, pitfalls (Cloudflare protection, captcha, dynamic loading).
2. Development — full pipeline: parser → normalization → import into Bitrix → admin panel for management.
3. Testing — run on full catalog volume, check edge cases (empty fields, malformed HTML, broken images).
4. Launch — configure cron, error monitoring via Telegram bot.
5. Support — competitor changed layout? Update CSS selectors in parser.

Get in touch for a free consultation — we will analyze your data sources and propose the optimal parser architecture. Request a project assessment today and get a fixed deadline. We guarantee stable parser operation and full support throughout the usage period.