News Feed Parsing for Automated Population in 1C-Bitrix

Imagine a news site on Bitrix with 10,000 daily visitors, but content is updated once a week. Editors spend 20 hours a week copying articles from 15 sources. As a result, search positions drop by 30% in a quarter. We implement RSS feed parsing that automatically collects news, filters duplicates, and publishes them in an information block. On one project, this reduced time spent from 15 to 1 hour per week and increased the citation index by 45%. Moreover, this approach completely eliminates human copying errors.

According to the RSS Board specification, RSS 2.0 supports all necessary fields for automated publishing.

How to choose a data source for parsing?

News feeds are available in several formats. Comparison by key parameters:

RSS/Atom is the optimal start. They are standardized, easy to parse, and require no special permissions.

How to organize the import process?

The news parser for Bitrix consists of three layers.

Fetcher. Retrieves RSS feeds from a URL list. Uses file_get_contents with context or cURL with timeouts. Each feed is parsed via SimpleXMLElement or the SimplePie library.

$xml = simplexml_load_string($rssContent);
foreach ($xml->channel->item as $item) {
    $title = (string)$item->title;
    $link  = (string)$item->link;
    $date  = strtotime((string)$item->pubDate);
    $desc  = (string)$item->description;
}

Processor. Cleans HTML tags, downloads images, normalizes dates, and determines category by keywords.

Importer. Creates elements in the information block via CIBlockElement::Add(). Checks duplicates by XML_ID (article URL or GUID).

Content storage and processing

Recommended RSS→infoblock mapping:

XML_ID is mandatory. Use an md5 hash of the article URL — this guarantees uniqueness.

Raw HTML from RSS is unsuitable for publication. Typical issues:

• External images — download to /upload/ during import.
• Third-party scripts and iframes — use strip_tags() with a whitelist or HTMLPurifier.
• Relative links — convert to absolute by substituting the source domain.
• Encoding — detect via mb_detect_encoding() and convert to UTF-8.

Why cron instead of agents?

Cron is 3 times more reliable than standard Bitrix agents for long-running tasks — agents abort if they exceed the execution time, while cron runs to completion. A cron job calls a PHP script:

$_SERVER['DOCUMENT_ROOT'] = '/home/bitrix/www';
require $_SERVER['DOCUMENT_ROOT'] . '/bitrix/modules/main/include/prolog_before.php';
CModule::IncludeModule('iblock');

Frequency: breaking news — every 15–30 minutes, industry news — 1–2 hours, analytics — 1–2 times a day.

How to set up the parser in 5 steps?

1. Select RSS/API sources.
2. Write a fetcher using SimplePie or cURL.
3. Configure the processor: HTML cleanup, image download, category mapping.
4. Implement the importer with XML_ID verification.
5. Set up cron and an error logging system.

Quality control and categorization

In addition to XML_ID, use:

• Date filter — do not import news older than N days.
• Minimum description length — discard entries shorter than 100 characters.
• Stop words — filter out keywords irrelevant to the topic.
• Source limit — no more than N news per day from a single feed.

The simplest categorization is "source → infoblock section" mapping. A more flexible approach is keyword-based classification:

$rules = [
    'Technology' => ['AI', 'blockchain', 'startup', 'app'],
    'Finance'    => ['stocks', 'exchange rate', 'investments', 'IPO'],
];

For 10+ categories, integrate external classifiers (OpenAI, Yandex GPT).

Legal side

Publishing others' news "as is" violates copyright. Options:

• Publish title + 2–3 sentences with a link (fair quoting).
• Automated rewriting via LLM — legally questionable.
• Use feeds with open licenses (Creative Commons, government sources).

What's included in the work

• Audit of the current catalog — infoblock structure, duplicate check.
• Parser development — fetcher, processor, importer.
• Integration with cron — schedule, logging.
• Deduplication and filters.
• Documentation and editor training.
• 30-day post-release support.

Experience and expertise

We have been developing on Bitrix for over 7 years and have completed 200+ automation projects. Our engineers are 1C-Bitrix certified. We use the full stack: CommerceML, REST API, Bizproc. Contact us to discuss the parser architecture for your site — we'll prepare a proposal within 1-2 days. Get a consultation on automating your news section.

Parser Development for 1C-Bitrix: Where to Start?

XMLReader, not SimpleXML — the choice of tool determines the project's fate. SimpleXML loads the entire XML into memory, and with an 800 MB supplier file, PHP will crash with a fatal error on a 512 MB limit. XMLReader processes streamingly, node by node, consuming 20–30 MB — 30 times more efficient. This detail starts any parser development for Bitrix. With over 10 years of Bitrix development and 50+ parser projects delivered, we know the pitfalls. Contact us to start your parser development today.

What Problems Does Parsing Solve?

• Primary catalog filling — 15,000 cards with descriptions, characteristics, photos. Manually, that's three months of content manager work; a parser takes a week with debugging.
• Competitor price monitoring — collecting data from Ozon, Wildberries, competitor sites. A competitor drops the price on a hot item — you find out in two hours, not two weeks.
• Supplier aggregation — five price lists in different formats (CSV with CP1251, XML in CommerceML, Excel with merged cells) become a single catalog with a unified property system.
• Card enrichment — pulling characteristics, instructions, 3D models from manufacturer sites. Without this, a product card is an SEO empty shell.
• Assortment update — products missing from the supplier feed are deactivated via CIBlockElement::Update($ID, ['ACTIVE' => 'N']). New ones are created. The catalog stays synchronized.

What Tools Do We Use in Parser Development?

Static websites — PHP (Goutte, Symfony DomCrawler) or Python (Scrapy, lxml). Speed: 50–100 pages/sec. Sufficient for catalogs without JS rendering.

SPA and dynamic websites — Puppeteer or Playwright. Infinite scroll, AJAX filters, lazy-load images — headless browser handles it all. Speed drops to 1–10 pages/sec, but there is no alternative: data exists only after JavaScript execution.

Supplier files:

• Excel (XLS, XLSX) — PhpSpreadsheet. Beware of merged cells and formulas — they break automatic mapping.
• CSV — fgetcsv() with correct encoding. Suppliers love CP1251, BOM in UTF-8, and semicolons instead of commas. All need detection and handling.
• XML/YML — XMLReader for large files, SimpleXML for feeds up to 50 MB.
• CommerceML — standard exchange format with 1C. We parse import.xml and offers.xml, map to information block structure.

API — Supplier REST endpoints, marketplace APIs (Ozon Seller API, Wildberries API). We work within rate limits, handle pagination.

How Is the Auto-Population Pipeline Structured?

Four stages. Each can break in its own way.

1. Collection. Parser crawls sources via cron schedule. Raw data goes to an intermediate table — not directly into b_iblock_element. Log everything: pages visited, elements parsed, where we got 403 or timeout. Without logs, debugging a parser is like fortune-telling.

2. Normalization. Main work here:

   • Clean HTML tags, extra spaces, Unicode garbage
   • Units: "mm" → "mm", "millimeters" → "mm", "миллиметр" → "mm"
   • Map supplier categories to Bitrix information block sections. One supplier has "Notebooks", another "Notebooks and tablets", third "Laptops" — all into one section
   • Deduplication by SKU, EAN/GTIN. One product from three suppliers should not appear three times

3. Load into Bitrix. Via CIBlockElement::Add() for new elements, CIBlockElement::Update() for existing. Images: download, resize via CFile::ResizeImageGet(), convert to WebP. Properties via CIBlockElement::SetPropertyValuesEx(). SEO meta via \Bitrix\Iblock\InheritedProperty\ElementValues. SEF URLs generated from name transliteration.

4. Update. Key point — not overwrite manual edits by content manager. Update only price, stock, activity. Description and photos manually edited are flagged with UF_MANUAL_EDIT property and skipped during import. Products missing from feed are deactivated, not deleted.

Why Is Competitor Price Monitoring Necessary?

A separate subsystem with its own specifics:

Result — dashboard: your product vs competitors, price history, trends. The manager sees where to raise price without losing position, and where to react.

CSV/XML Import Module: Customization for Your Format

For supplier files — custom module with admin panel:

• Configurable mapping: "column B in file → BRAND property of information block"
• Auto-detect encoding (CP1251, UTF-8, UTF-16) via mb_detect_encoding() with validation
• Download images from URL with queue — to avoid channel saturation
• Incremental update by row hash: row changed — update, no — skip
• Cron schedule, report: created 145, updated 892, errors 3 (with details)

Large files: CSV processed in batches of 1000 rows via fgetcsv() (10 times faster than row-by-row), XML streamed via XMLReader, background execution via Bitrix agent queue — no PHP timeouts.

Legal Aspects to Consider

• robots.txt — respect it. Crawl-delay — comply.
• Request frequency — 1–2 per second, no more. Don't DDoS someone else's site.
• Manufacturer content — use it. Unique author texts — don't copy.
• Personal data — don't collect.

What Is Included in a Turnkey Parser Development?

How We Work and Deadlines

1. Prototype — parser for 1–2 sources in 2–3 days. Assess data quality, pitfalls (Cloudflare protection, captcha, dynamic loading).
2. Development — full pipeline: parser → normalization → import into Bitrix → admin panel for management.
3. Testing — run on full catalog volume, check edge cases (empty fields, malformed HTML, broken images).
4. Launch — configure cron, error monitoring via Telegram bot.
5. Support — competitor changed layout? Update CSS selectors in parser.

Get in touch for a free consultation — we will analyze your data sources and propose the optimal parser architecture. Request a project assessment today and get a fixed deadline. We guarantee stable parser operation and full support throughout the usage period.