ETL Process Development for 1C-Bitrix

We often encounter a situation where the standard import via the 1C-Bitrix admin panel no longer handles regular synchronization. ERP, 1C, warehouse systems, marketplaces — each source pulls data in its own format. Without a full-fledged ETL, within a month you find that 3% of products have incorrect stock balances, and no one knows about it. We develop turnkey ETL processes: with transformation, error handling, and monitoring, so you can sleep soundly. Contact us for a preliminary assessment of your project.

How Does an ETL Process Solve the Data Inconsistency Problem?

ETL (Extract, Transform, Load) based on 1C-Bitrix is built around three layers. Extract — obtaining data from sources: files (CSV, XML, JSON, YML) via FTP/SFTP/HTTP, REST APIs of external systems (1C, SAP, Salesforce), direct database connections (MySQL, MSSQL, PostgreSQL) via PDO, message queues (RabbitMQ, Kafka). Transform — converting data to Bitrix structure: field mapping, format normalization, validation. Load — writing to Bitrix via D7 API or direct SQL queries for high volumes. This approach ensures data is always up-to-date and aligned with business logic.

Product Load Architecture

For loading products via the standard Bitrix API, we use \Bitrix\Iblock\ElementTable and CCatalogProduct. For volumes from 10,000 products, key settings:

// Отключаем ненужные обработчики на время импорта
define('STOP_STATISTICS', true);
define('NO_AGENT_STATISTIC', 'Y');
define('DisableEventsCheck', true);

// Отключаем поисковый индекс — перестроим в конце
\CSearch::DisableIndex();

// Загрузка элемента инфоблока
$el = new \CIBlockElement();
$result = $el->Add([
    'IBLOCK_ID' => CATALOG_IBLOCK_ID,
    'NAME' => $item['name'],
    'CODE' => $item['code'],
    'ACTIVE' => 'Y',
    'PROPERTY_VALUES' => [
        'VENDOR_CODE' => $item['vendor_code'],
        'WEIGHT' => $item['weight'],
    ],
]);

For volumes > 50,000 items, direct calls to CIBlockElement::Add degrade due to event cascade. We switch to direct INSERTs into tables b_iblock_element, b_iblock_element_property, b_catalog_product with full index rebuild afterward. This yields 3-5x performance improvement.

Why Incremental Synchronization Is Critical

Full overwrite every N hours is expensive and server-intensive. Incremental ETL processes only changed records:

// Фиксируем момент начала синхронизации
$syncStartTime = new \Bitrix\Main\Type\DateTime();

// Запрашиваем из источника только изменённые с прошлой синхронизации
$changedItems = $source->getChangedSince($this->getLastSyncTime());

// После успешной синхронизации обновляем метку
$this->setLastSyncTime($syncStartTime);

Table for storing sync state:

Data Transformation: Typical Challenges

Transformation is the most complex part. Category mapping: source may have a flat list with parent_id, Bitrix uses a tree of sections. We build a tree, map by code or external_id. Price normalization: source gives price with VAT, without VAT, in different currencies. Recalculate using exchange rates. HTML cleanup: descriptions from 1C often contain unreadable formatting — run through DOMDocument, remove unwanted tags. Deduplication: if source does not guarantee unique SKUs, implement merge logic for duplicates.

Row-Level Error Handling

ETL should not stop due to one invalid record:

foreach ($items as $item) {
    try {
        $transformed = $this->transform($item);
        $this->load($transformed);
        $this->stats->incrementSuccess();
    } catch (\Bitrix\Main\ArgumentException $e) {
        // Ошибка валидации — логируем и продолжаем
        $this->logger->warning('Validation failed', [
            'external_id' => $item['id'],
            'error' => $e->getMessage(),
        ]);
        $this->stats->incrementError($item['id'], $e->getMessage());
    } catch (\Exception $e) {
        // Неожиданная ошибка — логируем, но продолжаем
        $this->logger->error('Load failed', ['item' => $item['id'], 'error' => $e->getMessage()]);
        $this->stats->incrementError($item['id'], $e->getMessage());
    }
}

After sync, a report shows how many created, updated, skipped with errors. If errors > 5%, alert is triggered.

Memory Management for Large Volumes

PHP easily exhausts memory when processing 100,000 records. Rules:

• Read data in chunks, do not load entire file into array
• Use generators for CSV/XML iteration
• Explicitly call unset() after processing a chunk
• Reset Bitrix ORM cache: \Bitrix\Main\ORM\Data\DataManager::cleanCache()
• Monitor memory_get_usage() — log if approaching limit

// Генератор для чтения большого CSV
function readCsvChunks(string $file, int $chunkSize = 500): \Generator {
    $handle = fopen($file, 'r');
    $header = fgetcsv($handle);
    $chunk = [];
    while (($row = fgetcsv($handle)) !== false) {
        $chunk[] = array_combine($header, $row);
        if (count($chunk) >= $chunkSize) {
            yield $chunk;
            $chunk = [];
        }
    }
    if ($chunk) yield $chunk;
    fclose($handle);
}

Agents vs Cron vs Queue

• Bitrix agents (b_agent) — for small tasks (up to 1000 records per run). Unstable under low traffic.
• Cron — more reliable for regular syncs: */30 * * * * php -f /var/www/bitrix/etl/sync_products.php >> /var/log/etl.log 2>&1
• Queue (RabbitMQ/Redis) — for event-driven ETL when source publishes change events. Allows handling high-frequency changes without losses.

ETL Monitoring

What's Included

• Source analysis: data structure, formats, schedule
• Extract connector development for each source
• Transformation: mapping, normalization, validation
• Load into Bitrix: API or direct SQL, performance optimization
• Error handling and monitoring: logging, alerts, reports
• Documentation: ETL process description, admin guide
• Training: knowledge transfer to your team
• Post-launch support: stable operation guarantee

Development Stages

Total: 6–12 weeks depending on complexity. Contact us to discuss your task. Order ETL process development and get an engineer consultation.

More about synchronization implementation via CommerceML.

1C-Bitrix Module Development and Setup

The main trap of Bitrix is init.php. You add an OnBeforeIBlockElementUpdate handler there, then another one — a year later the file is 2000 lines, and on every hit all that code executes. We move business logic into full-fledged modules with D7 ORM, custom tables, and administrative interface. The module can be disabled, transferred to another project, covered with tests — none of that is possible with init.php. Our team has 10+ years of Bitrix experience, certified specialists, and a 6-month code guarantee. Request a consultation — we'll explain how to migrate legacy code to a modular architecture.

Why is init.php the worst place for business logic?

Init.php does not support class autoloading, lacks an isolated namespace, cannot be unit tested, and cannot be disabled without editing the file itself. Every handler written there runs on every request, even if not needed. In a module, you register handlers through EventManager, and they only execute when the event occurs. Performance difference: up to 3x with 10+ handlers.

Standard Modules: Typical Problems and Solutions

Information blocks. IBlock architecture is the first thing we review on any project. A classic mistake: one catalog infoblock with 80 properties, 30 of which are multiple. The b_iblock_element_property table swells to millions of rows, and CIBlockElement::GetList with filtering on three properties does a full scan. We move reference data to Highload-blocks, eliminate multiple properties where possible, and design the structure for 5x growth.

e-Store (sale). Cart business rules are a separate story. We set discount priorities to prevent two campaigns from giving 60% instead of 30%, connect payment handlers, and write custom validation via OnSaleOrderBeforeSaved.

Search. The built-in search module with morphology works up to 10–15 thousand elements. Beyond that — Elasticsearch. We configure it via the Bitrix search module API, indexing through CSearchFullText or custom indexers.

Highload-blocks for dictionaries, logs, user data — instead of bloated IBlocks. Direct queries via Bitrix\Highloadblock\HighloadBlockTable, custom tables instead of the EAV structure of standard infoblocks. A million records — no degradation.

Mail events. Configuration is not just templates in b_event_message. The key is SPF, DKIM, DMARC on the DNS, otherwise transactional emails go to spam. We check deliverability and set up bounce handling.

How to Design Infoblocks for Performance?

We use Highload-blocks for reference data (colors, sizes, manufacturers) that are not involved in complex queries. For SKUs — a separate infoblock with linking via IBLOCK_ELEMENT_PROPERTY. Enable INDEX_PROPERTY for frequently filtered properties. Tagged caching: when an element changes, only the related cache is cleared. Highload-blocks process up to 10x faster than infoblocks with multiple properties on volumes of 100,000 records.

Custom Module Development

Each module follows the structure /local/modules/vendor.modulename/:

• install/index.php — setup class, create tables via $DB->RunSQLBatch()
• lib/ — D7 ORM classes, extending Bitrix\Main\ORM\Data\DataManager
• admin/ — administrative pages using CAdminList, CAdminForm
• include.php — autoloading, event handler registration via EventManager::getInstance()->registerEventHandler()
• REST API endpoints via \Bitrix\Rest\RestManager

The module registers in the system, appears in the "Installed Solutions" list, and has its own settings at /bitrix/admin/settings.php?mid=vendor.modulename. It can be enabled, disabled, and updated through UpdateSystem or custom migration mechanics.

Examples of implemented tasks:

• Campaign management — visual condition builder via CAdminCalendar, timers via agents (CAgent::AddAgent), analytics linked to the sale module
• Cost calculator — React widget on the frontend, REST API in the module, formulas stored in a Highload-block
• Booking system — real-time calendar, locking via $DB->StartTransaction() / $DB->Commit() on concurrent requests, integration with channel manager via webhook

Components and Composite Cache

Component customization via result_modifier.php and component_epilog.php, not by editing template.php of the standard template. This way core updates are painless.

Composite cache ("Composite Site" technology) — the server sends ready HTML, bypassing PHP routing. Dynamic areas (cart, authorization) are loaded via CBitrixComponent::setFrameMode(true) and AJAX. TTFB drops to 30–50 ms. But there are caveats: not all components are compatible, $APPLICATION->ShowPanel() breaks composite, and careful markup of <div id="bx-composite-..."> is required.

What to Check Before Installing a Marketplace Module?

Before installing a module from the marketplace, an audit is mandatory. We check: SQL queries without prepared statements (hello SQL injection), direct use of $_REQUEST without filtering, use of outdated kernel API instead of D7, conflicts with the composite cache module. A module with no updates for over a year and a few dozen installations is likely a problem on the next PHP update. A typical case: a module calls CIBlockElement::GetList with no cache reset — the site crashes with 5000 elements.

Migration to D7

When upgrading PHP or switching to a new edition — refactor outdated calls:

• CIBlockElement::GetList() → Bitrix\Iblock\Elements\ElementTable::getList()
• CSaleOrder::GetList() → Bitrix\Sale\Order::getList()
• CModule::IncludeModule() → Bitrix\Main\Loader::includeModule() Testing on staging, rollback via git on issues.

According to official 1C-Bitrix documentation, D7 ORM is the recommended tool for working with data, providing type safety and automatic query generation.

Comparison: Init.php vs Module

Module Development Scope and Cost

What is included in module development?

• Technical specification and architectural plan
• Code following PSR-4 and Bitrix code style
• Unit tests (PHPUnit) for business logic
• Integration tests for events and REST API
• Installation, configuration, and API documentation
• Repository and documentation access
• Administrator training for module usage
• 6-month warranty support

Estimated timelines and complexity:

Cost is calculated individually — contact us for a project estimate.

Module Testing

Unit tests via PHPUnit cover business logic: discount calculation, validation, document generation. Mocks for Bitrix\Main\Application::getConnection() allow tests to be DB-independent. Integration tests verify event handlers on a real database — OnAfterIBlockElementAdd, OnSaleOrderSaved, etc. REST API endpoints are tested via curl or PHPUnit HTTP client. Critical for modules working with b_sale_order, b_catalog_price — where errors cost money.

Compatibility is checked on PHP 7.4, 8.0, 8.1, 8.2 and editions: Standard, Small Business, Business. We check conflicts with popular marketplace modules — they often intercept the same events. Load testing: measurements on 10K, 100K, 1M records, profiling via Xdebug for memory leaks and N+1 queries.

Practical Examples

Campaign module for an electronics chain. The built-in sale module discounts did not cover scenarios like "2+1", a gift with purchase over a certain amount, or combined conditions. We built a visual builder: marketers create rules via drag-and-drop without development tickets. Campaign calendar, auto-deactivation via agents, analytics linked to b_sale_order — conversion, average check, usage count. Time to launch a new campaign dropped from two days to half an hour.

Calculator for builders. Parameters (area, materials, number of floors) → formula → preliminary estimate → lead to CRM via CRest::call('crm.lead.add'). Regional coefficients and seasonal markups from a Highload-block, material prices from 1C exchange. The number of target leads increased by a third: clients see a breakdown before calling a manager.

Booking for a hotel chain. Real-time availability via AJAX requests to a custom table vendor_booking_slots, seasonal tariff calculation, synchronization with Booking.com via channel manager API. Room locking on concurrent booking via SELECT ... FOR UPDATE in transactions. Timezones handled via \DateTimeZone — a guest from Vladivostok and a manager from Moscow see the same picture.

We will evaluate your project within one day. Write to us — we'll tell you what is included in turnkey development. Contact us for a consultation on your project. Order a custom module development — get a ready solution with documentation and support.