Apache Kafka for 1C-Bitrix: Configuring Data Exchange

We have encountered projects where standard RabbitMQ queues became overwhelmed: event volumes exceeded 100,000 per minute, and each external system required its own processing order. In such cases, we implemented Apache Kafka — a distributed event log that stores data streams and gives multiple consumers access. For example, an online store with a catalog of 500,000 products and 10,000 orders per day: RabbitMQ introduced delays up to 30 seconds, while Kafka handled it in under 2 ms. Here we explain how to set up data exchange between 1C-Bitrix and external services via Kafka, what to focus on, and what pitfalls await you. Bitrix Kafka integration requires careful design of topics and partitions.

What problems does Apache Kafka solve for 1C-Bitrix?

Primarily scaling. If your system generates more than 50,000 events per minute (orders, catalog updates, user actions), classical RabbitMQ starts to fail: queues overflow, consumers fall behind. Kafka distributes the load, stores events with retention up to 30 days, and allows replay when needed. Typical scenarios:

• Event sourcing: every change in the system is saved as an event, from which state can be reconstructed at any point.
• Multi-channel processing: the same event is consumed independently by CRM, warehouse, analytics.
• Integration with 1C: Kafka enables continuous exchange without direct connections.

According to our data, on projects with loads above 100,000 events per minute, switching from RabbitMQ to Kafka reduces latency by 40% and eliminates data loss. Infrastructure cost savings reach 30% due to fewer servers.

How to set up the producer and consumer in Bitrix?

There is no official PHP client from Apache. We use arnaud-lb/php-rdkafka (bindings to librdkafka):

# Install librdkafka (Ubuntu) apt-get install librdkafka-dev # Install PHP extension pecl install rdkafka # PHP wrapper cd /local && composer require arnaud-lb/php-rdkafka 

Producer: publishing events from Bitrix

class KafkaProducer { private \RdKafka\Producer $producer; public function __construct() { $conf = new \RdKafka\Conf(); $conf->set('metadata.broker.list', COption::GetOptionString('site', 'kafka_brokers', 'kafka:9092')); $conf->set('security.protocol', 'PLAINTEXT'); // For production with SSL: // $conf->set('security.protocol', 'SSL'); // $conf->set('ssl.ca.location', '/etc/kafka/certs/ca-cert'); $this->producer = new \RdKafka\Producer($conf); } public function publish(string $topic, string $key, array $payload): void { $topic = $this->producer->newTopic($topic); $topic->produce( \RD_KAFKA_PARTITION_UA, // auto partition selection 0, json_encode($payload), $key // partitioning key — e.g., user_id for ordering user events ); $this->producer->flush(1000); // wait 1 sec for acknowledgment } } // Use in event handlers AddEventHandler('sale', 'OnSaleOrderSaved', function($order) { $kafka = new KafkaProducer(); $kafka->publish('bitrix.orders', (string)$order->getUserId(), [ 'event' => $order->isNew() ? 'order.created' : 'order.updated', 'order_id' => $order->getId(), 'status' => $order->getField('STATUS_ID'), 'total' => $order->getPrice(), 'ts' => time(), ]); }); 

Consumer: event processing

The consumer runs as a separate daemon (not in Bitrix context—in PHP-CLI):

// kafka_consumer.php $conf = new \RdKafka\Conf(); $conf->set('group.id', 'crm-sync-group'); $conf->set('metadata.broker.list', 'kafka:9092'); $conf->set('auto.offset.reset', 'latest'); // read from end, not beginning $consumer = new \RdKafka\KafkaConsumer($conf); $consumer->subscribe(['bitrix.orders', 'bitrix.products']); while (true) { $message = $consumer->consume(5000); // timeout 5 sec if ($message->err === \RD_KAFKA_RESP_ERR_NO_ERROR) { $payload = json_decode($message->payload, true); try { EventDispatcher::dispatch($message->topic_name, $payload); // Kafka manages offsets automatically when using group.id } catch (\Throwable $e) { // Log, do not commit offset — message will be re-read error_log("Kafka consumer error: " . $e->getMessage()); } } } 

Why does consumer performance drop when lag grows?

Lag is the difference between the last published and last read message. If lag grows, the consumer is not keeping up. Causes: insufficient consumer performance, incorrect number of partitions, slow message processing. Solutions: increase the number of consumers in the group (but no more than partitions), optimize processing logic, add server capacity. Our experience shows that the typical cause is inefficient database queries inside the consumer. Check indexes and use batch inserts. Load testing shows that Kafka is 5 times faster than RabbitMQ at 100,000 events per minute.

Comparison of Kafka and RabbitMQ for Bitrix

Topics and partitions

Number of partitions = maximum consumer parallelism. Start with 3–6 partitions per topic.

Monitoring Kafka

Consumer lag is the key metric. Monitor via Kafka UI (Provectus) or CMAK, alerts via Telegram through Alertmanager. We set up alerts when lag exceeds 1000 messages. We guarantee your system will be under control.

Apache Kafka Documentation: https://kafka.apache.org/documentation/

What is included in Kafka setup work

• Audit of current architecture and data flows
• Deploying Kafka infrastructure (brokers, topics, partitions, replication)
• Writing producer code to publish events from Bitrix
• Developing consumer scripts for external systems
• Configuring monitoring (lag, errors) and alerts
• Documentation on topics and data schemas
• Team training on Kafka

During the audit phase, we determine the exact topic topology and the number of partitions based on peak load. This is critical for scaling.

Project stages

1. Analytics — study current integrations and event volume.
2. Design — define topics, partitions, keys.
3. Implementation — write producers and consumers, configure infrastructure.
4. Testing — verify under load, measure lag.
5. Deployment — launch into production, connect monitoring.

Turnkey setup takes 3 to 5 working days. Contact us to discuss your project and get a timeline estimate. Request a consultation — we will help determine if Kafka suits your task. Our experience includes over 50 integration projects, 10+ with Kafka, and we provide a warranty on completed work. Our engineers hold certifications in Kafka and Bitrix.