Microservices grow, synchronous HTTP requests turn the system into a web. One failure — and the entire stack goes down. Event-Driven Architecture (EDA) is the only way to decouple this connectivity. We have implemented EDA for projects with loads up to 10,000 events per second and know how to avoid common mistakes.
A client with an online store processing 50,000 orders per day faced timeouts during peak loads, blocking inventory and notifications. After implementing EDA with Apache Kafka and the Outbox Pattern, response time dropped by 40%, throughput tripled, and support costs decreased by 30% (over $5,000 per month). Average event processing latency is 10ms.
Why EDA is better than synchronous interaction?
Synchronous HTTP calls are like phone conversations: you need an immediate answer. EDA works like mail: send a letter and forget. Comparison:
| Parameter | Synchronous | Asynchronous (EDA) |
|---|---|---|
| Time dependency | Client waits | Client non-blocking |
| Fault tolerance | Failure breaks chain | Queue isolates failures |
| Load | Direct RPS proportion | Buffering + backpressure |
| Development complexity | Easier to understand | Harder debugging (idempotency, monitoring) |
How we implement EDA: stack and case
We use Apache Kafka as the primary broker due to its high throughput and long-term storage.
Event structure
interface DomainEvent<T = unknown> {
id: string; // UUID — for idempotency
type: string; // 'user.registered', 'order.placed'
version: string; // '1.0' — for schema evolution
source: string; // 'order-service'
correlationId: string; // cross-service correlation ID
causationId?: string; // ID of the causing event
occurredAt: string; // ISO 8601
data: T;
}
// Concrete event
interface OrderPlacedEvent extends DomainEvent<{
orderId: string;
customerId: string;
items: Array<{ productId: string; quantity: number; price: number }>;
total: number;
shippingAddress: Address;
}> {
type: 'order.placed';
}
Apache Kafka — primary broker
import { Kafka, Partitioners } from 'kafkajs';
const kafka = new Kafka({
clientId: 'order-service',
brokers: process.env.KAFKA_BROKERS.split(',')
});
// Producer
const producer = kafka.producer({
createPartitioner: Partitioners.LegacyPartitioner
});
async function publishOrderPlaced(order: Order): Promise<void> {
await producer.send({
topic: 'order.events',
messages: [{
key: order.id, // partitioning by order ID
value: JSON.stringify({
id: uuidv4(),
type: 'order.placed',
version: '1.0',
source: 'order-service',
correlationId: context.correlationId,
occurredAt: new Date().toISOString(),
data: {
orderId: order.id,
customerId: order.customerId,
items: order.items,
total: order.total
}
} satisfies OrderPlacedEvent),
headers: {
'content-type': 'application/json',
'schema-version': '1.0'
}
}]
});
}
// Consumer — Inventory Service
const consumer = kafka.consumer({ groupId: 'inventory-service' });
await consumer.subscribe({ topics: ['order.events'], fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
const event = JSON.parse(message.value.toString()) as DomainEvent;
// Idempotency: check if this event has already been processed
const processed = await idempotencyRepo.exists(event.id);
if (processed) return;
try {
if (event.type === 'order.placed') {
await inventoryService.reserveStock(event.data.orderId, event.data.items);
}
await idempotencyRepo.mark(event.id);
} catch (error) {
// Publish to Dead Letter Topic for analysis
await deadLetterProducer.send({
topic: 'order.events.dlq',
messages: [{
value: message.value,
headers: { 'failure-reason': error.message }
}]
});
}
}
});
Outbox Pattern — guaranteed delivery
Common mistake: save to DB then publish to Kafka — risk of event loss. Correct approach — Transactional Outbox:
// Within a single database transaction
async function createOrder(dto: CreateOrderDto): Promise<Order> {
return db.transaction(async (trx) => {
// 1. Save order
const order = await trx('orders').insert({ ...orderData }).returning('*');
// 2. Save event in outbox table (same transaction!)
await trx('outbox_events').insert({
id: uuidv4(),
aggregate_id: order.id,
event_type: 'order.placed',
payload: JSON.stringify(orderPlacedEvent),
status: 'pending',
created_at: new Date()
});
return order;
});
}
A separate Outbox Poller reads pending events and publishes them to Kafka:
// Cron job or background worker
async function processOutbox(): Promise<void> {
const events = await db('outbox_events')
.where({ status: 'pending' })
.orderBy('created_at')
.limit(100)
.forUpdate()
.skipLocked();
for (const event of events) {
try {
await kafka.producer.send({
topic: getTopicForEventType(event.event_type),
messages: [{ key: event.aggregate_id, value: event.payload }]
});
await db('outbox_events')
.where({ id: event.id })
.update({ status: 'published', published_at: new Date() });
} catch {
await db('outbox_events')
.where({ id: event.id })
.update({ retry_count: db.raw('retry_count + 1') });
}
}
}
Alternative — Debezium CDC: reads PostgreSQL WAL and publishes changes to Kafka without code.
How to guarantee delivery without losses?
Key techniques:
- Transactional Outbox
- Consumer idempotency (check by event.id)
- Dead Letter Queue for failed messages
- Latency monitoring via Prometheus + Grafana
We set up a system handling 1000 msg/sec with no loss during a single Kafka node failure.
Efficient coordination: choreography vs orchestration
| Characteristic | Choreography | Orchestration |
|---|---|---|
| Coordination | Services react to events | Central orchestrator |
| Coupling | Low | Medium |
| Flow visibility | Hard to track | Explicit in code |
| Testing | Harder | Easier |
Choreography fits loosely coupled scenarios; orchestration when strict sequence is required. We combine: inside a service — orchestration, between services — choreography.
What are Event Sourcing and CQRS?
In Event Sourcing, all changes are stored as events, which are published to both event store and broker. CQRS (Command Query Responsibility Segregation) separates write and read operations, often using the same events to build projections. EDA and CQRS/ES work well together but require thoughtful design.
Example production Kafka configuration
For reliable production setup, configure replication factor 3, retention 7 days, compaction for critical topics. Minimum 3 brokers, monitor consumer lag via Burrow. For cloud environments — Confluent Cloud or AWS MSK.
What is included in the EDA implementation work?
When ordering, we provide:
- Architectural documentation (diagrams, broker choice)
- Development of producers and consumers with Outbox Pattern
- Kafka/RabbitMQ setup with persistency and replication
- Idempotency and Dead Letter Queue implementation
- Load testing up to 1000 msg/sec without loss
- Monitoring and alerting (latency, throughput)
- Team training
Get a consultation on EDA implementation for your project — we will assess your architecture and propose an optimal solution.
How long does EDA implementation take?
| Stage | Duration |
|---|---|
| Single scenario (producer + 2–3 consumers) | 1–2 weeks |
| + Outbox Pattern, idempotency, DLQ | +1 week |
| Full EDA for 5–10 services + monitoring | 4–8 weeks |
Broker choice depends on load and requirements. Kafka — for high throughput and long storage. RabbitMQ — for complex routing. Redis Streams — for low latency. Google Pub/Sub — for cloud. In most projects, we use Kafka.
EDA implementation in 4 steps
- Analysis and design: identify context boundaries, define events.
- Broker setup: deploy Kafka with replication, configure topics.
- Implement producers and consumers: write code with Outbox and idempotency.
- Monitoring and debugging: set up metrics, DLQ, alerts.
Our team has 10+ years of experience in distributed systems, with over 20 EDA projects. Order turnkey EDA implementation — get a reliable and scalable architecture. Contact us to evaluate your project.