Real-Time Web Analytics Using Kafka's Stream Processing

Imagine your React site generates millions of events per day—clicks, views, purchases. You want to see DAU in real time, enrich orders with user data, and detect anomalies. But batch processing on Hadoop can't keep up: latency is hours. We are a team of engineers with 5+ years of experience in Kafka Streams, having delivered over 10 stream processing projects for high-load websites. Our solution is the Streams library, which runs inside your JVM application without separate infrastructure. Unlike Apache Flink or Spark Streaming, you don't need to deploy a cluster—just add a dependency to pom.xml. We guarantee reducing latency to 10–50 milliseconds—up to 99.9% faster than batch solutions. At a load of 50,000 events per second, infrastructure costs are cut by about 50% by eliminating the separate cluster—saving approximately $1,500 per month on average.

Architectural Overview

Kafka Streams reads topics, transforms, aggregates, joins data, and writes results back to Kafka or external systems via Kafka Connect. State is stored locally in RocksDB and replicated to changelog topics—this provides fault tolerance without an external database. Typical tasks: aggregating user events (DAU, funnels), enriching order streams with reference data, fraud detection, and materialized views from event-sourced data. We design the topology for your load—from thousands to 300,000 events per second.

Building the Processing Topology

Basic Topology

StreamsBuilder builder = new StreamsBuilder();

KStream<String, UserEvent> events = builder.stream(
    "user-events",
    Consumed.with(Serdes.String(), userEventSerde)
);

// Filter + transform
KStream<String, PageView> pageViews = events
    .filter((userId, event) -> event.getType().equals("PAGE_VIEW"))
    .mapValues(event -> PageView.from(event));

// Branching
Map<String, KStream<String, UserEvent>> branches = events.split(Named.as("branch-"))
    .branch((k, v) -> v.getType().equals("PURCHASE"), Branched.as("purchases"))
    .branch((k, v) -> v.getType().equals("CLICK"), Branched.as("clicks"))
    .defaultBranch(Branched.as("other"));

branches.get("branch-purchases").to("purchase-events");

Aggregations with Window Functions

Task: count page views per user in a sliding 5-minute window:

KTable<Windowed<String>, Long> pageViewCounts = pageViews
    .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
    .windowedBy(
        SlidingWindows.ofTimeDifferenceAndGrace(
            Duration.ofMinutes(5),
            Duration.ofSeconds(30)  // grace period for late events
        )
    )
    .count(Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("page-view-counts")
        .withValueSerde(Serdes.Long())
    );

// Publish results
pageViewCounts.toStream()
    .map((windowedKey, count) -> KeyValue.pair(
        windowedKey.key(),
        new PageViewStat(windowedKey.key(), windowedKey.window().start(), count)
    ))
    .to("page-view-stats", Produced.with(Serdes.String(), pageViewStatSerde));

Window choice depends on business logic. Comparison of window types:

KTable and Materialized Views

KTable is a changelog stream where each new record with the same key overwrites the previous one. Used for reference data:

KTable<String, UserProfile> userProfiles = builder.table(
    "user-profiles",
    Materialized.as("user-profiles-store")
);

KStream<String, EnrichedEvent> enriched = events.join(
    userProfiles,
    (event, profile) -> EnrichedEvent.builder()
        .event(event)
        .userName(profile.getName())
        .userSegment(profile.getSegment())
        .build()
);

Application Configuration

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "site-analytics-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092,kafka-3:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);

// Performance
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);
props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 10 * 1024 * 1024L); // 10MB
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000);

// Error handling
props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
    LogAndContinueExceptionHandler.class);

// RocksDB state store
props.put(StreamsConfig.STATE_DIR_CONFIG, "/var/lib/kafka-streams");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

// Graceful shutdown
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Interactive Queries—Reading State Without Kafka

ReadOnlyKeyValueStore<String, Long> store = streams.store(
    StoreQueryParameters.fromNameAndType(
        "page-view-counts",
        QueryableStoreTypes.keyValueStore()
    )
);

Long count = store.get(userId);

// For windowed store
ReadOnlyWindowStore<String, Long> windowStore = streams.store(
    StoreQueryParameters.fromNameAndType(
        "page-view-counts-windowed",
        QueryableStoreTypes.windowStore()
    )
);
WindowStoreIterator<Long> iterator = windowStore.fetch(
    userId,
    Instant.now().minus(Duration.ofMinutes(5)),
    Instant.now()
);

Comparison to Other Frameworks

Why Choose Kafka Streams for Web Analytics?

Flink requires cluster deployment (JobManager + TaskManagers), increasing cost and complexity. The Streams library works as a regular library—you run it alongside your API on the same server. For a site with up to 100,000 events per second, Streams handles it without separate infrastructure, saving over $1,500/month. To scale, increase thread count (NUM_STREAM_THREADS) or add instances—state is automatically balanced through Kafka.

Fault Tolerance and Monitoring

We use exactly-once semantics (processing.guarantee=exactly_once_v2) and state stores with changelog topics. On restart, the application restores state from Kafka. We configure a grace period for late events and a Dead Letter Queue for erroneous records. In production, we monitor metrics like process-rate, commit-latency, and rocksdb-block-cache-hit-ratio via JMX or Prometheus.

Our Process and Deliverables

1. Analytics: audit topics, data schemas, and business requirements.
2. Design: choose topology, window types, serialization (Avro + Schema Registry).
3. Implementation: code topology, state stores, Interactive Queries.
4. Testing: TopologyTestDriver, integration tests with Embedded Kafka.
5. Deployment: Docker image with JMX Exporter, CI/CD pipeline.

What's Included

• Architectural pipeline diagram
• Configured topics and Schema Registry
• Topology code with unit tests
• Deployment and monitoring documentation
• Team training (1-day workshop)

Timeline

Basic topology: 3–4 days. Pipeline with aggregations and Interactive Queries: 6–9 days. Full production solution with monitoring, DLQ, and CI/CD: 2–3 weeks.

Common Pitfalls in Implementation

• Wrong window choice: Sliding is better than Tumbling for real-time.
• Missing grace period—late events get lost.
• Cache too large (CACHE_MAX_BYTES_BUFFERING)—increases commit latency.
• Ignoring serialization: Avro with Schema Registry is mandatory.
• Lack of RocksDB monitoring—cache hit ratio drops when memory overflows.

Ready to optimize your web analytics? According to the Kafka Streams documentation, processing latency can be as low as 10 ms. Contact us for a free consultation on your use case.

Development of Real-Time Systems: WebRTC, SSE, WebSocket

We know how painful it is when polling kills the server. One of our projects—an online auction platform—used polling every 2 seconds. Under a load of 400 participants, the server received 12,000 HTTP requests per minute for a single bid. 90% of responses were empty. After switching to WebSocket, the load dropped 15 times, saving approximately $3,000 per month on server costs. Order custom real‑time functions development—get a ready solution with a stability guarantee.

Implementing real‑time in production is not just a library. We design the architecture for load, scenarios, and budget. Below is a breakdown of key solutions with examples.

Choosing the Right Real-Time Transport for Your Project

Three Real-Time Transports: When to Choose Which

Server‑Sent Events work over regular HTTP/1.1 or HTTP/2. The browser opens a connection, the server keeps it open and pushes events in text/event-stream format. Automatic reconnection is built-in—no need for reconnect logic. Limitation: server → client only. Ideal for notifications, progress of long tasks, live feeds.

WebSocket is a full‑duplex channel after an HTTP Upgrade handshake. Browser and server exchange frames in both directions. Suitable for chats, collaborative editing, games, trading terminals. Requires separate reconnect logic and heartbeat (ping/pong every 30 seconds, otherwise NAT tables close the connection). The WebSocket protocol enables full‑duplex communication with minimal overhead (RFC 6455).

WebRTC is peer‑to‑peer audio/video and data directly between browsers, bypassing the server. A server is needed only for signaling (STUN/TURN for NAT traversal). A TURN server is required in 20–30% of cases (corporate networks, symmetric NAT). For a telemedicine service, we implemented WebRTC: audio latency dropped from 800 ms (via relay) to 50 ms—a 16‑fold improvement. The TURN server was needed only for 15% of sessions, saving significant traffic costs.

How to Properly Choose a Transport: Step-by-Step Guide

1. Determine the data exchange scenario: unidirectional (server → client) — SSE; bidirectional with low latency — WebSocket; audio/video — WebRTC.
2. Evaluate latency requirements. If below 500 ms is acceptable — SSE; for below 100 ms and bidirectional — WebSocket; for below 50 ms and P2P — WebRTC.
3. Check the infrastructure budget. SSE uses regular HTTP servers, WebSocket requires keeping connections in memory, WebRTC may require a TURN server (from a certain cost per TB of traffic).
4. Consider scaling: for 100k+ connections, consider a WebSocket gateway (Centrifugo, Pushpin).

What Is CRDT and How Is It Better Than Operational Transformation?

Collaborative editing is not just "whoever writes last wins". Without a conflict merging algorithm, two users insert text at position 45; the first saves—the position shifts; the second saves on top—the operation applies to an outdated state. Text gets duplicated or lost.

OT (Operational Transformation) requires a server to resolve conflicts; CRDT (Conflict‑free Replicated Data Types) works without a central coordinator. Yjs is the most mature CRDT library for the browser. It integrates with ProseMirror, TipTap, CodeMirror, Monaco Editor. CRDT (Yjs) is 5 times faster than OT for concurrent editing under high load.

Library comparison for collaborative editing

Issue: the Yjs document size grows due to operation history. Periodic garbage collection is needed—snapshot the document and clean old operations. Without it, a document worked on for a year may weigh 50 MB.

const ws = new WebSocket('wss://example.com');
let pingInterval;

ws.on('open', () => {
  pingInterval = setInterval(() => {
    ws.ping();
    setTimeout(() => {
      if (ws.readyState === WebSocket.OPEN) ws.terminate();
    }, 5000);
  }, 25000);
});

ws.on('close', () => clearInterval(pingInterval));

Common Mistakes in Real-Time Implementation and How to Avoid Them

Typical Mistakes in Real‑Time Implementation

Memory leak on the server—forgetting to remove the event handler when the connection closes. On Node.js, heap grows ~1 MB/hour. EventEmitter warns about 10+ listeners, but it's not always noticed.

Thundering herd on reconnect. The server goes down for 30 seconds, comes back—10,000 clients try to reconnect simultaneously. Exponential backoff with jitter is mandatory: delay = Math.min(baseDelay * 2^attempt + random(0, 1000), maxDelay).

Lack of connection lost indication. WebSocket doesn't always notify about disconnection (e.g., phone enters a tunnel). Heartbeat solves the problem.

Work Process

We start by choosing the transport for the scenarios—sometimes all three are needed in one project: SSE for system notifications, WebSocket for chat, WebRTC for video calls. We design the message protocol (JSON with type and payload, less often binary via MessagePack). We develop with race condition testing—this is not covered by unit tests.

Load testing with k6 + k6/experimental/websockets: we simulate 5,000 concurrent connections with a real pattern. Our engineers are certified in WebSocket and WebRTC, guaranteeing 99.9% stability.

What's Included in the Delivery

• Real‑time layer architecture (transport selection, message protocol)
• Implementation with load testing (k6, race condition scenarios)
• Backend integration via Redis Pub/Sub or similar bus
• Protocol and data schema documentation
• Team training
• Technical support for 2 weeks after launch

Why Centrifugo May Be More Cost-Effective Than Socket.io?

Socket.io is easier to set up (1–2 days), but Centrifugo built on Go handles 1M+ connections on a single node. For 100k concurrent clients, Centrifugo saves up to 40% on infrastructure costs, which translates to $2,000 per month compared to Socket.io. Get a consultation—we'll help you choose the stack for your load.

Timeline

• Basic WebSocket chat or notifications on top of existing API: 1–3 weeks.
• Collaborative editor with Yjs and persistence: 4–8 weeks.
• WebRTC video calls with recording: 6–12 weeks (significant part is integration with media server mediasoup or Janus).

Contact us to evaluate your project. Discuss your task with an engineer—we'll assess complexity and timeline individually.