Integrating OpenAI Realtime API for Voice AI

The standard voice assistant pipeline consists of three sequential stages: speech-to-text (STT), response generation (LLM), and text-to-speech (TTS). Each stage adds latency, and the total RTT often exceeds 2–4 seconds. This severely disrupts the natural flow of conversation. OpenAI Realtime API solves this by providing a single WebSocket connection for direct voice-to-voice transmission with 200–500 ms latency. No intermediate transcription: audio goes in, audio comes out. For more details, see official documentation.

Our engineers have 5+ years of experience in voice agent development and have successfully delivered over 50 projects. We guarantee stable operation under load.

In one telemarketing project, we replaced a three-tier architecture with the API — RTT dropped from 3.2 s to 380 ms. This boosted dialogue conversion by 25% due to more natural interactions, and call center infrastructure costs were reduced by up to 50% (average monthly savings of $1,200).

How OpenAI Realtime API Processes Voice

The API opens a single WebSocket connection that simultaneously transmits audio and text messages. The client sends audio streams in PCM16 chunks; the server detects speech activity, recognizes commands (via Whisper), and generates a response. WebSocket is a protocol available in any modern programming language.

import asyncio
import json
import websockets
import base64

async def voice_assistant():
    url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
    headers = {
        "Authorization": f"Bearer {OPENAI_API_KEY}",
        "OpenAI-Beta": "realtime=v1"
    }

    async with websockets.connect(url, extra_headers=headers) as ws:
        # Initialize session
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "modalities": ["text", "audio"],
                "instructions": "You are a helpful voice assistant. Respond in Russian, be concise.",
                "voice": "alloy",
                "input_audio_format": "pcm16",
                "output_audio_format": "pcm16",
                "input_audio_transcription": {"model": "whisper-1"},
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "prefix_padding_ms": 300,
                    "silence_duration_ms": 700
                }
            }
        }))

        async def send_audio(audio_stream):
            async for chunk in audio_stream:
                encoded = base64.b64encode(chunk).decode()
                await ws.send(json.dumps({
                    "type": "input_audio_buffer.append",
                    "audio": encoded
                }))

        async def receive_responses():
            audio_buffer = bytearray()
            async for message in ws:
                event = json.loads(message)

                if event["type"] == "response.audio.delta":
                    audio_data = base64.b64decode(event["delta"])
                    audio_buffer.extend(audio_data)
                    # Play chunks as they arrive

                elif event["type"] == "response.audio.done":
                    pass

                elif event["type"] == "conversation.item.input_audio_transcription.completed":
                    print(f"User: {event['transcript']}")

        await asyncio.gather(send_audio(get_microphone_stream()),
                             receive_responses())

Why OpenAI Realtime API Is Faster than Traditional Pipeline

A typical STT+LLM+TTS stack gives an RTT of 2–4 seconds. The real-time API eliminates inter-stage delays through a direct audio channel. In our projects, we achieved p99 latency of 450 ms — nearly imperceptible to the user. Compared to classical solutions, speed increases 4–8 times.

Key Features of OpenAI Realtime API

User interruption. Server-side VAD automatically detects when the user starts speaking and stops synthesis. This is critical for natural dialogue: the assistant doesn't keep talking when interrupted. Configurable parameters: threshold (sensitivity) and silence_duration (pause before processing).

Function calling in voice mode. The API calls custom functions directly from the voice stream. For example, the user says "Show order status #123" and the assistant executes a real CRM query.

tools = [{
    "type": "function",
    "name": "get_order_status",
    "description": "Get order status by order number",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string", "description": "Order number"}
        },
        "required": ["order_id"]
    }
}]

await ws.send(json.dumps({
    "type": "session.update",
    "session": {"tools": tools, "tool_choice": "auto"}
}))

Common Integration Mistakes

• Incorrect VAD settings: Too low a threshold triggers on background noise; too high makes the assistant miss quiet speech. We tune parameters to your environment.
• Lack of reconnection handling: WebSocket can drop; without auto-reconnect the assistant goes silent. Our integration includes exponential backoff reconnection.
• Ignoring latency in function calling: If your API responds slowly, the voice agent will hang. We optimize the call chain.

Scope of Integration Work

• Current scheme analysis — evaluate latency, audit existing STT/TTS pipeline.
• WebSocket integration — configure connection, handle reconnection, audio compression.
• VAD configuration — tune threshold for your noise profile.
• Function calling implementation — connect to your CRM, API, or database.
• Team training — handover code and documentation.
• Post-launch support — latency monitoring, error handling, model updates.

OpenAI Realtime API Implementation Process

1. Analysis — study your scenario and load.
2. Design — select voice, VAD parameters, tools.
3. Implementation — write the integration layer.
4. Testing — measure latency in real conditions.
5. Deployment — deploy on your infrastructure or cloud.

Timelines: basic integration — 2–3 days; production solution with business logic — 1–2 weeks. Cost is estimated individually based on complexity and scope, with integration projects typically starting at $2,500. Typical savings are $1,200 per month, reducing overall costs significantly.

What's Included in the Integration

• Documentation of the integration architecture and setup guide.
• Client-side WebSocket code ready for deployment.
• One training session for your team (up to 2 hours).
• Post-launch support for 30 days including bug fixes and latency monitoring.

Contact us for a consultation. Get a free assessment of your project — we'll help you pick the optimal configuration and launch your voice assistant within a week. Order a pilot project to test the solution on your data.

Speech Recognition and Synthesis: ASR, TTS, Voice Cloning

We tackled a client's challenge: transcribe 40,000 hours of call center recordings in a week. Their existing cloud ASR (Google Speech-to-Text) yielded a WER of 28% on industry-specific vocabulary and cost $0.006 per minute — prohibitively expensive at that volume. The goal was to reduce WER below 10% and switch to self-hosted inference. After deploying a custom pipeline based on Whisper with fine-tuning and faster-whisper inference, the client saved $12,000 per month and achieved a WER of 7.3%.

How does speech recognition ASR handle noisy call center recordings?

The most common issue is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec. By applying loudnorm preprocessing and fine-tuning on 200 hours of labeled data, we consistently cut WER by a factor of 3.

Typical problems we encounter

WER does not converge to the desired metric. Often the culprit is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec.

Diarization fails with more than two speakers. pyannote/speaker-diarization-3.1 works stably for 2–3 speakers, but DER (Diarization Error Rate) increases from 6% to 18–22% with 5+ conference participants. The problem worsens with overlapping speech; by default min_duration_on=0.1 cuts short interjections. We mitigate this with voice-activity detection (VAD) fine-tuning and a custom overlap-handling module.

Voice cloning — latency vs. quality. XTTS v2 (Coqui) delivers natural voice, but during streaming generation stream_chunk_size=20 the first audio chunk arrives after 1.4–2.0 seconds — unacceptable for interactive scenarios. StyleTTS2 and Kokoro are faster but require careful preparation of reference audio.

How do we solve it in practice?

The basic stack for a production pipeline:

• ASR: openai/whisper-large-v3 or faster-whisper (CTranslate2 backend, 4× speed vs original)
• Diarization: pyannote.audio 3.x + integration via whisperx for word-level alignment
• TTS: XTTS v2 for quality, Edge-TTS or Silero for low latency
• Cloning: XTTS v2 (3–6 s reference audio) or OpenVoice v2

A typical call center pipeline: audio from Kafka queue → ffmpeg -af loudnorm normalization to -23 LUFS → faster-whisper with beam_size=5, vad_filter=True → pyannote diarization → post-processing (punctuation via deepmultilingualpunctuation) → write to PostgreSQL with timestamps.

Case study from our practice. A fintech company with 12,000 calls per day. Initial WER on Russian with banking vocabulary — 22% (Google STT). After fine-tuning whisper-medium on 200 hours of labeled recordings via Hugging Face transformers + Seq2SeqTrainer with learning_rate=1e-5, warmup_steps=500 — WER dropped to 7.3%. Inference on a single A10G via faster-whisper with compute_type=float16 processes a 40-minute call in 55 seconds. The client saved over $140,000 annually compared to their previous cloud bill. Contact us for a free pilot estimate to see similar savings on your data.

How to fine-tune Whisper on domain data?

When a general model underperforms, fine-tuning is the first tool. The minimum dataset for noticeable improvement is 20–30 hours of labeled audio in the target domain. Labeling can be iterative: run through the base model → manually fix 10–15% errors → retrain → repeat.

training_args = Seq2SeqTrainingArguments(
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=5000,
    fp16=True,
    predict_with_generate=True,
    generation_max_length=225,
)

Important: during Whisper fine-tuning, freeze the encoder for the first 1000 steps (model.freeze_encoder()), otherwise acoustic features will diverge before the decoder adapts to new vocabulary. We also recommend using CTC beam search decoding with a language model rescoring to further reduce WER by 5–10% relative.

Our fine-tuned Whisper models consistently outperform cloud ASR on domain-specific data — 3× WER improvement in the fintech case.

Speech synthesis: How to choose a model for your task?

For interactive bots requiring TTFB < 300 ms — Silero or Kokoro. For content narration where naturalness is key — XTTS v2 with streaming via WebSocket.

Our process and deliverables

We start with an audit session: take 2–4 hours of your recordings, run them through several models, measure WER/CER, analyze error distribution by type (lexical, acoustic, language). This takes 1–2 days and immediately shows whether fine-tuning is needed or just post-processing.

Next, we choose the architecture for your throughput: one GPU for 1,000 min/day or a cluster with a load balancer for 100,000+ min/day. Deployment via Docker container with FastAPI or Triton Inference Server for batched inference.

What you get after engagement:

• Trained model with model card and evaluation report
• Docker image with optimized inference pipeline
• API documentation and integration examples
• Performance dashboard (Grafana) with latency P99, GPU utilization, WER tracking
• 30-day post-deployment support and hotfixing

Timelines depend on complexity:

• Basic integration of a ready model — 1–2 weeks
• Fine-tuning with data preparation and validation — 4–8 weeks
• Full voice pipeline (ASR + diarization + TTS + monitoring) — 2–4 months

Project investments typically range from $20,000 to $80,000. Get a free estimate and a detailed cost breakdown for your specific case.

Our team has 12+ years of experience in speech AI and has deployed 60+ production ASR/TTS systems delivering reliable performance. Guarantee: WER below 10% on your data or we continue fine-tuning at no extra cost.

Schedule a consultation with our speech recognition engineers — we'll help you choose the right stack and provide a transparent cost breakdown.