Problem: Dialogue latency kills UX
We've seen projects where the voice assistant took 3–4 seconds to respond — users simply abandoned the conversation. End-to-end latency is the key metric. Our experience shows: for a natural dialogue, you need to stay within 1.5 seconds from the end of user speech to the start of the response. We solve this with a Speech-to-Speech (S2S) architecture without text breaks. This architecture is critical for call centers, retail voice assistants, and medical systems — every second of downtime reduces conversion or even endangers patient health. Additionally, we implement latency monitoring by percentiles p50, p95, and p99 to guarantee stability even under load.
What problems does a Speech-to-Speech voice AI assistant solve?
The main technical challenges in the S2S pipeline:
- VAD + endpointing — detecting the end of a phrase with minimal delay (600–800 ms). Incorrect threshold leads to speech clipping or missing silence.
- STT latency — Whisper API gives 300–600 ms but adds network delay. Optimize via streaming mode and buffering.
- TTS streaming — synthesizing the first chunk in 200–400 ms, but the client must play on the fly. We use PCM stream with preloading.
- LLM reasoning — GPT-4o-mini responds in 200–500 ms, but complex queries take longer. We limit the context window and use few-shot examples.
Each of these problems is solved by choosing the right tool and tuning for the specific scenario. For example, in a telemedicine project we achieved p99 latency of 1.2 s by combining Silero VAD with local Whisper on GPU and streaming TTS. According to the official OpenAI Realtime API documentation, end-to-end latency does not exceed 800 ms when using server-side VAD.
How we build the Speech-to-Speech architecture
We build the architecture on streaming components to minimize buffering. The basic pipeline:
Microphone → VAD → STT → NLU/LLM → TTS → Speaker
↑ ↓
Endpointing First audio chunk
(600–800ms) (<300ms after TTS start)
Key insight: we start TTS after the first STT chunk, not after full transcription. This approach reduces overall latency by 20–30%.
Full pipeline on OpenAI
import asyncio
from openai import AsyncOpenAI
import sounddevice as sd
import numpy as np
client = AsyncOpenAI()
class VoiceAssistant:
def __init__(self):
self.conversation_history = []
self.system_prompt = "You are a helpful voice assistant. Answer briefly, 1–3 sentences."
async def listen_and_respond(self):
# Record via VAD
audio = await self.record_speech()
# STT
transcript = await client.audio.transcriptions.create(
model="whisper-1",
file=("audio.wav", audio, "audio/wav"),
language="en"
)
user_text = transcript.text
print(f"User: {user_text}")
# LLM
self.conversation_history.append({"role": "user", "content": user_text})
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "system", "content": self.system_prompt}]
+ self.conversation_history,
)
assistant_text = response.choices[0].message.content
self.conversation_history.append({"role": "assistant", "content": assistant_text})
print(f"Assistant: {assistant_text}")
# TTS streaming
async with client.audio.speech.with_streaming_response.create(
model="tts-1",
voice="alloy",
input=assistant_text,
response_format="pcm",
) as tts_response:
async for chunk in tts_response.iter_bytes(1024):
# Play chunks as they arrive
audio_data = np.frombuffer(chunk, dtype=np.int16)
sd.play(audio_data.astype(np.float32) / 32768.0, samplerate=24000)
sd.wait()
OpenAI Realtime API (optimal for production)
import websockets
async def realtime_voice_assistant():
url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
headers = {
"Authorization": f"Bearer {OPENAI_API_KEY}",
"OpenAI-Beta": "realtime=v1"
}
async with websockets.connect(url, extra_headers=headers) as ws:
# Configuration
await ws.send(json.dumps({
"type": "session.update",
"session": {
"voice": "alloy",
"instructions": "You are a voice assistant. Answer in Russian.",
"turn_detection": {"type": "server_vad"}
}
}))
# ...event handling
How we reduce end-to-end latency
The critical factors are parallel processing and choosing the right endpointing algorithm. We use webrtcvad with aggressiveness 1 and dynamic timeout. In production with Realtime API, server-side VAD is 100–200 ms faster than client-side. Additionally, we cache embeddings for frequent commands (p99 latency drops by 15%). According to official OpenAI documentation, end-to-end latency does not exceed 800 ms. Savings on call center operators can reach 70%.
Which technologies do we use?
| Component | Tools | Typical Latency |
|---|---|---|
| VAD | webrtcvad, Silero VAD | 50–100 ms |
| STT | Whisper-1, Wav2Vec 2.0 | 300–600 ms |
| LLM | GPT-4o-mini, LLaMA 3 8B | 200–500 ms |
| TTS | OpenAI TTS-1, ElevenLabs | 200–400 ms |
| Total | classic pipeline | 1.3–2.3 s |
| Total | OpenAI Realtime API | 500–800 ms |
For local inference we use ONNX Runtime and vLLM — GPU utilization reaches 85%. Comparison: classic pipeline is 2–3 times slower than Realtime API, but gives more control over the voice. The cost of processing one minute of audio in the cloud is fractions of a cent.
Process of work
- Analysis — measure current infrastructure, voice requirements, SLA (from 2 weeks).
- Design — select components (OpenAI/local), design integration (from 3 days).
- MVP implementation — basic chain VAD→STT→LLM→TTS with streaming (1 week).
- Testing — A/B tests with users, measure p99 latency, adjust endpointing (3 days).
- Deployment — set up CI/CD, monitoring in Grafana, alerts on latency (2 days).
- Optimization — fine-tune Whisper for accents, LoRA for LLM, TTS voice for brand (optional).
What's included in the work
- Architecture and API documentation.
- Source code of the assistant with comments and tests.
- Integration with your CRM/telephony via REST.
- Team training (2–3 hour workshop).
- 1 month post-launch support with bug fix guarantee.
Estimated timeline
MVP voice assistant — from 1 week. Full production with Realtime API — 2–3 weeks. The cost is calculated individually based on integration complexity and customization scope. Get a consultation for your project — we'll evaluate it turnkey.
Performance metrics
| Component | Latency |
|---|---|
| VAD + Endpointing | 600–800 ms |
| Whisper-1 API | 300–600 ms |
| GPT-4o-mini | 200–500 ms |
| TTS-1 first chunk | 200–400 ms |
| Total | 1.3–2.3 sec |
OpenAI Realtime API: end-to-end latency ~500–800 ms.
Contact us to get a consultation and a detailed implementation plan. We guarantee: your voice assistant will respond faster than 1.5 seconds.