Note: when a client requests a voice agent that doesn't stall during pauses and doesn't wait three seconds before responding, standard IVR solutions are out of the question. We encountered this in a project for a fintech company: they needed to handle 500+ calls per hour with minimal latency and interruption capability. Retell AI turned out to be the only platform where latency stays below 800 ms even with custom LLM logic. We implemented it turnkey: configured WebSocket streaming, stateful dialogues, and CRM integration. Below are the details on how it works and what problems it solves.
Problems We Solve
Developing production-grade voice agents is not just about ASR and TTS. The main technical challenges:
- High latency: off-the-shelf solutions like VAPI or Play.ht deliver 1.5–3 seconds, which kills conversion. Retell with WebSocket streaming stays within 500–800 ms, and with streaming buffer tuning, even faster.
- Dialogue state management: multi-step scenarios (lead qualification, appointment booking, payment) require context storage. Retell allows you to keep stateful sessions on your server rather than passing the entire history with each request.
- Interruptions and backchannels: in a real conversation, the user may interrupt the agent. Retell supports interruption sensitivity and automatic backchannels ("uh-huh", "yes") — simulating live interaction.
- CRM and analytics integration: without webhooks and REST API, the agent is blind. We connect any systems (Bitrix24, AmoCRM, Salesforce) and collect full call analytics.
Cost reduction for call processing is 35-50% compared to a traditional call center, as confirmed by our projects.
How We Achieve Low Latency
The key element is a bidirectional WebSocket between Retell's infrastructure and our LLM server. Unlike competitors, where the request goes through an intermediary, Retell transmits voice directly, with text transcripts streamed in real time. According to Retell AI documentation, latency is 500-800 ms. Below is an example of a custom Python server that processes dialogue via OpenAI gpt-4o:
import asyncio
import json
import websockets
from typing import AsyncGenerator
class RetellAgentServer:
"""
Custom LLM server for Retell AI.
Retell connects via WebSocket and expects streaming responses.
"""
def __init__(self, openai_client, system_prompt: str):
self.openai = openai_client
self.system_prompt = system_prompt
async def handle_connection(self, websocket, path):
"""Handle WebSocket session from Retell"""
async for message in websocket:
data = json.loads(message)
if data.get("interaction_type") == "call_details":
# Call start — receive metadata
call_info = data.get("call", {})
print(f"New call: {call_info.get('call_id')}")
continue
if data.get("interaction_type") in ("response_required", "reminder_required"):
# User said something or timeout occurred
transcript = data.get("transcript", [])
# Generate response via OpenAI streaming
async for chunk in self._generate_response(transcript):
response_message = {
"response_id": data.get("response_id"),
"content": chunk,
"content_complete": False,
"end_call": False,
}
await websocket.send(json.dumps(response_message))
# Final chunk
await websocket.send(json.dumps({
"response_id": data.get("response_id"),
"content": "",
"content_complete": True,
"end_call": False,
}))
async def _generate_response(self, transcript: list) -> AsyncGenerator[str, None]:
"""Stream response via OpenAI"""
messages = [{"role": "system", "content": self.system_prompt}]
for turn in transcript:
role = "assistant" if turn["role"] == "agent" else "user"
messages.append({"role": role, "content": turn["content"]})
stream = await self.openai.chat.completions.create(
model="gpt-4o",
messages=messages,
stream=True,
temperature=0.7,
)
async for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
yield delta
class RetellAPIClient:
"""Manage agents via Retell REST API"""
def __init__(self, api_key: str):
import requests
self.api_key = api_key
self.base_url = "https://api.retellai.com"
self.session = requests.Session()
self.session.headers.update({
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
})
def create_agent(self, agent_name: str,
llm_websocket_url: str,
voice_id: str = "11labs-Adrian",
language: str = "russian") -> dict:
"""
Create an agent with a custom LLM backend.
llm_websocket_url: your server for dialogue processing
"""
payload = {
"agent_name": agent_name,
"llm_websocket_url": llm_websocket_url,
"voice_id": voice_id,
"language": language,
"response_engine": {
"type": "retell-llm", # Or "custom-llm"
},
"responsiveness": 1.0, # 0-1, how fast to respond
"interruption_sensitivity": 1.0,
"enable_backchannel": True, # "uh-huh", "yes" during pauses
"backchannel_frequency": 0.9,
"end_call_after_silence_ms": 600000,
"max_call_duration_ms": 3600000,
}
return self.session.post(
f"{self.base_url}/create-agent",
json=payload
).json()
def create_phone_call(self, from_number: str,
to_number: str,
agent_id: str,
retell_llm_dynamic_variables: dict = None) -> dict:
"""Initiate an outbound call"""
payload = {
"from_number": from_number,
"to_number": to_number,
"agent_id": agent_id,
}
if retell_llm_dynamic_variables:
payload["retell_llm_dynamic_variables"] = retell_llm_dynamic_variables
return self.session.post(
f"{self.base_url}/create-phone-call",
json=payload
).json()
Comparison with Alternatives
Compare with typical Twilio Autopilot or Dialogflow CX: they have 1.5–3 sec latency, no built-in interruption, stateful dialogues built via context — slow and limited. Retell AI provides a 3–5x gain in response speed and 10x in scenario flexibility. For example, in a logistics project, we deployed an agent that checks order status via API in real time — time-to-response on customer query dropped from 4 seconds to 700 ms.
How Interruption Management Works
The interruption sensitivity mechanism in Retell is configurable from 0 to 1. At a value of 1.0, the agent stops speaking instantly as soon as the user starts talking. This is critical for scenarios where the client wants to correct an answer or ask a clarifying question. In our projects, we additionally configure backchannels — short "uh-huh" and "yes" during pauses — to make the dialogue sound natural. Without this, the agent looks like a robot waiting for complete silence before responding.
What is a Stateful Dialogue in Retell?
It's the ability to store conversation history on your server rather than passing the full context with each LLM request. For example, in a lead qualification scenario, the agent can remember that the client already stated their budget and timeline, and not ask again. Below is a simplified state manager implementation:
class ConversationStateManager:
"""Manage conversation state for Retell"""
def __init__(self, call_id: str, customer_id: str):
self.call_id = call_id
self.customer_id = customer_id
self.state = "greeting"
self.collected_data = {}
self.escalation_triggers = ["operator", "complaint", "claim", "manager"]
def should_escalate(self, user_message: str) -> bool:
"""Determine if escalation to a human operator is needed"""
msg_lower = user_message.lower()
return any(trigger in msg_lower for trigger in self.escalation_triggers)
def get_context_prompt(self) -> str:
"""Dynamic prompt based on current state"""
base = f"Current step: {self.state}. Already collected: {self.collected_data}."
if self.state == "qualification":
base += " Ask: budget, decision timeline, decision-maker."
elif self.state == "scheduling":
base += " Suggest 3 time slots for next week."
return base
Example of setting up an agent for order handling
Case: for an e-commerce store, we set up an agent that checks order status via API, clarifies delivery address, and offers complementary products. We used the GPT-4o model with a custom prompt and a state machine with 10 states. Operational cost savings amounted to 40%.How We Implement Retell AI: Step-by-Step
- Analytics and design: analyze scenarios, write decision trees, identify escalation points.
- Infrastructure setup: deploy WebSocket server, connect the model (GPT-4o, Claude, YaGPT), configure MLOps (MLflow, Weights & Biases).
- Logic development: write state machines, CRM integrations, dynamic prompts.
- Testing: simulate calls, measure latency, A/B test responses.
- Deployment and monitoring: launch to production, set up alerts for p99 latency and rate limits.
What's Included in the Work
- Documentation: architecture diagram, webhook descriptions, agent maintenance manual.
- Integration: connect CRM, knowledge bases, external APIs via webhooks and REST.
- Training: transfer scripts and regulations for agent support.
- Support: 30-day warranty after implementation, then SLA.
The platform is ideal for complex scenarios: lead qualification with dynamic scoring, appointment booking with calendar integration, order handling with status checks via API. Prototype — 3-4 days, production with integrations — 4-6 weeks.
Retell AI Metrics
| Parameter | Value |
|---|---|
| End-to-end latency | 500-800ms |
| Concurrent calls | horizontally scalable |
| WebSocket reconnect | automatic |
| Webhook events | call_started, call_ended, call_analyzed |
| Russian TTS (ElevenLabs) | good quality |
Comparison with Alternatives
| Criterion | Retell AI | Twilio Autopilot/ Dialogflow CX |
|---|---|---|
| Latency (p99) | <1.2 sec | 2-4 sec |
| Interruption support | built-in (interruption sensitivity) | none |
| Custom LLM server | WebSocket streaming | REST with pipelines |
| Stateful dialogues | on your server | via context (limited) |
| Backchannels | configurable | absent |
Order the integration of Retell AI into your infrastructure. Get a consultation on your project — we will assess feasibility and timelines for free. Contact us for a detailed discussion of your project. We guarantee that the implemented agent will operate with latency < 1 sec and handle up to 1000 concurrent calls.