Developing a Voice AI Agent for Call Processing
Manual handling of repetitive calls—order status checks, appointment bookings, delivery rescheduling—clogs channels and pushes AHT up to 8 minutes. Operators burn out, customers switch to competitors after 3 minutes of waiting. An AI voice assistant takes over up to 80% of such dialogues, working in real time with LLM, STT/TTS, and tools. With over 7 years of experience in NLP and 50+ successful voice agent deployments, we have the expertise to build high-performing agents.
We build a voice assistant that conducts full-fledged conversations: understands context, asks clarifying questions, makes decisions, queries CRM and databases, and wraps up with a result.
What pain points does it address?
The main business pain is low call center throughput during peak hours. An AI-powered voice agent processes inbound calls without human involvement, reducing operator load and cutting customer wait time. Moreover, the dialogue system doesn't follow a rigid script but adapts to the request. For example, on a call about a delayed delivery, the agent checks the status in CRM, offers to reschedule, and creates a task for the courier—all in one conversation. As a result, the automation rate jumps from typical 20–30% to 55–65%, and average handling time drops by 60%.
In one logistics project, the agent handled 1,500 calls per day, with 68% resolved without transferring to a human agent, and AHT dropped from 6 to 2 minutes. The average saving per call reaches ₽35, and at a load of 10,000 calls per month—₽350,000 compared to a traditional contact center. For a 15,000 call per month volume, the savings reach ₽525,000 per month. Our agent is 3x more cost-effective than human operators, and ROI of 300% is achievable within 3 months.
Architecture of the Voice AI Agent
Telephony (Twilio/Voximplant)
↓
WebSocket Bridge
↓
STT (Deepgram/Whisper) with VAD, diarization, noise suppression
↓
Dialog Manager
├── State Machine (dialog state tracking)
├── LLM (GPT-4o) with few-shot prompting, function calling, chain-of-thought
├── Tool Registry (CRM, DB, APIs)
└── Context Window
↓
TTS (ElevenLabs/OpenAI) with barge-in handling
↓
Audio Back to Call
Component details
The **WebSocket Bridge** converts audio stream to text and back, supporting up to 100 simultaneous calls per instance. STT includes voice activity detection and diarization to handle multiple speakers, achieving <5% WER. The **Dialog Manager** runs a state machine with intent classification and slot filling, using NLU confidence thresholds. The Tool Registry registers external functions available for LLM calling.Why GPT-4o instead of an open model?
GPT-4 ensures natural dialogue and low hallucination rates. In tests on a set of 500 calls, it showed an 18% higher self-service rate compared to LLaMA 3. For scenarios with high query variability (e.g., tech support), this is critical. Open models are suitable for narrow scenarios with a rigid script—in that case we use a fine-tuned Mistral or Qwen with INT4 quantization. According to our measurements, GPT-4o is 2x better than LLaMA-3 at reducing False Transfer Rate.
How do we implement telephony integration?
The basic glue layer is a WebSocket Bridge between the telephony provider (Twilio/Voximplant) and the Dialog Manager. At this stage, audio stream is converted to text (STT) and back. Example integration with Twilio:
from twilio.rest import Client
from twilio.twiml.voice_response import VoiceResponse, Start, Stream
twilio_client = Client(TWILIO_SID, TWILIO_AUTH)
def handle_incoming_call(call_sid: str, ws_url: str) -> str:
response = VoiceResponse()
start = Start()
start.stream(url=f'wss://api.example.com/stream/{call_sid}')
response.append(start)
response.say("Welcome! How can I help you?",
voice="alice", language="en-US")
response.pause(length=60)
return str(response)
Dialog Manager with tools
This is the heart of the agent. It manages dialogue state, calls the LLM, and executes external tools on demand. Implementation in Python:
from openai import AsyncOpenAI
from dataclasses import dataclass, field
import json
client = AsyncOpenAI()
@dataclass
class AgentState:
call_id: str
history: list = field(default_factory=list)
collected_data: dict = field(default_factory=dict)
current_intent: str = None
class VoiceAgent:
def __init__(self):
self.tools = [
{
"type": "function",
"function": {
"name": "lookup_order",
"description": "Find customer order by phone number or order ID",
"parameters": {
"type": "object",
"properties": {
"phone": {"type": "string"},
"order_id": {"type": "string"}
}
}
}
},
{
"type": "function",
"function": {
"name": "reschedule_delivery",
"description": "Reschedule delivery to another date",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"new_date": {"type": "string", "description": "YYYY-MM-DD"}
},
"required": ["order_id", "new_date"]
}
}
}
]
async def process_turn(self, state: AgentState, user_text: str) -> str:
state.history.append({"role": "user", "content": user_text})
response = await client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": self._get_system_prompt()},
*state.history
],
tools=self.tools,
tool_choice="auto"
)
message = response.choices[0].message
# Handle function calls
if message.tool_calls:
tool_results = await self._execute_tools(message.tool_calls)
state.history.append(message)
state.history.extend(tool_results)
# Second call for final response
final = await client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "system", "content": self._get_system_prompt()}]
+ state.history
)
reply = final.choices[0].message.content
else:
reply = message.content
state.history.append({"role": "assistant", "content": reply})
return reply
What business metrics does the agent improve?
| Metric | Typical Contact Center | With Voice AI Agent | Improvement |
|---|---|---|---|
| Containment Rate | 20–30% | 55–65% | +35% |
| Average Handle Time (AHT) | 5–8 min | 2–3 min | -60% |
| Cost per call | ₽30–50 | ₽5–10 | -80% |
| Data collection errors | 5–8% | <2% | -75% |
Model comparison for different scenarios
| Model | Application | Latency (p95) | Containment |
|---|---|---|---|
| GPT-4o | Tech support, complex contexts | <1.5 sec | 65% |
| Mistral fine-tune | Narrow scenarios (booking, status) | <0.8 sec | 55% |
| Qwen INT4 | High-load campaigns | <0.5 sec | 50% |
Process of engagement
- Scenario analysis: collect typical dialogues, define up to 5 key scenarios (booking, order status, rescheduling, complaint, consultation).
- Dialogue design: create a state machine, define transitions and required tools.
- Implementation: write agent code, integrate telephony and CRM, configure models.
- Testing: run 100+ test calls, measure TCR, Containment, False Transfer.
- Launch: deploy to production, set up monitoring (Weights & Biases, MLflow).
What's included in the work
- Documentation of agent architecture and API
- Source code with comments (Git repository)
- Access to metrics monitoring dashboard
- Client team training (2–3 sessions of 1 hour each)
- Support for 1 month after launch
Timelines and how to start
MVP agent with basic scenarios — 3–4 weeks. Production system with monitoring — 2–3 months. Our MVP starts at ₽300,000 and a full production system from ₽1,000,000. We guarantee a minimum 30% increase in automation rate or your money back. With over 7 years in AI development and 50+ successful deployments, we bring proven E-A-T to your project. Our solutions are GDPR compliant. Get an estimate for your scenario — we'll send an implementation plan within 2 business days. Order a demonstration of the agent on your data. Contact us for a technical audit of your scenarios.