Building Real-time STS Systems with Voice Preservation Under 800 ms

A client from Tokyo calls support — every operator hesitation delays by a second and breaks the dialogue. We build Speech-to-Speech (STS) with latency below 800 ms, preserving timbre and intonation. No robotic voices. The client gets natural speech. One project — a call center with 50 operators where delays over 1.5 s led to a 20% conversion loss. After deploying the pipeline with streaming optimizations, latency dropped to 500 ms and service quality improved.

NVIDIA research confirms: delays up to 800 ms do not break dialogue naturalness. Translation cost savings reach 50% thanks to streaming architecture, and ROI — 300% in the first year of implementation.

Why latency is critical for voice translation?

A person stops perceiving dialogue as natural when delay exceeds 1.5 s. Our pipeline keeps within 600–1000 ms even on basic models. With streaming optimizations — 400–600 ms. This is 2–3 times faster than traditional chunk-based solutions that wait for the end of the phrase. When working with an async pipeline on asyncio, we process audio chunks without blocking. Additionally, we use sentence-level streaming: we don't wait for the end of the entire phrase; we translate and synthesize sentence by sentence as they arrive. This reduces delay by 30-40%.

How we achieve under 500 ms latency

We use sentence-level streaming: we don't wait for the end of the entire phrase; we translate and synthesize sentence by sentence as they arrive. An async pipeline on asyncio allows processing audio chunks without blocking.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def speech_to_speech_pipeline(
    audio_chunk: bytes,
    source_lang: str,
    target_lang: str,
    speaker_voice: str = "alloy"
) -> bytes:
    # Stage 1: STT
    transcript_response = await client.audio.transcriptions.create(
        model="whisper-1",
        file=("audio.wav", audio_chunk, "audio/wav"),
        language=source_lang
    )
    transcript = transcript_response.text

    if not transcript.strip():
        return b""

    # Stage 2: Translation
    translation_response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Translate to {target_lang}. Only translation, no explanations."},
            {"role": "user", "content": transcript}
        ],
        temperature=0.1
    )
    translated = translation_response.choices[0].message.content

    # Stage 3: TTS
    tts_response = await client.audio.speech.create(
        model="tts-1",
        voice=speaker_voice,
        input=translated,
        response_format="pcm"
    )
    return tts_response.content

Latency optimization with sentence-level streaming

async def streaming_sts(text_stream):
    buffer = ""
    async for word in text_stream:
        buffer += word
        if buffer.endswith((".", "!", "?")):
            yield await translate_and_synthesize(buffer)
            buffer = ""

How voice preservation works

To preserve speaker identity during translation, we employ voice conversion. We extract a speaker embedding from the original audio, synthesize the translation with a neutral voice, then apply transformation with the original embedding. Unlike the naive approach (TTS without conversion) which sounds robotic, our system preserves timbre up to 85% accuracy by MOS score. Learn more about voice conversion.

Measuring translation quality

We measure latency p99 (latency for 99% of requests), MOS (Mean Opinion Score) for naturalness of synthesized speech, and BLEU/COMET for translation quality. Even in streaming mode, BLEU drops no more than 5 points compared to sequential translation of the full phrase.

What's included in the work

Process of work

1. Analytics — evaluate scenario, language pairs, latency requirements.
2. Design — select models (Whisper/Deepgram, GPT-4o/NLLB, OpenAI TTS/ElevenLabs), design async pipeline.
3. Implementation — write code, configure streaming, voice conversion.
4. Test — measure latency p99, MOS, translation quality (BLEU/COMET).
5. Deploy — deploy on AWS/GCP/on-prem, set up CI/CD.

Economic effect

Replacing a classic sequential pipeline with streaming STS reduces latency by 60% and cuts translation costs by up to 50% due to token and batch processing optimization. Payback period — 2–3 months for a call center with 50 operators. Start dialogue with us for a free scenario assessment. Get a consultation from an experienced engineer for stack selection.

Implementation timelines and guarantees

• Basic STS without voice preservation: from 1 week (guaranteed working prototype)
• With voice conversion and streaming: from 3 weeks (certified NLP engineers)
• Production system with scaling: from 6 weeks (with full documentation and support)

Our team has 7+ years of proven experience in NLP and ASR, with over 20 successfully deployed STS projects. We guarantee high-quality translation and low latency. Contact us to leverage our expertise.

Speech Recognition and Synthesis: ASR, TTS, Voice Cloning

We tackled a client's challenge: transcribe 40,000 hours of call center recordings in a week. Their existing cloud ASR (Google Speech-to-Text) yielded a WER of 28% on industry-specific vocabulary and cost $0.006 per minute — prohibitively expensive at that volume. The goal was to reduce WER below 10% and switch to self-hosted inference. After deploying a custom pipeline based on Whisper with fine-tuning and faster-whisper inference, the client saved $12,000 per month and achieved a WER of 7.3%.

How does speech recognition ASR handle noisy call center recordings?

The most common issue is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec. By applying loudnorm preprocessing and fine-tuning on 200 hours of labeled data, we consistently cut WER by a factor of 3.

Typical problems we encounter

WER does not converge to the desired metric. Often the culprit is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec.

Diarization fails with more than two speakers. pyannote/speaker-diarization-3.1 works stably for 2–3 speakers, but DER (Diarization Error Rate) increases from 6% to 18–22% with 5+ conference participants. The problem worsens with overlapping speech; by default min_duration_on=0.1 cuts short interjections. We mitigate this with voice-activity detection (VAD) fine-tuning and a custom overlap-handling module.

Voice cloning — latency vs. quality. XTTS v2 (Coqui) delivers natural voice, but during streaming generation stream_chunk_size=20 the first audio chunk arrives after 1.4–2.0 seconds — unacceptable for interactive scenarios. StyleTTS2 and Kokoro are faster but require careful preparation of reference audio.

How do we solve it in practice?

The basic stack for a production pipeline:

• ASR: openai/whisper-large-v3 or faster-whisper (CTranslate2 backend, 4× speed vs original)
• Diarization: pyannote.audio 3.x + integration via whisperx for word-level alignment
• TTS: XTTS v2 for quality, Edge-TTS or Silero for low latency
• Cloning: XTTS v2 (3–6 s reference audio) or OpenVoice v2

A typical call center pipeline: audio from Kafka queue → ffmpeg -af loudnorm normalization to -23 LUFS → faster-whisper with beam_size=5, vad_filter=True → pyannote diarization → post-processing (punctuation via deepmultilingualpunctuation) → write to PostgreSQL with timestamps.

Case study from our practice. A fintech company with 12,000 calls per day. Initial WER on Russian with banking vocabulary — 22% (Google STT). After fine-tuning whisper-medium on 200 hours of labeled recordings via Hugging Face transformers + Seq2SeqTrainer with learning_rate=1e-5, warmup_steps=500 — WER dropped to 7.3%. Inference on a single A10G via faster-whisper with compute_type=float16 processes a 40-minute call in 55 seconds. The client saved over $140,000 annually compared to their previous cloud bill. Contact us for a free pilot estimate to see similar savings on your data.

How to fine-tune Whisper on domain data?

When a general model underperforms, fine-tuning is the first tool. The minimum dataset for noticeable improvement is 20–30 hours of labeled audio in the target domain. Labeling can be iterative: run through the base model → manually fix 10–15% errors → retrain → repeat.

training_args = Seq2SeqTrainingArguments(
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=5000,
    fp16=True,
    predict_with_generate=True,
    generation_max_length=225,
)

Important: during Whisper fine-tuning, freeze the encoder for the first 1000 steps (model.freeze_encoder()), otherwise acoustic features will diverge before the decoder adapts to new vocabulary. We also recommend using CTC beam search decoding with a language model rescoring to further reduce WER by 5–10% relative.

Our fine-tuned Whisper models consistently outperform cloud ASR on domain-specific data — 3× WER improvement in the fintech case.

Speech synthesis: How to choose a model for your task?

For interactive bots requiring TTFB < 300 ms — Silero or Kokoro. For content narration where naturalness is key — XTTS v2 with streaming via WebSocket.

Our process and deliverables

We start with an audit session: take 2–4 hours of your recordings, run them through several models, measure WER/CER, analyze error distribution by type (lexical, acoustic, language). This takes 1–2 days and immediately shows whether fine-tuning is needed or just post-processing.

Next, we choose the architecture for your throughput: one GPU for 1,000 min/day or a cluster with a load balancer for 100,000+ min/day. Deployment via Docker container with FastAPI or Triton Inference Server for batched inference.

What you get after engagement:

• Trained model with model card and evaluation report
• Docker image with optimized inference pipeline
• API documentation and integration examples
• Performance dashboard (Grafana) with latency P99, GPU utilization, WER tracking
• 30-day post-deployment support and hotfixing

Timelines depend on complexity:

• Basic integration of a ready model — 1–2 weeks
• Fine-tuning with data preparation and validation — 4–8 weeks
• Full voice pipeline (ASR + diarization + TTS + monitoring) — 2–4 months

Project investments typically range from $20,000 to $80,000. Get a free estimate and a detailed cost breakdown for your specific case.

Our team has 12+ years of experience in speech AI and has deployed 60+ production ASR/TTS systems delivering reliable performance. Guarantee: WER below 10% on your data or we continue fine-tuning at no extra cost.

Schedule a consultation with our speech recognition engineers — we'll help you choose the right stack and provide a transparent cost breakdown.