Streaming TTS: Implementation from Scratch

Note: when a voice bot responds with a 2-second pause, the user leaves. In production, we encountered projects where 300 ms latency determined conversion fate. For example, in one call center, reducing TTFA from 1.2s to 250ms improved retention by 35%. In another project, optimizing from 1.2s to 200ms saved the company $45,000 annually in operational costs. Streaming speech synthesis (Streaming TTS) is not an optimization—it's a basic necessity for real-time voice interfaces.

We implement Streaming TTS with time-to-first-audio (TTFA) from 100 ms. Below is the technical side: how chunking, buffering, and parallel synthesis work. Our experience: 5+ years in speech technologies and over 30 projects with TTS.

How streaming speech synthesis works?

Text is split into logical chunks—usually by sentences or phrases of 10–20 words. The first chunk goes directly to synthesis, while the rest are prepared in parallel. The client receives the audio stream via WebSocket or HTTP chunked encoding and starts playback immediately.

Implementation with OpenAI TTS Streaming:

from openai import AsyncOpenAI
import asyncio

client = AsyncOpenAI()

async def stream_tts(text: str):
    async with client.audio.speech.with_streaming_response.create(
        model="tts-1",
        voice="alloy",
        input=text,
        response_format="pcm",
    ) as response:
        async for chunk in response.iter_bytes(chunk_size=4096):
            yield chunk

WebSocket server for real-time TTS

For self-hosted solutions (e.g., Coqui XTTS), we use WebSocket:

from fastapi import FastAPI, WebSocket
from TTS.api import TTS
import numpy as np
import asyncio

app = FastAPI()
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")

def split_into_sentences(text: str) -> list[str]:
    import re
    sentences = re.split(r'(?<=[.!?])\s+', text)
    return [s.strip() for s in sentences if s.strip()]

@app.websocket("/tts-stream")
async def tts_websocket(websocket: WebSocket):
    await websocket.accept()
    try:
        while True:
            text = await websocket.receive_text()
            sentences = split_into_sentences(text)
            for sentence in sentences:
                wav = await asyncio.get_event_loop().run_in_executor(
                    None,
                    lambda s=sentence: tts.tts(text=s, language="en", speaker_wav="default.wav")
                )
                audio_bytes = (np.array(wav) * 32767).astype(np.int16).tobytes()
                await websocket.send_bytes(audio_bytes)
            await websocket.send_json({"type": "done"})
    except Exception:
        await websocket.close()

Why minimal latency matters?

Microsoft Research shows that latency >1s reduces user retention by 20%. For voice assistants, the critical threshold is 400 ms—beyond that, the dialogue feels unnatural. In our projects, we achieve p95 TTFA <300 ms even on self-hosted solutions.

TTFA comparison of popular engines

ElevenLabs Turbo is twice as fast in TTFA compared to Coqui XTTS on GPU.

TTFA optimization steps

How we do it

We use the optimal approach for each task:

1. Cloud APIs: OpenAI, ElevenLabs, Azure—for fast integration (2–3 days).
2. Self-hosted: Coqui XTTS, Silero on GPU—for full control and offline capabilities.
3. Hybrid: cache template phrases (greetings, hold messages), stream dynamic content.
4. Monitoring: log TTFA, latency p99, inference FLOPS—for proactive alerting.

Choosing a TTS engine

The choice depends on latency requirements, quality, and infrastructure. If minimal TTFA is needed—ElevenLabs Turbo. For custom voices and offline—Coqui XTTS. Cloud APIs (OpenAI, Azure) suit typical scenarios. We help you select the stack and optimize it for your use case.

Measuring TTFA

TTFA (Time To First Audio) is the time from text submission to the first audio packet arriving at the client. Measured as the difference between timestamps. We use built-in engine metrics and monitoring tools (Prometheus). For some projects, TTFA is critical; for others, overall dialogue latency matters.

Want to implement streaming TTS? Contact us for a preliminary assessment of your project.

What is included in the implementation

• Server-client architecture (WebSocket or HTTP Streaming)
• Integration with chosen TTS (OpenAI, ElevenLabs, self-hosted)
• Chunk and buffer optimization
• Testing on your scenario (N+ hours of conversations)
• Integration and monitoring documentation
• Stability guarantee under load (up to 1000 concurrent sessions)
• Monitoring and alerting (TTFA, p99, GPU utilization)

Estimated timelines

• Cloud TTS integration: from 2 to 5 days
• Self-hosted server with GPU: from 1 to 2 weeks
• Full implementation with monitoring and optimization: from 3 weeks

Cost is calculated individually—depends on volume and chosen stack.

Our team has 5+ years of experience in speech technologies and has implemented over 30 TTS projects. Order streaming TTS implementation—get a consultation and project assessment. Contact us—we will evaluate your project and propose the optimal solution.

Speech Recognition and Synthesis: ASR, TTS, Voice Cloning

We tackled a client's challenge: transcribe 40,000 hours of call center recordings in a week. Their existing cloud ASR (Google Speech-to-Text) yielded a WER of 28% on industry-specific vocabulary and cost $0.006 per minute — prohibitively expensive at that volume. The goal was to reduce WER below 10% and switch to self-hosted inference. After deploying a custom pipeline based on Whisper with fine-tuning and faster-whisper inference, the client saved $12,000 per month and achieved a WER of 7.3%.

How does speech recognition ASR handle noisy call center recordings?

The most common issue is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec. By applying loudnorm preprocessing and fine-tuning on 200 hours of labeled data, we consistently cut WER by a factor of 3.

Typical problems we encounter

WER does not converge to the desired metric. Often the culprit is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec.

Diarization fails with more than two speakers. pyannote/speaker-diarization-3.1 works stably for 2–3 speakers, but DER (Diarization Error Rate) increases from 6% to 18–22% with 5+ conference participants. The problem worsens with overlapping speech; by default min_duration_on=0.1 cuts short interjections. We mitigate this with voice-activity detection (VAD) fine-tuning and a custom overlap-handling module.

Voice cloning — latency vs. quality. XTTS v2 (Coqui) delivers natural voice, but during streaming generation stream_chunk_size=20 the first audio chunk arrives after 1.4–2.0 seconds — unacceptable for interactive scenarios. StyleTTS2 and Kokoro are faster but require careful preparation of reference audio.

How do we solve it in practice?

The basic stack for a production pipeline:

• ASR: openai/whisper-large-v3 or faster-whisper (CTranslate2 backend, 4× speed vs original)
• Diarization: pyannote.audio 3.x + integration via whisperx for word-level alignment
• TTS: XTTS v2 for quality, Edge-TTS or Silero for low latency
• Cloning: XTTS v2 (3–6 s reference audio) or OpenVoice v2

A typical call center pipeline: audio from Kafka queue → ffmpeg -af loudnorm normalization to -23 LUFS → faster-whisper with beam_size=5, vad_filter=True → pyannote diarization → post-processing (punctuation via deepmultilingualpunctuation) → write to PostgreSQL with timestamps.

Case study from our practice. A fintech company with 12,000 calls per day. Initial WER on Russian with banking vocabulary — 22% (Google STT). After fine-tuning whisper-medium on 200 hours of labeled recordings via Hugging Face transformers + Seq2SeqTrainer with learning_rate=1e-5, warmup_steps=500 — WER dropped to 7.3%. Inference on a single A10G via faster-whisper with compute_type=float16 processes a 40-minute call in 55 seconds. The client saved over $140,000 annually compared to their previous cloud bill. Contact us for a free pilot estimate to see similar savings on your data.

How to fine-tune Whisper on domain data?

When a general model underperforms, fine-tuning is the first tool. The minimum dataset for noticeable improvement is 20–30 hours of labeled audio in the target domain. Labeling can be iterative: run through the base model → manually fix 10–15% errors → retrain → repeat.

training_args = Seq2SeqTrainingArguments(
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=5000,
    fp16=True,
    predict_with_generate=True,
    generation_max_length=225,
)

Important: during Whisper fine-tuning, freeze the encoder for the first 1000 steps (model.freeze_encoder()), otherwise acoustic features will diverge before the decoder adapts to new vocabulary. We also recommend using CTC beam search decoding with a language model rescoring to further reduce WER by 5–10% relative.

Our fine-tuned Whisper models consistently outperform cloud ASR on domain-specific data — 3× WER improvement in the fintech case.

Speech synthesis: How to choose a model for your task?

For interactive bots requiring TTFB < 300 ms — Silero or Kokoro. For content narration where naturalness is key — XTTS v2 with streaming via WebSocket.

Our process and deliverables

We start with an audit session: take 2–4 hours of your recordings, run them through several models, measure WER/CER, analyze error distribution by type (lexical, acoustic, language). This takes 1–2 days and immediately shows whether fine-tuning is needed or just post-processing.

Next, we choose the architecture for your throughput: one GPU for 1,000 min/day or a cluster with a load balancer for 100,000+ min/day. Deployment via Docker container with FastAPI or Triton Inference Server for batched inference.

What you get after engagement:

• Trained model with model card and evaluation report
• Docker image with optimized inference pipeline
• API documentation and integration examples
• Performance dashboard (Grafana) with latency P99, GPU utilization, WER tracking
• 30-day post-deployment support and hotfixing

Timelines depend on complexity:

• Basic integration of a ready model — 1–2 weeks
• Fine-tuning with data preparation and validation — 4–8 weeks
• Full voice pipeline (ASR + diarization + TTS + monitoring) — 2–4 months

Project investments typically range from $20,000 to $80,000. Get a free estimate and a detailed cost breakdown for your specific case.

Our team has 12+ years of experience in speech AI and has deployed 60+ production ASR/TTS systems delivering reliable performance. Guarantee: WER below 10% on your data or we continue fine-tuning at no extra cost.

Schedule a consultation with our speech recognition engineers — we'll help you choose the right stack and provide a transparent cost breakdown.