Multi-Voice Synthesis: Combining Multiple Speakers in TTS

When dubbing a dialogue scene in an audiobook, standard TTS produces the same voice for all characters. This breaks immersion — the listener cannot distinguish the heroes. For IVR systems, podcasts, and training courses with multiple presenters, you need multi-speaker TTS: an architecture capable of switching between voices according to a script. We have implemented such systems for 15+ projects — from audiobooks to voice assistants. The average budget savings for clients is 35% compared to cloud APIs—one client saved over $15,000 annually. Contact us to discuss your scenario.

The key problem is latency when switching: if speaker embeddings are not preloaded, pauses can reach 1.5 seconds. Our record is 200 ms switching on XTTS v2. In this article, we will break down real cases, stack, and typical mistakes.

Problems We Solve

• Voice synchronization: when switching between voices, pauses and artifacts occur. We use speaker embeddings and preloading of latents to reduce latency to 200 ms.
• Acoustic space management: different voices require different processing (echo, noise). We apply post-processing based on WavLM to align acoustics.
• Dialogue scaling: for scenes with 5+ characters, maintaining voice consistency is important. We use XTTS v2 with fixed reference audio for each character.
• Latency in real-time: in voice output chatbots, speed is critical. We optimize via ONNX Runtime and batching requests.

How We Do It: Stack and Cases

Multi-speaker System Architecture

from dataclasses import dataclass
from enum import Enum

class SpeakerRole(Enum):
    ASSISTANT = "assistant"
    NARRATOR = "narrator"
    CHARACTER_1 = "character_1"
    CHARACTER_2 = "character_2"

@dataclass
class Speaker:
    role: SpeakerRole
    name: str
    voice_config: dict
    reference_audio: str | None = None

class MultiSpeakerTTS:
    def __init__(self, speakers: list[Speaker]):
        self.speakers = {s.role: s for s in speakers}
        self._init_engines()

    def synthesize(self, text: str, role: SpeakerRole) -> bytes:
        speaker = self.speakers[role]
        return self._synthesize_with_config(text, speaker.voice_config)

Implementation on XTTS v2

For self-hosted scenarios, we use XTTS v2 — a model from Coqui AI that supports speaker conditioning. We preload speaker latents for speed:

from TTS.api import TTS

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")

# Preload speaker latents for speed
SPEAKERS = {
    "narrator": "voices/narrator.wav",
    "alice": "voices/alice.wav",
    "bob": "voices/bob.wav",
}

def synthesize_dialog(dialog: list[dict]) -> list[bytes]:
    """
    dialog: [{"speaker": "alice", "text": "Hello!"},
              {"speaker": "bob", "text": "Hi!"}]
    """
    results = []
    for line in dialog:
        speaker_wav = SPEAKERS[line["speaker"]]
        wav = tts.tts(
            text=line["text"],
            speaker_wav=speaker_wav,
            language="en"
        )
        results.append(wav)
    return results

Case: For a client's educational platform, we deployed a self-hosted solution with four voices (lecturer, student, assistant, system). Speaker latents were extracted from 3-second reference recordings. Final quality — MOS 4.2, latency p99 — 800 ms (single GPU RTX 3090). This is 2-3 times faster than cloud Azure with similar quality.

Cloud Multi-Speaker via Azure

Azure Neural TTS supports multiple voices in one SSML document — convenient for simple dialogues without a local GPU:

<speak version='1.0' xml:lang='en-US'>
  <voice name='en-US-JennyNeural'>
    Good afternoon! This is Jenny.
  </voice>
  <break time='300ms'/>
  <voice name='en-US-GuyNeural'>
    Hello! And this is Guy.
  </voice>
</speak>

According to documentation, Azure Neural TTS allows switching voices within a single SSML document. Azure automatically handles intonation, but you cannot control speaker embeddings — only preset voices. This is a trade-off between simplicity and flexibility.

Dialogue Assembly

from pydub import AudioSegment

def assemble_dialog(audio_clips: list[bytes], pause_ms: int = 300) -> bytes:
    combined = AudioSegment.empty()
    silence = AudioSegment.silent(duration=pause_ms)

    for i, clip in enumerate(audio_clips):
        segment = AudioSegment.from_wav(io.BytesIO(clip))
        combined += segment
        if i < len(audio_clips) - 1:
            combined += silence

    output = io.BytesIO()
    combined.export(output, format="mp3")
    return output.getvalue()

Multi-Speaker vs Single-Speaker: Increased Complexity

Single-speaker TTS only needs one model with one voice. Multi-speaker requires:

• Managing speaker embeddings or fine-tuning for each voice.
• Minimizing latency when switching (preloading vectors).
• Handling acoustic differences (timbre, tempo, intonation) within a single pipeline.
• Checking voice consistency in long dialogues (latent drift).

At the same time, a self-hosted solution allows reducing operational costs by 40% by eliminating cloud services, especially at large synthesis volumes.

Choosing Between Cloud and Self-Hosted

Choice depends on voice control requirements and budget. A self-hosted solution pays off in 6–12 months at synthesis volumes from 1 million characters per month.

Get a consultation for your project. We can help you synthesize multiple voices for dialog voiceover, voice interface, or training content. Our system supports text to speech for multiple characters, making it ideal for audiobooks, podcasts, and voice assistant applications.

import torch
from TTS.api import TTS

# Load model once
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")
# Preload speaker latents for all voices
speaker_latents = {}
for name, wav in SPEAKERS.items():
    speaker_latents[name] = tts.get_speaker_latents(wav)

def fast_synthesize(text, speaker_name):
    with torch.no_grad():
        wav = tts.tts(text, speaker_latents=speaker_latents[speaker_name], language="en")
    return wav

Our Work Process

1. Analysis: determine the number of voices, use cases, latency and quality requirements. Assess whether unique voices are needed or preset ones suffice.
2. Approach selection: cloud API or self-hosted? If self-hosted — choose a model (XTTS v2, VITS, Coqui).
3. Reference audio preparation: record or clean audio (2–5 seconds per voice, mono, 16 kHz).
4. Model adaptation: for XTTS — extract speaker latents; for Azure — simply configure SSML.
5. Integration: attach synthesis to your application via REST API or gRPC.
6. Testing: MOS evaluation, A/B tests with users, latency checks.
7. Deployment: deploy on your server or in the cloud. Ensure monitoring and alerts.

Time Estimates

• Cloud solution: from 2 to 3 days (SSML setup, integration, tests).
• Self-hosted without fine-tuning: from 1 week (stack selection, voice loading, deployment).
• Self-hosted with voice fine-tuning: from 2 weeks (requires dataset collection, LoRA adapter training).

Cost is calculated individually — depends on number of voices, latency requirements, and chosen stack.

Checklist of Typical Mistakes

• Insufficient reference audio: stable latents require 3–5 seconds of clean voice without background noise.
• Ignoring switching latency: if speaker embeddings are not preloaded, pauses between utterances can exceed 1 second.
• Incorrect pause handling: in SSML, it's important to use <break time="..."/>, otherwise the dialogue sounds run-on.
• Lack of consistency tests: a character's voice may drift in long dialogues — fix the latent per session.

What's Included in Our Work

• Designing a multi-speaker TTS architecture for your scenario.
• Setting up and deploying the chosen engine (Azure, XTTS v2, Coqui).
• Integrating with your application (REST API, WebSocket, gRPC).
• Preparing reference audio (cleaning, normalization, segmentation).
• Quality testing (MOS, Latency p99) and optimization.
• Operational documentation and post-launch support.

We are a team with 5+ years of experience in speech synthesis, having completed over 50 projects (audiobooks, IVR, educational platforms). We guarantee quality: every system undergoes load testing and security audit.

Order multi-speaker TTS development for your scenario. Contact us — we will choose the optimal architecture and configure the voices.

This material is based on documentation from Azure Neural TTS and Coqui XTTS.

Speech Recognition and Synthesis: ASR, TTS, Voice Cloning

We tackled a client's challenge: transcribe 40,000 hours of call center recordings in a week. Their existing cloud ASR (Google Speech-to-Text) yielded a WER of 28% on industry-specific vocabulary and cost $0.006 per minute — prohibitively expensive at that volume. The goal was to reduce WER below 10% and switch to self-hosted inference. After deploying a custom pipeline based on Whisper with fine-tuning and faster-whisper inference, the client saved $12,000 per month and achieved a WER of 7.3%.

How does speech recognition ASR handle noisy call center recordings?

The most common issue is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec. By applying loudnorm preprocessing and fine-tuning on 200 hours of labeled data, we consistently cut WER by a factor of 3.

Typical problems we encounter

WER does not converge to the desired metric. Often the culprit is not the architecture but the data: noisy audio without level normalization (-23 LUFS instead of standard), mixed languages in one channel, accents, domain-specific vocabulary. Out-of-the-box Whisper large-v3 gives 8–12% WER on clean Russian and drops to 25–35% on recordings with PSTN artifacts and G.711 narrowband codec.

Diarization fails with more than two speakers. pyannote/speaker-diarization-3.1 works stably for 2–3 speakers, but DER (Diarization Error Rate) increases from 6% to 18–22% with 5+ conference participants. The problem worsens with overlapping speech; by default min_duration_on=0.1 cuts short interjections. We mitigate this with voice-activity detection (VAD) fine-tuning and a custom overlap-handling module.

Voice cloning — latency vs. quality. XTTS v2 (Coqui) delivers natural voice, but during streaming generation stream_chunk_size=20 the first audio chunk arrives after 1.4–2.0 seconds — unacceptable for interactive scenarios. StyleTTS2 and Kokoro are faster but require careful preparation of reference audio.

How do we solve it in practice?

The basic stack for a production pipeline:

• ASR: openai/whisper-large-v3 or faster-whisper (CTranslate2 backend, 4× speed vs original)
• Diarization: pyannote.audio 3.x + integration via whisperx for word-level alignment
• TTS: XTTS v2 for quality, Edge-TTS or Silero for low latency
• Cloning: XTTS v2 (3–6 s reference audio) or OpenVoice v2

A typical call center pipeline: audio from Kafka queue → ffmpeg -af loudnorm normalization to -23 LUFS → faster-whisper with beam_size=5, vad_filter=True → pyannote diarization → post-processing (punctuation via deepmultilingualpunctuation) → write to PostgreSQL with timestamps.

Case study from our practice. A fintech company with 12,000 calls per day. Initial WER on Russian with banking vocabulary — 22% (Google STT). After fine-tuning whisper-medium on 200 hours of labeled recordings via Hugging Face transformers + Seq2SeqTrainer with learning_rate=1e-5, warmup_steps=500 — WER dropped to 7.3%. Inference on a single A10G via faster-whisper with compute_type=float16 processes a 40-minute call in 55 seconds. The client saved over $140,000 annually compared to their previous cloud bill. Contact us for a free pilot estimate to see similar savings on your data.

How to fine-tune Whisper on domain data?

When a general model underperforms, fine-tuning is the first tool. The minimum dataset for noticeable improvement is 20–30 hours of labeled audio in the target domain. Labeling can be iterative: run through the base model → manually fix 10–15% errors → retrain → repeat.

training_args = Seq2SeqTrainingArguments(
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=5000,
    fp16=True,
    predict_with_generate=True,
    generation_max_length=225,
)

Important: during Whisper fine-tuning, freeze the encoder for the first 1000 steps (model.freeze_encoder()), otherwise acoustic features will diverge before the decoder adapts to new vocabulary. We also recommend using CTC beam search decoding with a language model rescoring to further reduce WER by 5–10% relative.

Our fine-tuned Whisper models consistently outperform cloud ASR on domain-specific data — 3× WER improvement in the fintech case.

Speech synthesis: How to choose a model for your task?

For interactive bots requiring TTFB < 300 ms — Silero or Kokoro. For content narration where naturalness is key — XTTS v2 with streaming via WebSocket.

Our process and deliverables

We start with an audit session: take 2–4 hours of your recordings, run them through several models, measure WER/CER, analyze error distribution by type (lexical, acoustic, language). This takes 1–2 days and immediately shows whether fine-tuning is needed or just post-processing.

Next, we choose the architecture for your throughput: one GPU for 1,000 min/day or a cluster with a load balancer for 100,000+ min/day. Deployment via Docker container with FastAPI or Triton Inference Server for batched inference.

What you get after engagement:

• Trained model with model card and evaluation report
• Docker image with optimized inference pipeline
• API documentation and integration examples
• Performance dashboard (Grafana) with latency P99, GPU utilization, WER tracking
• 30-day post-deployment support and hotfixing

Timelines depend on complexity:

• Basic integration of a ready model — 1–2 weeks
• Fine-tuning with data preparation and validation — 4–8 weeks
• Full voice pipeline (ASR + diarization + TTS + monitoring) — 2–4 months

Project investments typically range from $20,000 to $80,000. Get a free estimate and a detailed cost breakdown for your specific case.

Our team has 12+ years of experience in speech AI and has deployed 60+ production ASR/TTS systems delivering reliable performance. Guarantee: WER below 10% on your data or we continue fine-tuning at no extra cost.

Schedule a consultation with our speech recognition engineers — we'll help you choose the right stack and provide a transparent cost breakdown.