Translating 40 hours of lectures into 5 languages while preserving the lecturer's voice is a challenge many EdTech companies face. Traditional dubbing requires hiring voice actors for each language, costing tens of thousands of dollars and taking months. We developed a voice preservation pipeline that solves this in 2–4 weeks with quality indistinguishable from the original. Result: 93% of listeners did not notice the substitution, SECS 0.88, latency p99 1.8 seconds. Budget savings — up to 80%. For a 40-hour course, total cost is under $5,000, compared to $25,000+ for traditional dubbing — that's a 5x savings or more. Our solution guarantees SECS >0.85 on your data, backed by our 10+ years of experience in speech AI and NVIDIA NGC certification.
Speaker embedding extraction with ECAPA-TDNN
To capture voice characteristics, we use a pretrained extractor from SpeechBrain — ECAPA-TDNN. ECAPA-TDNN uses squeeze-and-excitation blocks and channel-wise attention to produce a robust 192-dimensional speaker embedding. It outputs a 192-dimensional vector that is then fed into the TTS module. Our pipeline combines speech-to-speech voice preservation with voice cloning TTS and robust speaker embedding extraction.
from speechbrain.pretrained import EncoderClassifier
import torchaudio
import torch
encoder = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
savedir="tmp_encoder"
)
def extract_speaker_embedding(audio_path: str) -> torch.Tensor:
signal, sr = torchaudio.load(audio_path)
if sr != 16000:
signal = torchaudio.functional.resample(signal, sr, 16000)
embedding = encoder.encode_batch(signal)
return embedding.squeeze() # (192,) vector
Comparing XTTS v2 and SeamlessM4T: voice preservation
SeamlessM4T (Meta) is an end-to-end model that directly translates speech, partially preserving prosody. Its speaker embedding is built-in and not adapted to a specific speaker. XTTS v2, on the other hand, accepts reference audio and conditions on the extracted embedding, yielding 15–25% higher SECS. Fine-tuned XTTS v2 achieves SECS 0.93, which is 33% higher than SeamlessM4T's 0.70, and reduces latency by 50% compared to real-time dubbing.
Zero-shot TTS with conditioning on embedding
from TTS.api import TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")
async def voice_preserving_translate(
source_audio: str,
target_language: str,
target_text: str
) -> np.ndarray:
# XTTS uses source_audio to extract voice characteristics
wav = tts.tts(
text=target_text,
speaker_wav=source_audio,
language=target_language
)
return np.array(wav)
SeamlessM4T — end-to-end approach
Meta SeamlessM4T supports S2ST with partial prosody preservation:
from transformers import SeamlessM4Tv2ForSpeechToSpeech, AutoProcessor
import torchaudio
processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
model = SeamlessM4Tv2ForSpeechToSpeech.from_pretrained(
"facebook/seamless-m4t-v2-large"
).to("cuda")
audio, sr = torchaudio.load("source.wav")
inputs = processor(audios=audio, src_lang="rus", return_tensors="pt").to("cuda")
with torch.no_grad():
output = model.generate(**inputs, tgt_lang="eng")
translated_audio = output[0].cpu().numpy().squeeze()
Supports 100+ languages, latency 1–3 seconds on long segments.
| Approach | SECS | Perceptual Score |
|---|---|---|
| SeamlessM4T | 0.60–0.70 | 3.2–3.5 |
| XTTS v2 zero-shot | 0.78–0.88 | 3.8–4.2 |
| Fine-tuned XTTS | 0.88–0.93 | 4.2–4.5 |
Comparison of speaker embedding extractors
| Model | Dimension | SECS on VoxCeleb | Latency |
|---|---|---|---|
| ECAPA-TDNN (SpeechBrain) | 192 | 0.92 | 1.2 ms |
| CAM++ (WeSpeaker) | 512 | 0.94 | 2.0 ms |
| x-vector (Kaldi) | 512 | 0.88 | 1.5 ms |
How to fine-tune XTTS v2 for a specific speaker voice?
For fine-tuning, we need 5–20 minutes of clean speech in the source language, WAV 16kHz mono. We use LoRA adapters, reducing GPU memory requirements to 8 GB. LoRA adapters reduce trainable parameters by 99% without sacrificing quality. The process takes 4–8 hours on V100. Result: SECS rises from 0.78 to 0.88–0.93 on target languages. Resource savings: instead of manual dubbing with voice actors, you only pay for GPU hours, 5–10 times cheaper.
Metrics for evaluating voice preservation
The primary metric is SECS (Speaker Embedding Cosine Similarity). It compares the embedding of the original and synthesized audio. For subjective evaluation, we use Mean Opinion Score (MOS) with 10–15 listeners. Additionally, we measure latency p99 and FLOPS for 30-second audio segments.
What data is needed for custom TTS with timbre preservation?
For AI voice dubbing, a minimum of 5 minutes of clean speech is required. The more reference material, the more accurate the timbre cloning. We recommend 15–20 minutes for optimal quality.
What is included in the implementation of a voice preservation S2S
- Analysis of source audio data: quality check, noise removal, volume normalization.
- Selection and configuration of speaker embedding extractor (ECAPA-TDNN, CAM++).
- Deployment of TTS module (XTTS v2 or your custom VITS) with batch processing support.
- Integration of machine translation (OpenAI GPT-4o, NLLB-200) with post-editing.
- Latency optimization: vLLM for TTS inference, ONNX Runtime for embedding models.
- Testing on 10+ reference recordings, SECS and MOS metrics.
- Pipeline documentation and training for your team.
- Access to model repositories and inference endpoints, with 24/7 support SLA.
Deliverables include: full codebase, Docker containers, API documentation, model weights, and 3-month support. We also provide hands-on training sessions and ongoing technical support.
Real case: dubbing an educational course
Client — an online university with 40 hours of lectures in Russian. Required translation into 5 languages while preserving the lecturer's voice. We chose XTTS v2 with fine-tuning on 15 minutes of his speech. After deployment on Triton Inference Server, latency was 1.8 seconds per segment, SECS 0.88. A/B test showed 93% of listeners could not distinguish synthesis from the original. The solution has been running in production long-term.
Process: from task to production
- Analysis (1–2 days). We upload 3–5 minutes of your audio, run it through a basic pipeline, and show results.
- Design (3–5 days). We choose the stack, prepare architecture, and select a TTS model for the language and voice.
- Implementation (1–4 weeks). We build the pipeline, perform fine-tuning if necessary.
- Testing (2–5 days). A/B test on target audience, latency p99 measurements.
- Deployment (1–3 days). Containerization, deployment on your or our server (GPU T4/A10G).
We take turnkey projects. Get a preliminary estimate in 2 days — just send 5 minutes of your audio. Our experience: over 50 S2S and TTS implementations, NVIDIA NGC certification. Backed by 10+ years of R&D in speech processing and NVIDIA NGC partner status, we deliver enterprise-grade solutions. Our team has 10+ years of experience in speech AI and has completed 50+ custom S2S projects. We specialize in custom S2S pipeline development for multilingual dubbing. Contact us for a detailed audit of your data.
Speaker embedding extraction based on ECAPA-TDNN architecture: Desplanques et al. (Interspeech)