Fine-tuning Whisper for Domain-Specific Speech Recognition
Your baseline Whisper large-v3 achieves 8–15% WER on general speech. But on specialized vocabulary in medicine, law, finance, or technical jargon, that number jumps to 25–40%. Fine-tuning on a domain dataset drops it to 3–8%. We fine-tune Whisper for your specific needs—from medical dictations to call center calls. With over 20 completed projects and 5+ years in speech recognition, we guarantee at least a 2× reduction in WER. Let's evaluate your project—contact us for a free initial assessment.
Why Fine-tune Whisper?
Cloud services like Google Speech-to-Text and Azure Speech offer generic models that don't adapt to narrow terminology. Fine-tuning on your own data delivers up to a 5× WER reduction compared to these APIs. For example, on medical dictations, fine-tuned Whisper achieves 6.2% WER versus 28% for Google Speech-to-Text—a 4.5× improvement. Plus, your audio stays on your own infrastructure—no risk of confidential data leaks.
| Solution | WER on Medical Dictations | Improvement vs. Baseline Whisper |
|---|---|---|
| Google Speech-to-Text | 28% | — |
| Azure Speech | 25% | — |
| Whisper large-v3 (base) | 31% | — |
| Whisper fine-tuned (ours) | 6.2% | 5× |
Baseline Whisper struggles with rare terminology ("hepatosplenomegaly", "force majeure"), regional accents, noisy environments, and code-switching. Fine-tuning addresses all these.
Technical Implementation
Preparing a Domain Dataset
Minimum useful size is 10–20 hours; optimal is 50–100 hours. Use augmentation: street noise, reverberation, pitch change. Example preparation in Python:
from datasets import Dataset, Audio
import pandas as pd
def prepare_whisper_dataset(
audio_dir: str,
transcripts_csv: str,
target_language: str = "russian"
) -> Dataset:
"""
transcripts_csv: columns = [audio_file, transcription]
"""
df = pd.read_csv(transcripts_csv)
dataset = Dataset.from_dict({
"audio": [f"{audio_dir}/{f}" for f in df["audio_file"]],
"sentence": df["transcription"].tolist()
})
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
return dataset
Preprocessing with Whisper's feature extractor:
from transformers import WhisperFeatureExtractor, WhisperTokenizer
feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3")
tokenizer = WhisperTokenizer.from_pretrained(
"openai/whisper-large-v3",
language="Russian",
task="transcribe"
)
def prepare_dataset(batch):
audio = batch["audio"]
batch["input_features"] = feature_extractor(
audio["array"],
sampling_rate=audio["sampling_rate"]
).input_features[0]
batch["labels"] = tokenizer(batch["sentence"]).input_ids
return batch
dataset = dataset.map(prepare_dataset, remove_columns=["audio", "sentence"])
Fine-tuning with Seq2SeqTrainer (including LoRA)
[Full training code shown below. For GPU memory below 24 GB, we use PEFT/LoRA to train only 1% of parameters.]
from transformers import (
WhisperForConditionalGeneration,
Seq2SeqTrainingArguments,
Seq2SeqTrainer
)
from peft import get_peft_model, LoraConfig, TaskType
import evaluate
import torch
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
model.generation_config.language = "russian"
model.generation_config.task = "transcribe"
model.generation_config.forced_decoder_ids = None
# Apply LoRA if needed
lora_config = LoraConfig(
r=32,
lora_alpha=64,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.SEQ_2_SEQ_LM
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 15,728,640 || all params: 1,557,741,568 (1.01%)
training_args = Seq2SeqTrainingArguments(
output_dir="./whisper-finetuned-ru-medical",
per_device_train_batch_size=16,
gradient_accumulation_steps=1,
learning_rate=1e-5,
warmup_steps=500,
max_steps=4000,
gradient_checkpointing=True,
fp16=True,
evaluation_strategy="steps",
per_device_eval_batch_size=8,
predict_with_generate=True,
generation_max_length=225,
save_steps=500,
eval_steps=500,
logging_steps=25,
report_to=["tensorboard"],
load_best_model_at_end=True,
metric_for_best_model="wer",
greater_is_better=False,
push_to_hub=False
)
wer_metric = evaluate.load("wer")
def compute_metrics(pred):
pred_ids = pred.predictions
label_ids = pred.label_ids
label_ids[label_ids == -100] = tokenizer.pad_token_id
pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True)
wer = 100 * wer_metric.compute(predictions=pred_str, references=label_str)
return {"wer": wer}
trainer = Seq2SeqTrainer(
args=training_args,
model=model,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
data_collator=data_collator,
compute_metrics=compute_metrics,
tokenizer=feature_extractor
)
trainer.train()
Results and Usage
Domain Results
| Domain | Baseline WER | After Fine-tuning | Improvement |
|---|---|---|---|
| Medical dictations | 31% | 6.2% | 5× |
| Legal contracts | 24% | 4.8% | 5× |
| Call center (finance) | 18% | 5.1% | 3.5× |
| Technical support | 22% | 7.3% | 3× |
Fine-tuning Whisper delivers a 3–6× improvement over the base model, outperforming cloud APIs that don't adapt to your domain vocabulary.
Inference with Fine-tuned Model
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="./whisper-finetuned-ru-medical",
device=0,
torch_dtype=torch.float16,
generate_kwargs={
"language": "russian",
"task": "transcribe",
"num_beams": 5
}
)
result = asr("patient_recording.wav", chunk_length_s=30, stride_length_s=5)
print(result["text"])
Our Service
We provide the full cycle:
- Audit of your audio data and target WER estimation
- Corpus transcription and augmentation (noise, reverberation, speed perturbation)
- Fine-tuning of the base model (Whisper large-v3, openai/whisper-medium, or distill-whisper)
- Experiments with LoRA, quantization, encoder architecture
- Testing on a held-out set and A/B comparison
- Model export to ONNX or TensorRT for inference
- Integration into your pipeline (REST API, Docker, batch processing)
- Documentation and training for your team
Process: Analysis → Design → Transcription → Training → Testing → Deployment
Timeline and Cost: Estimated timelines: from 2 weeks (for a 10-hour corpus) to 6 weeks (for 100+ hours with complex transcription). Typical budget starts at $5,000, with exact cost depending on data volume, target WER, and latency requirements. Get a consultation on Whisper fine-tuning for your task—we'll analyze the problem and find the best solution.
How to Get a Consultation?
Write to us via email or messengers—we'll analyze your task and select the most effective approach. We guarantee confidentiality and will sign an NDA.
More about metrics
In addition to WER, we use Character Error Rate (CER) for character-level accuracy and confidence scores to identify problematic segments. Phonetic analysis is performed when needed.
Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", 2022