A mobile neural network must be lightweight yet accurate. Knowledge Distillation is a method that compresses a large model without critical quality loss. We have applied this approach on dozens of projects: from image classification to on-device NLP. The result — a model 10 times smaller with only a 1–2% accuracy drop. This approach is especially in demand in mobile ML, where every megabyte and millisecond count. Contact us — we will audit your model and propose a distillation plan.
Why Soft Labels Work Better
Standard training: correct class = 1.0, others = 0.0. Hard labels. A teacher on an image of a cat outputs: cat 0.85, lynx 0.08, tiger 0.04, dog 0.02 ... These "soft" labels carry information that a lynx is more similar to a cat than to an airplane. A student trained on such labels learns the structure of feature space, not just a binary classifier decision. This is the essence of the student-teacher paradigm.
import torch
import torch.nn.functional as F
def distillation_loss(student_logits, teacher_logits, true_labels, temperature=4.0, alpha=0.7):
"""
alpha — weight of distillation vs hard label loss
temperature — smooths teacher distribution
"""
# Soft targets loss (KL divergence between student and teacher)
soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
soft_student = F.log_softmax(student_logits / temperature, dim=-1)
distill_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (temperature ** 2)
# Hard label loss (standard cross-entropy)
hard_loss = F.cross_entropy(student_logits, true_labels)
return alpha * distill_loss + (1 - alpha) * hard_loss
temperature ** 2 — normalizing factor that compensates for gradient scale at high temperature. Without it, distill_loss and hard_loss are at different scales.
Choosing Student Architecture
The student must be smaller than the teacher, but not arbitrarily. Good base architectures for mobile:
- MobileNetV3-Small — 2.5 MB, designed for mobile from scratch, depthwise separable convolutions
- EfficientNet-Lite0/1 — good accuracy/speed balance
- MobileViT-XXS — hybrid CNN+Transformer, 1.3 MB
- DistilBERT (for NLP) — already distilled from BERT, 66 MB vs 440 MB
For object detection on mobile: student based on YOLOv8n (8 MB) distilled from YOLOv8l (87 MB).
| Architecture | Parameters | Top-1 ImageNet (distillation) | Size (FP32) | Latency (iPhone 13) |
|---|---|---|---|---|
| MobileNetV3-Small | 2.5M | 71–72% | 10 MB | 15 ms |
| EfficientNet-Lite0 | 3.5M | 74–75% | 14 MB | 20 ms |
| MobileViT-XXS | 1.3M | 69–70% | 5.2 MB | 18 ms |
| DistilBERT (NLP) | 66M | ~97% of BERT | 264 MB | 30–80 ms |
How Distillation Methods Compare?
There are several distillation approaches, differing in complexity and outcome. Main ones: logit distillation (on outputs), intermediate layer distillation, and data-free distillation. Choice depends on data availability and accuracy requirements.
| Method | Student training | Required data | Complexity | Typical accuracy drop |
|---|---|---|---|---|
| Logit distillation | On teacher's soft labels | Full dataset | Low | 1-2% |
| Feature (intermediate) distillation | On outputs of intermediate layers | Full dataset | Medium | 0.5-1% |
| Data-free distillation | On synthetic data | None | High | 2-5% |
Distillation Steps
- Teacher preparation: fine-tune or quantize existing model, freeze weights.
- Student architecture selection: fit resource budget (RAM, CPU, latency).
- Hyperparameter tuning: temperature (usually 2-8), alpha (0.5-0.9), choice of layers for intermediate distillation.
- Student training: minimize combined loss, monitor validation accuracy.
- Conversion to Core ML/TFLite with int8 quantization for final size.
- Validation on real devices: check latency and accuracy.
Distillation Process: Example for Classification
# Assume: teacher — ResNet-50, student — MobileNetV3-Small
teacher = torchvision.models.resnet50(pretrained=True).eval()
student = torchvision.models.mobilenet_v3_small(pretrained=False)
# Freeze teacher
for param in teacher.parameters():
param.requires_grad = False
optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
for epoch in range(100):
student.train()
for images, labels in train_loader:
with torch.no_grad():
teacher_logits = teacher(images)
student_logits = student(images)
loss = distillation_loss(student_logits, teacher_logits, labels,
temperature=4.0, alpha=0.7)
optimizer.zero_grad()
loss.backward()
optimizer.step()
scheduler.step()
val_acc = evaluate(student, val_loader)
print(f"Epoch {epoch}: student_acc={val_acc:.4f}")
Typical results: MobileNetV3-Small trained normally — 67–68% top-1 on ImageNet. After distillation from ResNet-50 — 71–72%. Increase of 3–4% due to knowledge transfer.
How Does Intermediate Layer Distillation Work?
Distillation only on outputs (logits) is the basic version. A stronger version: add correspondence of intermediate feature maps.
# FitNets / PKT: student learns teacher's feature maps
class DistillationHook:
"""Hook to capture intermediate activations"""
def __init__(self):
self.output = None
def __call__(self, module, input, output):
self.output = output
teacher_hook = DistillationHook()
student_hook = DistillationHook()
# Register on corresponding layers
teacher.layer3.register_forward_hook(teacher_hook)
student.features[9].register_forward_hook(student_hook) # Analogous layer
# In training loop add feature distillation loss
with torch.no_grad():
teacher(images)
teacher_features = teacher_hook.output
student(images) # with grad
student_features = student_hook.output
# If dimensions differ — need adapter (1x1 Conv)
if teacher_features.shape[1] != student_features.shape[1]:
student_features = adapter_conv(student_features) # adapter trained jointly
feature_loss = F.mse_loss(student_features, teacher_features.detach())
This approach requires alignment of feature map dimensions between teacher and student — via 1×1 convolutional adapters. The adapter adds a few parameters to the student but remains small.
How to Distill a Model if Original Data Is Unavailable?
Sometimes the original dataset is unavailable (IP restrictions, privacy). Data-free distillation — generate synthetic data that maximizes teacher activations:
# DAFL (Data-Free Learning): generator creates "samples" for distillation
generator = Generator(latent_dim=256, img_channels=3)
optimizer_G = torch.optim.Adam(generator.parameters(), lr=1e-4)
for step in range(1000):
z = torch.randn(batch_size, 256)
fake_images = generator(z)
# Losses: maximize teacher confidence + minimize BatchNorm statistics mismatch
teacher_out = teacher(fake_images)
activation_loss = -teacher_out.max(dim=1)[0].mean() # teacher should be confident
# BN statistics matching
bn_loss = compute_bn_statistics_loss(teacher, fake_images)
total_loss = activation_loss + 0.1 * bn_loss
optimizer_G.zero_grad()
total_loss.backward()
optimizer_G.step()
The quality of data-free distillation is lower than the full-data version, but sometimes it is the only option.
Distillation for NLP Tasks on Mobile
For mobile apps with NLP (sentiment classification, intent detection, summarization): distill from GPT-4 / Claude API responses into a small BERT/DistilBERT.
# Collect soft labels from teacher (GPT-4 API)
# For each training example request class probabilities
# Store as training labels for student
# Student — DistilBERT fine-tuned on these soft labels
DistilBERT (66 MB, ONNX int8 — 18 MB) runs on-device in 30–80 ms on iOS/Android. GPT-4 in the cloud — hundreds of ms, costs per request.
What's Included in Our Work
- Resource budget analysis: RAM, CPU, GPU, power consumption
- Student architecture selection based on your constraints
- Teacher preparation (fine-tuning, quantization)
- Distillation with hyperparameter tuning (temperature, alpha, intermediate layers)
- Validation on test set and on real devices
- Conversion to Core ML / TFLite with int8 quantization
- Integration and support (updates for new OS versions)
Our team has 10+ years of experience in mobile development and ML. We guarantee the model will run stably on iOS 14+ and Android 10+. Get a free engineer consultation on your model. Order distillation — we'll prepare a preliminary estimate in 1 day.
Timeline Estimates
Basic logit distillation for a classification task — 2–4 weeks (GPU time plus hyperparameter tuning). Full distillation with intermediate layers, non-standard architectures, data augmentation — 5–10 weeks. Custom estimate upon request.
Further reading: Knowledge Distillation on Wikipedia, PyTorch tutorial.