ML Edge Optimization: 4-8x Compression Techniques

A typical scenario: a model trained on an 80 GB GPU won't run on a Raspberry Pi — latency in seconds, OOM at every inference. Optimization for edge devices is a set of techniques that reduce model size and latency while maintaining acceptable quality. This article covers the key methods we apply in projects and shows how to achieve 4–8x compression without significant accuracy loss.

A common mistake is believing that simply converting the model to TFLite suffices. Without device adaptation, accuracy drops by 10–15% and latency remains high. We use a combination of quantization, pruning, and distillation to achieve maximum acceleration on specific hardware.

Real case: face recognition on Jetson Nano. Original ResNet-50 (98 MB, Float32) ran at 800 ms per frame. After PTQ INT8, size dropped to 25 MB, latency to 150 ms. Accuracy loss was 0.3%. We then applied structured pruning (50% channels) — latency fell to 90 ms. Final speedup: 9x. This demonstrates that a pipeline of structured pruning followed by quantization yields a 9x better performance than the unoptimized model.

How to Optimize an ML Model for Edge?

Key techniques include quantization, pruning, knowledge distillation, and neural architecture search. Let's examine each.

Quantization

The most impactful method. Converting weights from Float32 to INT8 reduces size 4x and speeds up inference 2–4x on supporting hardware. INT4 yields 8x compression but higher accuracy loss. Post-Training Quantization (PTQ) requires a calibration dataset (100–1000 samples) and takes hours. Quantization-Aware Training (QAT) trains the model with quantization in mind, giving 1–3% better accuracy. We recommend QAT for critical tasks. See also Quantization in ML.

Pruning

Removing unimportant weights. Unstructured pruning can achieve 80%+ sparsity but is hard to accelerate on standard hardware without specialized libraries. Structured pruning (removing entire filters or heads) provides direct speedup on any device. In practice, we combine both: first unstructured pruning to 50%, then fine-tuning. Structured pruning is typically 2x better than unstructured in terms of latency reduction.

Distillation

A small student model is trained to reproduce the outputs of a large teacher model. Example: BERT → TinyBERT runs 7.5x faster while retaining 96% of GLUE score. Distillation is often combined with quantization for maximum effect. See Knowledge Distillation.

Neural Architecture Search

Finding the optimal architecture under target latency and memory constraints. MobileNetV2, found via NAS, became a standard for mobile devices. For edge projects, we use lightweight NAS based on regression models.

Operator Fusion

Merging sequential operations (Conv+BN+ReLU) into one. Implemented in TFLite converter, ONNX Runtime, TensorRT. Provides speedup without changing weights. This fusion yields a 10-20% better latency compared to unfused execution.

Which Techniques Give the Biggest Impact?

Compare the main approaches:

Model compression examples:

When Should You Use Quantization-Aware Training?

QAT is justified when accuracy is critical and PTQ losses exceed 1%. For instance, in medical diagnostics or autonomous driving. We guarantee that with QAT, accuracy degrades no more than 0.5%. If a 2% loss is acceptable, PTQ is sufficient — faster and doesn't require access to training data. For a typical project, using QAT instead of PTQ costs an extra $500–$1000 but preserves an additional 1–2% accuracy — a worthwhile trade-off for high-stakes applications.

How to Combine Techniques for Maximum Speedup?

Often one method isn't enough. For a typical CV pipeline, we apply: structured pruning (remove 30% filters) → PTQ INT8 → operator fusion. For NLP: distill BERT into TinyBERT → QAT INT8. For detection: TensorRT with FP16 and INT8. Cost savings: instead of cloud inference, an edge device costing $100 once. We estimate payback in 3–6 months through reduced infrastructure costs. For example, switching from cloud to edge inference saves approximately $3,600 per year per device – a significant recurring reduction.

Process

The step-by-step process includes: 1) Model analysis: profile latency, memory, bottlenecks on the target device using layer-wise profiling. 2) Strategy selection: combine techniques for the specific task. For NLP, distillation + quantization. 3) Optimization: apply QAT or PTQ, pruning, fusion. 4) Validation: check accuracy on a representative dataset, compare with baseline. 5) Deployment: convert to TFLite, ONNX, or TensorRT format, integrate into the pipeline.

What's Included

• Optimized model in the format for your device (TFLite, ONNX, TensorRT)
• Profiling report with metric comparison
• Recommendations for further optimization
• Integration support

Timeline and Cost

Timelines: 2 to 4 weeks depending on model complexity and accuracy requirements. We'll assess your project for free — just contact us. Typical project cost ranges from $2,000 to $5,000, depending on the scope.

Our experience: 5+ years in AI/ML solution development, over 20 projects in edge optimization. We guarantee preservation of key quality metrics.

Order optimization — get a model that runs on Raspberry Pi, Jetson Nano, or any other edge device. Get a consultation for your project — we'll help select the optimal combination of techniques.

Edge AI and Optimization: How to Deploy Models Without Cloud?

Imagine: your face recognition model has 4 seconds latency on Jetson Orin, the battery runs out in an hour, and the model crashes with OOM. We are a team of Edge AI engineers with 5+ years in production — we have optimized over 150 models for edge devices. Without profiling and proper choice of quantization or distillation, the project is doomed. The gap between research code and edge deployment is a separate engineering discipline; we help you master it in 2–16 weeks turnkey. Edge AI and model optimization services are not just export, but systematic work with hardware.

Why Simply Exporting a Model Doesn't Work?

A PyTorch model with float32 and batch_size=32 is not ready for edge. Typical problems:

• ResNet-50 in fp32 occupies 98 MB, inference on Cortex-A78 — 380 ms. After INT8 quantization via torch.ao.quantization — 24 MB, 95 ms. Export to ONNX + TensorRT on Jetson — 28 ms.
• YOLOv8m on Raspberry Pi 5 in fp32 — 2.8 fps. TFLite INT8 — 9.4 fps. With XNNPACK delegate — 14 fps (1.5× faster than pure INT8).
• Transformer encoder on mobile CPU: MobileBERT in fp16 via CoreML on iPhone 15 — 18 ms/inference. distilbert-base-uncased in ONNX — 42 ms.

The problem is not choosing "quantize or not" — the right path is determined by the device, task, and acceptable metric degradation. We offer an assessment of your project: within 24 hours we will tell you how feasible it is to speed up the model.

How to Choose Quantization Method for Your Task?

PTQ (Post-Training Quantization) — a quick path. Take a trained model, run a calibration dataset (200–1000 samples), get INT8 or INT4 weights. Tools: torch.ao.quantization, ONNX Runtime quantization tool, bitsandbytes. Accuracy degradation: 0.5–2% on classification. Red zone — small object detection and segmentation, where PTQ gives -4–8% mAP.

QAT (Quantization-Aware Training) — training with simulated quantization noise. More expensive (retraining), but degradation 0.1–0.5%. Justified when PTQ is unacceptable. In PyTorch — torch.ao.quantization.prepare_qat().

GPTQ / AWQ — for LLMs. AWQ better preserves quality at 4-bit quantization. llm-compressor from Neural Magic or autoawq are the main libraries.

Potential savings from choosing the right method can be substantial — for example, reducing cloud inference costs by up to 70% when deploying to edge. Project cost is calculated individually based on model complexity and target platform.

When to Use Pruning vs Distillation?

Structural pruning removes channels or layers. torch.nn.utils.prune — basic tool. For transformers — attention head pruning (LTP, movement pruning). Result: ResNet-50 after removing 40% of channels with fine-tuning — -35% size, -28% latency, -1.2% top-1 accuracy.

Knowledge distillation — train a small student to mimic a large teacher. Classic via KLDivLoss on soft labels. Feature distillation on intermediate layers is more effective. Hugging Face DistilBERT: 66M vs 110M parameters, -40% latency, -3% on GLUE. This is a model compression technique.

Combined approach: distillation → pruning → QAT. Gives maximum effect on limited hardware. We recorded a case where a client achieved 70% reduction in cloud compute spend after moving to edge with this pipeline.

Target Platforms and Tools

TensorRT — the main tool for NVIDIA edge. TRT builds a graph with operator fusion, selects optimal kernels. On Jetson AGX Orin YOLOv8m in TRT INT8 gives 78 fps vs 22 fps in fp16 PyTorch — 3.5× improvement.

Practical Case: How We Detected Defects on a Production Line (Our Client)

Task: real-time scratch detection on metal, 30 fps, camera to Jetson Xavier NX (16GB). Original model YOLOv8l mAP50 0.91, server inference 28 ms, on Jetson in fp16 — 110 ms (9 fps). Not suitable.

Optimization steps we performed for our client:

1. Switch to YOLOv8m — mAP50 0.887 (-2.3%), 68 ms
2. Export to TensorRT FP16 via yolo export format=engine half=True — 31 ms (32 fps)
3. INT8 calibration on 500 frames — 22 ms (45 fps), mAP50 0.879

Result: 3.5% degradation at 5× speedup. Client received engine and documentation. We guarantee metric will not drop below agreed threshold — specified in contract.

What is Included in the Work?

• Report on model profiling on target device (layer latency, bottlenecks)
• Selection and justification of optimization methods (quantization / pruning / distillation)
• Optimized model (TensorRT engine / TFLite / CoreML / ONNX)
• Configs for reproducibility (scripts, Docker image, instructions)
• Testing on real device (at least 10,000 inferences)
• Training of your team (2 hours online)
• 1 month support after delivery

How to Order Model Optimization

1. Submit a request on the website or contact us in any convenient way.
2. We perform free profiling of your model on the target device within 24 hours.
3. We prepare an optimization plan with trade-off estimates (speed vs quality).
4. You approve the plan — we start work.
5. After completion, we deliver the optimized model, configs, and documentation.
6. We train your team and provide monthly support.

Timeline: optimization of an existing model — 2–4 weeks. Development from scratch for edge — 6–16 weeks.

Get a consultation — we will evaluate your model for free and offer a plan within 24 hours. Order free profiling now. For complex projects, contact our engineering team to discuss custom optimisation strategies.