Private and Fast On-Device Machine Learning Locally

Your mobile app processes biometrics on the server — every request goes through the network, data accumulates in the cloud, and latency stretches to a second? This is a classic problem for medical, financial, and enterprise applications. On-device ML radically changes the approach: the model lives directly on the user's device, training and inference happen locally. We build such systems turnkey — from prototype to production. No data transmission, no leaks, p99 latency in single-digit milliseconds (typically 5–20 ms). In 4–8 weeks you get a working solution. Our certified engineers have 6+ years of experience in on-device ml and have completed 30+ projects for FinTech, MedTech, and enterprise clients, serving over 100,000 users across 50+ device types.

We implement on-device ML using Core ML for iOS and TFLite for Android, enabling private compute and edge AI. Our certified engineers specialize in federated learning and on-device inference, ensuring data privacy and low latency.

Why choose On-Device ML?

On-Device ML is the only way to comply with HIPAA, GDPR, and corporate security policies. It enables private compute and edge AI, using Core ML or TFLite for on-device inference and federated learning for locally trained models. This ensures data privacy and fast mobile ML. Modern mobile chips with NPU (Neural Engine, Google Tensor) deliver performance comparable to cloud. On-device inference is 10x faster than cloud at p99 latencies, and on-device training allows personalization per user without sending data. Cloud infrastructure cost reduction reaches 70%, saving $10,000–$50,000 annually for medium-scale apps. For example, a fintech client saved $35,000 per year by switching to on-device inference. Another client with 100,000 users reduced cloud costs from $5,000/month to $500/month, saving $54,000 annually.

How we implement on-device inference

Inference is a relatively straightforward task. The model is trained on the server, then deployed to the device with optimization for specific hardware.

Typical case: face unlocking on a smartphone. We trained MobileFaceNet on the server, converted to Core ML with INT8 quantization (model size 2.3 MB, down from 10 MB), and integrated into the app. Inference takes 15–20 ms on modern iPhones, achieving 98% accuracy. For mobile ML, we leverage Core ML and TFLite for efficient on-device inference.

Comparison of on-device inference and training

How federated learning solves the privacy problem?

On-device training is more complex: backpropagation requires ~3x memory compared to inference, plus power consumption. We use federated learning: devices fine-tune the model on local data, send only gradient updates (not data), the server aggregates them via FedAvg. Stack: TensorFlow Federated, PySyft, FATE.

Example: a keyboard engine with personalized typing style. The device trains a small transformer on top of Federated EMNIST — only last layer, Adam with gradient clipping. The process runs during charging. Result: +15% accuracy without user data leakage, with additional power consumption not exceeding 5% of battery overnight.

What's included in your On-Device ML solution

• Project requirements analysis and feasibility study
• Model architecture selection and optimization (quantization, pruning, LoRA)
• Framework integration (core ml, tflite, ONNX Runtime)
• Testing on target devices (iPhone, Pixel, Samsung, etc.)
• Documentation (API docs, deployment guide, troubleshooting)
• Post-deployment support (3 months free)
• Training for your team (2 hands-on sessions)
• Access to model versioning system (MLflow)

Process (what is included)

1. Analysis — determine whether on-device training or inference is needed, select architecture (MobileNet, TinyBERT, etc.), estimate memory budget and FLOPS. At this stage we provide a detailed technical report.
2. Design — develop pipeline: server training → quantization/pruning → device deployment; for training — configure federated cycle. We use MLflow for model versioning.
3. Implementation — integrate with core ml / tflite, support batching, async inference. Write unit tests and integration tests on target devices.
4. Testing — measure p99 latency, power consumption, accuracy on real data. Use Xcode and Android Studio profilers for fine-tuning.
5. Deployment — release model via CDN or within APK/IPA. Document error handling and graceful degradation (fallback to cloud inference when memory is low).

Timelines: 4–8 weeks depending on complexity and number of platforms. Cost is calculated individually — contact us for an evaluation of your case. Typical project cost ranges from $15,000 to $50,000. Get a consultation: our engineers will assess the project in 1 day. Reach out to discuss your project.

On-device ML is transforming industries by enabling private compute, edge ai, and data privacy. Whether you need on-device inference for real-time applications or on-device training with federated learning, we deliver locally trained models that respect user privacy. Our expertise in core ml, tflite, and federated learning ensures your mobile ml solutions are fast, efficient, and secure.

Edge AI and Optimization: How to Deploy Models Without Cloud?

Imagine: your face recognition model has 4 seconds latency on Jetson Orin, the battery runs out in an hour, and the model crashes with OOM. We are a team of Edge AI engineers with 5+ years in production — we have optimized over 150 models for edge devices. Without profiling and proper choice of quantization or distillation, the project is doomed. The gap between research code and edge deployment is a separate engineering discipline; we help you master it in 2–16 weeks turnkey. Edge AI and model optimization services are not just export, but systematic work with hardware.

Why Simply Exporting a Model Doesn't Work?

A PyTorch model with float32 and batch_size=32 is not ready for edge. Typical problems:

• ResNet-50 in fp32 occupies 98 MB, inference on Cortex-A78 — 380 ms. After INT8 quantization via torch.ao.quantization — 24 MB, 95 ms. Export to ONNX + TensorRT on Jetson — 28 ms.
• YOLOv8m on Raspberry Pi 5 in fp32 — 2.8 fps. TFLite INT8 — 9.4 fps. With XNNPACK delegate — 14 fps (1.5× faster than pure INT8).
• Transformer encoder on mobile CPU: MobileBERT in fp16 via CoreML on iPhone 15 — 18 ms/inference. distilbert-base-uncased in ONNX — 42 ms.

The problem is not choosing "quantize or not" — the right path is determined by the device, task, and acceptable metric degradation. We offer an assessment of your project: within 24 hours we will tell you how feasible it is to speed up the model.

How to Choose Quantization Method for Your Task?

PTQ (Post-Training Quantization) — a quick path. Take a trained model, run a calibration dataset (200–1000 samples), get INT8 or INT4 weights. Tools: torch.ao.quantization, ONNX Runtime quantization tool, bitsandbytes. Accuracy degradation: 0.5–2% on classification. Red zone — small object detection and segmentation, where PTQ gives -4–8% mAP.

QAT (Quantization-Aware Training) — training with simulated quantization noise. More expensive (retraining), but degradation 0.1–0.5%. Justified when PTQ is unacceptable. In PyTorch — torch.ao.quantization.prepare_qat().

GPTQ / AWQ — for LLMs. AWQ better preserves quality at 4-bit quantization. llm-compressor from Neural Magic or autoawq are the main libraries.

Potential savings from choosing the right method can be substantial — for example, reducing cloud inference costs by up to 70% when deploying to edge. Project cost is calculated individually based on model complexity and target platform.

When to Use Pruning vs Distillation?

Structural pruning removes channels or layers. torch.nn.utils.prune — basic tool. For transformers — attention head pruning (LTP, movement pruning). Result: ResNet-50 after removing 40% of channels with fine-tuning — -35% size, -28% latency, -1.2% top-1 accuracy.

Knowledge distillation — train a small student to mimic a large teacher. Classic via KLDivLoss on soft labels. Feature distillation on intermediate layers is more effective. Hugging Face DistilBERT: 66M vs 110M parameters, -40% latency, -3% on GLUE. This is a model compression technique.

Combined approach: distillation → pruning → QAT. Gives maximum effect on limited hardware. We recorded a case where a client achieved 70% reduction in cloud compute spend after moving to edge with this pipeline.

Target Platforms and Tools

TensorRT — the main tool for NVIDIA edge. TRT builds a graph with operator fusion, selects optimal kernels. On Jetson AGX Orin YOLOv8m in TRT INT8 gives 78 fps vs 22 fps in fp16 PyTorch — 3.5× improvement.

Practical Case: How We Detected Defects on a Production Line (Our Client)

Task: real-time scratch detection on metal, 30 fps, camera to Jetson Xavier NX (16GB). Original model YOLOv8l mAP50 0.91, server inference 28 ms, on Jetson in fp16 — 110 ms (9 fps). Not suitable.

Optimization steps we performed for our client:

1. Switch to YOLOv8m — mAP50 0.887 (-2.3%), 68 ms
2. Export to TensorRT FP16 via yolo export format=engine half=True — 31 ms (32 fps)
3. INT8 calibration on 500 frames — 22 ms (45 fps), mAP50 0.879

Result: 3.5% degradation at 5× speedup. Client received engine and documentation. We guarantee metric will not drop below agreed threshold — specified in contract.

What is Included in the Work?

• Report on model profiling on target device (layer latency, bottlenecks)
• Selection and justification of optimization methods (quantization / pruning / distillation)
• Optimized model (TensorRT engine / TFLite / CoreML / ONNX)
• Configs for reproducibility (scripts, Docker image, instructions)
• Testing on real device (at least 10,000 inferences)
• Training of your team (2 hours online)
• 1 month support after delivery

How to Order Model Optimization

1. Submit a request on the website or contact us in any convenient way.
2. We perform free profiling of your model on the target device within 24 hours.
3. We prepare an optimization plan with trade-off estimates (speed vs quality).
4. You approve the plan — we start work.
5. After completion, we deliver the optimized model, configs, and documentation.
6. We train your team and provide monthly support.

Timeline: optimization of an existing model — 2–4 weeks. Development from scratch for edge — 6–16 weeks.

Get a consultation — we will evaluate your model for free and offer a plan within 24 hours. Order free profiling now. For complex projects, contact our engineering team to discuss custom optimisation strategies.