On-Device ML (Training and Inference on Device Without Data Transfer)
On-Device ML — data never leaves the device. Critical for: medical data (HIPAA), biometrics, corporate documents, personalization without privacy concerns. Apple, Google, Samsung actively pushing this direction.
On-Device Inference
Simpler task — model pre-trained on server, deployed on device:
- iOS: Core ML + Neural Engine. Excellent performance on iPhone 12+
- Android: TFLite + NNAPI/GPU/Hexagon
- Embedded: TFLite Micro, ONNX Runtime Mobile
On-Device Training
Significantly harder. Requires: sufficient memory, adaptive optimizer, efficient backward pass.
Federated Learning: Standard approach for on-device training. Device fine-tunes model on local data → sends only gradient updates (not raw data) → server aggregates via FedAvg → updated model returns. TensorFlow Federated, PySyft, FATE.
Continual Learning on Device: Model adapts to specific user without centralized training. NLP: adaptation to typing style. Computer Vision: personalized face recognition.
Apple Private Cloud Compute: new Apple approach — computing in cloud, but with cryptographic guarantees that data inaccessible to Apple or third parties.
Technical Limitations
Battery: training — energy-intensive operation. Only during charging. Memory: backpropagation requires ~3× memory vs. inference. Typically: only fine-tuning last layers.