Preserve Accuracy in ML Model Conversion to TFLite for Android

Imagine you trained an object detector on TensorFlow with mAP 0.85, converted to TFLite with Full INT8 quantization—and on-device mAP dropped to 0.6. The culprit: the representative dataset didn't cover dark scenes. Or runtime crashes on Android 9 due to missing Einsum operation. Here's how to avoid these pitfalls and preserve accuracy when porting to Android.

TFLite isn't just weight conversion. It's choosing quantization format, graph optimization, selecting an operation set compatible with target Android versions, and verifying numerical match with the original. Each step has specific pitfalls. Our team has over 10 years of production experience and has deployed ML models on 40+ Android projects—we know how to circumvent typical issues. This proven track record guarantees accurate conversions.

Typical Problems in Converting ML Models to TFLite

Quantization without a representative dataset is a common mistake. If the dataset isn't representative, scale factors shift and the model errs on real data. We use a dataset of 200–500 examples covering all edge cases.

Operation incompatibility—about 15% of modern TF operations (Einsum, RaggedTensor, SparseSegmentSum) are missing from TFLite Builtin ops. TensorFlow Lite ops compatibility shows that SELECT_TF_OPS solves this but adds ~5 MB to runtime size and reduces performance. We rewrite such operations as TFLite-compatible or implement custom ones via C++.

Different results across delegates—the same quantized model can produce different numbers on CPU, GPU, and NNAPI. We benchmark on 5–10 real devices and pick the delegate with the best speed/accuracy trade-off.

Case Study: Converting YOLOv5

A client recently asked to convert YOLOv5 for Android without NMS in the graph. Target: 30 FPS on Snapdragon 855 devices. We removed NMS from the model, implemented it in Kotlin with threshold 0.5 and IoU 0.45, used Full INT8 with calibration on 300 COCO images. Result: 35 FPS on GPU delegate, mAP dropped 2% relative to FP32—an acceptable compromise. Without custom NMS, it would have been 40 FPS but with multiple-box artifacts. This saved the client $3,000 per month in cloud compute costs.

How to Convert an ML Model to TFLite for Android?

Conversion Paths

The direct TF path gives minimal deviations. The ONNX path introduces additional potential incompatibilities—use only when the direct path is unavailable. Full INT8 quantization reduces model size 4x compared to FP32, while FP16 provides 2x size reduction with 5–10× speedup on GPU delegate. See the TensorFlow Lite GitHub repository for conversion parameters.

Quantization

# FP16 — minimal degradation, 2× smaller model, speedup on GPU delegate
converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir/")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]
tflite_fp16 = converter.convert()

# Dynamic INT8 — int8 weights, float32 activations. No calibration dataset needed.
converter2 = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir/")
converter2.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_dynamic_int8 = converter2.convert()

# Full INT8 — both weights and activations. Requires calibration dataset. Needed for Hexagon DSP.
def representative_dataset():
    dataset = load_calibration_data()  # 100-500 examples
    for sample in dataset:
        yield [sample[np.newaxis, :].astype(np.float32)]

converter3 = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir/")
converter3.optimizations = [tf.lite.Optimize.DEFAULT]
converter3.representative_dataset = representative_dataset
converter3.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter3.inference_input_type = tf.uint8
converter3.inference_output_type = tf.uint8
tflite_full_int8 = converter3.convert()

Which TFLite Delegate to Choose?

Delegate choice affects both performance and accuracy. We test the model on multiple delegates and select the optimal one. For instance, GPU delegate is 5× faster than CPU for FP16 models, while XNNPACK is 2× faster but supports more ops.

What to Do with Unsupported Operations?

Not all TF/PyTorch operations exist in TFLite builtin ops. Check:

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir/")
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS,
    tf.lite.OpsSet.SELECT_TF_OPS  # fallback to TF ops
]
tflite_model = converter.convert()

How to Verify TFLite Model Accuracy?

import numpy as np

# TF original
tf_output = tf_model(test_input).numpy()

# TFLite
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(input_details[0]['index'], test_input)
interpreter.invoke()
tflite_output = interpreter.get_tensor(output_details[0]['index'])

print(f"Max abs diff: {np.max(np.abs(tf_output - tflite_output))}")
print(f"MSE: {np.mean((tf_output - tflite_output)**2)}")
# FP32: < 1e-5, FP16: < 1e-2, INT8: < 0.05

If the difference exceeds norms—check input normalization, incorrect quantization parameters, or an operation where TFLite uses a different algorithm. We guarantee accuracy within these bounds.

Special Considerations for Object Detectors

YOLO, SSD, EfficientDet—all include NMS (Non-Maximum Suppression) post-processing. TFLite doesn't have built-in NMS (unlike Core ML Detection Output). Options:

1. Remove NMS from the model, implement in Java/Kotlin after inference.
2. Use TFLite Task Library—it provides a ready ObjectDetector API with NMS.

// TFLite Task Library: ObjectDetector (includes NMS)
val options = ObjectDetector.ObjectDetectorOptions.builder()
    .setScoreThreshold(0.5f)
    .setMaxResults(20)
    .build()
val detector = ObjectDetector.createFromFileAndOptions(context, "detector.tflite", options)
val image = TensorImage.fromBitmap(inputBitmap)
val results: List<Detection> = detector.detect(image)
for (detection in results) {
    val box = detection.boundingBox
    val label = detection.categories.first().label
    val score = detection.categories.first().score
}

Why Add TFLite Metadata?

from tflite_support.metadata_writers import image_classifier
from tflite_support.metadata_writers import writer_utils

writer = image_classifier.MetadataWriter.create_for_inference(
    writer_utils.load_file("model.tflite"),
    input_norm_mean=[0.0],
    input_norm_std=[255.0],
    labels_file_paths=["labels.txt"])
tflite_with_metadata = writer.populate()
writer_utils.save_file(tflite_with_metadata, "model_with_metadata.tflite")

Without metadata, the TFLite Task Library works poorly—no automatic normalization, no output mapping. With metadata, everything is handled automatically.

The calibration dataset for Full INT8 must reflect the real input distribution. For example, for a cat classification model, use 300 cat images under varying conditions—noise, darkness, rotations. This reduces quantization error by 10-20%.

What's Included in the Work?

• Analysis of the original model and selection of the optimal conversion path.
• Conversion with quantization type selection (FP16, Dynamic INT8, Full INT8).
• Accuracy verification on a representative dataset with a report.
• Adding TFLite Model Metadata for the Task Library.
• Testing on a device park (at least 5 devices) via Benchmark Tool (CPU, GPU, NNAPI).
• Integration into an Android app (Kotlin/Java) with error handling.
• Documentation on building, using, and maintaining the model.
• One month of support during deployment.
• Guaranteed accuracy preservation within tolerances.

The Process

1. Analytics—evaluate the model and conversion paths.
2. Design—choose quantization, decide on custom operations.
3. Implementation—convert, write custom code (NMS, preprocessing).
4. Testing—verify accuracy, benchmark on devices.
5. Deployment—integrate into the app, publish to Google Play.

Time Estimates

Direct conversion of a TF/Keras model with verification—from 3 to 7 days. Conversion via ONNX, custom ops, metadata addition, full testing—from 2 to 4 weeks. Our clients save up to 40% on cloud computing costs after switching to on-device ML, average savings range from $3,000 to $15,000 per month. Project cost is calculated individually based on your model and requirements. Contact us for a free TFLite compatibility audit of your model. Request a consultation to optimize your model for your target device fleet.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.