AI Food Photo Recognition & Calorie Counting in Mobile Apps

Food Recognition and Calorie Counting from Photos: AI in Mobile Apps

A user snaps a bowl of borscht—the app must output the macros. In practice, this is a chain: detection, classification, portion estimation, database lookup. Each link introduces error. Recently, a US startup came to us: their MVP recognized only 10 dishes with 70% accuracy. After implementing our pipeline, accuracy rose to 92%, and response time dropped from 2 seconds to 400 ms. We've accumulated experience in this domain across dozens of projects—from MVPs for startups to full fitness apps for iOS and Android, using Swift, Kotlin, Flutter, CoreML, and TFLite. Our track record: 10+ years in mobile AI, 40+ projects delivered.

When properly configured, recognition accuracy stays above 85% for the top 50 dishes. Delivery time for a basic version: 1 to 2 weeks. In this article, we break down the technical architecture, compare APIs, show code for iOS and Android, explain how to avoid common pitfalls, and integrate HealthKit with Health Connect.

How Is the Pipeline Built?

The right architecture isn't "one model to rule them all"—it's a pipeline of specialized steps.

Step 1: Detection and Classification

CoreML on iOS (model based on EfficientDet or YOLOv8), TFLite on Android. For an MVP, cloud APIs are viable: Clarifai Food Model, Google Cloud Vision with food tags, or the dedicated Logmeal API.

Step 2: Portion Estimation

This is harder. Without a reference object in the frame (coin, hand, standard plate), estimating grams is nearly impossible. Two practical solutions: ask the user to specify the container type (20 cm plate, 200 ml glass) or use ARKit/ARCore for depth estimation. Depth estimation via ARKit gives acceptable results for voluminous dishes—error margin 15–25%, which beats manual user input (they usually underestimate).

Step 3: Nutritional Data Lookup

USDA FoodData Central is a free API with 700,000+ products. Open Food Facts is an open-source database, good for packaged foods. For Russian-language markets, it's critical to have local dishes: borscht, pelmeni, Olivier—they aren't in USDA in the expected format. We use a combination of databases and supplement with custom data.

// iOS: Full recognition pipeline
struct FoodRecognitionPipeline {

    func analyze(image: UIImage, portionContext: PortionContext?) async throws -> MealAnalysis {

        // 1. Dish recognition via Logmeal API
        let foodItems = try await logmealClient.recognizeFood(image: image)

        // 2. Portion estimation
        let portionEstimates: [PortionEstimate]
        if let context = portionContext {
            portionEstimates = estimatePortionFromContext(foodItems, context: context)
        } else {
            portionEstimates = try await estimatePortionWithAR(image: image)
        }

        // 3. Nutritional data
        let nutritionData = try await withThrowingTaskGroup(of: NutritionResult.self) { group in
            for (item, portion) in zip(foodItems, portionEstimates) {
                group.addTask {
                    try await self.fetchNutrition(food: item, grams: portion.estimatedGrams)
                }
            }
            return try await group.reduce(into: []) { $0.append($1) }
        }

        return MealAnalysis(
            items: foodItems,
            portions: portionEstimates,
            nutrition: nutritionData.aggregate(),
            confidence: foodItems.map(\.confidence).min() ?? 0
        )
    }
}

Parallel nutritional requests via TaskGroup are crucial: with 3 dishes in the frame, sequential requests cause 3× latency.

How to Estimate Portion Without a Reference Object?

If there's no coin or standard plate in the frame, use ARKit to estimate scene depth. On iOS, an ARSession and depth image suffice. The error margin of 15–25% is acceptable for daily tracking. An alternative is to ask the user to select a container type from presets. With proper calibration, portion estimation accuracy stays above 70%.

Composite Dishes: Approaches to Recognition

A photo of borscht contains beets, cabbage, carrots, potatoes, meat, and sour cream in unknown proportions. Two solutions:

Recipe database. An LLM or custom model breaks the dish into ingredients based on a recipe. Works for standard dishes, poorly for home cooking with variations.

User correction. After automatic recognition, the user sees the presumed composition and can remove or add ingredients. Swipe-to-remove on an ingredient, slider for grams. This UX is fundamentally better than "98% accuracy" without editability.

// Android: Dish composition UI with editing
@Composable
fun MealCompositionEditor(
    items: List<FoodItem>,
    onItemRemoved: (FoodItem) -> Unit,
    onPortionChanged: (FoodItem, Float) -> Unit
) {
    LazyColumn {
        items(items, key = { it.id }) { item ->
            SwipeToDismiss(
                state = rememberDismissState { if (it == DismissValue.DismissedToStart) {
                    onItemRemoved(item); true } else false
                },
                background = { DeleteBackground() },
                dismissContent = {
                    FoodItemRow(
                        item = item,
                        onPortionChange = { grams -> onPortionChanged(item, grams) }
                    )
                }
            )
        }
    }
}

Integration with HealthKit and Health Connect

A logged meal should enter the health ecosystem. We follow Apple's recommendations for HealthKit.

// iOS: Log to HealthKit
func logMealToHealthKit(_ meal: MealAnalysis) async throws {
    let store = HKHealthStore()
    let caloriesType = HKQuantityType(.dietaryEnergyConsumed)
    let proteinType = HKQuantityType(.dietaryProtein)
    let carbsType = HKQuantityType(.dietaryCarbohydrates)
    let fatType = HKQuantityType(.dietaryFatTotal)

    let metadata: [String: Any] = [
        HKMetadataKeyFoodType: meal.primaryItem?.name ?? "Mixed Meal"
    ]

    let samples = [
        HKQuantitySample(type: caloriesType,
                        quantity: .init(unit: .kilocalorie(), doubleValue: meal.nutrition.calories),
                        start: .now, end: .now, metadata: metadata),
        HKQuantitySample(type: proteinType,
                        quantity: .init(unit: .gram(), doubleValue: meal.nutrition.protein),
                        start: .now, end: .now)
        // + carbs, fat
    ]

    try await store.save(samples)
}

Permissions are requested in advance via HKHealthStore.requestAuthorization. A common mistake is to request permissions on first app launch, before the user sees value. We recommend requesting at the moment of the first meal log—conversion increases by 30%.

Accuracy Barriers and How to Honestly Display Them

Even a good model fails on non-standard dishes, poor lighting, and unusual angles. Hiding uncertainty is a mistake. Show a confidence score next to the result.

struct NutritionDisplayView: View {
    let analysis: MealAnalysis

    var body: some View {
        VStack(alignment: .leading, spacing: 12) {
            if analysis.confidence < 0.6 {
                ConfidenceWarningBanner(
                    message: "Low recognition confidence. Please verify the dish composition."
                )
            }
            CalorieSummaryCard(nutrition: analysis.nutrition)
            MacroBreakdownChart(nutrition: analysis.nutrition)
            IngredientList(items: analysis.items, editable: true)
        }
    }
}

How to Integrate Recognition: 5 Steps

1. Choose a recognition API. Determine which service best fits your audience: Logmeal for high accuracy, Clarifai for quick start, Google Cloud Vision for versatility.
2. Adapt the model to the platform. Convert to CoreML (iOS) or TFLite (Android) for on-device operation. Provide a fallback to cloud API when confidence is low.
3. Implement portion estimation. Integrate ARKit/ARCore for automatic estimation or contextual container presets for manual input.
4. Connect a nutritional database. Link USDA FoodData Central for basic products, supplement with custom data for regional dishes.
5. Add HealthKit/Health Connect. Automate meal logging, requesting permissions at the moment of the first log.

What's Included in the Work

• Architecture design and API selection: analyze existing codebase, choose optimal recognition services and nutritional databases.
• Pipeline development: integrate detection, portion estimation, and data lookup—on iOS (Swift) and Android (Kotlin).
• UI/UX adaptation: customizable input screens, composition editing, confidence display.
• HealthKit/Health Connect integration: automatic meal logging into the health system.
• Documentation and support: code delivery, deployment instructions, test reports.

Time Estimates

Contact us for an evaluation of your project—we'll calculate exact timelines and cost. Request a consultation to discuss integration details with your app. Budget savings on development are achieved by using ready-made pipeline modules. Reducing AI function development costs is possible by choosing off-the-shelf APIs instead of training custom models.

Get a consultation—we'll help you select the optimal architecture for food recognition.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.