On-Device Facial Expression Analysis for Video Calls

Why Local Analysis is Essential for Video Call Privacy

During a video call, you want to gauge your counterpart’s reaction, but sending video to the cloud breaches privacy and increases latency. On-device analysis solves both: all processing happens locally, latency under 10 ms. You retain full control over data, complying with App Store Review Guidelines and GDPR. Academic research on FACS (Facial Action Coding System) shows that facial expressions do not map one-to-one to emotions. Therefore, we avoid labels like ‘angry’ or ‘happy’ and use neutral metrics—engagement level, facial activity. The system should not influence HR or legal decisions. Modern mobile emotion recognition solutions run locally, which is especially important for applications with high privacy requirements. Our certified mobile emotion AI solution integrates SwiftUI for intuitive indicators.

Why We Don’t Use Emotion Categorization

Emotion categorization (happiness, sadness) is an oversimplification that leads to errors. Action Units from FACS capture specific muscle movements, providing objective data. For example, a smile can be polite or genuine—we don’t assume, we deliver numerical metrics. The user sees only aggregated engagement scores, not emotional labels. In real projects, we combine Action Units with machine learning to improve recognition accuracy. We use engagement metrics derived from Action Units to provide objective feedback.

How We Implement Analysis on iOS and Android

Technology Stack

• Face detection: MediaPipe Face Detection (iOS/Android), Apple Vision (iOS)
• Expression recognition: Apple Vision VNDetectFaceExpressionsRequest, FER+ (CoreML/TFLite)
• Call integration: WebRTC data channel or Agora Video SDK

Approach Comparison

For video calls, on-device is superior in latency and privacy. No cloud costs is an extra benefit: you save $500–$1,500 per month for 10,000 calls. On-device is 10x faster than cloud in transmission latency, critical for real-time communication.

How We Do It on iOS

// iOS: face expression analysis via Vision
class FaceExpressionAnalyzer {

    func analyze(sampleBuffer: CMSampleBuffer) async throws -> ExpressionResult? {
        guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return nil }

        let faceRequest = VNDetectFaceLandmarksRequest()
        let expressionRequest = VNDetectFaceExpressionsRequest()

        let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
        try handler.perform([faceRequest, expressionRequest])

        guard let faceObs = faceRequest.results?.first as? VNFaceObservation,
              let exprObs = expressionRequest.results?.first as? VNFaceExpressionObservation else {
            return nil
        }

        return ExpressionResult(
            faceBox: faceObs.boundingBox,
            browLower: exprObs.browLowerQuirk,
            browRaise: exprObs.browRaiseRight + exprObs.browRaiseLeft,
            eyesClosed: exprObs.eyeBlinkLeft + exprObs.eyeBlinkRight,
            mouthSmile: exprObs.mouthSmileLeft + exprObs.mouthSmileRight,
            mouthFrown: exprObs.mouthFrownLeft + exprObs.mouthFrownRight,
            mouthOpen: exprObs.mouthOpen,
            jawOpen: exprObs.jawOpen
        )
    }
}

VNDetectFaceExpressionsRequest returns Action Units—basic muscle movements per FACS. This is more correct than interpreting a smile as happiness.

Time Aggregation

A single frame is noise. We use a sliding window of 15 frames (~0.5 sec):

class ExpressionAggregator {
    private var history: [ExpressionResult] = []
    private let windowSize = 15

    func update(_ result: ExpressionResult) -> AggregatedExpression {
        history.append(result)
        if history.count > windowSize { history.removeFirst() }

        return AggregatedExpression(
            averageSmile: history.map { $0.mouthSmile }.average(),
            averageBrowRaise: history.map { $0.browRaise }.average(),
            averageJawOpen: history.map { $0.jawOpen }.average(),
            smileTrend: computeTrend(history.map { $0.mouthSmile })
        )
    }
}

Integrating Analysis into an Existing Video Call

SDK with custom processor—Agora Video SDK allows frame interception before sending:

class EmotionVideoProcessor: AgoraVideoFrameDelegate {
    func onCapture(_ videoFrame: AgoraOutputVideoFrame,
                   sourceType: AgoraVideoSourceType) -> Bool {
        if let pixelBuffer = videoFrame.pixelBuffer {
            Task {
                let result = try? await expressionAnalyzer.analyze(buffer: pixelBuffer)
                await MainActor.run {
                    emotionDelegate?.didUpdateExpression(result)
                }
            }
        }
        return true
    }
}

Peer-to-peer via data channel—both participants analyze themselves and transmit results (not video):

struct EmotionDataPacket: Codable {
    let timestamp: Double
    let smile: Float
    let browRaise: Float
    let eyesClosed: Float
}

func sendEmotionData(_ expression: AggregatedExpression) {
    let packet = EmotionDataPacket(
        timestamp: Date().timeIntervalSince1970,
        smile: expression.averageSmile,
        browRaise: expression.averageBrowRaise,
        eyesClosed: expression.averageJawOpen
    )
    let data = try! JSONEncoder().encode(packet)
    dataChannel.sendData(RTCDataBuffer(data: data, isBinary: false))
}

Private and clean: each sees only own data and the counterpart’s aggregate. In contrast, cloud emotion AI services require sending video off-device, increasing latency and risk.

UX and Common Mistakes

Show Engagement, Not Emotions

Proper indicators—not emotions, but engagement:

@Composable
fun EngagementIndicator(score: Float) {
    Box(
        modifier = Modifier
            .size(12.dp)
            .clip(CircleShape)
            .background(
                when {
                    score > 0.7f -> Color(0xFF4CAF50)
                    score > 0.4f -> Color(0xFFFFC107)
                    else -> Color(0xFF9E9E9E)
                }
            )
    )
}

No verbal labels—only neutral color indicators.

What to Avoid

• Using cloud APIs without user consent—leads to App Store rejection.
• Naive interpretation of a single emotion—causes user distrust.
• Lack of time aggregation—noisy data.

Implementation Steps

Follow these steps to integrate on-device facial expression analysis into your video call app:

1. Audit your current stack: Identify where video frames are processed and assess privacy requirements. (1-2 days)
2. Choose your platform tools: Select Vision Framework (iOS) or MediaPipe (Android) based on your target OS. (1 day)
3. Implement local analysis: Write code using VNDetectFaceExpressionsRequest or MediaPipe to extract Action Units from each frame. (3-5 days)
4. Add temporal aggregation: Use a sliding window of 15 frames to smooth noise and compute engagement metrics. (1-2 days)
5. Integrate with call SDK: If using Agora, implement AgoraVideoFrameDelegate to intercept frames before sending. Alternatively, set up a WebRTC data channel to transmit computed metrics peer-to-peer. (2-4 days)
6. Build the UX: Create an engagement indicator (color dot or bar) and a consent dialog that explains privacy. (2-3 days)
7. Test on real devices: Verify latency (<10 ms), accuracy, and privacy compliance. (2-3 days)
8. Document and hand over: Provide API reference, deployment guide, and a training session for your team. (1-2 days)

Implementation Process and Timelines

Work Stages

Estimated Timelines and Costs

Basic version: from 1 to 2 weeks. Full system (iOS + Android + data channel): from 2 to 4 weeks. Typical project cost ranges from $5,000 to $15,000 depending on complexity. Additionally, on-device solution eliminates cloud costs of $500–$1,500 per month for 10,000 calls, providing significant long-term savings.

Deliverables

• Ready-to-use emotion analysis module (iOS/Android) with source code (available upon NDA).
• Integration and configuration documentation (API reference, deployment guide).
• Example usage with an engagement indicator.
• Consultation on passing App Store / Google Play moderation.
• Training session (up to 2 hours) for your team.
• 30-day support after handover.
• Guaranteed data privacy compliance.

Conclusion

Local facial expression analysis is an ethical and technically efficient solution for video calls. With 10+ years of experience and over 50 successful projects, we ensure smooth integration. Our certified solution guarantees privacy and low latency. Use on-device emotion detection to maintain privacy and reduce latency.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.