Face Tracking in AR: Precision, Animation, and Performance

Why Face Tracking Is a Bottleneck in AR

A user opens an AR filter, but the mask doesn't match their expression — typical when face tracking is misconfigured. We encounter this in almost every second project. Face tracking is a mature technology, but its implementation requires precise calibration and the right stack choice. A wrong platform decision turns the app into a laggy mess. Our 5+ years of experience with 15+ face tracking projects let us guarantee latency under 50ms. The right stack choice cuts rework costs in half.

Problems We Solve

Three main challenges: expression latency, tracking loss on head rotation, and lack of response to facial expressions. We solve them by selecting the right technology and tuning parameters. Every project includes load testing on 5+ devices.

Stack and Precision: ARKit vs ARCore vs MediaPipe

According to Apple ARKit documentation, the TrueDepth camera achieves submillimeter accuracy. ARKit with TrueDepth provides a face depth map with millimeter precision. ARCore AugmentedFace and MediaPipe work on RGB cameras but perform worse on motion. Key parameter comparison:

ARKit is 1.5x faster in FPS on flagships, while MediaPipe is a universal cross-platform solution. Depth map from ARKit is critical for realistic masks — it allows accurate texture overlay even on moving faces.

Performance on Different Devices

How We Ensure Stable Face Tracking: A Case Study

On a recent social media AR filter project, the client faced 120ms latency on Android and lost tracking when the head turned beyond 30 degrees. We switched to ARKit on iOS (TrueDepth) and kept MediaPipe on Android with a custom fallback prediction. After optimizing blend shape processing (dedicated thread, reduced sampling rate to 30Hz for non-critical expressions), latency dropped to 45ms on iPhone and 55ms on latest Android flagships. Tracking loss decreased from 15% to under 2% of frames. The filter achieved 60 FPS on iPhone 12+ and 30 FPS on mid-range Android devices.

What ARKit Face Tracking Specifically Provides

ARFaceTrackingConfiguration requires iPhone X or newer (TrueDepth front camera). Returns ARFaceAnchor:

• geometry — ARFaceGeometry with 1220 vertices and 2304 triangles. Real-time face mesh in meters. Updated ~30 times per second.
• blendShapes — dictionary of 52 AR face blend shape coefficients. Each is Float from 0 to 1. Basis for face-driven animation and expression recognition.
• leftEyeTransform, rightEyeTransform — position and orientation of each eye.

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }

    let blinkLeft = faceAnchor.blendShapes[.eyeBlinkLeft]?.floatValue ?? 0
    let jawOpen = faceAnchor.blendShapes[.jawOpen]?.floatValue ?? 0

    // Trigger UI actions on blink/jaw open
    if blinkLeft > 0.7 { triggerAction() }
}

How We Ensure Stable Face Recognition on Android

For Android, we combine ARCore and ML Kit. ARCore AugmentedFace provides 468 points and basic emotions. ML Kit Face Detection does contour detection. For avatar animation, we use a custom classifier on TensorFlow Lite. Testing on 10+ devices guarantees stability. Head rotation issues are solved by tracking orientation: if tracking loses the face, we fall back to predicting the last known coordinates.

Our Development Process

1. Analytics — define use cases, choose platform.
2. Design — prototype blend shape processing.
3. Implementation — integrate SDK, configure, code sign.
4. Testing — verify on 5+ devices including different iPhone and Android models.
5. Deployment — publish to App Store / Google Play with guideline compliance.

Common Implementation Mistakes

• Using ARKit on devices without TrueDepth — FPS drops.
• Missing isSupported check — crashes on older models.
• Ignoring camera resolution — latency due to format mismatch.
• Unoptimized blend shape handlers — FPS down to 20.

We avoid these through code review and load testing at every stage.

Timelines and Pricing

Base integration with blend shape triggers: 1-2 weeks. Adding 3D mask and video recording: 2-4 weeks. ML expression classifier: additional 2-3 weeks. Cost is determined after analysis — contact us for a project assessment. We guarantee transparent pricing with no hidden fees.

What You Get

• Source code with comments and documentation.
• Demo application for testing.
• Access to real-time logs and monitoring.
• 30 days of post-delivery support.

We ensure stable face tracking in your AR app. Request a tech audit — our engineers help from the first steps. Optimizing blend shapes early saves up to 40% development time. Contact us for a project review. Learn more about MediaPipe Face Landmark.

We develop AR applications on ARKit and ARCore that work stably even in challenging conditions. Our experience: 7+ years in mobile development and 30+ delivered AR projects. Guaranteed: tracking won't be lost, lighting will be realistic, and the user won't feel discomfort. Certified Apple and Google developers.

Why does tracking get lost and how to fix it?

ARKit and ARCore use VIO (Visual-Inertial Odometry) — a combined processing of camera data and IMU. Tracking fails in three scenarios: illumination below ~50 lux, texture-homogeneous surfaces (white wall, glass), and fast camera movements.

In practice, if the product is intended for furniture try-on, we add an explicit UI warning when ARCamera.TrackingState.limited(.insufficientFeatures). An app that silently loses tracking gets 2-star reviews — we don't allow that.

Plane detection is configured via ARWorldTrackingConfiguration.planeDetection = [.horizontal, .vertical]. Important: ARKit continues to refine plane geometry through ARSCNViewDelegate.renderer(_:didUpdate:for:) — if you don't handle updates, the object starts floating when the anchor is refined. Our team solves this at the architecture stage, not during testing.

AR Foundation: cross-platform with nuances

Unity AR Foundation is an abstraction layer over ARKit and ARCore. It reduces development time by 40% compared to separate native codebases. But some features (e.g., ARBodyTrackingConfiguration for body tracking) are unavailable and require a native plugin.

For React Native and Flutter, direct AR Foundation is missing. We use ViroReact (React Native) or ar_flutter_plugin for simple scenarios, but for production quality — native modules with a bridge. Hybrid approach: AR scene rendered in native ARKit/ARCore view, control from JS/Dart via method channel. Included in our standard delivery.

Platform comparison: ARKit outperforms ARCore in tracking stability and feature set (30% fewer failures in low-light scenarios), but ARCore is cheaper in device support. AR Foundation is a compromise: loses up to 20% performance on complex scenes but pays off with a single codebase.

Try-on: product fitting via AR

Fitting glasses, jewelry, cosmetics — a separate class of tasks. Here, face tracking is needed, not plane detection.

ARKit provides ARFaceTrackingConfiguration — 52 blend shape coefficients for expressions, 3D face mesh, position and orientation in space. Works only on devices with TrueDepth camera (iPhone with Face ID).

For Android, the equivalent is ML Kit Face Mesh Detection or Google ARCore Augmented Faces (Pixel and some flagships). For cross-platform try-on, we use Banuba Face AR SDK (Banuba Face AR SDK documentation) — covers both devices, provides ready-made masks and stable tracking even on mid-range Android.

Try-on quality critically depends on 3D product models. Models must be optimized for real-time: no more than 10-15K polygons for jewelry, PBR materials with correct roughness/metallic maps, LOD for long distances. Within our engagement, we provide ready-made optimization guides.

How to achieve realistic lighting in AR?

ARKit with modern iOS versions supports Environmental Texturing — automatic creation of an environment map from the camera for realistic reflections. Enabled via ARWorldTrackingConfiguration.environmentTexturing = .automatic. Without it, metallic and glass materials look plastic.

ARCore provides Light Estimation — intensity and color temperature of ambient light, applied to the shader of virtual objects. In practice, it's the difference between an object that blends into the scene and an obviously overlaid 3D model. We guarantee that the final image doesn't betray virtuality.

What's included

• AR solution architecture (stack choice, module design)
• 3D pipeline: model optimization for real-time, PBR materials, LOD
• Tracking integration (planes, faces, images, objects)
• Testing on 10+ real devices (iOS and Android)
• Documentation for SDK usage and ready components
• Post-launch support (1 month bug fixing)

Timeline and estimation

Simple AR scene with placing one 3D model on a plane — 1-2 weeks. Face try-on with product catalog — from 6 weeks (3D pipeline, tracking integration, selection and saving UI). Full AR shopping with cloud anchors and multiplayer — from 3 months. We'll estimate your project in 1 day — contact us to discuss your AR idea.