ARKit Integration in iOS: Tracking, Occlusion, Face Tracking

ARKit Integration: From Tracking to Face Tracking

Effective ARKit integration requires understanding of tracking, occlusion, and performance trade-offs. In our practice, ARKit isn't just about overlaying 3D objects on camera. It's a combination of a LiDAR scanner, Visual Inertial Odometry, and neural network models for scene understanding. Improper use leads to unstable tracking, occlusion artifacts, and ARSession crashes in the background. Below we break down key problems and solutions based on experience with ARKit development in real projects.

Why Does Tracking Fail in Dark Scenes?

Tracking degrades in poor lighting. ARWorldTrackingConfiguration builds a map from feature points — in dark rooms or on uniform surfaces, tracking drops to ARCamera.TrackingState.limited(.insufficientFeatures). Ignoring this state is dangerous: if you don't prompt the user to light the scene, anchors drift and the 3D object flies away. The correct approach is to subscribe to session(_:cameraDidChangeTrackingState:) and show appropriate instructions.

Speeding Up Plane Detection

Plane detection has a delay. ARPlaneDetection.horizontal finds the floor in 2–4 seconds with good lighting. If the user tries to place an object earlier, it misses. A common solution: make plane detection optional and add a raycast against ARMeshAnchor with LiDAR — on iPhone 12 Pro and later, this gives ~1 cm accuracy without waiting. Comparison: raycast is 40% more stable than the deprecated hitTest and 10x faster to deliver a result.

What to Do About Occlusion Without LiDAR?

Occlusion requires LiDAR. ARView.environment.sceneUnderstanding.options with .occlusion — only on LiDAR-equipped devices (Pro series from iPhone 12). On iPhones without LiDAR, the object always stays on top of the real world. Check ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) at startup and degrade gracefully: disable occlusion and use semi-transparent shadows.

Setting Up Image Tracking

ARImageTrackingConfiguration with maximumNumberOfTrackedImages = 4 — for marker-based AR. Load reference images from ARReferenceImage with physical size. physicalWidth is critical — ARKit uses it to compute distance and scale. An incorrect size means the object appears in the wrong place.

let referenceImage = ARReferenceImage(
    cgImage,
    orientation: .up,
    physicalWidth: 0.15 // 15 cm in real world
)
config.trackingImages = [referenceImage]

Using Face Tracking with ARKit

ARFaceTrackingConfiguration works only on devices with a TrueDepth camera (Face ID). It provides 52 blend shape coefficients via ARFaceAnchor.blendShapes — from .jawOpen to .eyeBlinkLeft. This powers virtual try-ons, AR avatars, and Liveness Detection.

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
    let jawOpen = faceAnchor.blendShapes[.jawOpen]?.floatValue ?? 0
    // Control character animation
}

Boosting AR App Performance

ARKit + RealityKit load both GPU and CPU simultaneously. Diagnostic tools: Xcode Reality Composer Pro for scene preview, GPU Frame Capture for analyzing draw calls, Metal System Trace in Instruments for detecting GPU bubbles.

Typical bottlenecks:

• USDZ with >100K polygons on mid-range devices — frame rate drops to 30 FPS.
• Multiple ModelEntity without instancing duplicates geometry in memory (memory usage increases by 50%).
• arView.renderOptions — disabling .disableDepthOfField and .disableMotionBlur gives +15% FPS on older devices.

Tracking Configuration Comparison

Performance Optimizations

Step-by-Step Object Placement

1. Start an ARWorldTrackingConfiguration session with planeDetection and environmentTexturing enabled.
2. Wait for session(_:didAdd:) or perform a raycast from the screen center.
3. Create an ARAnchor from the raycast result.
4. Load a USDZ model via ModelEntity.loadModel.
5. Create an AnchorEntity bound to the anchor.
6. Add the AnchorEntity to the scene.

Raycast Placement Code Example

let query = arView.makeRaycastQuery(
    from: arView.center,
    allowing: .estimatedPlane,
    alignment: .horizontal
)

if let query = query,
   let result = arView.session.raycast(query).first {
    let anchor = ARAnchor(name: "placed-object", transform: result.worldTransform)
    arView.session.add(anchor: anchor)

    let modelEntity = try! ModelEntity.loadModel(named: "model.usdz")
    let anchorEntity = AnchorEntity(anchor: anchor)
    anchorEntity.addChild(modelEntity)
    arView.scene.addAnchor(anchorEntity)
}

What's Included in ARKit Integration

• Audit of requirements and device compatibility (target models, LiDAR availability).
• Designer of AR architecture with graceful degradation on older devices.
• Development of AR core: session, anchors, plane/image/face detection.
• Integration of 3D content: conversion to USDZ, PBR materials setup.
• Testing on physical devices (minimum 5 models).
• Performance optimization for target devices.
• Source code delivery, documentation, and team training.

Integration Timelines and Cost

Basic integration with USDZ object placement on a plane: 3–5 days (from $1,500). Image tracking with markers and content overlay: 4–7 days (from $3,000). Comprehensive solution with face tracking, LiDAR occlusion, and custom Metal shaders: 3–6 weeks (from $12,000). Cost is calculated individually based on 3D content complexity and device support requirements. We guarantee high-quality results with over 5 years of experience and 20+ AR projects completed.

We have been developing iOS apps for over 5 years and have completed 20+ AR projects across various industries. If you want to implement augmented reality in your app, contact us — we'll evaluate your project and propose the optimal solution. Evaluate your project for AR capabilities — get a consultation from our engineer.

We develop AR applications on ARKit and ARCore that work stably even in challenging conditions. Our experience: 7+ years in mobile development and 30+ delivered AR projects. Guaranteed: tracking won't be lost, lighting will be realistic, and the user won't feel discomfort. Certified Apple and Google developers.

Why does tracking get lost and how to fix it?

ARKit and ARCore use VIO (Visual-Inertial Odometry) — a combined processing of camera data and IMU. Tracking fails in three scenarios: illumination below ~50 lux, texture-homogeneous surfaces (white wall, glass), and fast camera movements.

In practice, if the product is intended for furniture try-on, we add an explicit UI warning when ARCamera.TrackingState.limited(.insufficientFeatures). An app that silently loses tracking gets 2-star reviews — we don't allow that.

Plane detection is configured via ARWorldTrackingConfiguration.planeDetection = [.horizontal, .vertical]. Important: ARKit continues to refine plane geometry through ARSCNViewDelegate.renderer(_:didUpdate:for:) — if you don't handle updates, the object starts floating when the anchor is refined. Our team solves this at the architecture stage, not during testing.

AR Foundation: cross-platform with nuances

Unity AR Foundation is an abstraction layer over ARKit and ARCore. It reduces development time by 40% compared to separate native codebases. But some features (e.g., ARBodyTrackingConfiguration for body tracking) are unavailable and require a native plugin.

For React Native and Flutter, direct AR Foundation is missing. We use ViroReact (React Native) or ar_flutter_plugin for simple scenarios, but for production quality — native modules with a bridge. Hybrid approach: AR scene rendered in native ARKit/ARCore view, control from JS/Dart via method channel. Included in our standard delivery.

Platform comparison: ARKit outperforms ARCore in tracking stability and feature set (30% fewer failures in low-light scenarios), but ARCore is cheaper in device support. AR Foundation is a compromise: loses up to 20% performance on complex scenes but pays off with a single codebase.

Try-on: product fitting via AR

Fitting glasses, jewelry, cosmetics — a separate class of tasks. Here, face tracking is needed, not plane detection.

ARKit provides ARFaceTrackingConfiguration — 52 blend shape coefficients for expressions, 3D face mesh, position and orientation in space. Works only on devices with TrueDepth camera (iPhone with Face ID).

For Android, the equivalent is ML Kit Face Mesh Detection or Google ARCore Augmented Faces (Pixel and some flagships). For cross-platform try-on, we use Banuba Face AR SDK (Banuba Face AR SDK documentation) — covers both devices, provides ready-made masks and stable tracking even on mid-range Android.

Try-on quality critically depends on 3D product models. Models must be optimized for real-time: no more than 10-15K polygons for jewelry, PBR materials with correct roughness/metallic maps, LOD for long distances. Within our engagement, we provide ready-made optimization guides.

How to achieve realistic lighting in AR?

ARKit with modern iOS versions supports Environmental Texturing — automatic creation of an environment map from the camera for realistic reflections. Enabled via ARWorldTrackingConfiguration.environmentTexturing = .automatic. Without it, metallic and glass materials look plastic.

ARCore provides Light Estimation — intensity and color temperature of ambient light, applied to the shader of virtual objects. In practice, it's the difference between an object that blends into the scene and an obviously overlaid 3D model. We guarantee that the final image doesn't betray virtuality.

What's included

• AR solution architecture (stack choice, module design)
• 3D pipeline: model optimization for real-time, PBR materials, LOD
• Tracking integration (planes, faces, images, objects)
• Testing on 10+ real devices (iOS and Android)
• Documentation for SDK usage and ready components
• Post-launch support (1 month bug fixing)

Timeline and estimation

Simple AR scene with placing one 3D model on a plane — 1-2 weeks. Face try-on with product catalog — from 6 weeks (3D pipeline, tracking integration, selection and saving UI). Full AR shopping with cloud anchors and multiplayer — from 3 months. We'll estimate your project in 1 day — contact us to discuss your AR idea.