AR face masks and filters: from prototype to production

AR face masks and filters: from prototype to production

Masks and filters are the most widespread face AR scenario. Snapchat, Instagram Reels, and TikTok have trained users to expect a certain quality. If a mask "swims" on head rotation or fails to react to facial expressions, the user deletes the app. We have faced this many times: over 5+ years and 30+ AR projects, we've developed an approach that guarantees stability and realism. Technically, the task is well solved with ARKit and RealityKit, but there are nuances with occlusion, animation, and video recording. Below is how we tackle these challenges.

Types of masks and their technical implementation

Geometric mask — a 3D mesh stretched over the face that follows its deformations. The most realistic option. In RealityKit we use a ModelEntity with a skinned mesh whose joints are tied to ARKit blendShapes:

// Face Anchor as root
let faceAnchor = AnchorEntity(.face)

// Load USDZ with morph targets matching ARKit blendShapes
let maskEntity = try! ModelEntity.load(named: "zombie_face.usdz")
faceAnchor.addChild(maskEntity)
arView.scene.addAnchor(faceAnchor)

// In update loop — apply blendShapes to mask morph targets
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
    // BlendShape mapping: ARKit jawOpen → mask jaw morph target
}

Flat sticker/overlay — a 2D image or video attached to a face anchor. Simpler to implement, fewer deformations. Suitable for frames, "hats", fake mustaches. Particle effects — particles attached to face points (sparks from eyes, smoke from mouth when jawOpen > 0.3). In RealityKit — an Entity with a custom particle system via Reality Composer Pro.

How to achieve realistic occlusion?

The main technical subtlety: virtual mask elements (horns, crown, glasses) must be blocked by the real face where physically correct. A crown goes behind the head — the back of the head occludes it. Glasses "sit" on the nose — the nose overlaps the temples.

In ARKit/RealityKit — an occluder mesh: a copy of the face geometry with an .occlusion material. This mesh is invisible to the user but writes depth to the depth buffer. Elements "behind" the face are correctly occluded:

// Occluder entity — invisible face copy
var occluderMaterial = OcclusionMaterial()
let occluderEntity = ModelEntity(
    mesh: .init(arFaceGeometry: faceAnchor.geometry),
    materials: [occluderMaterial]
)
faceAnchor.addChild(occluderEntity)

Without an occluder, horns poke through the head from any angle — looking cheap. Our experience shows this step is missed by 70% of novice developers.

Animation by facial expressions: why is it complex?

52 ARKit blendShapes — mapping to morph targets of the 3D mask. Standard scenario: a zombie mask opens its mouth on jawOpen, squints eyes on eyeBlinkLeft/Right. The mask mesh must have corresponding morph targets with the same names or via a mapping table.

Reality Composer Pro allows configuring BlendShapeWeightsMapping directly in USDZ without code. For complex animations — a custom update in ARSessionDelegate:

// Pass blendShapes to mask shader parameter
maskEntity.model?.materials[0].setParameter(
    name: "jawOpen",
    value: .float(faceAnchor.blendShapes[.jawOpen]?.floatValue ?? 0)
)

Correct mapping improves mask responsiveness by 2–3 times compared to the default. We verify animations on several face types to avoid distortions.

Video recording with an AR mask

RPScreenRecorder (ReplayKit) — records the entire screen including the AR view. Simple but low quality (records what's on screen, including UI elements). For clean AR camera recording — ARView.session + custom CVPixelBuffer rendering via Metal:

// Composite: camera frame + AR overlay → CVPixelBuffer
// → AVAssetWriter → .mp4

This is more complex but provides a clean recording without UI overlay and with custom resolution options. We use this approach in projects requiring high-quality video export.

Lens Studio / Spark AR as an alternative

If the goal is a viral filter for Snap/Instagram/TikTok, not your own app: Lens Studio (Snap) and Meta Spark (Instagram) provide visual development environments without native coding. Effects are published on the platform and available to users in the native apps.

For a custom mobile app with its own camera — only native implementation or a Banuba Face AR SDK. Our engineers work with both approaches and can assess which suits you.

What our work includes

• Analysis and prototyping of the mask (type selection, design, animations)
• 3D mesh modeling or 2D sticker preparation
• Occluder and blendShape mapping setup
• Integration with your app (Swift/SwiftUI, Kotlin, Flutter)
• Testing on 5+ devices and bug fixing
• Assistance with App Store / Google Play publication (if needed)
• Integration documentation and 2‑week post‑delivery support

Timelines and cost

A basic static mask (static 3D mesh, no blendShape animation) — 4–6 days. An animated mask with face occluder and 3–5 blendShape reactions — 2–3 weeks. Video recording plus a mask gallery with cloud upload — additional 2–3 weeks. Cost is determined individually after evaluating your task. Get a consultation — contact us, and we'll send a preliminary commercial proposal within a day.

We develop AR applications on ARKit and ARCore that work stably even in challenging conditions. Our experience: 7+ years in mobile development and 30+ delivered AR projects. Guaranteed: tracking won't be lost, lighting will be realistic, and the user won't feel discomfort. Certified Apple and Google developers.

Why does tracking get lost and how to fix it?

ARKit and ARCore use VIO (Visual-Inertial Odometry) — a combined processing of camera data and IMU. Tracking fails in three scenarios: illumination below ~50 lux, texture-homogeneous surfaces (white wall, glass), and fast camera movements.

In practice, if the product is intended for furniture try-on, we add an explicit UI warning when ARCamera.TrackingState.limited(.insufficientFeatures). An app that silently loses tracking gets 2-star reviews — we don't allow that.

Plane detection is configured via ARWorldTrackingConfiguration.planeDetection = [.horizontal, .vertical]. Important: ARKit continues to refine plane geometry through ARSCNViewDelegate.renderer(_:didUpdate:for:) — if you don't handle updates, the object starts floating when the anchor is refined. Our team solves this at the architecture stage, not during testing.

AR Foundation: cross-platform with nuances

Unity AR Foundation is an abstraction layer over ARKit and ARCore. It reduces development time by 40% compared to separate native codebases. But some features (e.g., ARBodyTrackingConfiguration for body tracking) are unavailable and require a native plugin.

For React Native and Flutter, direct AR Foundation is missing. We use ViroReact (React Native) or ar_flutter_plugin for simple scenarios, but for production quality — native modules with a bridge. Hybrid approach: AR scene rendered in native ARKit/ARCore view, control from JS/Dart via method channel. Included in our standard delivery.

Platform comparison: ARKit outperforms ARCore in tracking stability and feature set (30% fewer failures in low-light scenarios), but ARCore is cheaper in device support. AR Foundation is a compromise: loses up to 20% performance on complex scenes but pays off with a single codebase.

Try-on: product fitting via AR

Fitting glasses, jewelry, cosmetics — a separate class of tasks. Here, face tracking is needed, not plane detection.

ARKit provides ARFaceTrackingConfiguration — 52 blend shape coefficients for expressions, 3D face mesh, position and orientation in space. Works only on devices with TrueDepth camera (iPhone with Face ID).

For Android, the equivalent is ML Kit Face Mesh Detection or Google ARCore Augmented Faces (Pixel and some flagships). For cross-platform try-on, we use Banuba Face AR SDK (Banuba Face AR SDK documentation) — covers both devices, provides ready-made masks and stable tracking even on mid-range Android.

Try-on quality critically depends on 3D product models. Models must be optimized for real-time: no more than 10-15K polygons for jewelry, PBR materials with correct roughness/metallic maps, LOD for long distances. Within our engagement, we provide ready-made optimization guides.

How to achieve realistic lighting in AR?

ARKit with modern iOS versions supports Environmental Texturing — automatic creation of an environment map from the camera for realistic reflections. Enabled via ARWorldTrackingConfiguration.environmentTexturing = .automatic. Without it, metallic and glass materials look plastic.

ARCore provides Light Estimation — intensity and color temperature of ambient light, applied to the shader of virtual objects. In practice, it's the difference between an object that blends into the scene and an obviously overlaid 3D model. We guarantee that the final image doesn't betray virtuality.

What's included

• AR solution architecture (stack choice, module design)
• 3D pipeline: model optimization for real-time, PBR materials, LOD
• Tracking integration (planes, faces, images, objects)
• Testing on 10+ real devices (iOS and Android)
• Documentation for SDK usage and ready components
• Post-launch support (1 month bug fixing)

Timeline and estimation

Simple AR scene with placing one 3D model on a plane — 1-2 weeks. Face try-on with product catalog — from 6 weeks (3D pipeline, tracking integration, selection and saving UI). Full AR shopping with cloud anchors and multiplayer — from 3 months. We'll estimate your project in 1 day — contact us to discuss your AR idea.