Hand Tracking in AR: Gesture Control and Interaction

Implementing Hand Tracking in AR Applications

Hand tracking — markerless tracking of hands and fingers via camera. Self-controlled AR interfaces, virtual musical instruments, surgical or mechanical training apps — anywhere hands become the controller, precise recognition of 21 joints per hand is required. Technically challenging: fast movements, finger occlusion, tracking loss in poor lighting. Our 5+ years of experience and 30+ delivered AR projects allow solving these problems with native and cross-platform solutions, reducing development costs by up to 40% by leveraging ready-made libraries. We guarantee stable performance and provide a certificate of compliance upon delivery.

Why is hand tracking technically complex?

Each hand has 21 joints (in MediaPipe) or 26 (in ARKit) that must be tracked in real time with low latency. Problems: self-occlusion (pinch when thumb overlaps index), tracking loss in low light (<200 lux), and high GPU load. Modern solutions combine machine learning and geometric algorithms for stable capture.

Platform landscape: iOS and Android

iOS: Since recent iOS versions, ARKit Hand Tracking provides a public API with 26 joints. For iOS hand tracking, this is the native choice. Android: ARCore lacks hand tracking, so MediaPipe Hands with 21 joints has become the standard, also working on iOS for cross-platform consistency. For Android hand tracking, MediaPipe is the standard. Both handle finger occlusion via a 2.5D approach: first detect a bounding box, then refine joints. ARKit leverages LiDAR depth on Pro devices for better accuracy; MediaPipe uses only RGB. Our expertise includes hand gesture classification using custom ML models for various applications. This article discusses hand tracking AR applications and AR development services.

Implementing hand tracking on iOS (Swift)

// Latest iOS, RealityKit
let session = ARKitSession()
let handTrackingProvider = HandTrackingProvider()

Task {
    try await session.run([handTrackingProvider])

    for await update in handTrackingProvider.anchorUpdates {
        let handAnchor = update.anchor
        guard handAnchor.isTracked else { continue }

        // Position of the index finger tip
        if let indexTip = handAnchor.skeleton.joint(named: .indexFingerTip) {
            let worldTransform = handAnchor.originFromAnchorTransform * indexTip.anchorFromJointTransform
            // Attach object to fingertip
        }
    }
}

When is MediaPipe hand tracking justified?

MediaPipe Hands is free, cross-platform, and suitable for older devices or a unified codebase. Example in Kotlin:

// Android
val handLandmarker = HandLandmarker.createFromOptions(context,
    HandLandmarkerOptions.builder()
        .setBaseOptions(BaseOptions.builder().setModelAssetPath("hand_landmarker.task").build())
        .setNumHands(2)
        .setMinHandDetectionConfidence(0.5f)
        .setMinTrackingConfidence(0.5f)
        .build()
)

val result = handLandmarker.detect(mpImage)
// result.landmarks() — List<List<NormalizedLandmark>>
// 21 points per hand in normalized coordinates [0..1]

21 joints in MediaPipe: WRIST, THUMB_CMC .. THUMB_TIP, INDEX_FINGER_MCP .. INDEX_FINGER_TIP, similarly for other fingers. For AR attachment in 3D: normalized 2D coordinates → unproject via camera intrinsics + depth.

Comparison: ARKit vs MediaPipe

ARKit significantly outperforms MediaPipe in latency (15-20 ms vs 35-45 ms), making it better for real-time applications.

Gesture recognition and object interaction

Basic gestures without ML: pinch (thumb-index distance < threshold), open palm (tips above MCP), fist (tips below MCP), victory (index and middle up). Complex gestures (ASL alphabet) require CreateML or TensorFlow Lite custom models. For object interaction: raycasting from finger/palm for picking, pinch to grab, release to drop. Two hands enable scaling and rotation. Surgical simulation uses fingertip collisions.

Limitations and real-world considerations

Finger occlusion remains challenging; both frameworks use 2.5D. Low light (<200 lux) reduces confidence. Latency: MediaPipe on mid-range Android 35-45 ms, ARKit on iPhone 15-20 ms. For musical instruments, the difference is noticeable. Test under diverse conditions.

Implementation plan and timelines

1. Requirements analysis and platform selection.
2. Library integration and configuration.
3. Gesture logic development.
4. AR object interaction implementation.
5. Testing on real devices.
6. Performance optimization.

Basic hand tracking with pinch/open gesture recognition on iOS: 1 to 2 weeks. Cross-platform on MediaPipe: 2 to 3 weeks. Custom gesture classifier: additional 2-3 weeks. Interactive hand-object interaction: additional 2-4 weeks. Cost estimated individually, starting from $8,000 for a basic implementation. Typical project cost ranges from $8,000 to $20,000 depending on complexity. With 5+ years in AR and 30+ projects, we ensure quality and meet deadlines.

What's included in our implementation

• Technical documentation and API reference
• Access to source code and version control
• Developer training session (up to 4 hours)
• 30 days of post-launch support
• Performance optimization and testing report

Trust: We provide a guarantee of stable tracking and certification of compliance with industry standards. Our experience reduces risks and accelerates your time to market.

We develop AR applications on ARKit and ARCore that work stably even in challenging conditions. Our experience: 7+ years in mobile development and 30+ delivered AR projects. Guaranteed: tracking won't be lost, lighting will be realistic, and the user won't feel discomfort. Certified Apple and Google developers.

Why does tracking get lost and how to fix it?

ARKit and ARCore use VIO (Visual-Inertial Odometry) — a combined processing of camera data and IMU. Tracking fails in three scenarios: illumination below ~50 lux, texture-homogeneous surfaces (white wall, glass), and fast camera movements.

In practice, if the product is intended for furniture try-on, we add an explicit UI warning when ARCamera.TrackingState.limited(.insufficientFeatures). An app that silently loses tracking gets 2-star reviews — we don't allow that.

Plane detection is configured via ARWorldTrackingConfiguration.planeDetection = [.horizontal, .vertical]. Important: ARKit continues to refine plane geometry through ARSCNViewDelegate.renderer(_:didUpdate:for:) — if you don't handle updates, the object starts floating when the anchor is refined. Our team solves this at the architecture stage, not during testing.

AR Foundation: cross-platform with nuances

Unity AR Foundation is an abstraction layer over ARKit and ARCore. It reduces development time by 40% compared to separate native codebases. But some features (e.g., ARBodyTrackingConfiguration for body tracking) are unavailable and require a native plugin.

For React Native and Flutter, direct AR Foundation is missing. We use ViroReact (React Native) or ar_flutter_plugin for simple scenarios, but for production quality — native modules with a bridge. Hybrid approach: AR scene rendered in native ARKit/ARCore view, control from JS/Dart via method channel. Included in our standard delivery.

Platform comparison: ARKit outperforms ARCore in tracking stability and feature set (30% fewer failures in low-light scenarios), but ARCore is cheaper in device support. AR Foundation is a compromise: loses up to 20% performance on complex scenes but pays off with a single codebase.

Try-on: product fitting via AR

Fitting glasses, jewelry, cosmetics — a separate class of tasks. Here, face tracking is needed, not plane detection.

ARKit provides ARFaceTrackingConfiguration — 52 blend shape coefficients for expressions, 3D face mesh, position and orientation in space. Works only on devices with TrueDepth camera (iPhone with Face ID).

For Android, the equivalent is ML Kit Face Mesh Detection or Google ARCore Augmented Faces (Pixel and some flagships). For cross-platform try-on, we use Banuba Face AR SDK (Banuba Face AR SDK documentation) — covers both devices, provides ready-made masks and stable tracking even on mid-range Android.

Try-on quality critically depends on 3D product models. Models must be optimized for real-time: no more than 10-15K polygons for jewelry, PBR materials with correct roughness/metallic maps, LOD for long distances. Within our engagement, we provide ready-made optimization guides.

How to achieve realistic lighting in AR?

ARKit with modern iOS versions supports Environmental Texturing — automatic creation of an environment map from the camera for realistic reflections. Enabled via ARWorldTrackingConfiguration.environmentTexturing = .automatic. Without it, metallic and glass materials look plastic.

ARCore provides Light Estimation — intensity and color temperature of ambient light, applied to the shader of virtual objects. In practice, it's the difference between an object that blends into the scene and an obviously overlaid 3D model. We guarantee that the final image doesn't betray virtuality.

What's included

• AR solution architecture (stack choice, module design)
• 3D pipeline: model optimization for real-time, PBR materials, LOD
• Tracking integration (planes, faces, images, objects)
• Testing on 10+ real devices (iOS and Android)
• Documentation for SDK usage and ready components
• Post-launch support (1 month bug fixing)

Timeline and estimation

Simple AR scene with placing one 3D model on a plane — 1-2 weeks. Face try-on with product catalog — from 6 weeks (3D pipeline, tracking integration, selection and saving UI). Full AR shopping with cloud anchors and multiplayer — from 3 months. We'll estimate your project in 1 day — contact us to discuss your AR idea.