Technical Implementation of Accurate Glasses Try-On in a Mobile App
A client approaches with a task: implement glasses try-on in a mobile app so users can see how a model fits on their face. The first attempt with WebAR via three.js results in jitter and inaccurate positioning — the model 'floats' 2–3 mm when the head turns. ARKit and ARCore solve this: a face mesh of 1220 points provides stability down to 0.1 mm. Virtual try-on for accessories, including glasses and headphones, is becoming the retail standard. In one project for an optics chain, we integrated 40 glasses models on both platforms — conversion to purchase increased by 23%. Glasses are an ideal candidate for AR: fixed shape, attachment to the nose bridge and ears, no mesh deformation required. But 'well-fitting' glasses require correct pivot, proper scaling to IPD, and stable animation during head movement. Our accumulated experience includes 6+ AR try-on projects for retail. One client noted: "After implementing AR try-on, return rates dropped by 30%".
How to Achieve Accurate Glasses Placement on the Face
ARFaceTrackingConfiguration requires iPhone X or newer with a TrueDepth camera. It returns ARFaceAnchor with:
-
transform— position and orientation of the face in world coordinates -
geometry— 1220-point face mesh -
blendShapes— 52 coefficients for expressions (blink, smile, brow raise)
For glasses, we use specific face mesh points: nose bridge (~vertex 9 in ARKit mesh), temples. The glasses model's pivot point is strictly on the nose bridge. With a correct pivot, the frame sits on the face without additional position corrections.
// Attach glasses model to face anchor let faceAnchorEntity = AnchorEntity(.face) let glassesEntity = try! ModelEntity.load(named: "glasses.usdz") glassesEntity.position = SIMD3(0, 0.005, 0.065) // fine Y, Z correction faceAnchorEntity.addChild(glassesEntity) arView.scene.addAnchor(faceAnchorEntity) How to Scale the Model to Real Face Size
A standard glasses model in USDZ is for an 'average' face. Real faces vary. Two approaches:
- Automatic. Measure face width via distance between left and right ears in
ARFaceAnchor.geometry. ARKit provides these points in meters — compare with the frame's reference width, compute scaling factor. Apply toModelEntity.scale. - Manual. A slider in the UI. Simpler to implement, worse UX.
Interpupillary distance (IPD) via eye tracking is available on iPad Pro (ARKit, ARFaceAnchor.leftEyeTransform, rightEyeTransform). On iPhone, face tracking does not directly measure IPD, but it can be approximated via face geometry.
Anchor Points for Different Accessories
Same principles, different anchor points:
- Headphones/earrings →
leftEarTransform,rightEarTransformfromARFaceAnchor. ARKit provides these transforms directly. - Hats →
ARFaceAnchor.transform(head center) + Y offset upward. Requires accounting for head tilt — otherwise the hat 'slides' when tilting. - Sunglasses with lenses. Tinted/gradient lenses use PBR with transmission and tint color. In RealityKit —
.PhysicallyBasedMaterialwithopacityThresholdand blending. Mirrored lenses use environment texture reflection.
ARKit vs ARCore vs MediaPipe
| Technology | Mesh Accuracy | Availability | Performance |
|---|---|---|---|
| ARKit (TrueDepth) | 1220 points (high) | iPhone X+ | Excellent, GPU |
| ARCore (depth) | 468 points (medium) | Pixel 3+, Galaxy S10+ | Good |
| MediaPipe ML Kit | ~468 points (low on motion) | Any Android | Average (CPU) |
ARKit is 2–3 times more accurate than ARCore in mesh point count. For glasses, MediaPipe accuracy is sufficient for static frames, but jitters on motion. We use MediaPipe only as a fallback for devices without depth sensors.
Performance and Memory
On iPhones older than XR, simultaneous face tracking + high-poly model + environment texturing loads the GPU. Metal frames with reflection require environment probes — additional draw calls.
Optimization: LOD (level of detail) for glasses — full model on static frame, simplified on motion. Textures 512×512 for small frame details are sufficient for mobile screens. We guarantee stable 30 FPS even on devices with A12 Bionic.
Step-by-Step Implementation Process
- Analytics and 3D model preparation: retopology, pivot setting, optimization for USDZ (iOS) and GLB (Android).
- Face tracking integration: ARKit ARFaceTrackingConfiguration / ARCore AugmentedFaces / MediaPipe with calibration for the specific model.
- UI/UX development for try-on: frame selection, size slider, 'Try On' button, photo sharing.
- Testing on 10+ real devices across generations.
- Documentation and training: source code handover, guide for adding new models.
Technical Implementation Details
We use PBR materials with environment reflections for metal frames. For mirrored lenses, we apply texture mapping based on cubemaps. LOD is calculated automatically — at distances >0.5 m, the model is replaced by a low-poly version with 300 polygons.Comparison of Scaling Approaches
| Approach | Accuracy | UX | Implementation Complexity |
|---|---|---|---|
| Automatic | High (based on ear distance) | Excellent | Medium |
| Manual | Medium (depends on user) | Good | Low |
The automatic approach improves placement accuracy by nearly 50% compared to manual. Budget is determined after an individual assessment.
Estimated Timelines
Face tracking + basic AR try-on for one glasses model — 5–7 days. Frame gallery, face scaling, sharing — 2–3 weeks. Android support via ARCore AugmentedFaces — plus 1 week. Contact us for a prototype demonstration on your device. Get a consultation from an engineer on integrating AR into your product.
Frame selection via AR allows clients to try on up to 40 models per minute. Virtual glasses try-on for iOS and Android is our specialty. Reach out for details.