Scene Reconstruction in AR: 3D Mesh, Surface Classification, Occlusion

We integrate Scene Reconstruction into AR applications when accurate interaction with the real environment is required. Without LiDAR on devices (iPhone 12 Pro and newer, iPad Pro 2020+) scene reconstruction is impossible—only plane detection is available. A typical problem: after integrating ARKit with sceneReconstruction = .mesh, the app starts lagging because every mesh vertex is recreated each frame. We solve this by working directly with Metal buffers and updating only the changed anchors. In one warehouse project, we processed up to 30,000 vertices per frame while maintaining 60 FPS. Let's dive into the technical details.

How scene reconstruction works?

ARMeshAnchor accumulates room geometry in real time. ARMeshGeometry stores vertices, normals, and triangle indices. Updates come through the delegate session(_:didUpdate:)—each frame ARKit can send dozens of updated anchors. A naive implementation that recreates a MeshResource on every update kills the main thread in seconds. The correct approach: update the mesh only for changed anchors, use MDLMesh as an intermediate format, and pass data directly into Metal buffers.

In RealityKit it looks like this:

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { for anchor in anchors.compactMap({ $0 as? ARMeshAnchor }) { updateMeshVisualization(for: anchor) } } func updateMeshVisualization(for anchor: ARMeshAnchor) { let geometry = anchor.geometry // Work with geometry.vertices, geometry.faces directly // Do not create a new MeshResource every time—patch the existing one } 

Why surface classification is critical for AR?

ARMeshClassification provides types: floor, ceiling, wall, door, window, seat, table, none. Using classification, you can filter out unnecessary surfaces—for example, ignore the ceiling for floor navigation. Classification works only when sceneReconstruction = .meshWithClassification and on LiDAR devices. Without checking ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification)—crash or silent ignore.

How we avoid performance drops?

Core stack: ARKit 5+ + RealityKit 2 + Metal. We don't use SceneKit for the mesh—it is not optimized for dynamic geometries. Session setup:

let config = ARWorldTrackingConfiguration() config.sceneReconstruction = .meshWithClassification arView.debugOptions = [.showSceneUnderstanding] // for debugging arView.session.run(config) 

For debugging mesh visualization, we draw wireframe via arView.debugOptions. In production, mesh visibility is turned off, but we use its data for:

• Occlusion—objects behind walls are not visible. Scene Reconstruction gives 5x more accurate occlusion than approximate plane detection.
• Physics—CollisionComponent interacts with real geometry.
• Raycast—precise hit on real surfaces, not just planes.

Case study: navigation AR app for a warehouse. Needed to detect obstacles (shelves, pallets) and build a route. Used Scene Reconstruction to build an occupancy grid: each mesh vertex with classification .none (unrecognized object) was added to the obstacle graph. NavMesh updated every 2 seconds—a balance between freshness and CPU load. On iPad Pro M2 this maintains 60 FPS without drops. For comparison: on devices without LiDAR, we would have to use simplified collisions, increasing false positives by 40%.

What's included in our work?

• Requirements analysis and stack selection (ARKit/RealityKit/Metal).
• Session configuration with scene reconstruction enabled.
• Mesh visualization development (debug and production).
• Integration of mesh data: occlusion, physics, raycast.
• Performance optimization (update pooling, buffering).
• Testing on LiDAR devices and fallback for non-LiDAR.
• Documentation and team training.

Process: from idea to deployment

1. Analysis—discuss usage scenarios and determine required data (occlusion, navigation, physics).
2. Design—choose architecture (component system, integration of ARView and Metal).
3. Implementation—write code in Swift, use delegate pattern for updates.
4. Testing—check on real devices in various conditions (lighting, mirrors).
5. Deployment—release to App Store, set up TestFlight for beta testers.

Timelines and how to start

Basic integration with mesh visualization takes 1–2 weeks. If surface classification, physics collisions, and mesh-based navigation are needed—4–6 weeks. The cost is calculated after a detailed discussion of requirements. Contact us for a consultation—get an estimate for your project. You can also order a preliminary analysis of your AR app.