AR Environment Occlusion Integration
We integrate realistic environment occlusion: virtual objects are occluded by real surrounding objects. Place an AR sofa against a wall, walk behind it — the sofa is partially hidden by your silhouette. Without occlusion, AR objects always render on top, breaking the illusion immediately. Technically, this is one of the most demanding effects because real-time depth information of the real world is required. Our experience shows that proper occlusion increases user engagement by 30–40% (based on A/B tests in our projects, consistent with Apple ARKit documentation). To achieve high-quality occlusion, device-specific characteristics must be considered and depth threshold properly tuned.
How LiDAR occlusion works
On iPhone 12 Pro and newer, iPad Pro, we use ARWorldTrackingConfiguration.sceneReconstruction = .mesh — a dense mesh of the environment. The mesh writes to the depth buffer, and AR objects behind the mesh are occluded. Accuracy is 1–2 cm at distances up to 5 meters, real-time with no latency. For devices without LiDAR, we use ML depth: a neural network predicts the depth map from RGB. ARKit's ARDepthMap / ARCore Depth API provide accuracy of 5–15 cm, with edge artifacts and 1–3 frame latency. For production: if the audience primarily uses iPhone Pro / iPad Pro, we use LiDAR. For mass-market apps, we recommend ML depth with LiDAR acceleration where available.
RealityKit: enable occlusion with one line
On devices with LiDAR and modern iOS:
arView.environment.sceneUnderstanding.options = [
.occlusion, // real objects occlude AR
.collision, // AR objects collide with real geometry
.physics, // physics relative to real surfaces
.receivesLighting // AR objects are lit like real ones
]
Without LiDAR on the same API: ARView uses person segmentation (A12 chips and newer) — only the human silhouette occludes AR objects. Limited, but better than nothing.
Why soft occlusion matters
At LiDAR mesh edges, occlusion can appear pixelated due to finite depth resolution. Soft occlusion via bilateral blur smooths the transition. In Metal: a custom kernel with depth-aware blur (blur only within the depth threshold). This is noticeable only on close inspection, but for premium AR products, it's worth implementing. We guarantee quality even on complex surfaces.
ARCore Depth API (Android)
On devices with a ToF sensor (Samsung Galaxy S21 Ultra, LG V60) — hardware depth. On others — ML depth from ARCore Depth API (Raw Depth Image):
val frame = session.update()
if (session.isDepthModeSupported(Config.DepthMode.AUTOMATIC)) {
val depthImage = frame.acquireDepthImage16Bits()
// depth in mm, 16-bit unsigned short
// ARCore recommends using via OpenGL texture
}
ARCore provides an OcclusionShader through their sample — an OpenGL fragment shader that discards fragments of AR objects deeper than real geometry.
Comparison of occlusion methods
| Method | Devices | Accuracy | Latency | Complexity |
|---|---|---|---|---|
| LiDAR mesh | iPhone 12 Pro+, iPad Pro | 1–2 cm | 0 frames | 2–3 days |
| ML depth (ARKit/ARCore) | All A12+ / Android with ARCore | 5–15 cm | 1–3 frames | 1–2 weeks |
| Person segmentation | A12+ (iPhone) | contour | 0 frames | 3–5 days |
Occlusion support on popular devices
| Device | LiDAR | ML Depth | Person Segmentation |
|---|---|---|---|
| iPhone 12 Pro | Yes | Yes | Yes |
| iPhone 11 | No | Yes | Yes |
| iPad Pro 2020 | Yes | Yes | Yes |
| Samsung S21 Ultra | No (ToF) | Yes | No |
| Pixel 4 | No | Yes | No |
We select the optimal method for your project, considering target audience and accuracy requirements.
Shadows from virtual objects on real surfaces
A bonus to occlusion: an AR sofa casts a shadow on the real floor. In RealityKit with sceneUnderstanding.receivesLighting, shadows are automatic. In a custom renderer: shadow map from the AR object projected onto real-world geometry from the LiDAR mesh. Without LiDAR, shadows are only on ARKit-detected planes (floor, table). The user sees a "flat" shadow that does not follow the real surface relief.
When should you use a custom shader instead of built-in API?
When soft occlusion or non-standard behavior (e.g., semi-transparent AR objects) is required, built-in APIs are insufficient. A custom Metal/GLSL shader gives full control over the depth buffer and blending. We implement such solutions with performance considerations for target devices.
Process
- Analysis: define target devices, accuracy requirements, and budget.
- Design: select the stack (LiDAR/ML) and design the rendering architecture.
- Implementation: API integration, custom shaders, testing on real scenes.
- Testing: A/B tests on different devices, checking for artifacts.
- Deployment: publish to App Store / Google Play, configure TestFlight.
Occlusion testing details
We test on a set of scenarios: different distances to objects, complex textures (carpet, grass), camera movement. We check for z-fighting and flickering at edges. For each device, we collect FPS and depth map latency metrics.With over 10 years of experience and 40+ AR projects delivered, we ensure robust integration.
What's included in the work
- Source code for occlusion integration (Swift / Kotlin / Dart).
- Custom shaders (Metal / OpenGL ES) if needed.
- Configured
ARWorldTrackingConfiguration/Config.DepthMode. - Build and deployment documentation.
- 30-day support after delivery.
Contact us to order occlusion integration for your AR application. Get a consultation for your project.
Checklist of common mistakes
-
.occlusionnot enabled — objects are not occluded. - Missing LiDAR handling — crash on older devices.
- Z-fighting due to too small offset in the shader.
- Ignoring person segmentation — humans do not occlude AR objects.
Timelines and pricing
LiDAR occlusion via RealityKit API — 2–3 days integration. Person segmentation — 3–5 days. ARCore Depth occlusion on Android — 1–2 weeks (custom shader). Soft occlusion with custom Metal/GLSL shading — plus 1–2 weeks. Pricing is individual. We estimate the project after discussing details. Reach out for a consultation.