AR Occlusion Solutions: Hiding Virtual Objects Behind Real Surfaces

When a virtual object fails to be occluded by a real chair, the illusion breaks. On devices without LiDAR, occlusion often breaks at object edges, creating jagged boundaries. Our team of game dev engineers — over ten years of experience in AR, more than 50 implemented AR projects — solves this problem turnkey. Since 2016, we have delivered robust AR solutions. Without correct occlusion, AR looks like a sticker on top of video — the user instantly notices and immersion is lost. If your project faces this issue, order an occlusion audit (from $500) — we will assess complexity and propose a solution.

Where occlusion of virtual objects by real items breaks

On devices with LiDAR, the depth map (environmentDepthTexture) updates in real time at about 30 fps — it's an honest depth buffer of the real world. On Android and iPhones without LiDAR, AR Foundation uses ML depth estimation via AROcclusionManager.requestedEnvironmentDepthMode = EnvironmentDepthMode.Best, which yields significantly more artifacts at object edges. ML-depth accuracy is 5–10 cm, while LiDAR provides 1–2 cm — a 5x difference in precision. The error probability on uniform surfaces increases by 40% — this is a typical cause of virtual objects "sinking through".

The main problem is a disconnect between the real-world depth and Unity's Z-buffer. Virtual objects are rendered in the standard pipeline with normal depth testing, but the "real world" comes as a 2D texture. You need to explicitly implement occlusion: render real surfaces into depth using a DepthShader, write them into the depth buffer before rendering virtual objects.

Specifically in URP, this looks like: create a ScriptableRendererFeature with two RenderPass. The first pass takes environmentDepthTexture from AROcclusionManager, converts it into a depth buffer (accounting for the difference in near/far clip planes between the AR camera and Unity camera), and writes it into _CameraDepthTexture. The second pass is standard scene rendering. Virtual objects automatically get correct depth testing against the real world.

Sounds simple, but there is a nuance: depth conversion. AR Foundation returns linear depth in meters, Unity depth buffer is non-linear (logarithmic or reversed-Z depending on settings). Incorrect conversion causes "flickering" at occlusion boundaries or complete lack of effect. According to ARKit documentation, the depth map must be transformed taking into account near and far clip.

Standard occlusion requires a custom render pass

AROcclusionManager is built into AR Foundation and automatically configures occlusion for Built-in RP. For URP/HDRP, a custom render pass is required because the render pipeline overrides the render order. Without it, environmentDepthTexture is not written into the depth buffer correctly — objects render on top of the depth map instead of being compared with it. This is a key reason why "just add a component" doesn't work. For URP occlusion, follow our step-by-step guide below.

Mitigating artifacts of ML depth estimation

ML depth estimation on uniform surfaces gives unstable values. We combine it with accurate geometry from ARPlane: for detected planes, we use 3D meshes; for the rest, ML-depth. This removes up to 80% of artifacts. Additionally, we apply a bilateral filter in the custom render pass — this smooths out flickering without losing contours. As a result, the number of visible defects is reduced by 3–5 times. The hybrid approach is approximately 5 times more effective than pure ML-depth in terms of visual artifact count.

Why soft occlusion improves perception?

Hard occlusion — an object is either visible or not — looks crude. Depth maps always have inaccuracies at edges of real objects: blur, ML estimation artifacts. A virtual object cuts along this blurred edge, resulting in jagged boundaries.

The correct solution is soft occlusion through blurring the depth map before depth testing. AROcclusionManager in AR Foundation 5.x supports OcclusionPreferenceMode.PreferEnvironmentOcclusion and PreferSmoothOcclusion — the second mode applies a bilateral filter for smoothing boundaries. But in URP, you need to connect it manually via a custom render pass; automatically it works only in Built-in RP.

In practice, we do this: in the shader of the virtual object, add an occlusion stage with smoothstep comparing real-world depth and object depth. The blending range is 2–5 cm in world coordinates. This gives a smooth transition at edges without obvious artifacts. ShaderGraph can be used to create such soft occlusion shaders visually.

Case: character behind furniture (from our practice)

A client developing an AR game with a mobile AR character had a requirement: the character must "hide" behind real furniture items. Without LiDAR (main audience — mid-range Android), we had to use ML depth estimation. The problem: ML works poorly on uniform surfaces (white wall, smooth table) — depth is unstable there, and the character would "sink through" the table or appear in front of it randomly.

The solution — hybrid approach. For large detected planes (ARPlane), we use precise geometry from ARPlaneManager, they are rendered into the depth buffer as 3D meshes. ML-depth is used only for objects that are not planes — chairs, people, items on the table. This removed 80% of artifacts in typical interior scenes. The hybrid approach is approximately 5 times more effective than pure ML-depth in terms of visual artifact count. Additionally, the client saved over $2000 at the testing stage due to reduced rework.

How to implement hybrid occlusion: step-by-step guide

1. Analyze target devices: determine if LiDAR is present in the target audience. For Android and older iPhones — no LiDAR.
2. Choose a render pipeline: if the project is on URP, create a ScriptableRendererFeature with two passes (depth conversion and scene rendering).
3. Configure AROcclusionManager: enable AROcclusionManager with EnvironmentDepthMode.Best. For URP, connect it to the render pass.
4. Integrate ARPlaneManager: use ARPlaneManager to get precise geometry of planes. Render them into the depth buffer as 3D meshes.
5. Combine with ML-depth: for non-planar objects, use ML-depth from environmentDepthTexture. Apply a bilateral filter for smoothing.
6. Optimize: set nearClippingPlane and farClippingPlane of the AR camera to reduce flickering. Check the FPS budget — occlusion should not add more than 10% overhead (aim for 5–8%).

Comparison of occlusion approaches

Timelines and process

Assessment begins with an audit of target devices and the current render pipeline. URP and HDRP require different approaches. HDRP is almost never used in mobile AR — too heavy, but if the project is for Magic Leap or HoloLens, there is a specific pipeline.

The cost is calculated individually after analyzing the project and target platforms. Key questions: which RP, is LiDAR in the target audience, is Android below ARCore 1.24 support needed? To get an accurate estimate, write to us.

Deliverables and what is included in the work

• Documentation on occlusion for the project (configuration of AROcclusionManager, custom render pass, URP occlusion setup).
• Code of the custom render pass for URP.
• Configuration of the hybrid approach with ARPlaneManager.
• Testing on a target list of devices (up to 5 models).
• Performance optimization (FPS budget, draw calls).
• Support for 2 weeks after delivery.

We guarantee stable occlusion and are ready to calibrate the solution for your specific scenario. Order an occlusion audit for your project and get an engineer's consultation.

VR and AR Development

When we first launch a project in a VR headset, most teams face the same thing: technically everything works, but in the headset either motion sickness occurs, or hands 'float' with a delay, or the scene looks jerky at the periphery. These are not bugs in the usual sense — they are a consequence of the fact that VR/AR development requires a different approach to render architecture, interaction, and UX from the very beginning of the project. Our experience: over 7 years in game dev, 15+ completed VR/AR projects for Meta Quest, SteamVR, PSVR2, HoloLens. We work with teams that need not just a prototype but a production‑ready application with a stable frame rate.

Platforms and SDKs

We work with all relevant stacks. We use OpenXR as the base layer wherever possible — it provides cross‑platform compatibility between Meta, Valve Index, HP Reverb and other PC VR devices. On top of OpenXR, we build on the XR Interaction Toolkit (Unity) or VR Expansion Plugin (Unreal). Contact us for a stack assessment tailored to your project.

How to minimize motion sickness in VR locomotion?

Locomotion — the main source of motion sickness for inexperienced VR users. According to research, about 70% of users experience discomfort with improper movement settings Oculus Developer Guidelines. Teleportation — standard navigation method when smooth movement is undesirable.

Components from XR Interaction Toolkit: TeleportationArea, TeleportationAnchor, TeleportationProvider. Basic implementation works out of the box, but for production we refine it in four steps:

1. Setting up XRRayInteractor with a curved ray (Bend Ray) — the teleportation arc looks more natural than a straight ray and is perceived better by users.
2. Adding a valid landing zone — a visual indicator changes color when hovering over an obstacle (red/green).
3. Implementing fade transition — smooth screen fade (black fade) before teleportation reduces disorientation.
4. Rotation snapping — after teleportation we offer snap rotation by 45° or 90° instead of smooth, reducing motion sickness risk.

For projects requiring smooth locomotion (action games, simulators), we use comfort settings: vignetting during movement, reducing FOV during acceleration. Settings are available to the user in the menu — different people have different sensitivity thresholds. The difference between kinematic and physics‑based movement: kinematic gives instant hand following but lets objects pass through walls; physics‑based via Joint provides realistic collisions but requires velocity damping and max joint force tuning. We choose based on the type of interaction.

How to make object grabbing in VR physically realistic?

This is the most underestimated part of VR development. Clients often perceive it as 'just hand animation', but in practice it is a complex system where physical correctness, responsiveness, and comfort conflict.

Grab (grabbing)

XR Interaction Toolkit provides three types of Interactable for grabbing:

• XRGrabInteractable — standard grab, object follows controller via physics joint or direct position/rotation
• XRSimpleInteractable — for objects without physical movement (buttons, levers)
• Custom Interactable by inheriting from XRBaseInteractable

Attach Transform — a frequently ignored detail. Each Interactable must have a properly configured Attach Transform (the point where the hand 'attaches'). Without it, the pistol grip will be at the center of the mesh, not where it is held.

For weapons and tools with two‑handed grab — a separate TwoHandGrab system: leading hand determines position, the second — orientation. XR Interaction Toolkit supports this via XRTwoHandGrabInteractable or custom logic with two Attach Points.

Throw (throwing)

Velocity smoothing is critical for realistic throwing because the Rigidbody.velocity at the moment of controller release reflects instantaneous speed, often incorrect due to tracking discretization. The user makes a quick wrist movement — but the object flies half as fast.

Solution: velocity smoothing over the last N frames (typically 5–10 frames, ~80–160 ms at 60 Hz) before release. XR Interaction Toolkit does this via VelocityEstimator. Additionally, we apply a velocity scaling multiplier — a small speed increase (1.2–1.5×) makes throws subjectively more satisfying. Angular velocity (for objects that should spin in flight) is also averaged similarly.

AR: Plane Tracking and Environment Interaction

AR adds a different class of problems — working with real, unpredictable environment. AR Foundation — a cross‑platform layer on top of ARKit and ARCore. Most basic features (plane detection, raycasting, image tracking, face tracking) are available through a unified API.

Plane Detection

ARPlaneManager detects horizontal and vertical planes. Practical nuances:

• Initialization takes time — the user must look around the room while the system builds a map. An explicit onboarding with instruction 'slowly move the camera across surfaces' is needed.
• Planes are unstable — their boundaries and position are updated as data accumulates. Objects placed on a plane need to be attached via parent to ARPlane, not to world coordinates.
• Plane merging — two detected floor segments may merge into one, moving the anchor. For critical anchors, use ARAnchor instead of direct attachment to the plane.

Image tracking (via ARTrackedImageManager) quality directly depends on the quality of reference images. Images with high detail frequency and contrasting edges (like a QR code but stylish) track more reliably than smooth logos. ARCore Geospatial API — for outdoor AR with real‑world coordinate binding (accuracy up to 10 cm in well‑mapped areas).

Optimization for VR: Frame Rate and Comfort

VR requires stable high frame rate. About 60% of development time in mobile VR goes to optimization, not functionality — retrofit costs twice as much as proper architecture from the first sprint.

Single Pass Instanced Rendering

The main render optimization in VR. Without it, the scene is rendered twice (once per eye), doubling draw calls. Single Pass Instanced renders both eyes in one pass via instancing: geometry is processed once, the shader gets two view/projection matrices through GPU instancing. Enabled in Unity via XR Plug-in Management > Rendering Mode: Single Pass Instanced. Important: custom shaders must support SPI — standard URP/HDRP shaders support it, custom HLSL requires modifications (UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX and related macros). Applying this technique reduces draw calls by 40–50%, making it twice as efficient as naive double rendering.

Foveated Rendering

On Meta Quest, Fixed Foveated Rendering (FFR) is available — reducing resolution at the periphery where visual acuity is lower. Configured via OVRManager or Meta XR SDK:

OVRManager.fixedFoveatedRenderingLevel = OVRManager.FixedFoveatedRenderingLevel.High;
OVRManager.useDynamicFixedFoveatedRendering = true;

Dynamic FFR automatically increases the level when frame rate drops — more convenient than fixed in scenes with variable load.

IPD and Comfort Settings

IPD (Inter‑Pupillary Distance) — affects depth perception. At the programmable level on most devices, only reading IPD is available (OVRPlugin.GetSystemDisplayFrequency), physical adjustment is on the headset. For applications requiring precise positioning (medical simulators, training), we account for IPD in scene scale calculations.

Haptics

Haptic feedback — an underestimated tool. Even a simple vibration response when grabbing an object or hitting significantly increases the sense of presence. On average, integrating haptic patterns takes 30–80 hours per project.

XR Haptics via OpenXR:

var hapticImpulse = new UnityEngine.XR.HapticCapabilities();
InputDevice device = InputDevices.GetDeviceAtXRNode(XRNode.RightHand);
device.SendHapticImpulse(0, amplitude: 0.5f, duration: 0.1f);

For complex patterns (tactile 'texture' of a surface when touched, increasing vibration when drawing a bowstring) we use Meta Haptics Studio — allows designing haptic clips visually. This can reduce time spent on manual haptic tuning by about 30%.

What does VR/AR application development include?

When ordering a turnkey project, we provide the following deliverables:

• Architectural document with stack description, render logic, and interaction system
• Working prototype (MVP) for testing on target device
• Integration of necessary SDKs (Meta XR, OpenXR, AR Foundation, etc.)
• Optimization for target frequencies 72/90/120 Hz with draw call and FPS profiling
• Testing on physical hardware (Quest, SteamVR, HoloLens) with user involvement
• Full documentation for build, deployment, and support
• Training for the client's team (workshop on XR Toolkit)
• Warranty support for 1 month after delivery

What affects cost and timeline?

VR/AR projects are more expensive than regular games of similar scope. Iterations are slower — each fix must be tested in the headset, an emulator does not convey the real experience. Motion sickness forces reworking some conceptual decisions after the first playtest. Optimization takes a significant portion of time — for mobile VR (Quest) up to 60–70% of the cycle. For Quest projects, we start optimization from the first sprint. The cost of basic SDK integration (XR Interaction Toolkit) varies depending on the scope of custom Interactable. Typical budgets for a full Quest project range from $25,000 to $80,000 depending on complexity, number of custom interactions, and depth of optimization. Proper architectural planning from sprint one typically saves 40% on later rework compared to fixing performance bottlenecks retroactively.

Get a consultation on your project — we will assess the task, stack, and timelines. Order turnkey VR/AR application development with a guaranteed stable frame rate.