Extension Development for Physical Stand Interaction in VR

id: 235 slug: physical-stand-vr-interaction-extension-development title_en: "Development of Extensions for Interaction with Physical Stands in VR" tags: [vr-ar]

Development of Extensions for Interaction with Physical Stands in VR

Physical stand in VR-installation — buttons, levers, pedals, touch panels, tactile devices — everything user touches with hands, virtual scene reacts. Bridge between hardware and Unity — not "connect Arduino and read Serial". This is synchronization of physical input with VR-space without lag, drift, and desynchronization on connection loss.

What Makes Physical Stand Integration Complex

Spatial positioning synchronization. If stand has physical steering wheel and VR has virtual one, they must coincide visually. Calibration error 5 mm at steering wheel rotation angle gives visible mismatch. Calibration built via reference markers: physical stand points (QR-codes or ArUco-markers) matched to virtual via AR Foundation Image Tracking, transformation matrix fixed and applied to stand coordinate system.

Connection protocol and delays. Between stand controller (Arduino, Raspberry Pi, industrial PLC) and Unity several options:

• USB HID — natively recognized as joystick via Input.GetJoystickNames(). Latency 1–8 ms, reliable. Limitation: up to 8 analog axes, 128 buttons in standard HID descriptor.
• Serial (COM-port) — universal for Arduino. System.IO.Ports.SerialPort in Unity works, but read blocking — must be separate thread with ConcurrentQueue<byte[]> for data transfer to main thread.
• UDP — for WiFi stands or multi-device installations. Latency 1–5 ms in local network, but no delivery guarantees — need own loss detection logic.
• WebSocket — if stand managed via browser interface or Node.js server. NativeWebSocket or websocket-sharp for Unity.

Physical feedback (haptics). Servos, vibromators, linear actuators on stand controlled by commands from Unity. This is bidirectional channel: Unity → stand (activate vibration on object touch), stand → Unity (resistance on lever turn). Architecture must account for round-trip latency: command from Unity → stand → mechanical reaction → tracking → Unity. On USB HID total latency 10–20 ms, noticeable with tactile interaction.

Extension Architecture

Build on Hardware Abstraction Layer pattern:

PhysicalStandInput (IStandInputProvider)
    ├── UsbHidProvider
    ├── SerialProvider
    └── UdpProvider

Top VR-app layer works only with IStandInputProvider — doesn't know data source. Allows testing VR-logic without physical stand via MockStandInputProvider and easy protocol switch on hardware change.

Stand State Machine — separate component tracking aggregated state of all physical elements. Don't process each event in Unity Update — collect state packet every 16 ms (1 frame at 60 FPS) and apply delta.

For physical and virtual object spatial matching use Transform.SetPositionAndRotation() with interpolation via Vector3.Lerp / Quaternion.Slerp — raw controller data gives jumps, interpolation over 2–3 frames gives smoothness without noticeable delay.

From case study: VR-trainer for drilling rig operators with physical control panel (12 toggle switches, 4 joysticks, manometers with servos). Connection via USB HID (panel as custom HID device). Problem — on fast sequential toggle switching, Unity skipped events because reading Input.GetAxis() in Update (polling, not event-driven). Switched to raw HID data reading with InputSystem.onEvent — migrated to Input System 1.x with custom InputDevice and InputControl per panel element.

Work Stages

Stand technical analysis. Study hardware scheme, connection protocol, latency requirements, feedback presence.

Communication layer prototype. Minimal data reading implementation + visualization in Unity for verification.

HAL development. Complete Hardware Abstraction Layer with mock-provider for testing.

Spatial calibration. Physical and virtual object matching system.

VR-scenario integration. HAL connection to game logic, haptic feedback.

Stand testing. Long-term stability test, reconnect verification on connection loss.

Cost calculated after studying stand technical documentation and integration requirements.