Server-Side Hand and Head Position Synchronization in Games

tags: [vr-ar]

Server synchronization configuration of hand and head positions in games

Positions in VR update at 72–120 Hz frequency. If sending every frame to all players, traffic for 8 people is about 50–100 KB/s for transforms alone — before game logic. The task of VR avatar network synchronization: maximum movement accuracy with minimum traffic and perceived latency.

Simple solution — NetworkTransform from Photon Fusion or Netcode for GameObjects — works, but not optimized for VR specifics. It sends data every FixedNetworkUpdate without checking if the object moved.

Dead Reckoning and hand prediction

Dead Reckoning technique came from maritime navigation and flight simulators. For VR avatars it works like this: instead of position every frame, client sends position + velocity + angular velocity. Receiver extrapolates next position based on last known state + movement physics model. When next packet arrives — compare with predicted position and correct.

In Photon Fusion this is implemented via custom NetworkBehaviour with [Networked] properties for position, rotation and derivatives. Interpolation and Extrapolation configured in NetworkTransform via InterpolationDataSource:

• Predicted — client-authoritative prediction, minimal perceived latency for local player
• Interpolated — delayed but smooth for remote players
• Auto — Fusion chooses based on State Authority

For VR hands it's crucial that Predicted mode applies only to StateAuthority object. Remote hands always interpolate, and latency is inevitable — question is how smoothly it's hidden.

Transform compression: quantization and delta encoding

Hand position in VR takes 3 × float32 = 12 bytes. Rotation as quaternion — 4 × float32 = 16 bytes. Total 28 bytes per hand, 56 for two + 28 for head = 84 bytes per player without headers.

Quantization reduces this to 3–4 bytes per component: position in VR workspace (3×3×3 m) with 1 mm accuracy encodes to 15 bits per axis. Rotation via compressed quaternion (three components, fourth recovered) — 10 bits per component. Total ~15–20 bytes per player instead of 84.

In Photon Fusion quantization enabled via NetworkTransform.Precision and RotationPrecision. For custom data use INetworkStruct with manual quantization via BitStream.

Delta encoding — next level: transmit only change from previous state. Hand that didn't move (player holding object still) sends almost zero delta, compressing to few bits. In practice for VR this gives 30–60% traffic reduction compared to full positions every tick.

Authoritative server vs client authority

Architectural choice with direct gameplay consequences. Client authority for hands — position accepted as-is from client — minimal latency, maximum responsiveness. But enables cheating: client can send hand in position physically impossible to reach.

Server authority — server validates every hand movement against physical constraints (max speed, reachability zone from torso) — resistant to cheating, but requires RTT compensation for responsiveness feeling.

For VR games without competitive component (shared experience, co-op) — client authority with basic server anomaly validation. For competitive VR — server authority with aggressive client prediction and rollback.

Case: handshake synchronization in social VR

In VR social space hand grip synchronization was required between players: players literally "shake hands", and it must look correct for both. Problem: at 80–120 ms RTT hand positions at moment of "contact" diverged by 5–12 cm — handshake looked like empty gesture in air.

Solution: special "snap" protocol. When one hand enters another's collider (checked on server), server notifies both clients of grab event. Both clients switch to "anchor" mode: one becomes leader, second — follower. Follower interpolates its position to leader's with Lerp over 80 ms. Visually — smooth connection without teleportation.

Cost calculated after audit of current network architecture and requirements for player count and game genre.