Building WebXR Games in Babylon.js

Step 1: Set Up a Babylon.js WebXR Project

Install the core Babylon.js packages via npm: @babylonjs/core for the engine, @babylonjs/loaders for glTF model loading, and @babylonjs/materials for additional material types. Set up a Vite project with HTTPS enabled since WebXR requires a secure context.

Create your scene with the standard Babylon.js setup: an Engine bound to a canvas element, a Scene, and a FreeCamera or UniversalCamera for the non-VR view. Add a ground plane using MeshBuilder.CreateGround and a HemisphericLight for basic illumination. Place some box or sphere meshes in the scene so you have objects to look at and interact with.

Now add WebXR support with a single method call: scene.createDefaultXRExperienceAsync({ floorMeshes: [ground] }). This creates the WebXRDefaultExperience, which includes the Enter VR button overlay, session management, teleportation on the specified floor meshes, controller model loading, and the stereoscopic render pipeline. When the player clicks Enter VR, Babylon.js requests an immersive-vr session, starts rendering to the headset, and loads 3D models of the player's controllers automatically.

The floorMeshes parameter tells the teleportation system which meshes the player can teleport onto. Without it, teleportation is disabled. You can pass multiple meshes if your scene has several walkable surfaces.

Step 2: Configure the XR Experience Helper

The default XR experience works out of the box, but most games need to customize it. The createDefaultXRExperienceAsync method accepts a configuration object that controls every aspect of the XR setup.

Teleportation configuration lets you set the teleport arc color, the landing indicator mesh, movement speed for smooth locomotion, and snap-turn angles. Snap turning rotates the player's view in discrete increments (commonly 30 or 45 degrees) and is essential for comfort since smooth rotation can cause motion sickness. Set the teleportation's rotationEnabled property to true and configure the snapAngle to enable this.

Controller configuration controls which controller models are loaded and how input is mapped. Babylon.js supports the standard WebXR input profile registry, automatically loading the correct 3D model for the player's specific controller hardware (Quest Touch Pro, Valve Index knuckles, etc.). You can override this to use custom controller models if your game has a specific visual style.

Feature requests are specified when creating the XR experience. If you need hand tracking, add "hand-tracking" to the optional features list. For AR sessions, request "immersive-ar" as the session mode and add features like "hit-test", "plane-detection", or "anchors" as needed. Babylon.js wraps each WebXR module in a feature manager that provides a clean, Observable-based API.

Access the XR helper's components through the returned experience object. The baseExperience gives you the underlying XRSession and reference space. The teleportation property controls movement. The input property manages controllers and hands. The featuresManager lets you enable and configure additional WebXR modules at runtime.

Step 3: Handle Controller and Hand Input

Babylon.js provides input handling through its WebXRInput system. When controllers connect, the system fires an onControllerAddedObservable event with a WebXRInputSource object. This object contains the motion controller (with button and axis data), a pointer mesh (the ray coming from the controller), and a grip mesh (the 3D controller model).

To detect button presses, access the motionController's components. Each component (trigger, grip, thumbstick, buttons A/B/X/Y) has an onButtonStateChangedObservable that fires when the button is pressed, released, or its analog value changes. The trigger component, for example, reports values from 0.0 (released) to 1.0 (fully pressed), letting you implement progressive squeeze mechanics for things like variable-force grabs or analog weapon charging.

For pointer-based interaction, Babylon.js extends its standard action system to work in VR. Any mesh with an ActionManager responds to the VR controller's ray pointer the same way it responds to mouse clicks on the flat screen. Trigger pull maps to a "pick" action. This means your existing non-VR interactions often work in VR without code changes.

Hand tracking in Babylon.js uses the WebXRHandTracking feature. Enable it through the features manager, and the system provides 25 joint meshes per hand that update each frame. You can query joint positions, compute distances between joints for pinch detection (thumb tip to index tip distance below a threshold), and build custom gesture recognizers. Babylon.js also includes a built-in near-interaction system that lets tracked hands push buttons and grab objects directly.

For grab mechanics, implement a pickup system that parents a mesh to the controller's grip pose when the grip button is pressed, and releases it when the button is released. Apply the controller's velocity to the released object for realistic throwing. Babylon.js's physics integration (Havok) makes this straightforward since you can read physics velocities directly from the motion controller.

Step 4: Build Interactive VR Game Mechanics

With input handling in place, you can build game mechanics specific to VR. The key difference from flat-screen games is that the player's body is part of the input system. Head position, hand positions, body orientation, and physical gestures all carry meaning.

3D UI panels in Babylon.js use the GUI3DManager from @babylonjs/gui. Create holographic buttons, sliders, and text panels that float in 3D space. These respond to both controller pointers and hand interaction. Position UI panels at a comfortable reading distance (about 1.5 to 2 meters from the player) and at a slight downward angle for ergonomic viewing. Avoid placing critical UI behind the player or requiring them to turn around frequently.

Physics-based interactions make VR games feel tangible. Use the Havok physics plugin to give objects mass, friction, and collision responses. When the player picks up and throws an object, it should arc realistically, bounce off surfaces, and interact with other physics bodies. Babylon.js integrates Havok tightly, so physics bodies can be attached to XR controllers for direct manipulation.

Spatial audio anchored to game objects creates immersive soundscapes. Babylon.js's Sound class supports spatial positioning, so an enemy behind the player sounds like it is behind them. Use spatial audio cues for game feedback: a UI click sound at the button's position, footstep sounds at the ground, ambient environment sounds spread around the scene. Set the maxDistance and rolloffFactor on each sound to control how it fades with distance.

Scene management for VR requires careful handling of transitions. Instant scene switches cause disorientation. Instead, fade to black, load the new scene, position the player, then fade back in. Babylon.js supports this through post-processing effects or by animating a sphere around the player's head from transparent to opaque and back.

Step 5: Add AR Features with Babylon.js

Babylon.js supports immersive-ar sessions through the same Experience Helper. Change the session mode to "immersive-ar" and request the appropriate features. The rendering pipeline automatically composites your 3D content over the device's passthrough or camera view.

Hit testing lets you find real-world surfaces. Enable the WebXRHitTest feature through the features manager. The hit test system casts rays from the controller or from the player's gaze direction and reports intersection points with detected surfaces. Use these intersection points to position a placement indicator, then anchor a game object at that location when the player confirms.

Plane detection reports horizontal and vertical surfaces in the player's environment. Each detected plane includes its position, orientation, and boundary polygon. Use detected floor planes as physics collision surfaces so virtual objects land on real tables and floors. Use wall planes as barriers or as surfaces for mounting virtual decorations, game boards, or UI panels.

Anchors persist object placement across tracking corrections and, on some devices, across sessions. When the player places a virtual chess board on their table, create an anchor at that position. The anchor maintains the object's location even as the device's spatial understanding improves, preventing drift. Babylon.js wraps the anchor API in its feature manager, providing Observables for anchor creation, updates, and deletion.

AR games in Babylon.js share the same scene graph, physics, and interaction systems as VR games. The primary differences are the transparent background (so the real world shows through), the availability of environment-sensing features, and the input model (which varies between devices). Design your game to detect which mode it is running in and adapt the visual style accordingly, since dark backgrounds work well in VR but look wrong in AR where the player can see their room.