Building WebXR Games in Three.js

Step 1: Enable WebXR in the Three.js Renderer

Start with a standard Three.js setup: a WebGLRenderer, a Scene, and a PerspectiveCamera. To enable WebXR, set renderer.xr.enabled = true. This tells Three.js to use the WebXR frame callback instead of the standard requestAnimationFrame, and to handle stereoscopic rendering (two eye views) automatically when an XR session is active.

Replace your manual requestAnimationFrame loop with renderer.setAnimationLoop(renderFunction). This is critical because the WebXR specification requires that rendering happen inside the XR session's frame callback, and setAnimationLoop handles this transition seamlessly. Your render function receives a timestamp and an XRFrame reference when in XR mode, or just a timestamp in normal mode.

Add an Enter VR button to your page by importing VRButton from three/addons/webxr/VRButton.js and calling VRButton.createButton(renderer). Append the returned DOM element to your page. The button handles feature detection (it appears only when WebXR is supported), session creation, and the visual state toggle. For AR, import ARButton instead and use ARButton.createButton(renderer, { requiredFeatures: ['hit-test'] }) to request specific AR capabilities.

Three.js manages the XR camera internally. When an immersive session starts, the renderer replaces your PerspectiveCamera with an ArrayCamera that renders separate viewports for each eye. Your scene, lights, and meshes remain unchanged. The XR camera's position and orientation are driven by the headset's tracking data automatically.

Step 2: Set Up Controllers and Controller Models

Three.js represents VR controllers as Object3D instances that you retrieve from the renderer. Call renderer.xr.getController(0) for the first controller and renderer.xr.getController(1) for the second. These objects update their position and orientation to match the physical controllers each frame. Add them to your scene so they participate in rendering.

For visual controller models, use the XRControllerModelFactory from three/addons/webxr/XRControllerModelFactory.js. Create a factory instance, then call factory.createControllerModel(controller) to get a mesh that automatically loads the correct 3D model for the player's hardware. The factory uses the WebXR input profile registry to select the right model, so Quest Touch controllers look like Quest controllers and Index controllers look like Index controllers.

You also need the controller's grip space, which represents the physical position of the controller in the player's hand (as opposed to the pointing ray). Get it with renderer.xr.getControllerGrip(0) and add the controller model to this grip group. The distinction matters: the controller (index 0) represents the pointing direction for ray interactions, while the grip (index 0) represents the physical hand position for grab interactions.

Add a visible line to each controller to show the pointing ray. Create a BufferGeometry with two points (the controller position and a point far ahead of it), add a Line mesh, and attach it as a child of the controller object. This line extends from the controller in the pointing direction, giving the player visual feedback for where they are aiming.

Step 3: Implement Raycasting Interaction

Three.js uses its Raycaster class for VR interaction, the same tool used for mouse picking in flat games. Each frame, set the raycaster's origin to the controller's position and its direction to the controller's forward vector (extracted from the controller's matrixWorld). Call raycaster.intersectObjects() with the list of interactive objects in your scene.

When the raycaster finds intersections, highlight the nearest intersected object by changing its material (emissive color, outline, scale pulse, whatever fits your game's visual style). Clear the highlight when the ray moves away. This gives the player visual feedback about what they are pointing at before they press the trigger.

Listen for controller events to handle selection. Three.js dispatches "selectstart" when the trigger is pressed, "selectend" when it is released, "squeezestart" for the grip button press, and "squeezeend" for the grip release. Register event listeners on the controller objects: controller.addEventListener('selectstart', onSelect). Inside the handler, check what the raycaster is currently intersecting and perform your game action.

For grab mechanics, detect the intersection on squeezestart, parent the intersected object to the controller's grip, and on squeezeend, reparent it back to the scene and optionally apply the controller's velocity for throwing. Calculate the controller's velocity by tracking its position over the last few frames and computing the delta.

The combination of per-frame raycasting for hover states and event-driven trigger handling for actions gives you a responsive interaction system that feels natural in VR. The hover highlight tells the player what will happen before they commit to the action.

Step 4: Add Movement and Locomotion

Three.js does not include a built-in locomotion system, so you implement movement yourself. The most common approaches are smooth locomotion (walking with the thumbstick), snap turning, and teleportation.

Smooth locomotion reads the thumbstick axes from the controller's gamepad. Access it through the XR session's inputSources array, which provides a gamepad property following the standard Gamepad API. The left thumbstick typically controls movement (forward, backward, strafe), while the right thumbstick controls rotation. Apply the thumbstick values as velocity to a movement group that contains the camera and controllers.

Create a "camera rig" group (a Group object) and add the camera to it. When you want to move the player, move the rig, not the camera directly. The camera's local position and rotation are controlled by the headset tracking, and your movement system controls the rig's world position. This separation prevents conflicts between tracking and locomotion.

Snap turning rotates the rig by a fixed angle (typically 30 or 45 degrees) when the right thumbstick is pushed left or right. Add a dead zone and a cooldown to prevent multiple snaps from a single flick. Snap turning is more comfortable than smooth turning for most players because it avoids the vection (perceived self-motion) that triggers motion sickness.

Teleportation requires a visible arc indicator, a landing point calculation, and a valid target surface check. Cast a parabolic arc from the controller's position in the pointing direction, find where it intersects the floor, show a ring or disc at the landing point, and move the camera rig to that position when the player releases the trigger. The arc visualization uses a series of line segments computed from a projectile motion equation with configurable gravity.

Step 5: Build AR Experiences with Three.js

Three.js supports immersive-ar sessions with automatic passthrough compositing. When you create the session with ARButton, the renderer's clear color becomes transparent, and your 3D scene is composited over the device's camera or passthrough view.

Hit testing requires requesting the hit-test feature when creating the session. In your render loop, use the XRFrame's getHitTestResults() method with an XRHitTestSource to find where the player's device is pointing at a real-world surface. The hit test returns a pose representing the intersection point and surface normal. Position a reticle mesh at this pose to show the player where objects will be placed.

When the player taps (selectstart event in AR), use the current hit test result to position a new game object. Create a mesh, set its position and orientation from the hit test pose, and add it to the scene. The object now appears to sit on the real-world surface.

Lighting estimation is available on some devices through the WebXR lighting estimation module. When supported, the browser provides a spherical harmonics light probe that represents the real-world lighting conditions. Apply this to your scene's environment map so that virtual objects are lit consistently with the physical environment, making them look more integrated into the real world.

Three.js AR development follows the same patterns as VR development, just with a transparent background and surface-aware placement. The raycasting, controller events, and scene management all work identically. The main design consideration is that AR objects should look physically plausible in the player's real environment, so use realistic materials, match the scale of real objects, and add shadows that ground virtual objects on real surfaces.