AR & VR Integration in TouchDesigner

TouchDesigner’s VR support is deep but requires careful setup — the TD VR pipeline imposes different constraints than a game engine renderer, and understanding those constraints determines what’s achievable at 90 fps.

SteamVR Support in TouchDesigner

VR rendering in TD requires a Pro licence. The runtime path uses SteamVR as the compositor, which means any SteamVR-compatible HMD works: Valve Index, HTC Vive Pro 2, Windows Mixed Reality headsets, and Meta Quest 2/3 via Oculus Link.

Prerequisites before opening TD with a VR intent:

1. Install SteamVR and launch it; verify the HMD is tracked
2. Disable the SteamVR dashboard on launch (Settings > General > uncheck “Use Desktop Game Theatre”)
3. Set SteamVR’s per-application resolution override to 100% for TD — super-sampling inside SteamVR and inside TD simultaneously is wasteful
4. Ensure TD is running on the same GPU as SteamVR’s compositor (check Windows Display Settings)

The VR Render COMP

The VR Render COMP (Operators > COMP > VR) is TD’s primary VR rendering operator. It wraps a standard Camera COMP and Geometry COMP network in a stereo rendering pipeline that submits frames to the SteamVR compositor.

Key VR Render COMP parameters:

• HMD: automatically detected; shows as SteamVR HMD when VR is running
• Resolution Per Eye: set to Tracked to use the HMD’s native resolution, or override to a lower resolution if maintaining 90 fps requires it
• Multisample AA: 2× is the maximum practical on most GPUs in VR; 4× will cost too much at 90 fps
• Near/Far Clip: set far clip as tight as possible — depth precision in VR is more sensitive to far-clip distance than in flat rendering

The VR Render COMP outputs a stereo texture (both eyes side-by-side) to the SteamVR compositor automatically. You also get a single-eye output TOP that you can route to a traditional Window COMP for spectator display.

Tracking Data as CHOP Channels

The VR Render COMP outputs all tracking data as CHOP channels on its child CHOP network. Access via op('vrrender1/vrtrack'):

# Common VR CHOP channel names
# HMD pose
hmd_tx = op('vrrender1/vrtrack')['hmd/tx'][0]   # X position in metres
hmd_ty = op('vrrender1/vrtrack')['hmd/ty'][0]   # Y
hmd_tz = op('vrrender1/vrtrack')['hmd/tz'][0]   # Z
hmd_rx = op('vrrender1/vrtrack')['hmd/rx'][0]   # Pitch (degrees)
hmd_ry = op('vrrender1/vrtrack')['hmd/ry'][0]   # Yaw
hmd_rz = op('vrrender1/vrtrack')['hmd/rz'][0]   # Roll

# Left controller — index 0
lc_tx  = op('vrrender1/vrtrack')['controller0/tx'][0]
trigger_l = op('vrrender1/vrtrack')['controller0/trigger'][0]  # 0-1

# Right controller — index 1
rc_tx  = op('vrrender1/vrtrack')['controller1/tx'][0]
trigger_r = op('vrrender1/vrtrack')['controller1/trigger'][0]
grip_r    = op('vrrender1/vrtrack')['controller1/grip'][0]

Route these channels through a Filter CHOP (Lag type, 1–2 frame lag) to smooth prediction jitter before driving physics or visual parameters.

Mixed Reality via Quest Passthrough

Meta Quest 2/3 via Oculus Link presents as a SteamVR device in TD. The passthrough camera feed (colour passthrough on Quest 3, monochrome on Quest 2) is accessible as a VR Render COMP special input when the OpenXR XR_FB_passthrough extension is active. Note: direct passthrough access from TD requires TD build 2023.11340 or later, and Meta’s OpenXR runtime with passthrough permissions granted in the Meta Quest app.

For mixed reality compositing:

1. Enable passthrough in VR Render COMP parameters: Passthrough → Colour (Quest 3) or Monochrome (Quest 2)
2. The passthrough image is available as input 0 to the VR Render COMP’s internal Render TOP
3. Composite your TD geometry over the passthrough using alpha blending in the Render TOP’s background settings

OpenXR Standard in TouchDesigner

TD 2023+ supports OpenXR as an alternative to the SteamVR runtime. OpenXR allows TD to connect to:

• SteamVR’s OpenXR runtime
• Meta’s OpenXR runtime (via Oculus software)
• Windows Mixed Reality (WMR) OpenXR runtime

Switch between runtimes by setting the OpenXR Runtime parameter on the VR Render COMP. The advantage of OpenXR over TD’s direct SteamVR integration is access to vendor-specific extensions — hand tracking (XR_EXT_hand_tracking), eye tracking (XR_EXT_eye_gaze_interaction), and foveated rendering (XR_FB_swapchain_update_state_foveation).

Leap Motion for Hand Tracking

Leap Motion (Ultraleap) provides markerless hand tracking without an HMD, making it ideal for interactive installations where users are not wearing headsets. Install the Ultraleap Gemini driver, then use the Leap Motion CHOP (Operators > CHOP > Leap Motion):

# Leap Motion CHOP channel structure
# For each detected hand (up to 2):
# handN/palm/tx, ty, tz     — palm centre position (mm from sensor)
# handN/palm/rx, ry, rz     — palm normal angles
# handN/finger0-4/tip/tx    — fingertip positions
# handN/finger0-4/mcpj/tx   — knuckle positions
# handN/pinch_strength       — 0-1 pinch detection
# handN/grab_strength        — 0-1 fist detection

leap = op('leapmotion1')
if leap.numChans > 0:
    palm_x = leap['hand0/palm/tx'][0]  # millimetres
    pinch  = leap['hand0/pinch_strength'][0]

Leap Motion works in desktop mode (sensor face-up on a table) or VR mode (sensor mounted on an HMD). In desktop mode the tracking volume is roughly 600mm wide × 400mm tall × 300mm deep, centred above the sensor.

Spatial Audio and Head Tracking

TD’s Audio Device Out CHOP supports HRTF-based binaural audio when the output device is a headphone output. To position audio sources in 3D relative to the listener’s head orientation:

# In an Execute DAT, frameStart callback
def frameStart(frame):
    # Get HMD yaw for audio rotation
    hmd_ry = op('vrrender1/vrtrack')['hmd/ry'][0]  # degrees

    # Route to a Spatial Audio CHOP or update audio source positions
    # TD's built-in 3D audio uses the Audio 3D CHOP
    audio_3d = op('audio_3d_source')
    # Position relative to world, TD rotates HRTF based on listener orientation
    audio_3d.par.tx = op('particle_centre')['tx'][0]
    audio_3d.par.ty = op('particle_centre')['ty'][0]
    audio_3d.par.tz = op('particle_centre')['tz'][0]

The Audio 3D CHOP handles HRTF convolution for the Head Rotation parameter driven by the HMD’s yaw angle. At 90 fps, the head-rotation-to-audio-update latency is 11 ms — perceptibly low for spatial accuracy.

Rendering Performance for VR: 90 fps Target

The 90 Hz target is not optional — SteamVR’s Asynchronous Spacewarp (ASW) kicks in when frames are missed, which introduces artefacts and motion discomfort. Design your TD network to sustain 90 fps with headroom.

Foveated rendering via OpenXR: reduce resolution in the peripheral areas of each eye’s render, where the lens distortion means peripheral pixels contribute less to perceived quality:

# Set via VR Render COMP parameter (OpenXR runtime required)
op('vrrender1').par.FoveatedRendering = True
op('vrrender1').par.FoveatedRenderingMode = 'FixedFoveated'
# Without eye tracking, fixed foveation uses the optical centre

Performance budget for VR (RTX 3080 baseline):

• Geometry SOP → Render TOP: 2–4 ms per 100k triangles with a simple GLSL shader
• GLSL TOP simulation: 1–2 ms per 1920×1080 feedback pass
• Total GPU budget: 10 ms (11 ms frame at 90 fps, leaving 1 ms overhead)
• Reserve 1–2 ms for SteamVR compositor overhead

Use TD’s Performance Monitor (Dialogs > Monitors > Performance Monitor) to identify cook hotspots. The VR Render COMP’s cook time includes the SteamVR submit latency — values above 8 ms indicate impending frame drops.

Full Example: Room-Scale Particle Attractor

A room-scale VR piece where physical controller positions drive particle attractors. Particles live in a GLSL TOP GPU simulation; their world-space attractor positions come from the controller tracking CHOP.

// VR particle update GLSL TOP
// Particle state: RG = position (world XZ, normalised to [-1,1])
//                BA = velocity

uniform vec3 uAttractorL;   // Left controller world position
uniform vec3 uAttractorR;   // Right controller world position
uniform float uStrengthL;   // Left trigger value 0-1
uniform float uStrengthR;   // Right trigger value 0-1
uniform float uDelta;

void main()
{
    vec2 uv = vUV.st;
    vec4 state = texture(sTD2DInputs[0], uv);
    vec2 pos = state.rg;
    vec2 vel = state.ba;

    // Attractor: controllers in XZ plane
    vec2 posL = uAttractorL.xz * 0.5 + 0.5;  // world → UV
    vec2 posR = uAttractorR.xz * 0.5 + 0.5;

    vec2 toL = posL - pos;
    vec2 toR = posR - pos;

    float dL = length(toL) + 0.001;
    float dR = length(toR) + 0.001;

    vec2 accel = normalize(toL) * (uStrengthL * 0.3 / (dL * dL))
               + normalize(toR) * (uStrengthR * 0.3 / (dR * dR));

    vel += accel * uDelta;
    vel *= 0.985;
    pos = fract(pos + vel * uDelta);

    fragColor = vec4(pos, vel);
}

In a Python Execute DAT, update the GLSL uniforms from the VR tracking CHOP:

def frameStart(frame):
    track = op('vrrender1/vrtrack')
    glsl  = op('particle_sim')

    glsl.par.Attractorluniformx = track['controller0/tx'][0]
    glsl.par.Attractorluniformy = track['controller0/ty'][0]
    glsl.par.Attractorluniformz = track['controller0/tz'][0]
    glsl.par.Strengthluniform   = track['controller0/trigger'][0]

    glsl.par.Attractorruniformx = track['controller1/tx'][0]
    glsl.par.Attractorruniformy = track['controller1/ty'][0]
    glsl.par.Attractorruniformz = track['controller1/tz'][0]
    glsl.par.Strengthruniform   = track['controller1/trigger'][0]

    # Spatial audio: particle cloud centre moves with average attractor
    avg_x = (track['controller0/tx'][0] + track['controller1/tx'][0]) * 0.5
    avg_z = (track['controller0/tz'][0] + track['controller1/tz'][0]) * 0.5
    speed = abs(track['controller0/tx'][0] - track['controller1/tx'][0])
    op('ambient_audio').par.Volume = min(speed * 2.0, 1.0)

The particle rendering step maps the UV-space simulation texture to a 3D point cloud using an Instanced Geometry COMP, transforming each particle’s position from UV-space back to world-space within the GLSL vertex shader.