Kinect & Depth Cameras in TouchDesigner

Depth cameras turn human presence into structured data. In TouchDesigner, the Kinect Azure (Azure Kinect DK) and Kinect v2 are both supported natively — you get colour, depth, and infrared images as TOPs, plus skeleton joint positions as a CHOP. Combined with the visual network, these become the inputs for body-driven generative art, interactive installations, and live performance instruments.

Hardware Requirements

The Kinect Azure (Azure Kinect DK) is the recommended device. It uses a Time-of-Flight depth sensor capable of tracking up to 6 bodies with 32 joints each. It connects via USB 3.0 (blue ports only — bandwidth matters). Install the Azure Kinect SDK and Azure Kinect Body Tracking SDK from Microsoft before launching TD — the operator will not appear in the palette without them.

The Kinect v2 (Xbox One Kinect with USB 3.0 adapter) is also supported via the older Kinect v2 SDK. Its joint set (25 joints, 6 bodies) is slightly different from the Azure.

Kinect Azure TOP

The Kinect Azure TOP streams camera imagery. In its parameters:

• Active must be On for the device to initialize.
• Output selects what the TOP outputs: Colour, Depth (16-bit normalized), IR, or Point Cloud (RGB encodes XYZ position).

For a typical setup, create three Kinect Azure TOPs with Output set to Colour, Depth, and IR respectively. They share the same hardware device — only one device connection is open, and the other TOPs reference the first one via the Select field.

The depth output is a single-channel 16-bit TOP where pixel brightness represents distance in millimetres (up to 3860mm in NFOV unbinned mode). Feed it into a Lookup TOP to apply a false-colour gradient, or into a Threshold TOP to isolate a depth range (useful for clipping out the floor and far background).

Kinect Azure CHOP

The Kinect Azure CHOP outputs skeleton joint data. Set it to reference the same device as your Kinect Azure TOP. Each tracked body gets a set of channels named:

body0_Pelvis_tx   body0_Pelvis_ty   body0_Pelvis_tz
body0_SpineNavel_tx  body0_SpineNavel_ty  body0_SpineNavel_tz
body0_SpineChest_tx  ...
body0_Neck_tx     ...
body0_Head_tx     ...
body0_ShoulderLeft_tx   body0_ShoulderLeft_ty   body0_ShoulderLeft_tz
body0_ElbowLeft_tx  ...
body0_WristLeft_tx  ...
body0_HandLeft_tx   ...
body0_HandTipLeft_tx ...
body0_ThumbLeft_tx  ...
body0_ShoulderRight_tx  ...
body0_ElbowRight_tx ...
body0_WristRight_tx ...
body0_HipLeft_tx    ...
body0_KneeLeft_tx   ...
body0_AnkleLeft_tx  ...
body0_FootLeft_tx   ...
body0_HipRight_tx   ...
body0_KneeRight_tx  ...
body0_AnkleRight_tx ...
body0_FootRight_tx  ...
body0_NoseLeft_tx   ...
body0_EarLeft_tx    ...
body0_EarRight_tx   ...
body0_EyeLeft_tx    ...
body0_EyeRight_tx   ...

Positions are in millimetres in the Kinect’s camera coordinate space. The Y axis points down from the Kinect’s perspective by default; compensate with a Math CHOP that multiplies ty channels by -1 to get a natural Y-up world.

Accessing Joint Positions in Python

chop = op('kinectazure1')

# Read left wrist position
lx = chop['body0_WristLeft_tx'][0]
ly = chop['body0_WristLeft_ty'][0]
lz = chop['body0_WristLeft_tz'][0]

# Read right hand tip
rx = chop['body0_HandTipRight_tx'][0]
ry = chop['body0_HandTipRight_ty'][0]

# Map right hand X (roughly -400 to 400mm) to noise amplitude (0–1)
amplitude = (rx + 400) / 800.0
amplitude = max(0.0, min(1.0, amplitude))
op('noise1').par.amplitude = amplitude

For robust code, check whether the body is tracked before reading its joints:

# body0_Pelvis_tx will be 0 if no body is detected
tracked = chop['body0_Pelvis_tz'][0] != 0.0
if tracked:
    # process joints
    pass

Driving Visuals from Hand Positions

A practical pattern: map both wrist positions to two LFO frequencies so hand height controls the visual rhythm:

kinectazure1 (CHOP)
  └─→ select1 (Select CHOP) ── channels: body0_WristLeft_ty, body0_WristRight_ty
        └─→ math1 (Math CHOP) ── From Range: -400 to 0, To Range: 0.5 to 8.0
              └─→ lfo_freq (Null CHOP) ── drives lfo1.par.rate and lfo2.par.rate

Export the two channels from lfo_freq to the Rate parameters of two LFO CHOPs. Those LFOs then modulate separate parameters in your visual network — for instance, the Amplitude and Roughness of a Noise TOP.

Depth Point Cloud

The Kinect Azure TOP in Point Cloud mode outputs a 512×512 (NFOV) or 1024×1024 (WFOV) TOP where each pixel’s RGB encodes the X, Y, Z world position of the corresponding depth pixel. Feed this into a Point Cloud SOP (SOP > Point Cloud):

1. Set its TOP input to the Point Cloud TOP.
2. The SOP generates one point per non-zero pixel.
3. Wire into a Point Sprite SOP or a Geo COMP with instancing for rendering.

Filter out points where Z = 0 (no depth data) using a Threshold TOP on the depth channel before passing to the Point Cloud SOP, or use a GLSL TOP to zero out pixels outside your desired depth range.

RealSense Alternative

If you do not have a Kinect, the RealSense TOP works with Intel RealSense D415 and D435 devices. It has the same Colour/Depth/IR output structure. RealSense does not include built-in skeleton tracking — for body joints you need a third-party solution such as the NuiTrack CHOP component or a Python script using the pyrealsense2 library piping data via OSC.

Full Example: Wrist Positions → Audio-Reactive Particle System

This network maps both wrists to the two frequencies of a dual-oscillator audio synthesis, which in turn drives a particle system’s birth rate and turbulence:

kinectazure1 (CHOP)
  └─→ select_wrists (Select CHOP) ── body0_WristLeft_ty, body0_WristRight_ty, body0_WristLeft_tx, body0_WristRight_tx
        └─→ lag1 (Lag CHOP) ── lag 0.1s to smooth jitter
              └─→ math_remap (Math CHOP) ── remap ty from -500..0 to 1..20 Hz
                    └─→ lfo_left, lfo_right (LFO CHOPs, rates driven by exports)
                          └─→ audio_filter (Audio Filter CHOP) ── bandpass at LFO frequencies
                                └─→ audio_level (Audio Spectrum CHOP)
                                      └─→ math_level (Math CHOP) ── overall level → particle birth rate

# Execute DAT onFrameStart — maps body state to particle parameters
def onFrameStart(frame):
    chop = op('kinectazure1')
    
    if chop is None:
        return
    
    # Check tracking
    if chop['body0_Pelvis_tz'][0] == 0.0:
        op('particles1').par.birthrate = 0
        return
    
    # Left wrist height → turbulence
    ly = chop['body0_WristLeft_ty'][0]
    turbulence = max(0, (-ly - 100) / 400.0)  # hand raised = more turbulence
    op('particles1').par.turbulence = turbulence * 5.0
    
    # Right wrist X → wind direction
    rx = chop['body0_WristRight_tx'][0]
    wind_x = rx / 400.0 * 2.0
    op('particles1').par.windx = wind_x
    
    return

The net effect: raise your left hand to increase turbulence; sweep your right hand side to side to change the wind direction, blowing the particle trail across the screen.