Machine Learning in TouchDesigner

TouchDesigner’s ML TOP treats an ONNX model the same way a GLSL TOP treats a shader: you give it inputs, specify the model file, and it runs inference on the GPU each frame. Paired with the MediaPipe plugin for skeleton tracking, you have the ingredients for a full ML-to-visuals pipeline without leaving TD.

The ML TOP

The ML TOP (Operators > TOP > ML) accepts up to four TOP inputs which it maps to named ONNX input tensors. Key parameters:

• Model File — path to a .onnx file; TD caches the model and doesn’t reload it on every cook
• GPU Device — which CUDA device to run inference on; must match the device holding your input textures to avoid cross-device transfers
• Execution Provider — CUDA or CPU; always use CUDA for real-time work
• Output Name — the ONNX output tensor name to display; for multi-output models you can daisy-chain ML TOPs

The output is always a TOP — for classification models this is a 1×N texture where each pixel’s red channel is one logit; for generative models it is the full output image.

MediaPipe Integration

Derivative ships a first-party MediaPipe plugin that wraps Google’s MediaPipe graph framework. After installing the plugin (available from the Derivative forum), you get three new TOPs: MediaPipe Face Mesh TOP, MediaPipe Hands TOP, and MediaPipe Pose TOP. Each takes a camera TOP input and produces both a visualisation TOP and a CHOP output.

Face Mesh: 468 Landmarks

The Face Mesh TOP exports a CHOP with channels named face0_pt0_x, face0_pt0_y, face0_pt0_z through face0_pt467_z. Coordinates are normalised 0–1 in X and Y, and depth-relative in Z. With 468 points × 3 axes × N faces, you end up with potentially thousands of channels — prune with a Select CHOP or rename with a Rename CHOP before routing downstream.

Key landmark indices (MediaPipe’s canonical numbering):

• 33 — left eye outer corner
• 133 — left eye inner corner
• 362 — right eye inner corner
• 263 — right eye outer corner
• 1 — nose tip
• 17 — chin centre

Hand Tracking: 21 Landmarks per Hand

# In a Script CHOP, extract fingertip world positions from MediaPipe CHOP output
# Landmark indices: 4=thumb tip, 8=index tip, 12=middle tip, 16=ring tip, 20=pinky tip
FINGERTIPS = [4, 8, 12, 16, 20]

def onCook(scriptOp):
    mp_chop = op('mediapipe_hands_chop')
    if mp_chop.numChans == 0:
        return

    scriptOp.clear()
    for finger_idx, landmark in enumerate(FINGERTIPS):
        for axis in ['x', 'y', 'z']:
            chan_name = f'hand0_pt{landmark}_{axis}'
            source = mp_chop[chan_name]
            if source:
                c = scriptOp.appendChan(f'finger{finger_idx}_{axis}')
                c[0] = source[0]

Pose Estimation: 33 Body Landmarks

Pose exports pose0_pt0_x through pose0_pt32_z plus a pose0_pt{N}_visibility channel (0–1, confidence). Useful indices: 11/12 = shoulders, 13/14 = elbows, 15/16 = wrists, 23/24 = hips, 27/28 = ankles.

Converting Custom Models to ONNX

Any PyTorch or TensorFlow model can be exported to ONNX for use in the ML TOP.

# PyTorch → ONNX export
import torch
import torch.onnx

model = MyStyleTransferModel()
model.load_state_dict(torch.load('weights.pth'))
model.eval()

dummy_input = torch.zeros(1, 3, 512, 512)  # NCHW format

torch.onnx.export(
    model,
    dummy_input,
    'style_transfer.onnx',
    opset_version=17,
    input_names=['input_image'],
    output_names=['stylized_image'],
    dynamic_axes={
        'input_image':   {0: 'batch', 2: 'height', 3: 'width'},
        'stylized_image': {0: 'batch', 2: 'height', 3: 'width'},
    }
)

After export, validate with onnxruntime:

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession('style_transfer.onnx',
                             providers=['CUDAExecutionProvider'])
dummy = np.zeros((1, 3, 512, 512), dtype=np.float32)
out = sess.run(['stylized_image'], {'input_image': dummy})
print(out[0].shape)  # Should be (1, 3, 512, 512)

TD’s ML TOP expects NHWC layout for textures, not NCHW. If your model uses NCHW internally (most PyTorch models do), add a transpose at the end of your export graph, or use the Transpose parameter in the ML TOP to handle the conversion automatically.

Real-Time Style Transfer via ONNX

For a 512×512 style transfer running at 30 fps on an RTX 3080, the pipeline is:

Camera TOP → Resize TOP (512×512) → ML TOP (style_transfer.onnx) → Level TOP → Out TOP

The ML TOP in this configuration expects a single input tensor named input_image of shape [1, 3, 512, 512] with values in [0, 1]. Set the ML TOP’s Input Normalisation to 0 to 1 and Channel Order to RGB to match the training data format. The output stylized_image is the same shape; the ML TOP unwraps it back to a 512×512 RGBA texture.

Full Example: Hand Tracking → Particle Attractor

This pipeline takes the MediaPipe index fingertip position and uses it to attract a GPU particle system written in GLSL.

Network structure:

Camera TOP
    └→ MediaPipe Hands TOP ──→ mediapipe_hands_chop
                                    └→ Script CHOP (extract tip) → fingertip_chop
                                                                        └→ GLSL TOP (particles)

The GLSL compute-style GPU particle system uses a feedback loop. The attractor position comes in via a Uniform:

// GLSL TOP — particle update pass
// uAttractor: xy in UV space [0,1]
// uDeltaTime: seconds since last frame

uniform vec2 uAttractor;
uniform float uDeltaTime;
uniform float uAttractStrength;

// Particle state texture layout:
// R=posX, G=posY, B=velX, A=velY (all in UV space)

void main()
{
    vec2 uv = vUV.st;
    vec4 state = texture(sTD2DInputs[0], uv); // previous frame state

    vec2 pos = state.rg;
    vec2 vel = state.ba;

    // Attract toward fingertip
    vec2 toAttractor = uAttractor - pos;
    float dist = length(toAttractor) + 0.001;
    vec2 accel = normalize(toAttractor) * (uAttractStrength / (dist * dist));

    vel += accel * uDeltaTime;
    vel *= 0.98; // drag
    pos += vel * uDeltaTime;

    // Wrap at boundaries
    pos = fract(pos);

    fragColor = vec4(pos, vel);
}

The Script CHOP that feeds uAttractor checks MediaPipe’s output each frame and normalises the coordinates:

def onCook(scriptOp):
    mp = op('mediapipe_hands_chop')
    scriptOp.clear()

    # Index fingertip = landmark 8
    cx = scriptOp.appendChan('attractor_x')
    cy = scriptOp.appendChan('attractor_y')

    if mp.numChans > 0 and mp['hand0_pt8_x']:
        # MediaPipe X is flipped relative to screen X for mirrored camera
        cx[0] = 1.0 - float(mp['hand0_pt8_x'][0])
        cy[0] = float(mp['hand0_pt8_y'][0])
    else:
        cx[0] = 0.5
        cy[0] = 0.5

In the GLSL TOP’s Vectors 1 parameter, reference op('fingertip_chop')['attractor_x'][0] and op('fingertip_chop')['attractor_y'][0] bound to uAttractor.x and uAttractor.y. Add a Filter CHOP before the GLSL TOP with a low-pass filter at 8 Hz to smooth finger jitter without introducing perceptible lag.

GPU ML Inference via Script TOP and CUDA

For models too complex for the ML TOP (multi-head attention, dynamic shapes, etc.), the Script TOP with a Python environment that has onnxruntime-gpu installed gives you full control:

import onnxruntime as ort
import numpy as np

# Module-level session — created once, reused every cook
_session = None

def onSetupParameters(scriptOp):
    global _session
    _session = ort.InferenceSession(
        'C:/models/depth_estimation.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
    )

def onCook(scriptOp):
    # Pull input texture to numpy
    input_top = op('camera_resized')
    frame = np.frombuffer(input_top.numpyArray(delayed=False),
                           dtype=np.float32)
    frame = frame.reshape(1, input_top.height, input_top.width, 4)
    frame = frame[:, :, :, :3]  # drop alpha
    frame = frame.transpose(0, 3, 1, 2)  # NHWC → NCHW

    result = _session.run(['depth'], {'image': frame})[0]
    result = result.squeeze()

    # Normalise and write to output
    result = (result - result.min()) / (result.max() - result.min() + 1e-8)
    scriptOp.copyNumpyArray(result[:, :, np.newaxis] * np.ones((1, 1, 4)))

Keep the ORT session at module scope — creating an InferenceSession is expensive (model compilation on first run takes 2–5 seconds on CUDA). The delayed=False flag on numpyArray() forces a synchronous GPU readback; for async pipelines use TD’s performCPUMemoryDownload() callback instead.