Article 9

Machine Learning with Wekinator

Connect Processing to Wekinator via OSC to build real-time gesture classifiers and regression models that map body movement to visual output.

⏱ 50 min Wekinator machine learning OSC gesture recognition regression

Wekinator is a standalone application by Rebecca Fiebrink that trains supervised machine learning models in real time. It receives feature vectors from Processing over OSC, trains on labeled examples you provide by clicking a button, and sends predictions back. No ML background is required, and the round-trip latency is typically under 5 ms on a local machine.

How Wekinator Connects to Processing

The communication flow is:

Processing extracts features from user input (mouse, keyboard, camera, sensor) and sends them to Wekinator’s OSC input port as /wek/inputs.
Wekinator runs the selected model (kNN, neural network, or SVR) and sends predictions to the OSC output port, typically at /wek/outputs.
Processing listens for /wek/outputs messages and maps prediction values to visual parameters.

Default OSC ports: Wekinator listens on 6448, sends on 12000. Processing sends on 6448 and listens on 12000.

Required Libraries

Install two Processing libraries from the Contribution Manager:

oscP5 by Andreas Schlegel
NetP5 (ships with oscP5, no separate install needed)

import oscP5.*;
import netP5.*;

OscP5   oscP5;
NetAddress wekinator;

void setup() {
  size(800, 600);
  oscP5       = new OscP5(this, 12000);          // listen on 12000
  wekinator   = new NetAddress("127.0.0.1", 6448); // send to Wekinator
}

Sending Feature Vectors to Wekinator

The OSC address /wek/inputs carries a flat array of floats. The number of floats must match the input count you configured in Wekinator’s “New Project” dialog.

void sendFeatures(float[] features) {
  OscMessage msg = new OscMessage("/wek/inputs");
  for (float f : features) msg.add(f);
  oscP5.send(msg, wekinator);
}

Call sendFeatures() every frame in draw(). Wekinator will continuously predict from whatever it last received; the model runs regardless of whether it has been trained yet (it returns the default output until trained).

Receiving Predictions from Wekinator

The oscEvent() callback fires for every incoming OSC message. Filter by address:

float[] outputs = new float[0];

void oscEvent(OscMessage msg) {
  if (msg.checkAddrPattern("/wek/outputs")) {
    int n = msg.typetag().length();
    outputs = new float[n];
    for (int i = 0; i < n; i++) {
      outputs[i] = msg.get(i).floatValue();
    }
  }
}

outputs is then available in draw() to drive visual parameters.

Building a Gesture Classifier

In Wekinator, create a new project with:

Inputs: 6 (two XY mouse positions sampled at three time steps)
Outputs: 1 output, type Classification, 3 classes

The sketch captures a sliding window of mouse positions as the feature vector:

import oscP5.*;
import netP5.*;

OscP5        osc;
NetAddress   wek;
float[]      features  = new float[6];   // 3 time steps × (x,y)
float[][]    window    = new float[3][2];
float[]      outputs   = {1};

int         predictedClass = 1;
color[]     classColors    = {color(255,80,80), color(80,255,80), color(80,80,255)};

void setup() {
  size(800, 600, P2D);
  osc = new OscP5(this, 12000);
  wek = new NetAddress("127.0.0.1", 6448);
}

void draw() {
  // Shift window
  window[0][0] = window[1][0]; window[0][1] = window[1][1];
  window[1][0] = window[2][0]; window[1][1] = window[2][1];
  window[2][0] = mouseX / (float)width;
  window[2][1] = mouseY / (float)height;

  // Build feature vector (normalised coordinates)
  for (int i = 0; i < 3; i++) {
    features[i*2]   = window[i][0];
    features[i*2+1] = window[i][1];
  }

  sendFeatures(features);

  if (outputs.length > 0) predictedClass = (int)outputs[0] - 1;

  // Visualise
  background(20);
  fill(classColors[constrain(predictedClass, 0, 2)]);
  noStroke();
  ellipse(width/2, height/2, 200, 200);

  fill(255);
  textSize(24);
  textAlign(CENTER);
  text("Class: " + (predictedClass + 1), width/2, height/2 + 5);
  text("Move mouse to draw gesture, then label in Wekinator", width/2, height - 30);
}

void sendFeatures(float[] f) {
  OscMessage m = new OscMessage("/wek/inputs");
  for (float v : f) m.add(v);
  osc.send(m, wek);
}

void oscEvent(OscMessage m) {
  if (m.checkAddrPattern("/wek/outputs")) {
    outputs = new float[m.typetag().length()];
    for (int i = 0; i < outputs.length; i++) outputs[i] = m.get(i).floatValue();
  }
}

Training workflow:

Launch Wekinator, set 6 inputs, 1 output, Classification, 3 classes.
Run the Processing sketch.
Draw a slow circular gesture with the mouse. In Wekinator, set output to class 1 and click Add Examples.
Draw a fast zigzag. Set to class 2, add examples.
Draw a diagonal line. Set to class 3, add examples.
Click Train in Wekinator.
The circle in Processing now changes colour based on gesture.

Building a Regression Model

Regression maps continuous inputs to continuous outputs — useful for driving visual parameters like colour, scale, or speed from hand position.

In Wekinator, create a project with:

Inputs: 2 (normalised XY)
Outputs: 3, type Continuous (Regression)

import oscP5.*;
import netP5.*;

OscP5      osc;
NetAddress wek;
float[]    outputs = {0.5, 0.5, 0.5};  // default: 3 regression outputs

void setup() {
  size(800, 600, P2D);
  colorMode(HSB, 1, 1, 1);
  osc = new OscP5(this, 12000);
  wek = new NetAddress("127.0.0.1", 6448);
}

void draw() {
  float nx = mouseX / (float)width;
  float ny = mouseY / (float)height;

  OscMessage m = new OscMessage("/wek/inputs");
  m.add(nx); m.add(ny);
  osc.send(m, wek);

  // Map regression outputs to colour
  float hue  = constrain(outputs[0], 0, 1);
  float sat  = constrain(outputs[1], 0, 1);
  float bri  = constrain(outputs[2], 0, 1);

  background(hue, sat, bri);
  fill(1 - bri, 0, bri > 0.5 ? 0 : 1);
  textSize(20);
  textAlign(CENTER);
  text(nf(hue,0,2) + "  " + nf(sat,0,2) + "  " + nf(bri,0,2), width/2, height/2);
}

void oscEvent(OscMessage m) {
  if (m.checkAddrPattern("/wek/outputs")) {
    for (int i = 0; i < min(outputs.length, m.typetag().length()); i++)
      outputs[i] = m.get(i).floatValue();
  }
}

Train the model by moving the mouse to different positions while clicking Add Examples with different target values set in Wekinator. After training, moving the mouse smoothly interpolates between the learned mappings.

Full Example: Real-Time Gesture-to-Visual Classifier

This sketch captures 10 mouse positions over half a second as a 20-float feature vector and maps the predicted class to a distinct visual style.

import oscP5.*;
import netP5.*;

OscP5      osc;
NetAddress wek;

final int  SAMPLES   = 10;
final int  DIMS      = 2;
float[][]  trail     = new float[SAMPLES][DIMS];
int        trailHead = 0;
long       lastSample = 0;
float[]    outputs   = {1};
int        styleIdx  = 0;

void setup() {
  size(1000, 700, P2D);
  osc = new OscP5(this, 12000);
  wek = new NetAddress("127.0.0.1", 6448);
}

void draw() {
  // Sample mouse position 20 times per second
  if (millis() - lastSample > 50) {
    trail[trailHead][0] = mouseX / (float)width;
    trail[trailHead][1] = mouseY / (float)height;
    trailHead           = (trailHead + 1) % SAMPLES;
    lastSample          = millis();

    // Send feature vector (ordered from oldest to newest)
    OscMessage m = new OscMessage("/wek/inputs");
    for (int i = 0; i < SAMPLES; i++) {
      int idx = (trailHead + i) % SAMPLES;
      m.add(trail[idx][0]);
      m.add(trail[idx][1]);
    }
    osc.send(m, wek);
  }

  if (outputs.length > 0) styleIdx = constrain((int)outputs[0] - 1, 0, 3);

  background(10);
  drawStyle(styleIdx);

  // Show mouse trail
  noFill(); stroke(255, 60);
  beginShape();
  for (int i = 0; i < SAMPLES; i++) {
    int idx = (trailHead + i) % SAMPLES;
    curveVertex(trail[idx][0]*width, trail[idx][1]*height);
  }
  endShape();

  fill(255); noStroke();
  textSize(18);
  text("Style: " + (styleIdx+1) + "  |  Move mouse, label in Wekinator, then Train", 10, 25);
}

void drawStyle(int s) {
  switch (s) {
    case 0:  // Concentric rings
      noFill(); stroke(0, 200, 255, 180); strokeWeight(1.5);
      for (int r=20; r<500; r+=20) ellipse(width/2, height/2, r, r);
      break;
    case 1:  // Grid lines
      stroke(255, 180, 0, 150); strokeWeight(1);
      for (int x=0; x<width; x+=30)  line(x, 0, x, height);
      for (int y=0; y<height; y+=30) line(0, y, width, y);
      break;
    case 2:  // Radial spokes
      stroke(180, 80, 255, 160); strokeWeight(1.2);
      for (int a=0; a<360; a+=12) {
        float rad = radians(a) + frameCount * 0.01;
        line(width/2, height/2, width/2+cos(rad)*350, height/2+sin(rad)*350);
      }
      break;
    case 3:  // Scattered dots
      noStroke(); fill(80, 255, 120, 200);
      randomSeed(42);
      for (int i=0; i<300; i++) ellipse(random(width), random(height), 6, 6);
      break;
  }
}

void oscEvent(OscMessage m) {
  if (m.checkAddrPattern("/wek/outputs")) {
    outputs = new float[m.typetag().length()];
    for (int i=0; i<outputs.length; i++) outputs[i] = m.get(i).floatValue();
  }
}

The 20-float trajectory feature gives Wekinator enough information to distinguish slow circular motions (class 1), fast horizontal sweeps (class 2), diagonal strokes (class 3), and vertical shakes (class 4). Each maps to a completely different visual aesthetic with no code changes required after the initial training session.