Documentation:

For this assignment, I created an interactive p5 + ml5 sketch that trains a small neural network in real time using the handpose model from ml5.js (ml5 v0.12.2). The idea was to let users record examples of different hand gestures and have the model classify them as calm, energy, or noise. Each frame extracts three live features from the hand: pinch distance, finger spread, and wrist height. After collecting a few examples per label, the sketch trains a classifier directly in the browser and visualizes its predictions as colored confidence bars and background changes.

Process and Challenges: I started by adapting code from the ml5.js handpose examples and the NeuralNetwork classification demo on the ml5 documentation site. One of my main challenges was version compatibility — the newest ml5@1 build didn’t support ml5.handpose() anymore, so I had to downgrade to ml5 0.12.2. Because that version doesn’t include ml5.setBackend(), I had to remove that line completely. Once the correct version was loaded, the hand-tracking landmarks finally appeared.

Another issue was slow prediction performance. The handpose model was processing every video frame, and when I kept calling classify() too quickly, the browser lagged. I fixed this by throttling predictions so they happen about every 120 milliseconds instead of every frame, and by reducing the webcam resolution from 480 × 360 to 320 × 240. That small change made the interaction feel smoother and more responsive.

I also had to troubleshoot UI feedback. At first, I couldn’t tell whether samples were being added correctly, so I built visual counters and status messages that update whenever data is collected or the model is trained. This helped me see how many examples each class had and whether the model was ready to predict.

Reflection: Even though the idea seems simple, I learned how sensitive real-time data collection can be. Small hand movements or lighting differences can throw off the model, which made me more conscious of how noisy live input actually is. I liked how intuitive it felt to train a model through gestures rather than code or spreadsheets — it made machine learning feel more tangible. If I had more time, I’d extend the output so that each class triggers different sound responses using the p5.sound library.

Overall, this project made me appreciate how approachable ml5.js makes machine learning for creative work. The references I used — ml5.js documentation (ml5js.org), the p5.js Web Editor examples, and notes from Daniel Shiffman’s ml5 tutorials — helped me understand how to connect real-time sensor data with neural network training in a playful and artistic way.

Picture/Screenshot: