Face Model Documentation:

Screen Recording 2025-10-09 at 9.09.19 PM.mov

Reflection (Process + Documentation):

For this project, I was interested to learn how face features might trigger visual and audio feedback simultaneously in real-time. I started by taking the ml5.js FaceMesh model as a base and then overlaying custom interactions with p5.js and p5.sound. The overall concept was to make the face itself come alive, the lips are an active, neon-like shape that slightly "breathes" when the mouth is open, and the cheeks respond with gentle, floating blush spheres that follow emotional expression. I used the standard indices for the outer and inner lips, and on the cheeks, I calculated a personal anchor from the outer eye to the corner of the mouth so that the blush would be directly positioned at the cheekbone. Once I had modeled the facial geometry, I had the mouth opening value control a triangle-wave oscillator, so when I opened my mouth or spoke, it would generate a soft tone that changed pitch and amplitude.

Geometry- and calibration-related challenges were some of the most intimidating tasks I had to deal with. At first, the blush position looked fine in still recordings, but when the video was played, it would keep adhering to my eyes or lips. I thought I needed to calculate a dynamic anchor instead of relying on one landmark. Getting the layering right between the video feed, lip contour, and fill transparency was also a matter of trial and error since shapes would flip or draw inside out if the order of contours was incorrect. One tricky part was handling the sound responsiveness: the oscillator would be on all the time or glitch on the openness value being approximately zero, so I added a soft threshold and amplitude ramping to make it sound more natural. Overall, it was a sequence of little adjustments — experimenting with different constants, tweaking the lerp values, and finding that sweet spot where everything felt smooth and expressive rather than noisy.

P5.js Sketch link: https://editor.p5js.org/Anna_Tang/sketches/VoiawEcf0