William Yao
Director, Tinkerer, Professional Side Quester.
Film & Media and Computer Science at Johns Hopkins, then five years as a software engineer at Bloomberg. Now at NYU’s ITP, making interactive music videos, immersive theater, and toys that remind him of a joyous time.

On the bench
More films and videos, all online
-

Shark Bait2018
-

Creator Culture2019
-

Love Like Movies2021
-

Small Talk2025
-

Heavenmade2024
-

A Short Film About Rainbows2020
-

Flowcode Music Videos2026
-
Neon LightsBaltimore Dance Crews ProjectFamily Set 2019
Neon Lights2019
-

Samsung x TwitchCon2023
Writing
Build notes from class, on whatever paper was there
-
Trying to Break Face ID

An intangible interaction that replaced a tangible one: Face ID.
-
Lovers & Enemies
Made with Jaye

Two mushroom figures spin like a Chinese rattle drum, fight, and stop to kiss when the love button is pressed.
-
The Watch that Captures the Sun

A handheld device that soaks up sunlight to allow the user to capture photos, encouraging time out in the sun.
-
Goodbye, Weiwei
With guidance from Emilee

An alter ego mask after the Red Mask of Beijing opera, sewn from salvaged fabric.
-
Controlling a Servo with My Hand
Serial communication between p5.js and an Arduino: a servo that follows the angle between my thumb and index finger…
-
Midterm Drafts: Soft Guitar and Butt Lamp

Two midterm ideas: a 3D-printed mold for a butt lamp, and a plush-like guitar that can be played by anyone…
-
Pressure Meter and a Greeting Robot
Greeting robot made with Aram

A pressure meter where more force lights more LEDs, and, with Aram, a time-of-flight sensor robot that turns to greet you.
-
Making My Arduino Talk to GarageBand

My Arduino as a USB MIDI controller for GarageBand: a potentiometer picks the note, a button plays it.
-
Robot Tennis and a Looping Machine

A pinball machine that turned into two robots playing tennis, and a Fly Me to the Moon looper where pressure sensors…
-
Castaway

A device you could build on a crash-landed island: shoes that turn walking into light.
The Watch that Captures the Sun

A handheld device that soaks up sunlight to allow the user to capture photos, encouraging time out in the sun.
The Watch that Captures the Sun

A handheld device that soaks up sunlight to allow the user to capture photos, encouraging time out in the sun.
A handheld device that soaks up sunlight to allow the user to capture photos, encouraging time out in the sun.
Inspiration (Little Sun)
Olafur Eliasson's Little Sun is a small, flower-shaped solar lamp, built to bring light to communities without electricity, but also a quiet object lesson in paying attention to the sun. You charge it by being outside with it.
I wanted to build a camera that worked in a similar way. Instead of focusing on the charging aspect, I wanted to also make being in the sun a ritual, a ritual with a reward. You have to face the light, wait, and earn the photo. When you take the photo, you feel more appreciative of the journey that got you there, and of the sunlight which made it possible.
Concept

A brass stopwatch that has a solar panel in the back, echoing Little Sun's design. I wanted it to be wearable, so the user can take it with them on, say, their belt loop, and soak up the sun in their everyday activities. The device shows an e-ink indication when it is ready to take a picture, or the crown at 12 o'clock glows when the watch is full. When the crown is pressed, it captures a photo. The face of the watch holds your last captured photo until the next one replaces it.
The reach is maybe that all the photos can then be saved by plugging into another device which rests at home. This device can both extract all the photos ever taken, and build another artwork, like a flower.
Build
Due to time constraints, I decided to make a p5.js, single-sketch, mobile-first prototype. The camera samples a luminance-weighted brightness average ten times a second.
I wrote two knobs to let the user calibrate to their circumstances and preferences:
- 1.Light Threshold sets the bar for what counts as "enough light"
- 2.Charge Speed multiplies how fast the petals fill
To make it work in our class, the hardest part was getting touch input to work on iOS Safari. p5's touchStarted abstraction has timing issues: mouseX/mouseY aren't always populated when the handler fires, so hit tests silently fail. The fix was bypassing p5 entirely and reading touch coordinates from the native event:
cnv.elt.addEventListener('touchstart', (e) => {
e.preventDefault();
const rect = e.currentTarget.getBoundingClientRect();
mouseX = e.touches[0].clientX - rect.left;
mouseY = e.touches[0].clientY - rect.top;
mousePressed();
}, { passive: false });
Here is the whole sketch:
The Watch that Captures the Sun, p5.js sketch (opens in a new tab)p5.js Web Editor
Reflections
I really love the physical commitment the piece asks for. You can't grab a photo casually. You have to go to the light. In the class demo, that meant everyone tilting toward windows, holding phones up to the sun together.
Though I do want to create a fully realized version that isn't just a p5 prototype. I also want to lean into the energy of the sun more. As some have suggested, playing with cyanotype and other forms of photography more closely related to the sun's effects.
All in all, I'm glad that Biophilic Experiences and Energy inspired me to think more about artworks that bring people closer to the world around them, as it has been one of my main goals here at ITP.
Lovers & Enemies
Made with Jaye

Two mushroom figures spin like a Chinese rattle drum, fight, and stop to kiss when the love button is pressed.
Concerned with whether I would have enough time to make the soft guitar or the butt lamp, I decided to come up with yet another project idea. This time I wanted something simpler on the fabrication. I also wanted to focus on just using what we have learned in class instead of new libraries. Lastly, I wanted it to feel more "artsy" than the other two "toy-like" projects.
Jaye and I came up with an art piece, temporarily named Lovers & Enemies. Inspiration: love is a constant choice.
Parts
- 1.Arduino Nano 33 IoT x 1
- 2.Motor driver x 2
- 3.DC motors x 2
- 4.DC power source x 1
- 5.Linear actuator motor x 2
- 6.Capacitor x 1 (powerful enough to run 4 motors)
- 7.White mushrooms x 2
- 8.King mushrooms x 2, or enoki mushrooms x 4
When Jaye and I watched Cybroc, the work by Ke Huang, Yue Zhou, Xi He, Weibo Chen and Botao Amber Hu, we felt a sense of wonder coupled with a meditation-like calmness. We were inspired to make something similar that could relate to our childhood memories. Jaye also grew obsessed with the idea of mushrooms.
We looked at a drum toy that every Chinese person remembers playing with as a child. We called it "Yao Yao Gu", which means "Shake Shake Drum". The toy itself does not involve much shaking. Rather, the player holds the wooden stick attached to the drum between two palms, then quickly spins it back and forth to make the stringy ends hit the drum faces.
We then came up with the idea to stick two rotating motors on two linear actuators. This way, we can spin the two "characters" while moving them forward and back. Jaye suggests that there is a much more elegant way to fabricate the movement, and I believe her 100%.

Like any good programmer/electrical engineer/artist, we drew out the entire circuit as well as the pseudocode needed to make this work. The fixture has 2 motor drivers controlling the pair of spinning motors and the linear actuators.

When the kiss/love button is not pressed, the mushroom drum figures keep spinning in opposite directions while moving together and further apart. When they come together, we don't want them to fully crash into each other, but we do want the suspense and release of them hitting each other with the enoki arms or the drum strings.
When the kiss/love button IS pressed, the figures stop spinning. Then they move towards the center, where they are finally close enough to kiss. Inside the white mushroom heads are two polarized magnets. We want the heads loose enough that when they come together, the mushrooms attach themselves to each other.
When the kiss/love button is released again, the figures move back "home" and begin spinning again, back to the neutral state.
Challenges
How do we make sure that the drums are flat (or flat enough) for the kiss, when we have no way of controlling how they spin and where they stop? One possible solution is to attach the drum's wooden stick to the motor through a bearing, so the stick itself is loose. When we push them together, it will be loose enough to be flattened.
Will we break the linear actuators if we force them past their max and min limits? Since we have no way of reading their position, we planned to push them forward for 3 seconds and back for 4 seconds, always making sure they return to their min. Or perhaps there is a better tool to move something to a predetermined position.
From linear actuators to servos
We tested the linear actuators and found that they are too slow, so we made a version that turns rotation into linear motion, with servos.
Arduino code (no agentic coding AI was used; the concept is not too complicated this time). This is the final version, with one button for each figure:
#include "Servo.h" // include the servo library
Servo servoMotorBoy;
Servo servoMotorGirl;
//pins
const int switchBoyPin = 2; // switch input
const int switchGirlPin = 8; // switch input
const int motorBoy1Pin = 6;
const int motorBoy2Pin = 7;
const int motorGirl1Pin = 4;
const int motorGirl2Pin = 5;
const int motorBoyPWMPin = 11;
const int motorGirlPWMPin = 10;
const int servoBoyPin = 9;
const int servoGirlPin = 3;
// constants
const int spinSpeed = 35; // motor spin speed
const int fightAngle = 30;
const int kissAngle = 0;
const int kissPauseAngle = 20;
const int homeAngle = 135;
const int angleIncrement = 7;
const int kissIncrement = 5;
// variables
int lastSwitchBoyState = LOW;
int lastSwitchGirlState = LOW;
int currentBoyServoAngle = 0;
int currentGirlServoAngle = 0;
bool boyApproaching = true;
bool girlApproaching = true;
const unsigned long moveInterval = 80;
const unsigned long kissInterval = 800;
long lastBoyMove = 0;
long lastGirlMove = 0;
void setup() {
// set the switch as an input
pinMode(switchBoyPin, INPUT);
pinMode(switchGirlPin, INPUT);
// set all pins other as output
pinMode(motorBoy1Pin, OUTPUT);
pinMode(motorBoy2Pin, OUTPUT);
pinMode(motorGirl1Pin, OUTPUT);
pinMode(motorGirl2Pin, OUTPUT);
pinMode(motorBoyPWMPin, OUTPUT);
pinMode(motorGirlPWMPin, OUTPUT);
servoMotorBoy.attach(servoBoyPin);
servoMotorGirl.attach(servoGirlPin);
Serial.begin(9600);
}
void loop() {
currentBoyServoAngle = servoMotorBoy.read();
currentGirlServoAngle = servoMotorGirl.read();
// <----------------------------------- Girl ------------------------------------>
if (digitalRead(switchGirlPin) == LOW) {
// normal state motors, Girl spins counterclock-wise
digitalWrite(motorGirl1Pin, LOW);
digitalWrite(motorGirl2Pin, HIGH);
analogWrite(motorGirlPWMPin, spinSpeed);
if (girlApproaching) {
if (millis() - lastGirlMove >= moveInterval) {
servoMotorGirl.write(currentGirlServoAngle > fightAngle ? currentGirlServoAngle - angleIncrement : fightAngle);
lastGirlMove = millis();
}
} else {
if (millis() - lastGirlMove >= moveInterval) {
servoMotorGirl.write(currentGirlServoAngle < homeAngle ? currentGirlServoAngle + angleIncrement : homeAngle);
lastGirlMove = millis();
}
}
} else if (digitalRead(switchGirlPin) == HIGH) {
// kiss state
// if the love button is pressed, stop spinning and move Boy and Girl together
// stop the spinning
analogWrite(motorGirlPWMPin, 0);
if (currentGirlServoAngle > kissAngle + kissPauseAngle) {
if (millis() - lastGirlMove >= moveInterval) {
servoMotorGirl.write(currentGirlServoAngle - kissIncrement);
lastGirlMove = millis();
}
} else {
if (millis() - lastGirlMove >= kissInterval) {
servoMotorGirl.write(kissAngle);
lastGirlMove = millis();
}
}
lastSwitchGirlState = HIGH;
}
if (currentGirlServoAngle <= fightAngle) {
girlApproaching = false;
} else if (currentGirlServoAngle >= homeAngle) {
girlApproaching = true;
}
if (digitalRead(switchGirlPin) == LOW && lastSwitchGirlState == HIGH) {
lastSwitchGirlState = LOW;
girlApproaching = false;
if (millis() - lastGirlMove >= moveInterval) {
servoMotorGirl.write(currentGirlServoAngle < homeAngle ? currentGirlServoAngle + kissIncrement : homeAngle);
lastGirlMove = millis();
}
}
// <----------------------------------- BOY ------------------------------------>
if (digitalRead(switchBoyPin) == LOW) {
// normal state motors, Boy spins clock-wise
digitalWrite(motorBoy1Pin, HIGH);
digitalWrite(motorBoy2Pin, LOW);
analogWrite(motorBoyPWMPin, spinSpeed);
if (boyApproaching) {
// in normal state, boy mushroom approaches and...
if (millis() - lastBoyMove >= moveInterval) {
servoMotorBoy.write(currentBoyServoAngle > fightAngle ? currentBoyServoAngle - angleIncrement : fightAngle);
lastBoyMove = millis();
}
} else {
// goes apart
if (millis() - lastBoyMove >= moveInterval) {
servoMotorBoy.write(currentBoyServoAngle < homeAngle ? currentBoyServoAngle + angleIncrement : homeAngle);
lastBoyMove = millis();
}
}
} else if (digitalRead(switchBoyPin) == HIGH) {
// kiss state
// if the love button is pressed, stop spinning and move Boy and Girl together
// stop the spinning
analogWrite(motorBoyPWMPin, 0);
if (currentBoyServoAngle > kissAngle + kissPauseAngle) {
if (millis() - lastBoyMove >= moveInterval) {
servoMotorBoy.write(currentBoyServoAngle - kissIncrement);
lastBoyMove = millis();
}
} else {
if (millis() - lastBoyMove >= kissInterval) {
servoMotorBoy.write(kissAngle);
lastBoyMove = millis();
}
}
lastSwitchBoyState = HIGH;
}
if (currentBoyServoAngle <= fightAngle) {
boyApproaching = false;
} else if (currentBoyServoAngle >= homeAngle) {
boyApproaching = true;
}
if (digitalRead(switchBoyPin) == LOW && lastSwitchBoyState == HIGH) {
lastSwitchBoyState = LOW;
boyApproaching = false;
if (millis() - lastBoyMove >= moveInterval) {
servoMotorBoy.write(currentBoyServoAngle < homeAngle ? currentBoyServoAngle + kissIncrement : homeAngle);
lastBoyMove = millis();
}
}
}
Goodbye, Weiwei
With guidance from Emilee

An alter ego mask after the Red Mask of Beijing opera, sewn from salvaged fabric.
This week our assignment was to create an alter ego mask. I wanted to recreate the Beijing Opera (or Jing Opera) mask that my mom used to show me when she played in these shows as a musician. I sometimes wonder if there is a suppressed version of William deeply hidden in my own subconscious.
I often find myself teetering on the line between showmanship, drama and expressionism, and the other side: the good, the proper, the stable. I want to create this mask not to hide away, but to allow that first personality to flourish.
This was also a huge undertaking for me, making a mask out of soft material, because I've never sewn in my life. I'm glad I was able to get a lot of guidance from Emilee. We ended up making 2 masks that complemented each other's look.

The flesh of the piece is made with salvaged fabric from Materials for the Arts in Long Island City. I taped up a mask and drew out a pattern after the Red Mask in Jing Opera, which represents bravery and righteousness. This feels like a character trait that many have ascribed to me, even if I don't personally feel right wearing these titles.






Midterm Drafts: Soft Guitar and Butt Lamp

Two midterm ideas: a 3D-printed mold for a butt lamp, and a plush-like guitar that can be played by anyone…
Butt Lamp
This week I learned how to 3D model, so I can print a mold for my butt lamp idea.


I'm currently trying to reach the Soft Robotics teacher to get some pointers on how to make this butt out of silicone.
I'm thinking of using an individually addressable LED strip instead of multiple LEDs for this first iteration of the lamp, so that the physical construction inside the lamp can be much simpler.
Soft Guitar
While waiting for that idea to marinate, I wanted to test my other idea: soft instruments. The idea is rather simple. Create a plush-like guitar that can be played by anyone. The construction should be pretty simple, with 3 or 4 pressure sensors that tell the program where the player is pressing, and an IR sensor that tells the program when the player strums.
I might be thinking too much like a software engineer, because I want to keep my constructions easy and make the code do the heavy lifting. Though I am trying to remember that if my code gets too complicated, chances are there is better hardware to solve my problem.
My goal this week is to prototype playing adjustable chords with a pressure sensor and our VL53L0X time-of-flight sensor (I wonder why it's called that), with GarageBand and an Arduino Nano.

// Play a chord (multiple notes simultaneously)
void playChord(byte rootNote) {
for (int i = 0; i < CHORD_SIZE; i++) {
byte chordNote = rootNote + CHORD_INTERVALS[i];
// Ensure note is within valid MIDI range
chordNote = constrain(chordNote, 0, 127);
noteOn(MIDI_CHANNEL, chordNote, VELOCITY);
lastChordNotes[i] = chordNote;
}
}
// Check if guitar is strummed
bool strummed() {
int result = MAX_RANGE;
if (sensor.isRangeComplete()) {
result = sensor.readRangeResult();
}
// Determine if hand is present (with some stability)
bool handPresent = ((result < MAX_STRUM_RANGE && result > 0) && result != MAX_RANGE); //false
// Only trigger when hand comes back to sensor (not while hovering)
bool shouldTrigger = false;
if (handPresent && !handCurrentlyPresent) {
// Hand just arrived - trigger!
shouldTrigger = true;
handCurrentlyPresent = true;
} else if (!handPresent && handCurrentlyPresent) {
// Hand just left - reset for next trigger
handCurrentlyPresent = false;
}
lastSensorDistance = result;
return shouldTrigger;
}
Problems to solve
- 1.The pressure sensor is too sensitive and changes the chord way too easily. Some better mapping should help.
- 2.How will the 3 sensors on the neck work? Maybe we don't need an analog input for them; we really just need to know if each is pressed. However, I think more inputs means more fun.
- 3.I need to figure out why the states are not working in my code, because it keeps playing the note when I hover my hand over the sensor. It should just play once.
Lab work
Working with DC motors, drivers and transistors this week. I wrote a little bit of code to make the motor drop in speed and slowly build up whenever it switches direction.

const int switchPin = 2; // switch input
const int motor1Pin = 3; // Motor driver leg 1 (pin 3, AIN1)
const int motor2Pin = 4; // Motor driver leg 2 (pin 4, AIN2)
const int pwmPin = 5; // Motor driver PWM pin
void setup() {
// set the switch as an input:
pinMode(switchPin, INPUT);
// set all the other pins you're using as outputs:
pinMode(motor1Pin, OUTPUT);
pinMode(motor2Pin, OUTPUT);
pinMode(pwmPin, OUTPUT);
Serial.begin(9600);
}
const int minimalSpeed = 10;
const int maxSpeed = 80;
const int speedInterval = 100;
int speed = minimalSpeed;
int lastState = LOW;
long speedLastIncreased = 0;
void loop() {
if (digitalRead(switchPin) == HIGH) {
speed = lastState == LOW ? minimalSpeed : speed;
lastState = HIGH;
if (speed >= maxSpeed) {
analogWrite(pwmPin, maxSpeed);
} else if (millis() - speedLastIncreased >= speedInterval) {
speed++;
analogWrite(pwmPin, speed);
speedLastIncreased = millis();
} else {
analogWrite(pwmPin, speed);
}
digitalWrite(motor1Pin, HIGH); // set leg 1 of the motor driver high
digitalWrite(motor2Pin, LOW); // set leg 2 of the motor driver low
}
// if the switch is low, motor will turn in the other direction:
else {
speed = lastState == HIGH ? minimalSpeed : speed;
lastState = LOW;
analogWrite(pwmPin, speed >= maxSpeed ? maxSpeed : millis() - speedLastIncreased >= speedInterval ? speed++ : speed);
speedLastIncreased = millis() - speedLastIncreased >= speedInterval ? millis() : speedLastIncreased;
digitalWrite(motor1Pin, LOW); // set leg 1 of the motor driver low
digitalWrite(motor2Pin, HIGH); // set leg 2 of the motor driver high
}
}
Open questions
- 1.Why would the voltage of the switching power supply drop to zero while I turn up my potentiometer? This especially happens when I am supplying a low current.
- 2.I can "bypass" the potentiometer and transistor when I connect the ground wire of the motor to the resistor between the output pin and the transistor. Why is that?
- 3.Why does my MOSFET transistor circuit not work?
I could not for the life of me get this stepper motor lab to work.

Robot Tennis and a Looping Machine

A pinball machine that turned into two robots playing tennis, and a Fly Me to the Moon looper where pressure sensors…
My goal this week is to build things from what we learned and to think about midterm ideas at the same time.
Design
I drew up a circuit diagram for a simple pinball machine:
- 1.2 buttons that operate 2 servos
- 2.An ultrasonic distance sensor that senses when a ball is dropped into the game, starting or restarting the game
- 3.An FSR that, when the ball hits it, grants points based on how hard it was hit
- 4.A 7-segment display, which I don't know how to use yet

Problems and risks
- 1.If I use an ultrasonic sensor to detect the ball passing a certain point, I'll have to make sure the ball never reaches that point again during the game, or else the game will restart.
- 2.The ultrasonic sensor has a cone-shaped detection area, so maybe it would be smarter to put a pressure sensor in the ball's path?
- 3.I'm pretty sure most pinball machines use light sensors.
I realized that I can replace the human player with another servo and create a match of tennis between two robots. I can use a marble, so whenever the ball rolls over one of the two pressure sensors (attached underneath a wooden sheet), the servo motors, likely attached to some wooden planks, swing in that direction.

I felt like this wouldn't be very fun, and if I have to account for the servo motor resetting, I probably don't want Player 1 to swing when the ball is going towards Player 2. So I added two more pressure sensors.

The schematic would look something like this. I'm obviously being very idealistic about how the fabrication will look, and I'm also overestimating how much area each racket can cover.

To do this, I will build 1 player first, with 1 sensor. To make sure the pressure sensors can lie flat on the ground, I twisted some stripped wires around them.

The build

#include <Servo.h>
Servo tennisServo;
const int SERVO_P1 = 2;
int servo_p1_pos = 90;
int sensor_1_value = 0;
long lastMoveTime = 0;
void setup() {
Serial.begin(9600);
tennisServo.attach(SERVO_P1);
tennisServo.write(servo_p1_pos);
}
void loop() {
sensor_1_value = analogRead(A0);
if(sensor_1_value > 300 && millis() - lastMoveTime > 50) {
if (servo_p1_pos >= 90) {
tennisServo.write(0);
servo_p1_pos = 0;
} else {
tennisServo.write(179);
servo_p1_pos = 179;
}
lastMoveTime = millis();
}
}
The result
The servo was actually not getting any power. I found out that the Nano doesn't have its 5V out available: behind the 5V pin, the pads are separated, and they would need to be soldered together for the power to be supplied.

Instead, I moved everything so that the servo would get power from the 3.3V pin. The speed was extremely disappointing, and I realized I'll have to explore other motors and power supplies before I can build tennis players strong enough to play against each other.
It's probably a good time to talk about capacitors. As you can see, I didn't use one, mainly because I didn't find the 10 uF capacitor on the floor. However, I learned that they are supposed to help when the current spikes and dips.
A looping machine
Moving on to tone. I wanted to build a looping machine, or a beat-making machine; I don't have the perfect name for it. I want to play a series of notes, in this case the melody of the first 4 bars of Fly Me to the Moon, when I press a pressure sensor. But I don't want the sensor to start the song from the beginning every time I press it.
Rather, I want the song to always be "synced to time". I want the pressure sensor to act like a gate instead of a play-and-stop button. The song should keep "playing in the background", and whenever I press the sensor, I just hear it. This way, I can have another sensor with another speaker, which plays the walking bass line of the same 4 bars when I press it.
This design can then scale to as many layers and buttons as we want, and lets anybody choose which stems play, since they will always be synced. I assume I will just have to get clever with millis().

#include "notes.h"
// notes in the melody:
int melody[] = {
NOTE_C5, NOTE_B4, NOTE_A4, NOTE_G4, NOTE_F4, NOTE_G4, NOTE_A4, NOTE_C5,
NOTE_B4, NOTE_A4, NOTE_G4, NOTE_F4, NOTE_E4
};
// note durations: 4 = quarter note, 8 = eighth note, etc.:
float noteDurations[] = { 3, 8, 8, 4, 4, 4, 4, 4, 3, 8, 8, 4, 1};
const int NUM_NOTES = sizeof(melody) / sizeof(melody[0]);
const int SENSOR_PIN = A0;
const int SPEAKER_PIN = 8;
const int SENSOR_THRESHOLD = 300;
int currentNote = 0;
unsigned long noteEndTime = 0; // when the current note should advance
bool speakerOn = false;
bool lastGate = false; // remember previous gate state
unsigned int noteMs[NUM_NOTES];
void startNextNoteWindow(unsigned long now) {
// Advance to next note in a loop
currentNote = (currentNote + 1) % NUM_NOTES;
unsigned int dur = noteMs[currentNote];
noteEndTime = now + dur;
}
void setup() {
Serial.begin(9600);
pinMode(SPEAKER_PIN, OUTPUT);
// Build ms durations and initialize transport
for (int i = 0; i < NUM_NOTES; i++) {
noteMs[i] = (unsigned int)(2000.0 / noteDurations[i]); // your 120 BPM-ish formula
}
unsigned long now = millis();
// Start at note 0
noteEndTime = now + noteMs[0];
}
void loop() {
unsigned long now = millis();
// Advance the notes in the melody
if ((long)(now - noteEndTime) >= 0) { // time to move to next note window
startNextNoteWindow(now);
}
int gate = analogRead(SENSOR_PIN) > SENSOR_THRESHOLD;
// If gated, play whatever note window we're currently in
// We'll (re)trigger tone when:
// - the gate just turned on
// - the note changed
// - or tone isn't currently on
static int lastNoteWhenPlayed = -1;
if (gate) {
// Remaining time in this note window
unsigned long remaining = (noteEndTime > now) ? (noteEndTime - now) : 1;
if (!speakerOn || !lastGate || currentNote != lastNoteWhenPlayed) {
// Retrigger tone for the remaining slice of this note
tone(SPEAKER_PIN, melody[currentNote], remaining);
speakerOn = true;
lastNoteWhenPlayed = currentNote;
}
} else {
// Gate closed: ensure the speaker is muted
if (speakerOn) {
noTone(SPEAKER_PIN);
speakerOn = false;
}
}
lastGate = gate;
}
I'm giving each note in the melody array a calculated duration, then checking millis() against each note's duration to know when to move to the next note. When the gate is open, I play the note for the time it has left.
Moving on to the second speaker and sensor pair, I quickly ran into the issue that tone() only uses one timer, so it can only play into one speaker at a time. I then discovered toneAC(), which uses another timer. I had to resort to vibe coding with ChatGPT, which was rather gross, but I wanted to see how it would look. It turns out the Nano 33 IoT doesn't support toneAC(), so I might have to borrow someone's Uno.
In another world where I'm totally insane and have all the time, I would oscillate very quickly between two notes whenever two notes play at once, perhaps 10 ms per note. That's the only way I can see layering work on the Nano with a software workaround. It feels like a lot of work compared to just switching to another microcontroller.
Trying to Break Face ID

An intangible interaction that replaced a tangible one: Face ID.
This week's assignment is to identify an "intangible" interactive system that is replacing what used to be "tangible". I walked around my apartment to find such systems. Initially, I had very little luck. I couldn't find anything other than Face ID, Siri, and the myriad interactions that my phone has made somewhat intangible.
Then I got hungry, and stopped at my air fryer. I thought about how convenient we have made roasting meat. One could compare the ease of use to a traditional oven, but I think we could go even further back. Instead of a finicky construction of 3 wooden sticks, we've replaced it with a clean-edged, usually black metal box. And instead of an open fire, where one would have to constantly add firewood, fan the flame to keep it going, and turn the meat to make sure all sides cook evenly, we now have a button that just says "Airfry".
So I decided to go back to Face ID.
While flying in and out of Japan in the winter, I noticed that they no longer have a security officer checking your boarding pass and passport. Instead, there are facial recognition machines that ask you to scan your passport. Further down the security check, many gates automatically scan your face and open.
Facial recognition was first introduced to the western masses with smartphones, most notably when the iPhone switched from fingerprint (Touch ID) to Face ID with the iPhone X in 2017. Now unlocking your phone is almost entirely intangible. Since then, it feels like the world picked up on unlocking basically anything with your face. The technology isn't terribly difficult, and the sensor is usually a small camera.
There are 4 steps to this interaction:
- 1.Scanning waits to be turned on, or to detect a face
- 2.The user turns on Face ID, or walks into detection range
- 3.A signal tells the user that face scanning is taking place
- 4.It reports success and unlocks the device, or reports failure
If it fails, the user engages (if not enrages) in a loop of trying again, or, on most phones, is prompted to unlock with a password instead.

It turns out that, at least on the iPhone, the sensor is not a normal camera (the front selfie camera), but a TrueDepth camera system dedicated to Face ID. It includes the front camera, a dot projector, a flood illuminator and an infrared camera:
- 1.The dot projector projects a grid of 30,000 infrared dots
- 2.The infrared camera captures the distortion that the contours of our faces make in the dot grid
- 3.The flood illuminator adds infrared light to help capture faces in a dark place
However, this rather expensive approach is not what most facial recognition systems use. Rather, they use a high-resolution 2D camera and compare 2D images or video. The "Face Express" system used in Japan relies on NEC's Bio-IDiom technology. Instead of just a selfie, the system compares your live face to the high-quality biometric image stored on your passport's IC chip, a standard in about 180 countries.
I tried to "break" Face ID by making extremely exaggerated facial expressions: elated, depressed, enraged and smug. All of them failed to unlock the phone, which I assume is due to the massive distortion compared with the infrared grid stored in the phone.




Now, the logical thing to do is to create different infrared mappings for different emotions to cover all these "use cases". But the more interesting thing to do is to create an unlocking system that unlocks different things based on the emotion it detects.
Perhaps eventually I can walk up to my air fryer, and instead of pressing any buttons, it will either make me a crispy chicken thigh because I'm feeling good and light-hearted, or a slow-roasted chicken thigh because I am depressed and can't be bothered to bite into anything too hard.
Pressure Meter and a Greeting Robot
Greeting robot made with Aram

A pressure meter where more force lights more LEDs, and, with Aram, a time-of-flight sensor robot that turns to greet you.
Goal
To create a pressure-sensing meter with LEDs: the stronger the force, the more LEDs light up.
I can use the map() function and analog output to set the brightness of the LEDs from the pressure. For one LED, I can go from 0 to 255 up to a certain amount of force, let's say 200. Whenever the force goes past that, I light up another LED. This repeats until all 3 LEDs are fully lit.
Problems
I first tried to light up the small white LEDs in our kit. They would not light up. Initially I thought the resistors were too strong, but I didn't have any smaller resistors to test, and I really did not want to pop any light bulbs. So I switched in a red LED to see if it would work. It did. So I switched in my colored LEDs.

However, I could see that once the red and green LEDs hit maximum brightness, they went back down to really, really dim and climbed again. This made me think something was going on with the map() function.
Solution
I broke the logic up into very ugly if statements. That seemed to do the trick: map() doesn't actually keep its result between 0 and 255 when the input goes over 200.
void loop() {
sensorValue = analogRead(A0); // read the pot value
int brightness = 0;
if (sensorValue < 200) {
brightness = map(sensorValue, 0, 200, 0, 255);
analogWrite(firstLED, brightness);
} else if (sensorValue < 600) {
brightness = map(sensorValue, 0, 200, 0, 255);
analogWrite(firstLED, brightness);
brightness = map(sensorValue, 200, 600, 0, 255);
analogWrite(secondLED, brightness);
} else {
brightness = map(sensorValue, 0, 200, 0, 255);
analogWrite(firstLED, brightness);
brightness = map(sensorValue, 200, 600, 0, 255);
analogWrite(secondLED, brightness);
brightness = map(sensorValue, 600, 900, 0, 255);
analogWrite(thirdLED, brightness);
}
}
Tests
The potentiometer was pretty straightforward, other than the fact that I was using a rotary encoder by mistake for an hour.

With digital output, I tried to create a pulse-like light wave when I press a button. I tried to do the same with a pressure sensor, but I think because there are too many LEDs, it started to act up.
A robot that greets you
Aram and I also worked on a laser-sensor-based "robot". We were trying to test the limits of the laser sensor, which advertised that it could sense up to 1 meter, but in reality only really worked up to 250 mm. Our robot would "greet" the user if they were close enough to the sensor.
In the code you can see us trying to debug lastDistance, because we realized that the laser sensor would keep scanning the "ghost" of an object if that object was removed quickly. So it would keep thinking that a user was close to the sensor, even after they had left.
The hardware way to resolve this is to keep something within 250 mm of the sensor as a "reference wall", so the sensor sees the wall once the subject is removed. The software way was to keep a lastDistance.
#include <Wire.h>
#include <Adafruit_VL53L0X.h>
#include <Servo.h>
// --- Sensors & Actuators ---
Adafruit_VL53L0X lox; // VL53L0X ToF sensor
Servo head; // SG-5010 servo
// --- Pins ---
const int SERVO_PIN = 2; // D2: servo signal (power servo from 5–6V, share GND)
// --- Angles ---
const int CENTER_DEG = 90; // neutral
const int FORWARD_DEG = 125; // 90 + 35
// --- Thresholds with hysteresis for "close" detection ---
const int CLOSE_MM_ON = 180; // enter CLOSE when dist < 180 mm
const int CLOSE_MM_OFF = 180; // leave CLOSE when dist > 180 mm (used to re-arm)
// --- Smoothing for distance ---
float distMM = 400; // smoothed distance in mm
const float ALPHA = 0.35; // 0..1 (higher = snappier)
// --- Behavior timing ---
const unsigned long FORWARD_HOLD_MS = 300; // how long to stay at FORWARD before returning
const int RETURN_STEP_DEG = 1; // degrees per frame when easing back
const unsigned long LOOP_DELAY_MS = 20; // loop delay (also sets return speed)
// --- Simple state machine ---
enum State { FAR_IDLE, CLOSE_FORWARD, RETURNING };
State state = FAR_IDLE;
unsigned long stateSince = 0;
int lastDistance = 400;
void setup() {
Serial.begin(115200);
while (!Serial) { delay(1); } // wait for Serial Monitor
Wire.begin(); // I2C on A4/A5 (Nano 33 IoT default)
if (!lox.begin()) { // init VL53L0X
Serial.println(F("Failed to find VL53L0X. Check wiring."));
while (1) {}
}
head.attach(SERVO_PIN); // attach servo on D2
head.write(CENTER_DEG); // start centered
delay(300);
}
void loop() {
// --- 1) Read distance (mm) and smooth it ---
VL53L0X_RangingMeasurementData_t m;
lox.rangingTest(&m, false); // false = no library debug prints
if (m.RangeStatus == 0) { // valid reading
float mm = (float)m.RangeMilliMeter;
distMM = ALPHA * mm + (1 - ALPHA) * distMM;
}
// Helper flags for readability
bool closeEnter = (distMM < CLOSE_MM_ON); // under ON threshold
bool closeExit = (distMM > CLOSE_MM_OFF); // over OFF threshold
// --- 2) State machine ---
unsigned long now = millis();
int current = head.read();
switch (state) {
case FAR_IDLE:
// // We're centered, waiting for someone to come close.
head.write(CENTER_DEG);
if (closeEnter && lastDistance != (int)distMM) {
// // Trigger CLOSE: snap to forward and start hold timer.
head.write(FORWARD_DEG);
state = CLOSE_FORWARD;
stateSince = now;
lastDistance = distMM; // debug label
}
break;
case CLOSE_FORWARD:
// // Stay at FORWARD for a short moment to acknowledge the approach.
head.write(FORWARD_DEG);
if (now - stateSince >= FORWARD_HOLD_MS) {
// // Time to begin a slow return to center (no matter the distance).
state = RETURNING;
stateSince = now;
lastDistance = distMM; // debug label
}
break;
case RETURNING:
// // Ease back toward center in small steps to feel alive.
if (abs(current - CENTER_DEG) <= RETURN_STEP_DEG) {
head.write(CENTER_DEG); // // snap when close enough
} else if (current < CENTER_DEG) {
head.write(current + RETURN_STEP_DEG);
} else {
head.write(current - RETURN_STEP_DEG);
}
// // Re-arm only after user goes far again (prevents rapid retriggers).
if (head.read() == CENTER_DEG) {
state = FAR_IDLE;
stateSince = now;
lastDistance = distMM; // debug label
}
break;
}
delay(LOOP_DELAY_MS);
}
Controlling a Servo with My Hand
Serial communication between p5.js and an Arduino: a servo that follows the angle between my thumb and index finger…
This week I learned how to communicate between p5.js and my Arduino using asynchronous serial communication.
Using what I've learned in ICM and other coding classes, I tried to control my servo motor with my hand motion. With ml5.js, I used a pretty simple (perhaps clever) slope calculation between my thumb and index fingertips.
let handPose;
let hands = [];
let myVid;
function preload() {
handPose = ml5.handPose({flipped: true});
}
function setup() {
createCanvas(640, 480); // make the canvas
// ... p5.WebSerial port setup ...
myVid = createCapture(VIDEO, {flipped: true});
myVid.hide();
handPose.detectStart(myVid, gotHands);
}
function gotHands(results) {
hands = results;
}
function draw() {
image(myVid, 0, 0, width, height);
if (hands.length > 0) {
let hand = hands[0];
for (let j = 0; j < hand.keypoints.length; j++) {
let keypoint = hand.keypoints[j];
fill(0, 255, 0);
noStroke();
circle(keypoint.x, keypoint.y, 10);
}
const dy = (hand.index_finger_tip.y - hand.thumb_tip.y);
const dx = (hand.index_finger_tip.x - hand.thumb_tip.x);
const angle = atan2(dy, dx);
const mappedAngle = map(angle, -PI, PI, 0, 360);
const constrantedAngle = constrain(mappedAngle, 0, 180);
// only write to serial every `writeInterval` milliseconds
if (millis() - lastWriteTime >= writeInterval) {
serial.write(constrantedAngle);
lastWriteTime = millis();
}
fill(255, 0, 0);
textSize(32);
text("Mapped Angle Value: " + mappedAngle, 10, height - 20);
text("Constrained Angle Value: " + constrantedAngle, 10, height - 60);
}
text("incoming value: " + inData, 30, 30);
}
#include "Servo.h"
Servo servo;
const int servoPin = 5;
void setup() {
Serial.begin(9600); // initialize serial communications
servo.attach(servoPin);
Serial.begin(9600);
}
void loop() {
if (Serial.available() > 0) { // if there's serial data available
int inByte = Serial.read(); // read it
Serial.write(inByte); // send it back out as raw binary data
servo.write(inByte);
}
}
I don't really know why the incoming value, echoed back from the Arduino, is so jittery, and I wonder if that's why the motor is also not as responsive as I want it to be at times.
Then I did the Two-Way (Duplex) Serial Communication lab, in the hope of learning more about handshakes and perhaps solving my problems. Following the lab, I wrote code so that p5 and the Arduino shake hands before sending data.
Making My Arduino Talk to GarageBand

My Arduino as a USB MIDI controller for GarageBand: a potentiometer picks the note, a button plays it.
This week I tried to make my Arduino talk to GarageBand as a MIDI controller. I used a potentiometer and a button. Turning the potentiometer changes the signal and acts as different keys; pressing the button sends the note to GarageBand.
#include "MIDIUSB.h"
// Hardware Configuration
const int POTENTIOMETER_PIN = A0; // Analog pin for pitch control
const int BUTTON_PIN = 2; // Digital pin for note trigger (with internal pullup)
// MIDI Configuration
const byte MIDI_CHANNEL = 0; // MIDI channel (0-15, where 0 = channel 1)
const byte VELOCITY = 80; // Note velocity (0-127)
const int NOTE_OFFSET = 48; // Base note offset (C3 = 48, adjust as needed)
const int NOTE_RANGE = 24; // Range of notes (2 octaves)
// Variables
byte currentNote = 0; // Current MIDI note value
byte lastNotePlayed = 0; // Last note that was played
int lastButtonState = HIGH; // Previous button state (HIGH = not pressed due to pullup)
int currentButtonState = HIGH; // Current button state
bool notePlaying = false; // Track if a note is currently playing
void setup() {
// Configure pins
pinMode(BUTTON_PIN, INPUT_PULLUP); // Button with internal pullup resistor
}
void loop() {
// Read potentiometer value (0-1023) and convert to MIDI note (0-127)
int potValue = analogRead(POTENTIOMETER_PIN);
currentNote = map(potValue, 0, 1023, 0, NOTE_RANGE) + NOTE_OFFSET;
// Ensure note is within valid MIDI range
currentNote = constrain(currentNote, 0, 127);
// Read button state
currentButtonState = digitalRead(BUTTON_PIN);
// Check for button press (LOW because of pullup resistor)
if (currentButtonState == LOW && lastButtonState == HIGH) {
// Button just pressed - start playing note
if (!notePlaying) {
noteOn(MIDI_CHANNEL, currentNote, VELOCITY);
lastNotePlayed = currentNote;
notePlaying = true;
}
}
// Check for button release
if (currentButtonState == HIGH && lastButtonState == LOW) {
// Button just released - stop playing note
if (notePlaying) {
noteOff(MIDI_CHANNEL, lastNotePlayed, 0);
notePlaying = false;
}
}
// If note is playing and pitch has changed significantly, update the note
if (notePlaying && abs(currentNote - lastNotePlayed) > 0) {
// Turn off the old note
noteOff(MIDI_CHANNEL, lastNotePlayed, 0);
// Turn on the new note
noteOn(MIDI_CHANNEL, currentNote, VELOCITY);
lastNotePlayed = currentNote;
}
// Save button state for next iteration
lastButtonState = currentButtonState;
// Small delay to prevent overwhelming the MIDI output
delay(10);
}
// Send MIDI Note On message
void noteOn(byte channel, byte pitch, byte velocity) {
midiEventPacket_t noteOn = {0x09, 0x90 | channel, pitch, velocity};
MidiUSB.sendMIDI(noteOn);
MidiUSB.flush();
}
// Send MIDI Note Off message
void noteOff(byte channel, byte pitch, byte velocity) {
midiEventPacket_t noteOff = {0x08, 0x80 | channel, pitch, velocity};
MidiUSB.sendMIDI(noteOff);
MidiUSB.flush();
}
There is a good amount of tweaking that needs to happen. Firstly, I need to fix the button so that the audio only plays when I press it. I assume this has something to do with using the internal pull-up resistor. Secondly, I want multiple buttons to work at once (multiple inputs), so that it can act more like a MIDI keyboard.
While doing the labs, I came up with some concepts and wanted to know if they were "real" or "functional".


Castaway

A device you could build on a crash-landed island: shoes that turn walking into light.
This week's assignment is to come to class with a design for a device that can be made on a crash-landed island, preferably one that transforms kinetic energy into electricity.
I started imagining the sources of "sustained energy" that can be harnessed on an island. Two things came to mind: waves and gravity.
Waves are a little difficult to prototype and test at ITP, so I opted to think about how to harness gravity. Jeff sent me a company that makes what they call GravityLight. They lift a weight to a height with high potential energy, then convert that energy into kinetic energy as the weight travels down. The kinetic energy is then turned into electricity to power the light.
I want something that doesn't require me to do any more work than I already would on the island. What is something I must do every day that could be turned into electricity?
Walking.
Fitting nicely with the theme, I want to create a pair of shoes that power a battery, which can then be used to turn on an LED light display. Considering the varying pressure I can apply to these shoes, perhaps this LED display should vary with the pressure. Skipping the battery, maybe I can put these boots on and enjoy an LED panel that works both as a source of light and as an artistic outlet.


