AN-005 · Embedded & Robotics · In progress
Started Nov 2025

RoboOrchestra: robots that play music, and a conductor who waves at a camera

1.0Abstract

I lead RoboOrchestra, a 40+ member group at CMU that builds robotic musical instruments. My own work on it has been a full rewrite of the instrument firmware on ESP32 Feathers, a MIDI-based protocol so the instruments can talk to each other over a wireless link, and a conductor mode that plays the drum set from hand gestures on camera.

At a glance

Parameter	Value
Status	In progress
When and where	Nov 2025 – present, Carnegie Mellon
My part	Project lead of a 40+ member community; the firmware rewrite, the protocol and conductor mode are my own builds
Built with	ESP32 Feathers with W.FL antennas; MIDI; gesture detection on camera
Protocol	Platform-independent MIDI between instruments, carried over a wireless link between the boards
Conductor mode	A waved hand plays the drum set; the region of the screen the hand is in selects what gets hit

About twenty green RoboOrchestra handbell driver boards, Rev. 2, laid out in rows on a black tray beside bags of components — A run of Rev. 2 handbell boards, April 2026.

Why this exists

I play viola and piano and I build robots, so an instrument that plays itself is the obvious thing to make. RoboOrchestra is the CMU community doing exactly that: a room of people building instruments that hit, pluck and strike on command.

Seen from above, an orange robot arm drawing a bow across a violin held flat on a workbench — One of the instruments: a robot arm bowing a scale on a violin. Eight seconds of a take that was filed under “oops”.

What I built

Rewrote the instrument firmware completely. The instruments run on ESP32 Feathers.
Established the inter-instrument protocol: platform-independent MIDI messages, a language every sequencer and DAW already speaks, carried over a wireless link between the boards, with W.FL antennas added for potential longer-range operation.
A conductor mode. Gesture detection merged with drum-beat triggering: wave a hand at the camera and the drum set plays; which region of the screen the conducting hand is in decides which drum is hit.

One RoboOrchestra handbell board, Rev. 2: a shift register, a solenoid driver, a DIP switch for the board’s ID and an eight-pin connector at each end, with A4 written on its label — One handbell board: a solenoid driver with an ID switch, daisy-chained through the connector at each end. This one plays A4.

A drumstick held in a blue 3D-printed striker clamped to a snare drum, hitting a practice pad, with a laptop showing a score beside it — A drum striker on the bench: a stick in a printed mount, with the score open on the laptop beside it. Eight seconds of the rehearsal video.

← All projects