Great Balls of Fun

What was IDEO’s contribution? Giant beach balls. But not for the beach. More like magical floating orbs that made music when you touched them.

“We wanted to build an instrument with play as an interface,” said creative director Peter Hyer. “An interactive environment in which anyone can play music without giving it a thought.”

‍

How do they work? Here's the layperson’s version: The team fitted the balls with sensors that mapped the force of the impact to notes. Those notes were then matched to an algorithmically-generated backing track.

For the more technical version, keep on reading.

The process

Ideation

We knew that we wanted to channel the spirit of Bob Moog in our installation, so we began with a brainstorming and prototyping phase where we focused on bridging music and technology with creativity. We really wanted to break the barrier that prevents people from expressing themselves musically, and we also wanted to capture something of the Moogfest zeitgeist. Two ideas emerged—one around music, and one around narrative and synthesis based on the statements of the people at the festival. We loved both and wanted them to coalesce, but we were concerned they might be oil and water.

To find out, we began prototyping. We made a quadraphonic phase-music vocoder sampler in hopes of making spoken phrases musical; a computer-vision Post-it® Note step sequencer to bring musical life to an unintimidating object; and generative word games to try to tease out a crowd-sourced narrative arc of the festival. But nothing felt quite right, so we went back to the drawing board and asked ourselves, what is it that we want people to feel?

A Step Back

To get more perspective, we looked for reference points. We found inspiration in large scale installations from artists like Zimoun, where a simple idea like the sound of a cotton ball hitting a cardboard box took on new effect and meaning when scaled. It prompted us to wonder, what if we scaled the acoustic sound of balloons? What are the sounds the balloons should make, where should the sound come from, and how might that capture or express the narrative of the festival?

Iteration

Quickly and tragically, we discovered that combining a bunch of large balloons with an industrial fan was not quite as mind-blowing as we had expected. But, still, their innocuous beauty made balloons seem like a magical invitation to play.

We sought out people who had worked on similar projects, and who were not steeped in our brainstorming assumptions. We talked to Jack Gale from CCA, who had created an interactive balloon project, invited a group from San Francisco's Exploratorium to brainstorm and test out a rough prototype of balloon word games, and worked to find balloons that were big enough and robust enough for us to embed light elements and sensors in them.

Flexibility

Originally, we had based our design around an 8-foot by 16-foot box. Instead, we chose a covered outdoor basketball court. Huge! Central! Beautiful!

But the scale was drastically larger and the lights we had planned to embed couldn’t compete with daylight. To make it work, we scaled the beach ball diameter from 2.5 feet to 8 feet to better fit the space (we had already swapped the balloons for beach balls, since the materials are more robust), eliminated light elements, and put the emphasis on sound derived from beach ball interaction. Since we were at a music festival, it was poetic for sound to be the primary feedback mechanism.

Our friends at the Exploratorium invited us to do a trial install at their museum, which quickly brought the interaction questions to the forefront. You learn a lot quickly when the public kicks the tires. When people walked up to the beach balls, their first instinct was to hit them. But we didn’t want the beach balls to just work like a button—we wanted the installation to be more nuanced, and produce different sounds based on where and how each ball was hit. To do it, we used the nine degree of freedom inertial motion unit to map directional acceleration to pitch, which worked as though there was a keyboard wrapped around each beach ball.

How it works

The Beach Ball Sensor

Each beach ball has a sensor package in a custom made mechanical holder that is held captive by a carabiner. For sensing and communication, we used an Arduino with WiFi, a nine degree of freedom inertial motion sensor that detects acceleration in X, Y, and Z directions, rotation about X, Y, and Z directions, and magnetic field in X, Y, and Z directions. We also opted for an external antenna for added WiFi robustness.

This package communicates raw sensor data via OSC messages over WiFi to a computer that processes the sensor data. A USB battery designed to recharge your smartphone on-the-go powers the hardware and provides enough energy to run each system for about 14 hours—more than enough for an entire day of uninterrupted play.

Signal Processing and Gesture Recognition

On the computer, Max processes the streaming sensor data, recognizes gestures, and outputs a MIDI note with velocity whenever it detects a beach ball impact gesture. Ableton Live takes the MIDI data from all the beach balls, filters the notes to work with the current harmony, and triggers instrument sound.

Feedback

We wanted to be certain that participants could hear how they were changing the music, so while the background music played evenly throughout the installation, each individual beach ball generated a sound in the speaker closest to it—kind of like stage monitors that bands use playing live. That feedback encouraged everyone to play more.

Collaborative Music Design

To set the stage for musical confidence, we wrote algorithmic music that acted as a backing track, so no one felt the pressure to play the first note. Depending on where a beach ball was hit, one of 15 possible chromatic notes was generated. So, we filtered the beach ball notes so that they always fit consonantly with the backing track—you couldn’t play a “wrong” note. Even still, six different instruments playing over background texture can be too dense. So we opted for a section approach—three sets of two instruments. A trio with backing felt like the right balance of individual voice without cacophony.

Rigging

When we got to the venue, we had to invent a rigging system to get the beach balls to hang at the right place and height. In a divine moment of MacGyver-like inspiration, Diem Ho devised a hierarchical beach ball wiring system that could be lowered and raised quickly with precise control. When we were swarmed with little kids, this system allowed us to raise the beach balls just enough so people could see little kids behind a beach ball and avoid hitting them; things became a lot safer.

Improvisation

The most important design practice throughout the entire process was to stay nimble and improvise. We repurposed problems into constraints that catalyzed our evolving platform. In a sea of unknowns, improvisation allows you to find generative ways to translate even the simplest knowns into a compass toward meaning, and have fun doing it.

Lead engineer and composer Aaron Soloway couldn’t help but smile as he watched festival attendees interact with the balls. “Whether it was a huge bearded dude into modular synths, or a 5-year-old who just wanted to hit things, everyone was into it.”

For Hyer, that universal spirit of play was key. “It created an opportunity for people to connect with music that was unexpected and in the spirit of the festival,” he said. “Ultimately, it’s about celebrating the convergence of art and engineering. That’s what that Bob Moog was all about.”

Moog hit the big-time when his company started producing portable versions of his giant synthesizers (aka the Minimoog). For now however, the beach ball synth will remain enormous.

‍