How We Did It: Prototyping in Virtual Reality

Part of Project Literacy, a campaign founded and convened by Pearson, the installation is intended for large, influential gatherings like the General Assembly of the United Nations, the World Economic Forum annual meeting in Davos, and the Social Good Summit, so the concept needed to scale to different physical contexts. But because the goal was to reach a much wider audience, there had to be a way to distribute the work easily through digital channels.

IDEO had just eight weeks to research the topic, develop ideas, test them with users, and build both physical and digital prototypes. Along with Toronto-based content studio Secret Location, who executed the concept, IDEO and Pearson launched a stunning virtual reality installation at Davos in January. In conversation with Ed White, designers Martin Brown, Kate Burn, and Jure Martinec reveal the backstory of prototyping with the new technology, including software and storytelling tips for others beginning to work in the medium.

What inspired you to use VR?

Jure: It was clear that we would have to find an engaging way of stepping into the shoes of those challenged by illiteracy. It wasn’t just about showing some confusing scrambled text on a website, which is forgettable in seconds. The problem is often marginalized, so we had to do more than just show it. We had to make people really feel it. VR seemed like a natural choice.

Martin: Chris Milk and Spike Jonze have done some great stuff with their VR platform Vrse, particularly “Clouds Over Sidra,” which is the story of a girl living her life in a Syrian refugee camp. The feeling that you’re actually, physically there in the camp with her is intense. Chris Milk calls VR “the ultimate empathy machine,” and he’s right. No amount of statistics or explaining can substitute for the visceral empathy that VR is capable of creating. Though the medium is still very much in the experimental phase—a bit like the early days of film—we felt there was a very compelling opportunity to use it to create empathy for a situation like illiteracy, which many people find impossible to imagine.

What other avenues did you explore to express this story?

Martin: We looked at a lot of different possibilities. We really wanted people to experience illiteracy from the inside, so we thought about immersive theatre, in the vein of Punchdrunk’s Sleep No More, or You Me Bum Bum Train, or even physically immersive games or puzzles that the audience would need to solve, similar to Hinthunt. We really felt that we needed a physical aspect to the experience, because no matter how good a digital-only solution might be, the age of truly viral content seems to be over, and without a huge marketing push, it would be incredibly hard to cut through all the cat videos on the Internet.

Why was VR a good choice for the presentation and distribution needs of this project?

Martin: Because VR is so new, it really suits an exhibition context where users can experience it for the first time in a guided and controlled setting. At the same time, the availability of Google VR, Samsung Gear VR and even YouTube VR meant that it was also something that could spread beyond the initial exhibition and be experienced anywhere.

What was your process for prototyping in VR?

Jure: We tried many different experiences, from simple gimmicky Cardboard apps to quite strong Gear VR narratives and cutting-edge Oculus Rift examples, also using extensions such as Leap Motion. I spent a lot of time tripping over things and waving my hands around the office. Once we had a somewhat good understanding of the dos and don’ts of the medium, we made the first script, recorded some sound on a laptop, and played it over Google StreetView’s VR view to see what a first person, in-the-scene narrative feels like and got some quick feedback. Then we tested interactions to find out how the story would feel if we broke linearity. We took a basic Cardboard example in Unity and tweaked it to show a static scene with interactive hotspots. This let us finalize the script as well as define the assets we needed for final production.

Kate: We invited a few of the people we'd met during our research interviews to help us test, iterate, and develop the prototype. This was potentially very nerve-wracking! We were very conscious of how easy it might be to misrepresent, patronize, or trivialize the issues surrounding illiteracy, and we sought feedback from pre-literate people and our expert adult literacy teachers at a midpoint in the design to ensure we were communicating people's experiences accurately and respectfully.

What hardware, software, and coding languages did you use, and why?

Jure: The scene had two major requirements: It needed to feel realistic and the hotspots had to be in optimal positions. So although live-action video would have been great, it would be hard to iterate on it quickly. Instead, we went for a fully custom photorealistic 3D render to set the direction for the feeling and interactions of the story. We hit a bit of a wall with Unity, but luckily there was an Android developer floating around. Since at that point it was clear what we needed to make, we wrote a pretty dirty Cardboard/GearVR app directly in Java. Just to put it in context, I think it took us around a week and a half for all of this.

What were the technical challenges that came with designing in VR?

Jure: The medium is fairly new, so unlike when you design apps or websites, there are few approachable tools that help you prototype and learn quickly. None of us had really worked with VR before, but that was just another excuse to go for it. Luckily there is a lot of talk online about how to design for the medium, and pioneering companies like Google are sharing tools like Cardboard Design Lab. Their Cardboard SDK is also quite accessible, so it was easy to get started. If you’re familiar with game engines like Unity, you can quickly create advanced experiences for many different platforms, including the Oculus Rift. As WebVR is being pushed, frameworks are starting to pop up and many web developers are suddenly in the VR world as well.

What were the storytelling considerations you faced in VR?

Martin: Everything is so new that there is no rulebook. Google, among others, is starting to codify what the language of VR filmmaking should be, but no one really knows. For example, who or what is the viewpoint you’re looking from? Is it invisible, like in film? Possibly, but that somehow feels weird, like you’re a voyeur. We intuitively felt like you had to be experiencing the scene from the first person.

What are your technical tips for coders prototyping in VR?

Jure: Be smart and inventive. Yes, you can make very complex things, but it's really fun to think about how you can learn with smart interactions and storytelling. Also, be mindful of the platform you're designing for. A Google Cardboard app will be much less immersive than an Oculus Rift app, but on the other hand, it will be much more accessible to people. Think of who you're designing for and why.

What about filmmaking tips for budding VR directors and videographers?

Martin: Prototype and test everything. Traditional filmmaking took decades to figure out the basic tropes. I think that as VR filmmaking evolves, things will become standardized, but it’s the Wild West at the moment. For now, simplicity wins. The audience is so wowed just to be there, inside a movie, that you don’t need to resort to a lot of tricks to keep people entertained.

What do you see as the most exciting potential for the technology moving forward?

Jure: VR is becoming big. Technical standards are being established, making development and prototyping tools more powerful and accessible for developers. As platforms like Google Cardboard and Oculus grow, their distribution channels are lowering the barrier of adoption for developers and viewers. The medium is maturing, so the content is becoming much more engaging, with rich storytelling, which almost forces big production companies to move into the field. Things are definitely happening soon.

Martin: There’s that story about the Lumière brothers in 1895, screening a film of a train arriving at a station. People in the theatre, who had never seen moving pictures before, ran away because they thought they’d get hit. That’s where we’re at with VR. It’s raw and unsophisticated but still incredibly visceral, and because of that, it’s powerful. I think it has the capacity to be the dominant art form of the 21st century—like film was for the 20th. I really think it will be that big. We just might need a few decades to figure out how to make it work best.