Institute For The Future

Rewriting the Rules of Reality

The future of VR storytelling

[Download article as a PDF]

Standing on the desert surface of Tatooine, you instinctively duck as the Millennium Falcon swoops in for a thunderous and dramatic landing beside you. Through the lenses of your virtual-reality headset, it looks real. That’s because it is, in the sense that it’s the same 3D computer model that appeared in “Star Wars: The Force Awakens.” After you help Han and Chewie complete an urgent repair, R2D2 presents you with a light saber. A squad of Stormtroopers appears on the distant ridge. Not to worry, The Force is strong with you.

The Force is also strong with Industrial Light & Magic’s Experience Lab (ILMxLAB), the supergroup of artists, engineers, sound designers, and storytellers prototyping the future of interactive, immersive cinema. This is their latest demonstration, titled “Trials on Tatooine.” The ILMxLAB is Lucasfilm’s R&D arm, leveraging graphics technology invented for traditional filmmaking and applying it to virtual reality, augmented reality—via a new partnership with Magic Leap—and theme-park attractions. (The latter is no surprise given that the Walt Disney Company acquired Lucasfilm in 2012.)

The ILMxLAB leverages the tools of traditional filmmaking and the principles of game design, but are wedded to neither, or perhaps both. John Gaeta, the group’s executive creative director who is best known for his dazzling special effects work on the “Matrix” trilogy, frames their efforts as first-person immersive storytelling, in which the story itself, not the game mechanics, sucks you in. The goal is to place compelling characters within a strong narrative that generates a natural pull to participate.

Indeed, this is the digital dream of all of the interdisciplinary auteurs who are experimenting with today’s virtual-reality systems.

“We want to make it plausible for storytellers to imagine allowing the audience inside these worlds as if they exist for real, not limited to fantasy,” Gaeta says.

And doing that requires rethinking a century of film grammar.

Here are four of the tensions, techniques, and strange behaviors emerging as pioneering artists, designers, and engineers write the future of VR storytelling.

Too Close for Comfort In a regular movie, a close-up shot focuses the audience’s attention on the actor’s emotional state. An unexpected close-up when you’re wearing VR goggles gives the distinct feeling that you’re invading someone’s personal space.

“[If you are sitting very] close to someone who is nearly about to cry, that is not comfortable,” says Saschka Unseld, co-founder of the Oculus Story Studio. “But if the character sits somewhere back there and is about to cry, you actually have a lot of empathy for him.”

That’s why every possible moment needs to be in service of the story, says Lucasfilm Story Group’s Diana Williams, or at least anticipated by the storyteller.

“Yes, a close-up in VR can be jarring, but if the director has created a good story, you’ll know if that person in your space is a friend or there to kill you,” says Williams who spoke at the recent IFTF Technology Horizons conference, Everything is Media.

Builds not Cuts The creators of virtual reality experiences rarely use the word “cut,” as in “director’s cut,” or “edit,” when referring to a particular version of a story. Why? Because “cut” is a very linear term, implying a singular way that the story was edited.

“Editing as we know it doesn’t really work in VR,” says Optimist Design founder Tino Schaedler who created 3D experiences for musical artist The Weeknd and others. “The viewer creates its own subjective cut by his head movement.”

On the other hand, “build” is a term from software, meaning a compiled version of a program that the user interacts with.

According to Motionographer co-founder JustinCone, this distinction is key to understanding VR storytelling, as “every viewer has a different experience of a project, one shaped by their own curiosity and sense of pacing. The experiences react to the viewer— and vice versa—forming an emotional feedback loop that is radically different than traditional filmmaking.”

Dissolving the Fourth Wall In dramatic television or cinema, it’s rare for an actor to acknowledge the audience directly, known as “breaking the fourth wall.” The phrase comes from theater where a three-walled box set contained the action that the audience would watch through the imaginary wall in front of the stage. In virtual reality, the “fourth wall” is generally not the default state. The action surrounds you, beckoning you to participate. Of course, the storyteller can decide to instill a kind of fourth-wall detachment so that the first-person experience of the story is entirely voyeuristic, or perhaps he or she acts as a puppet-master.

“Some ideas in the sphere of storytelling and gameplay can only be implemented provided the gamer impacts the game world directly from the “outside,” writes Olga Peshé, chief operating officer of educational VR firm Cerevrum. “That said, VR now gives developers another powerful tool to interact with users, and the power to use it lies in the hands of the developers.”

No Disbelief to Suspend In 1817, philosopher Samuel Taylor Coleridge coined the phrase “suspension of disbelief” to describe sacrificing our sense of what’s possible in order to fully appreciate a fictional story. Virtual reality, at its best, can flip that to the point that you are so immersed in the experience that belief in the simulation becomes your default state.

Nearly 20 years ago, media theorist Matthew Lombard explored the physical and psychological effects of this state, sometimes known as “presence.”

“A number of emerging technologies including virtual reality, simulation rides, video conferencing, home theater, and high-definition television are designed to provide media users with an illusion that a mediated experience is not mediated,” Lombard wrote in a technical paper.

Two decades later, we are finally at a point where virtual reality can make us digitally delusional. Presence is ready for prime time, says VR pioneer Chris Milk, creator of the critically acclaimed experiences “Walking New York” and “The Displaced.”

“Our brain is no longer translating an approximation of the story…” Milk says. “Instead of suspending your disbelief, you actually have to remind yourself not to believe.”

Beyond that, to quote The Beatles, nothing is real. 

[Download article as a PDF]