With Oscar season upon us, the nominations were announced and along came the stories of directors and best picture snubs. But this year, the unsung hero is real-time visual effects and the role it plays in giving moviemakers the ability to collaborate across teams for innovative movie-making.

The Wizard Behind the Curtain

Up until recently, traditional, big-budgeted movies like Avengers: Infinity War (nearly $400M production budget) would have a visual effects supervisor on set to give input to directors on what can be accomplished behind the scenes, after filming has ended, to make the movie shine.

Whether that's creating a digital world or a computer-generated character, the magic happened post-filming, with the visual effects teams. 

"Visual effects used to be a highly technical, massive investment that was hard for filmmakers to be fully in control of. It was like a 'magical curtain,' cloaking hundreds of visual effects technicians on the other side of filming. They stood between the director's vision and what ended up on the screen" commented Kevin Baillie, visual effects supervisor for films including two Star Trek movies, two Transformers features and, most recently, Welcome to Marwen.

There was no real-time way of seeing what the end product could look like, so directors and everyone on set would have to take the visual effects supervisor's word for it; that in the end, the visual effects will look like they envisioned.

Today, real-time visual effects has turned that way of making movies on its head. Now entire teams can collaborate on the look of the film to create a more beautiful and visually believable story.

Bringing Teams Together to Build Whole New Worlds

The best way to see the value of real-time data, and its impact on movie-making, is in the beautiful visual effects that went into making Welcome to Marwen.

The film, directed by Robert Zemeckis, is based on the true story of Mark Hogancamp, a man who loses his memory after being physically assaulted to within an inch of his life. Unable to afford therapy and suffering from PTSD, Hogancamp takes recovery into his own hands, building a fictional World War II-era world in his backyard, equipped with live-action dolls as alter egos representing himself, his friends, family, and attackers--all to enact epic battles and recreate memories.

It's a dark, beautiful story that required dazzling visual effects that would captivate audiences, and feel authentic. As Baillie shared, "The challenge is that the story is made up of dolls, in a made-up world. And Zemeckis envisioned a film where audiences could connect to the dolls, and feel part of the fictional world."

To achieve these emotionally-relatable doll performances, Baillie and his team utilized a new way of bringing the director's vision to the screen and collaborating with everyone on set.

To start, actors like Steve Carell and Leslie Mann would squeeze into their tight gray motion capture suits, with reflective balls stuck all over them, to track the body's motion. Filmed in a motion capture stage, the actor typically would have little to interact with and, and in a traditional process, everyone is forced to simply imagine the scene unfolding in the fictional world.

For a more inclusive and collaborative process, Baillie and the team set up monitors on set to go beyond the blank stage and show in real-time, how the scene would look in final production after visual effects would be applied.

Using game technology from compies like Epic Games and Unity, the entire film crew could collaborate on the look of the scene to create a more beautiful and visually believable story.

"It was inspiring to show the director, actors, and everyone on set what the scene will look like. We could adjust lighting, shoot the characters' interactions in new ways, and see in real-time how it would all unfold."

Unlike before, this gave everyone on the physical production team - a group of people who are usually kept out of the visual effects process - a visual representation for how they might want to rethink the filming, as they went.

"Real-time technology allows people with decades worth of non-digital filmmaking experience to contribute immense value to the digital process," said Baillie. "It proved to be the ultimate fuel for spontaneous collaboration."

The result is undeniable--the visual effects of Welcome to Marwen are cutting-edge and stunning. The ability to leverage real-time visual effects undoubtedly shortens the gap between filming and post-production, creating a more collaborative effort and in the end, beautiful stories for the audience.