Back to selection
Worlds Within Screens
Indiana University-Purdue University Indianapolis's LED facility (photo by Mike Lulgjurai, IU School of Informatics and Computing, IUPUI) In the late 1990s and early aughts, film schools moved away from film itself as digital cameras (and editing) became the main tools. What’s happening today may not be quite as seismic but will still change film schools’ DNA: the movie and TV industry is moving toward virtual production. Popularized by The Mandalorian, virtual production essentially takes green/blue screen to the next level, and in some ways, it reverses traditional workflows. Instead of cast and crew finishing principal photography and then handing it off to an army of VFX techies, the techies create that VFX before anyone steps on set. This is not a fad. “Remember when 3D came in and everyone said this is the real thing? I was like, ‘No, it’s not,’” remembers Rosanne Limoncelli, who oversees New York University’s first virtual production class. “‘If it didn’t work in the ’50s, it’s not going to work now.’ But VP is a whole different ballgame.”
To make The Mandalorian, video game designers created immersive 3D backgrounds, which appeared on a massive wall of LED screens in a shooting space known as a volume. If that still sounds like green screen, it’s not: with green screen—long used by the likes of Marvel—the backdrop is composited in later. With VP, the background is visible to the performers and, like the screen of a first-person game, can move, with the camera moving in tandem with it. The result is the illusion that actors are inside fantastical (or mundane) photorealistic worlds. (Green screen’s not going away—it can still be used with VP to add VFX after principal photography, and it’s also much, much cheaper.)
“It’s really just shifting all of the CG work from post- to pre-production,” says Max Thomas, who teaches game design, including how it applies to VP, at Georgia State University. “I think directors who accept this process will really like it because it puts the control back in their hands, and they have an example of exactly what they want the film to look like before they even have to roll the first cameras.”
VP also makes the actors’ jobs easier. “If we have a live-action actor, or even someone in a motion capture suit, they can respond directly to what they see on the LED wall,” says Olaiya Gardner, who also teaches at GSU. “If you have, let’s say, a bullet train going by, you can see it coming and move your body to react to that. With green screen, you can’t see that.”
Bullet Train, the forthcoming Brad Pitt action movie, is one of a number of productions that have adopted, in part or whole, VP. (Others include Star Trek: Discovery, Thor: Love and Thunder and other Star Wars shows, like Obi-Wan Kenobi.) Not only is it cost-effective, but it’s made production easier and safer during the pandemic. Right now, universities are in the early days of adapting to this new (and expensive) technology, with some drawing from both their film and game design departments—divisions that, historically, had only minimal overlap. “I look at it as not necessarily a mesh between the film department and the game design department, more so as the game design department taking a cinematic turn into the industry,” says James Clayton Martin, who teaches VFX and motion capture at GSU.
In 2021, NYU debuted “Introduction to Virtual Production,” a graduate-level course that’s also open to experienced undergrads (i.e., not first years). Students spend the first half of the semester learning Unreal Engine, the 3D game design engine used by The Mandalorian. Eventually, they break up into small groups and collaborate on two-minute-long final projects. They can either download available assets to create their worlds or create them themselves, then integrate them with a live actor.
NYU doesn’t have a volume on campus. They will starting next summer, when the recently announced Martin Scorsese Institute of Global Cinematic Arts, boasting an LED wall, opens. For now, students remotely access a volume in California. That LED wall is nowhere near the size of the one for The Mandalorian, which eats up a huge wall as well as the ceiling—this one’s about 10-by-8 feet. Students have access to PTZ, robotic “pan-tilt-zoom” cameras, which require no physical—or in-person—contact. They can do everything via Zoom, even as the school increasingly returns to in-person learning. Because it’s still tricky to audition talent mid-pandemic, their actor is just the guy who runs the California studio.
The pandemic sped up NYU’s shift to VP, and the same happened at Indiana University-Purdue University Indianapolis (IUPUI) that fall. Zebulun M. Wood, co-director of the school’s Media Arts and Science program, says he and his colleagues had a lot of time to think about how to not only piece together what became a six-course certificate program, but also how to afford technology that he describes as “pretty prohibitive.” (One retailer sells a 25-by- 10-feet wall for about $35,000.)
One solution was to see what the university already had. For instance, there was an LED wall in a building for engineering and science labs. Last spring, Wood had five students on-site make a virtual production commercial for the school’s virtual production capabilities. They used Unreal to create a cityscape and a medieval realm with a dragon. The result was essentially a proof-of-concept to attract students to the school while teaching the students who made it the tech they could use to get jobs in a rapidly changing industry.
The LED wall at IUPUI is fairly modest, about 10-by-10 feet. The one built at Full Sail University is not. In March, the Florida school opened a $3 million VP studio, which includes an LED wall that’s 40-by-16 feet. They then did something unusual: they opened it up to a professional production, namely 9 Windows, a modernization of Rear Window featuring William Forsythe and Michael Paré. The film was mostly set either in a house or a basement, making it an ideal maiden voyage for Full Sail’s new volume—and a reminder that VP can be used for more than sci-fi and fantasy. Full Sail has already leased out its giant LED wall to other productions and has been taking meetings with producers. That way, it can make money that can go back into the school (and pay for expensive equipment) while allowing students to learn VP alongside real crews.
The LED wall installed at Georgia State University in Atlanta isn’t nearly as wide as the one at Full Sail, about 25-by-15 feet. But GSU has something else: a huge, ambitious MFA program devoted to VP, which launches in the fall, combining film and game design courses. The school is also right in Atlanta—as industry types call it, the Hollywood of the South. When he’s not teaching, Thomas works on major productions, like The Suicide Squad and the forthcoming Lyle, Lyle, Crocodile, which have embraced VP techniques, and says it’s a “total pipeline” from GSU to Atlanta productions, which students visit while in class. “We take students on set with us and put them to work,” he says. “They get hands-on onset. They’re there from sun-up to sundown, six days a week, sometimes. It’s straight into the fire.”
One reason these educators are confident that the industry will be moving, at least partially, toward VP is that it’s a great expense that winds up saving money. (The same goes for schools that adopt it.) Small teams can be sent to scan environments, which are then recreated inside a set space. With VP, studios “don’t have to send 600 people to New Zealand or anything like that,” says Thomas. “Your location comes to you.”
VP also makes sets safer. “When a student wants to go film on a rooftop location, it’s very difficult: the insurance, the risk, something—God forbid—happening,” says Sang-Jin Bae, who teaches the VP class at NYU. But with VP, dangerous work can be done safely, on a giant set. “We can easily fabricate a rooftop in Unreal or take photographs and project it onto a 3D space. Then, they’re just on an LED wall. They’re not up on a rooftop. Space becomes accessible to everyone.”
Bae calls VP a “great equalizer,” pointing out that Epic, the video game company that first launched Unreal in 1998,\makes it free to download, allowing anyone, even non-students, to get their feet wet. “Any time this type of technology comes out, right away it makes science fiction, fantasy and period pieces accessible to the filmmaker who doesn’t have the budget to do it.”
It’s also attracted people outside of either film or TV. At NYU, musical theater students have signed up for the class. “Their specialty is writing music, and all of a sudden they can make a previs[ual] for the soundtrack,” Bae says. “It helps them visualize what the story they’re trying to tell with their music. They can put visuals to it very quickly. It gives them creative control without them relying on somebody else to make the visuals for them.”
Limoncelli says VP jobs are “exploding,” and some of those jobs didn’t exist before. She points to the shift from film to digital two decades back, when all of a sudden there were new jobs, like a DIT (Digital Imaging Technician). “Instead of the film loader, you needed a DIT to work with the camera department and the post-production people to download everything and set everything up.”
It’s too early to tell whether VP will completely revolutionize film (and, for that matter, gaming) schools the way the shift from film to digital did, but educators see it as adding more variety. “It’s another tool in the shed,” says Wood. “And the shed is growing. There’[re] more tools, and the tools are getting cheaper.” More and more schools are preparing for VP classes and degrees, including University of Southern California, Florida State University and L.A. Film School. The virtual production initiative at Chapman University in Orange, California, has gone so well that they’re about to add a second LED wall.
“It just made sense for us to move alongside the industry so we know our students will be able to get jobs,” says Gardner about GSU. “We’ve already had quite a few students get jobs utilizing Unreal because of our training.” Universities can’t ignore the jobs that are opening up: “We just want to stay in line with the studios, march parallel to them.”
