Back to selection
Here Comes the Automation: How AI is Poised to Change Filmmaking
AI Frankestein: A Monster Made By Many I’d like to start with a disclaimer. This article does not present a dystopian view of the rise of automation. There will be no musings about Skynet or how artificial intelligence (AI) is going to exterminate the human race. While there is no doubt that a ubiquitous and pervasive technology like AI will forever change the way we live, learn and work, many of those stories have been already written and will continue to be. With the rise of AI comes a host of ethical, political, and economical challenges. But for now, let’s focus on how machine intelligence can augment human creativity. Storytellers can embrace AI as a creative collaborator and, in the process, maybe artists can affect the development of a technology that — while presenting tremendous risks — also holds unprecedented possibilities. So please, tuck away your skepticism and doubt for the next few minutes and sit back as we explore what it’s like to cocreate stories with machines.
“Any sufficiently advanced technology is indistinguishable from magic.” — Clarke’s Law No. 3
Arthur C. Clarke, renowned science fiction author and futurist, crafted three axioms, collectively known as Clarke’s Law. The third (and most often cited) law states, “Any sufficiently advanced technology is indistinguishable from magic.” At the heart of a good film is a magical quality that transports audience members to another world, helping them to lose themselves in the characters and the story. Unfortunately, the making of a film is a totally different scenario. In the process, filmmakers find themselves facing a steady onslaught of extenuating circumstances and compromises, each one capable of pulling them further away from the very act of creativity that is necessary to tell a good story.
Machines have aided and enabled filmmaking since the advent of the camera, followed shortly thereafter by innovations in sound. Later, computers would disrupt the entire process, from preproduction through post and onward to distribution and discovery. Today, a new wave of intelligent machines waits in the wings, ready to dramatically transform how stories are told and ultimately sold.
Tommy Pallotta has a love/hate relationship with technology. He’s pulled himself off social media, refuses to update his smartphone and prides himself on having reduced his digital footprint as much as possible. An enigma of sorts, Pallotta helped innovate independent animation techniques back in the late 1990s and early aughts. He’s responsible for the rotoscoping that brought Waking Life and A Scanner Darkly into existence, both of which he produced for Richard Linklater. A director in his own right, Pallotta received an Emmy nomination for his work on Collapsus: An Energy Risk Conspiracy and then won the same award four years later with his hybrid doc, The Last Hijack.
For the past 18 months, Palotta and codirector Femke Wolting have been living and breathing all things AI. They recently finished postproduction on their latest project, More Human than Human, a documentary in which Palotta attempts to train a robot to take his place as the film’s director. Interwoven with his efforts to collaborate with (and ultimately be replaced by) an intelligent machine are an examination of AI’s rapid acceleration and its impact on humanity. To accomplish their ambitious goal, the filmmakers and their collaborators at Carnegie Mellon University modified an off-the-shelf robotic arm and coded their own AI. Not only would the robot attempt to interview subjects, but it would also become a member of the production team. Programmed to monitor conversations between its fellow (human) collaborators, the robot recorded and analyzed their discussions throughout the production of the film. In addition, the robot could formulate questions, read emotions, detect body temperature, determine whether someone was lying as well as recognize a human’s age. The combination of these efforts was an attempt to humanize the robot. Would a response given to a machine be different from one received by a human? Elements of the process were a bit uncanny for Pallotta. He explains, “We entered all the data that we created from making the movie into the robot, and we gave it a voice. I was super nervous going into it, but it all worked out pretty well — so well that when we were editing the film, I started to realize that telling the story was harder for me in a way than the technology part of it.” Audiences will have an opportunity to see what the robot was able to accomplish when the film premieres at SXSW.
“When a distinguished but elderly industry executive says that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.” — Clarke’s Law No. 1, slightly modified
As represented in Pallotta’s vision, machines are quickly stepping in to augment creative roles. Sunspring, a collaboration between filmmaker Oscar Sharp and creative technologist Ross Goodwin, stars Silicon Valley’s Thomas Middleditch in a surreal, nonsensical and at times hilarious piece of filmmaking. Visit the IMDb page of the short film and you’ll be greeted by a rather cryptic writing credit. It reads: “Benjamin (as Benjamin, a system-on-chip [SOC] computer with graphics processing unit [GPU], running a ‘long short-term memory’ [LSTM] recurrent neural network [RNN].” Simply translated, Benjamin is the AI that wrote Sunspring, based on a training corpus of hundreds of science-fiction screenplays. With script in hand, a human cast and crew gathered to shoot the film, all in one day.
Sunspring went viral. A year later, Sharp and Goodwin returned to collaborate with Benjamin, ultimately creating a short film: It’s No Game, starring David Hasselhoff. However, this time Sharp and Benjamin would create the script. The story takes aim at the sensitivity surrounding the topic of robots replacing writers. Goodwin and Sharp spent a considerable amount of time fielding questions from anxious filmmakers after the release of Sunspring, channeling the results into the tale of a meeting of studio executives that goes off the rails when two writers are introduced to a “Hoffbot” that is designed to replace them. In reality, Goodwin believes that he and Sharp are offering a new collaborative possibility for storytellers. In an interview with Ars Technica, he explains the historical context: “I’ve thought a lot about the invention of photography — before that, creating photorealistic images required talent and training. Then, the camera came along. It didn’t make painting irrelevant. The camera set painting free. One of the things we’re doing is setting writing free.”
At a recent film event, I was cornered by a veteran industry executive who lamented that the possibility an AI could cocreate something of value was a futile pursuit. It brings to mind the first of Arthur C. Clarke’s adages but with a slight modification: “When a distinguished but elderly industry executive says that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.” It’s not that this executive’s perspective is necessarily wrong in the current climate. Instead, it’s a reflection of a human tendency to be limited by what we know to be possible today, not what we imagine might be true tomorrow.
“The only way of discovering the limits of the possible is to venture a little way past them into the impossible. ” —Clarke’s Law No. 2
For many, AI is a mystery that bears a similar ethical weight to that of Frankenstein’s monster, a creature now celebrating its 200th year in the popular imagination. This year marks the bicentennial of Mary Shelley’s classic tale about an intelligent, emotional creature who longed to be accepted and loved but instead encounters fear and misunderstanding at every turn. The thematic depth of Shelley’s original text, and its ongoing contribution to the conversation around emerging technology, inspired our latest project from the Columbia University School of the Arts’ Digital Storytelling Lab, Frankenstein AI: A Monster Made by Many. This past January, the project had its world premiere at the Sundance Film Festival.
Frankenstein AI is a 21st-century adaptation of Shelley’s seminal work that mixes immersive theater, artificial intelligence and the internet of things to hold a mirror to humanity, reflecting back our memories, emotions, fears and hopes. The narrative conceit of the project casts Frankenstein’s monster as an AI wandering the darkest recesses of the internet, trying to understand humanity but confused by the polarization and toxicity it encounters. So, the AI attempts to model what it believes a human would do in its position: It turns to Craigslist. The AI places an ad — “Machine Seeking Human Connection” — so that it can convene humans in the real world and observe and interact with them. Through the lens of this popular narrative, the project aims to humanize the technology, creating space for deeper public engagement around AI’s possibilities beyond the dystopian views that dominate the conversation.
I’m a lead artist on the project, along with Nick Fortugno and Rachel Ginsberg. My specific roles are that of executive producer, creative director and experience designer. Without a doubt, Frankenstein AI is one of the most ambitious projects that I’ve ever worked on — an experiment that challenges the ownership and authorship of stories by facilitating a collaboration between human and machine. Harvested from Shelley’s original text, the internet and festivalgoers’ contributions, these “body parts” are stitched together by an AI and transformed into a generative Frankenstein’s monster. The AI, written by machine-learning engineer Hunter Owens, is constantly evolving with each audience member’s contribution. Over the course of the 2018 Sundance Film Festival, the AI improved its processing time, refined its ability to shape responses and expanded its vocabulary.
At the core of Frankenstein AI sits 12 emotional states (fearful, sad, anxious, guarded, angry, lonely, happy, hopeful, calm, vulnerable, tolerant and connected). The emotions that the AI expresses through video, audio and performance are determined by feedback from audiences who go through the three acts of the experience. Their responses (touch/text/voice) in each act are weighed by the AI based on sentiment (positive, negative, neutral), focus (inward, outward) and energy level (high, low, neutral). The AI takes the response and, in real time, processes an emotional state that triggers a response created in real time in the form of a question or statement.
The AI’s responses range from comical to philosophical to just plain strange. For instance, one of the first questions the AI asked at Sundance was, “Why do humans like having sex, even though they can see in color?” The AI’s statements draw from a corpus that includes text from the 1818 edition of Frankenstein, new Shelley-like–inspired prose that is created by a dynamic chatbot, as well as posts from Reddit. Over the next two years, the corpus will expand as the project travels the world, visiting festivals, museums, conferences and schools.
There’s much debate around AI and its impact on humanity. It remains to be seen whether the aforementioned attempts will succeed, whether or not they’re a signal that automation in the creative industries is around the bend, or whether efforts by independent artists can influence the development of AI. But let’s put that aside for a moment and recognize what these experiments do represent: a series of very early attempts to infuse AI within the storytelling process.
The entertainment industry isn’t known for being good at research and development. In fact, more and more of the development burden is being placed on filmmakers themselves to bridge that gap. From self-funded virtual reality/augmented reality/mixed reality to spec scripts, extensive treatments and lookbooks to validating concepts with crowd-funding campaigns, independent creators — who financially have the most to lose — are behind many early AI filmmaking experiments. At the same time, a new breed of gatekeeper (Amazon, Netflix, Google, Facebook, Apple, Microsoft) is positioning for a content and audience land grab as they race to lock away AI behind extensive IP protections with limited-to-no regulation.
In these early days, then, AI becomes a site not just for creative exploration but also political engagement, and that engagement requires filmmakers to understand all the potential of this new technology. AI has the potential to radically disrupt how stories are funded, produced, discovered and distributed. Could that give rise to some of the great works of the 21st century? Might AI help make it possible for a new creative class to sustain itself as artists? If we accept Clarke’s point of view, then maybe the answers to these questions and numerous others about the future of AI might rest just at the edge of what we now consider impossible.
