Back to selection
Among the Ashes: Iterating the Generative Documentary Where There’s Smoke
Photo by Phil Bradshaw I grew up in a firefighting culture full of pancake breakfasts, fire parades and beef and beers. For 20 years my dad was a volunteer firefighter and amateur fire scene photographer. He shot thousands of 35mm slides of blazes, often capturing moments of destruction that are disturbing and yet at times hauntingly beautiful.
However, my dad’s obsession with fire would eventually intersect with our lives in devastating ways. In the 1980s, while on a family vacation, our van erupted in flames. 11 months later, our house burnt to the ground. For over 30 years, I’ve wondered if those two fires were accidents or something more. At a certain point I resigned myself to accepting that I might never know the truth.
When my dad was diagnosed with stage four colon cancer, my wife and I became his primary caregivers, navigating a bureaucratic and often un-empathetic healthcare system while aiding my mom as she struggled with her own health limitations. As my dad was nearing the end of his battle with cancer, he asked if I would interview him. He felt the need to document his life and invited me to ask anything; in return, he would be candid in his responses. In the six months leading up to his death, we recorded 15 hours worth of interviews. As his body was failing, my courage to seek the truth about the fires was growing. Over the course of our discussions, skeletons began to emerge from the closet, and I discovered family secrets that shook my understanding of my dad and his upbringing.
The series of interviews eventually became the foundation for my immersive storytelling project, Where There’s Smoke. While I could have developed it as a traditional feature-length doc or even a limited series, since 2016 I’ve been experimenting with this story’s form and function in several ongoing emerging media adaptations. The interviews with my dad are the foundational element carrying across each version as the work has gone from a live performance onstage to a physical installation within a storefront to a virtual browser-based experience to, this past year, a generative, self-guided gallery exhibition using new tools that didn’t exist at the project’s launch.
Describing the overall work, I often find myself stating all the things that it is not. For instance, Where There’s Smoke is like a movie, but it is not experienced linearly. There are moments that feel like immersive theater, but the project has no actors. The setting resembles an escape room, but there is no “win scenario”—in other words, no escape.
Every time I develop a new adaptation of the work, I step through a prototyping process that is rooted within human experience and designed to evoke a certain aesthetic. Since my work is a form of spatial storytelling, I’ll carefully consider what someone might be feeling and thinking as they are physically moving through the various beats of the story. These early prototypes are fast and often make use of analog assets (paper/pen) or quickly coded mockups (web apps). Because the work is participatory, it is critical to engage an audience as early in the design process as possible. That’s why I consider each stage of the work to be iterative, meaning that festival showings are often live audience research and development opportunities.
A Pop-up Installation on Canal Street
After three years of work, Where There’s Smoke premiered at the Tribeca Film Festival in 2019. Over the course of 45 minutes, four guests investigated a 1,400-square foot set of a burned-out house. Within the charred remains they discovered a rotary phone, camera, cassette tape and walkie talkie, each an Internet of Things-enabled object that could alter the lighting, sound and score and determine which stories guests hear. For example, when one of the objects was picked up by a guest, a light table in the center of the room came to life. Its four light panels began to illuminate, and a nearby 35mm slide projector turned on. By placing the rotary phone, camera, cassette tape and walkie talkie in different positions on the table, guests unlocked a series of story fragments that played as slide shows with audio accompaniments. As a result, each showing of Where There’s Smoke was different, as guests’ interactions with the objects determined when and how many stories were unlocked and the order in which they played.
During the festival run of Where There’s Smoke, a new side of the project began to emerge. After my dad died, my mom stopped talking about him. She was struggling with the loss and was suffering from depression. I had invited my mother to the Tribeca installation, but she wasn’t interested. Then, two days prior to us closing the run, my mom called and asked if she could see the project. My mom, my wife, my son and I all went through the experience together. Afterward, my mom began to talk about my dad and continued to do so up until her death in 2022. The installation was transformative for her, and I started to wonder how it might be beneficial to others.
Through my work at Columbia University School of the Arts, I had done a number of collaborations between its Digital Storytelling Lab, which I oversee, and the Narrative Medicine program within the Medical School. I invited Dr. Rita Charon, a colleague at Columbia, to experience the Tribeca run. Dr. Charon helped to originate the field of Narrative Medicine, and she’s been a supporter of the work of the Digital Storytelling Lab for years. In her 2001 article “Narrative Medicine: A Model for Empathy, Reflection, Profession, and Trust,” originally published in the Journal of the American Medical Association, Dr. Charon writes, “The effective practice of medicine requires narrative competence—that is, the ability to acknowledge, absorb, interpret, and act on the stories and plights of others. Medicine practiced with narrative competence, called narrative medicine, is proposed as a model for humane and effective medical practice… With narrative competence, physicians can reach and join their patients in illness, recognize their own personal journeys through medicine, acknowledge kinship with and duties toward other health care professionals, and inaugurate consequential discourse with the public about health care.” Soon after Tribeca, Dr. Charon introduced me to Dr. Deborah Starr, a lecturer in the Narrative Medicine program and an end-of-life specialist. She encouraged Dr. Starr and I to start collaborating to see how narrative medicine protocols could help to inform the design of digital storytelling experiences like Where There’s Smoke.
Lessons learned from going virtual during a pandemic
The installation at the Tribeca Film Festival took two weeks to build and ran for 21 days, during which more than 500 people experienced it. The overall budget to design, produce, install and run the project was over $60,000, and these funds were privately raised with the hope that the project would be able to travel.
But when the pandemic halted Where There’s Smoke’s touring plans, I pivoted to developing a browser-based adaptation of the piece. I began to experiment with Miro, a whiteboarding/design tool for team work, and Zoom, which had grown in popularity during the lockdown. Through trial and error I subverted productivity tools and converted them into collaborative storytelling and sense-making tools. What emerged from the experimentation was a 90-minute immersive experience that scaled to an audience of 80 people. The virtual version consisted of guests having editorial access to Miro. Within the experience, guests were placed within Zoom breakout rooms. Together they helped each other to navigate Miro’s infinite canvas, along the way interacting with a series of layered digital collages consisting of text, images, audio and video assets. By design there is too much to investigate within the Miro board and not enough time. As a result, the guests have different journeys and through discussion with others begin to discover more of the story.
In the spring of 2020 I began to stage free weekly shows of Where There’s Smoke in collaboration with Dr. Deborah Starr. In a run that was extended for five months, showings of Where There’s Smoke would end with a special session moderated by Dr. Starr. This virtual version of the project went on to have special showings at IDFA DocLab and the Portland International Film Festival, in addition to being presented at various medical conferences around the world.
The combination of narrative medicine protocols and digital storytelling yielded some interesting results. A common response from viewers was the desire to interview a loved one. For example, a gentleman attended the virtual version of Where There’s Smoke and soon after travelled over eight hours to visit his father from whom he was estranged. Over time we started to notice that people were coming two or three times to experience the work. Viewers connected with each other while experiencing Where There’s Smoke and have continued to stay in touch. Based on what we’ve learned, Dr. Starr and I are developing a narrative medicine-focused workshop that uses Where There’s Smoke to encourage others to use digital storytelling to explore their own life, loss and memories.
The virtual browser-based experience had no production budget. Everyone involved volunteered their time, and all the shows that we ran were always open and free. Over a two-year period, we ran the virtual experience of Where There’s Smoke over 100 times and approximately 7,500 people participated.
Finding Space to Incubate
For years I’ve relied upon the festival circuit to incubate my immersive work. Sundance, Tribeca and the New York Film Festival have provided venues for projects combining storytelling, code and audience participation. However, over time these stagings have become less sustainable. Each work tends to be bespoke and require costly resources. There is never enough time to adequately test and build on site. Plus, festivals provide little in the way of support outside of guaranteeing space and audiences.
So, when I was thinking about staging a new adaptation of Where There’s Smoke, I looked for a potential partner able to provide a kind of residency that included access to an exhibition space and necessary resources, an institution that would grant the project the time necessary for it to evolve organically. That’s when I set my sights on the non-profit ArtYard, which describes itself as “an interdisciplinary alternative contemporary art center comprised of an exhibition space, theater, and residency program, dedicated to presenting transformative artwork, fostering unexpected collaborations, and incubating original new work.” Our missions were aligned, and ArtYard was located in Frenchtown, New Jersey—only a few miles from where my dad had been a firefighter.
The exhibition at ArtYard, which opened June 17 and runs through October 1, is the most ambitious adaption of Where There’s Smoke so far. Taking lessons learned from the Tribeca installation and the virtual experience, this newest version is both generative and self-guided. Whereas the previous incarnations have always needed docents to help facilitate the experience, at ArtYard guests are given a flashlight and a pair of headphones that set them on their way.
Within each flashlight is a microcontroller (similar to a tiny computer) and a geotag that helps us triangulate the position of the flashlight at any point within the experience. This custom piece of hardware enables us to mix up to eight channels of audio in real-time directly on each flashlight. The result is a seamless personalized experience where the stories you hear and the score that accompanies them is generative. For instance, if someone stays in a specific location for a period of time, the score will decay and fall apart, similar to how a memory fades. In other instances we have created sound portals. As a guest enters the doorway to a room, the story they’re listening to falls away and becomes an ambient soundscape, but once they walk back through the doorway the story picks up where they left off.
To execute the precision needed to track every user, we built a geo-fence around the gallery using a real-time location system from a company called Eliko. Eliko’s next-generation location tracking solution was designed to track vehicles, goods and people in real-time using a reliable indoor positioning system. We combined Eliko’s solution with a “state machine” that we’ve developed. The state machine keeps track of each guest in real-time, and when combined with Eliko’s tracking solution and the microcontrollers within each flashlight, we are able to gather a variety of data points. For instance, we can determine the exact path a guest takes, what stories they’ve listened to, the density of guests in specific areas of the exhibition and the time a guest spends within the experience overall.
The collected data provides a unique ability to shape what is known as “conditionals,” which are ways to control sensory and storytelling elements, such as lighting and score as well as the order and number of stories that a guest experiences. This means that guests’s interactions within the exhibition can have cause and effect or a trace that is visible or invisible to them depending on how the story is delivered to their headphones. As a result, each guest going through Where There’s Smoke has a personalized experience; no two are ever the same.
The data that we collect through the flashlight is also brought back into the exhibition and used to create generative pieces of art. In the lobby of ArtYard we’ve set up a “plotter” made by a company called AxiDraw. The device is like a drawing robot—imagine a metal arm on a track holding a pen as it receives data from the experience it draws. For instance, the pathway data of guests is collected and used by the AxiDraw to create an A1-film-poster-size line drawing on top of a print of one of my dad’s fire photos. The drawings are then added to the gallery, so with every guest the exhibition continues to grow.
The 2,500 square foot ArtYard adaption has been designed to travel. To date, over 3,500 people have gone through the experience. The overall budget to design, produce, install and run the project was $92,000. The funds come from a commission from ArtYard, a grant from the Solana Foundation and sales of digital artwork.
Through an extended development process, I’ve had the unique ability to “slow think” the project. Taking the time to let it organically grow has been an interesting way to approach emerging technology, which is in a constant state of flux. This past summer I started prototyping a new adaption of Where There’s Smoke, a live theatrical performance mixing storytelling and participatory artmaking. So, in 2024 Where There’s Smoke will continue to evolve as both a touring exhibition and a live show.
