The Dead Sea is one of the unknown casualties of the turbulent politics of the Middle East. Population growth since the founding of Israel has diverted much of its source water for human use. Mineral extraction companies have reduced it even further, and of course global warming is continually increasing temperatures and making the region more arid. Since the watershed basin is shared between Israel and Jordan it requires international cooperation to address, and though there have been attempts to do so they have not matched the challenge that the Sea is facing. The result is that the Sea shrinks by over three feet per year, leaving behind a barren landscape riddled with enormous sinkholes that make human activity dangerous. Large swaths of land that were once crowded beaches are now cordoned off for safety.

Documentary photographer Adi Lavy spent her childhood visiting the Sea and now as an adult documentary photographer has been chronicaling its decline. When she discovered virtual reality five years ago, however, she knew it would be the ideal medium to recreate this vanishing place. After several years of production she premiered the VR documentary film Once Upon a Sea at the Venice Film Festival last year, and it’s now showing as part of SXSW’s Virtual Cinema lineup. The film alternates between what are usually distinct forms of VR filmmaking, such as traditional 360-degree documentary footage, interactive spaces with six degrees of freedom (6DOF) where viewers can move around and interact with elements captured through photogrammetry, a process similar to scanning all of an object’s surfaces to digitally recreate it in three dimensions. The resulting film is more of a portrait than a political polemic, and is permeated with a melancholy nostalgia for what is already lost. Lavy’s goal, however, is that this three-dimensional recreation can convey to future audiences how the Dead Sea once was, and it’s possible, of course, that this emotional approach could spur political action now to save it before it is entirely gone.

Filmmaker: You grew up on the shores of the Dead Sea. On a personal level, how does it compare now to the memories that you have from then?

Lavy: I grew up in Jerusalem, which has no sea, so the Dead Sea is considered the sea of Jerusalemites and my mother and I spent a lot of time there. My father and sisters wouldn’t come near the Sea, it was too salty and hot for them.

Today, going to the Dead Sea is heartbreaking. Most beaches are forbidden access and everywhere you look you see ruins. It’s very hard to reach the water, the natural beaches, and even when you do, there are no facilities, no sweet water. Most of the places I’ve known since childhood are too dangerous to visit. When I was young, I remember sitting in the back seat of my parents’ car, nose glued to the window, looking out at the big blue of the Dead Sea. Back then the Sea reached the road, and it seemed that if I stretched out my hand, I could touch its salty waters. Today, driving down the same road, the water line has receded many miles, replaced with barren land and sinkholes, sprinkled with many warning signs forbidding access to the Sea.

Filmmaker: Why did you want to make this piece in virtual reality rather than a traditional film or other medium?

Lavy: Each person has a landscape that has been etched in his childhood and continues to be a special place for him, even if only as a memory. The shores of the Dead Sea were that special place for mother and I. The heavy heat, the salty water, the wild nature and the loneliness that prevailed there brought us closer together. But slowly our private paradise changed face. In the past thirty-five years the Sea’s water receded over thirty-five meters due to human intervention and political neglect. Due to the ecological crisis, the Sea receded, huge sinkholes appeared, roads collapsed and the shores of the Sea were abandoned.

For the past decade, since my mother has passed away, I’ve been devoted to documenting the Sea, in hopes of preserving whatever was left, a minute before it was also gone for good. But whatever technology I used, photography, video, audio, I was unhappy with the end results. I wasn’t able to capture the essence of being there. That changed in  2015 when I experienced  Way to Go by Vincent Morisset. It was the first VR experience I tried and it impacted me greatly. After years of failed attempts, a seemingly simple walk in the woods awarded me the creative vision to recreate the place I loved. Now I had to learn what these are and how to use them. Five years later I am glad to share the results of that inspiration. A new view of a haven that is gone, but still exists in my memories and here.

For many years I had an ominous feeling when visiting the Sea. From fear of losing my haven, I instinctively began photographing the Sea and those who visited it.  After a few years, once I realized the scope of the disaster and the Sea’s grim future, I felt that photography and video wasn’t enough. I looked for other technologies that were more immersive, that could freeze the place in time and let me visit it time and again. After Way to Go introduced me to the world of VR and its storytelling possibilities I instantly knew I found a way to tell the story of the Sea.

Because we have lost so much access to the Sea throughout the years, and people didn’t know what was really happening on the closed  beaches,  I wanted to create an experience as close to reality as possible and expose the magnitude, beauty, and horror of this phenomenon.  We used photogrammetry to create most of the 3D assets. With places that were already ruined, like Shimon’s garden, we used old photos and recreated his  once lush garden in CGI.

Filmmaker: Tell me a bit about the writing process, especially as you planned out how to combine the nonfiction footage, the photogrammetry, printed text, and spoken narrative.

Lavy: For a long time I looked for characters that had a special connection to the Sea, ones that were  dedicated to it, that their lives were dependent on the Sea, that they had a symbiotic relationship with the Sea, similiar to the relationship I have with the Sea. From the get go, I wanted the Sea to have a voice. The more the Sea receded and the more sinkholes appeared, the more I felt that this was the Sea’s way of trying to push man away, saying ‘you’ve done enough damage,’ keep away. I wanted to make sure that the Sea is heard, that its grand history history will be center stage. I tried many times to write the voice of the Sea, but I never quite got it right. I was very fortunate that the writer Heidi Miller came onboard and did such a wonderful job with capturing the essence of the Sea, the mood, its character and its rich history. Heidi wrote the story very poetically, yet was able to implement geological and scientific information about the ecological crisis. The Sea’s VO really helped combine all these elements together and set a poetic yet fierce tone to the experience.

Filmmaker: You mentioned the photogrammetry. Can you talk about the combination of that with the 360-degree live action footage? Not just the animation, but how did the interactive elements enhance your story?

Lavy: The basis of the project, from the get go, was photo based. As a trained photographer, it was important for me to have the closest to reality and highest quality environment. That is why we chose to use photogrammetry to create our worlds. The high resolution and the sharpness of the DSLR lens gives us a feeling of presence that isn’t there in the CGI or drone-based scans. Using photogrammetry meant putting our lives at risk, because we stepped into actual sinkholes to document it. The fact that the environments are photo/reality based enhances the users experience and really makes the user feel as if he’s actually in the Dead Sea visiting places that are inaccessible and dangerous, and today the only way to visit or access them is in VR.

We tried using real life objects and scan them into 3D assets in the project for a few reasons. As a documentary story, we actually had these objects and it added credibility to the story. For example, the letters Shimon wrote on toilet paper while he was stranded in the sinkhole. These are old letters that Eli keeps in a special box, since 1997. they are almost falling apart, but there is so much added value in holding a scan of the actual letter, seeing Eli’s handwriting, the smudged ink, the torn paper, it gives another layer of authenticity. Also, it was a way to bring together the visual language of the 360-degree films closer to the 6DOF environments. Another element that was important for us to scan was the lemon tree in Shimon’s garden. By the time we had enough budget to go on production, Shimon had to leave his home and his garden had already wilted.

When we arrived there, everything was wilted, grey, and the trees had already started to fall. The one exception was the lemon tree. It had dried so beautifully, with the lemon intact, and they hadn’t fallen down. All the lemons stayed on the tree, dried, but beautiful and yellow. The beauty within destruction. It’s a concept that followed us all along through the project. The destruction all over the coast is so massive, absolutely terrifying to witness. But nature is so majestic and powerful that it’s breathtaking and beautiful, even in its decline.

360 technology and 6DOF are very far apart tech wise. It’s not a natural match and most professionals we consulted with told us it’s a bad idea and that it won’t work. But the footage was very precious. We started shooting in 360 way back in 2015 and a lot of the places we shot, like Shimon’s garden, were gone. We didn’t want to give up this footage, it was very valuable, so we tried to match visual elements so it won’t look so foreign. 

Filmmaker: What do you hope to see in the near future of nonfiction VR, AR, and similar media? How are VR’s immersivity and interactivity advancing the documentary form?

Lavy: We are just at the beginning of the possibilities immersive media has to offer us. Technology is advancing so quickly and creators have so much to learn in order to create spaces for our memories to live in. Growing up my dream was to have my own personal Star Trek Holodeck device. Well, Once Upon a Sea is my first attempt at creating that: a visual space for my memories to live in, an alternative space that froze in time, that I can visit anytime. I hope that in the future I will be able to implement AI in these worlds, these digital spaces, so they can keep growing and evolving, not only forward to the future but also back in time. Teaching the machine to also take us back in time, so we can fix all the damage that we did, at least virtually. We are very excited to see what will come next.

VR pushes the documentary form further, giving the user unique access to components of a story that they wouldn’t have otherwise in a “flat” documentary film. For a project about the Dead Sea, this made sense as the experience of falling into a sinkhole, getting close to the Sea, is something quite unique. Not every story or subject lends itself to VR/AR or interactive media, so as creators and producers we need to choose very carefully which stories we want to tell this way.