Issue 37, Winter/Spring 2024

https://doi.org/10.70090/SS24EYVR

Abstract

This paper explores the Once Upon a Time in Palestine XR Documentary as a case study to investigate the potential of virtual and augmented reality technologies in reshaping historical narratives through immersive stories that engage youth with history. The documentary recounts the oral history of pre-1948 Palestine by providing an interactive virtual experience presented on a virtual reality (VR) headset and augmented reality (AR) app. Developed on the Unity3D Gaming Engine, the project employs cutting-edge AI technologies to transform archival photographs into a three-dimensional environment. The storytellers are seamlessly integrated into the XR documentary as 3D volumetric videos to deliver mini-scripted reenactments of oral accounts that were collected from interviews with Palestinian elders. The paper examines various facets of the production process and its outcome while exploring the implications for youth engagement in both the creation process and interaction with the finished project. At the core of this process lies an endeavor to present content with minimal technological mediation that fosters a sense of plausibility through the creation of ‘presence’ for the audience. The case study presents observations and articulates the process undertaken to produce the documentary, which underscores the significance of employing virtual and augmented reality in historical documentaries. Furthermore, it assesses how an interdisciplinary multimodal pedagogical design can facilitate collaborative and interactive projects for undergraduate students, especially when the creators and consumers of the content are the target audience.

Introduction

The Once Upon a Time in Palestine XR Documentary is a multi-phased and non-traditional research project that took five years to complete. Phase one involved training Palestinian youth on the art of interviewing. Phase two was the process of empowering the youth to collect the oral histories of their villages through interviewing village elders. Phases one and two took three years to complete and resulted in 105 interviews with elders, which resulted in well over 60 hours of oral history. This content was used to create the micro-Palestinian village heritage. The third phase, which is examined in this paper, involved the process of creating the XR documentary. This phase took a little over two years to complete. The process was undertaken by the primary author and accomplished in collaboration with undergraduate students from three different colleges at the researcher’s university. Industrial design students from the College of Architecture, Art, and Design provided the animation while computer programming students from the College of Engineering provided the software. Concurrently, the mass communication students from the College of Arts and Science produced the volumetric videos and videography, as well as conducted research and edited video. Students from the three colleges had complementary skills that were essential in producing this complex and technically challenging project. Phase three involved 16 undergraduate students—under the direct supervision of the primary researcher—who worked on the project over the course of two years. The purpose of this case study is three-fold. First, to explain the process of creating XR documentaries. Second, to evaluate the benefits of immersive technologies­—such as virtual reality (VR) and augmented reality (AR)—in retelling oral histories. Third, to demonstrate how interdisciplinary and multimodal pedagogical design can help undergraduate students create collaborative projects that reimagine history.

This case study took place at the American University of Sharjah (AUS), which is in the United Arab Emirates (UAE). The primary author and sole researcher of this project is a full-time faculty member in the Department of Media Communication at AUS. The co-author is a full-time faculty member in the Department of English at AUS. Moreover, 16 undergraduate students from AUS worked on the XR documentary. This research was supported by an AUS Faculty Research Grant.

Literature Review

Technology is an indispensable part of the contemporary human experience and inextricably integrated in modern living. As such, combining reality and virtual reality is not atypical. This is because media technologies shape and are shaped by cultural practices. McLuhan (1964) explains the concept of technological determinism, which explicated the importance of technology in our individual lives. Moreover, media technologies continually influence both human experiences and societal structures. Feiner argues that fusion and interplay between reality and VR/AR will become a fundamental medium for storytellers (2002, 55). Bucher also maintains the significance of VR as an innovative approach to communicate human thoughts and experiences while providing a new platform for creation (2018, 308). Feiner posits this is due to augmented reality systems heighten people’s sensory perceptions (2002). Interestingly, different technologies can be incorporated when creating VR and AR environments. For example, Fleisher and Anlen (2018) explain how Volume, which is a software toolkit, facilitates the reconstruction of archival/historical materials as volumetric renderings. Immersive stories, particularly those that present alternate realities, transform our experiences, and are assisted by technology, appeal to multiple senses. Gröppel-Wegener and Kidd explained that “immersive stories can be created through words, visuals, sounds, kinesthetic experiences, through touch and even taste. Using a different genre allows different ways of working with the material” (2019, 24). One approach to enhance immersive stories is through VR and AR as both can be effectively integrated depending on the creator’s intended purpose.

Using VR and AR in multiple genres is not a novel concept as extensive research has been conducted on the benefits of both technologies within various disciplines as a mode of learning. Fullerton (2008) and Raessens (2006) use the genre of video games—also referred to as docu-games—as an important medium to interpret and understand historical events. Moesgaard et al. (2015), Smith et al. (2019), and Camps-Ortueta et al. (2021) focus on game-based VR and/or AR to enhance learning experiences in museums, as well as to render these active learning spaces that enable knowledge transfer. Shehade and Stylianou-Lambert (2020) also examined the use of VR technology in museums. However, their research surveyed the attitudes of museum professionals as it relates to the increasing use of VR in museums worldwide. Meanwhile, Bartucca (2018) uses a case study of an exhibition in Prague to demonstrate how VR supports inclusivity and accessibility as it pertains to art exhibitions. Pavlik and Bridges (2013) examine AR’s ability to transform digital storytelling in journalism to increase the engagement and involvement of citizens. This research is especially pertinent because traditional mediums, such as news reports, are no longer as captivating nor the common means that people access information. Slater (2018) also discusses VR through examining the benefits for psychology research. These various sources all demonstrate the use and advantages of virtual reality in various academic disciplines and aspects of life.

Paul Debevec suggests that narratives are how we store information about history and learn from it (Bucher 2018, 308). Given the importance of historical knowledge and comprehension, it is interesting to note there are different modes of creating history or what Michael Renov (1993) terms the ‘historical real’. As such, using VR and AR expands the boundaries of how history is shared. AR narratives are created by leveraging both the physical and virtual world (MacIntyre et al. 2001), which may instigate pathos. In other words, they create real feelings and emotions that are like those experienced in real life. This is because VR and AR create a virtual presence:

“Amid all the technology and digital complexity, something real, organic, and meaningful emerges. Something that sparks a user’s imagination and sense of wonder. That which was never before possible suddenly appears before them in quite realistic fashion” (Bucher 2018, 199).

The concept of presence is not novel and hearkens back to the phenomenological theories of presence as discussed by Heidegger (1962/2010) and Merleau-Ponty (1964/1968). For example, VR can be conceived as an extension of Heidegger's being-in-the-world. That is say, the user in the virtual environment is also ‘there’ through genuine, context-specific interactions. Merleau-Ponty’s (1964/1968) concept of presence is explained through embodied perception. More specifically, sensory engagement connects the physical self to one’s environment, which is why the VR experience is both ‘real’ and immersive. VR and AR have the capacity to induce presence as the user’s perceptions and physical responses engage in what the user perceives of as a real-world experience.

Presence is the illusion of being in a particular time or place, which is created by immersive VR and AR experiences. Lombard and Ditton (1997) explain the concept of presence as it is advanced in media. Gröppel-Wegener and Kidd (2019, 27) propose a framework that includes the participant (agency of the people participating), process (development of the story, as well as the experience and role of people participating), creation (how the story is told through space and the senses) and story (properties of the conveyed narrative and subsequent experience). They posit the interplay of all four creates an immersive storytelling experience. Moreover, immersion allows for richly designed environments that allow “participants to explore an idea, a concept, or a whole world that makes immersive storytelling experiences special” (Gröppel-Wegener and Kidd 2019, 106). It is immersion that creates unique experiences. More importantly, onlookers learn and retain information because they experience presence through AR and VR. While AR and VR have been examined through a Middle Eastern context as it relates to science, healthcare, engineering, and design management (Abidin et al. 2020; Almarzouqi et al. 2022; Hiyasat et al. 2023), there is currently a literature gap pertaining to the integration of AR and VR in UAE educational learning environments. This is particularly true regarding virtual reality documentaries.

By focusing on one specific case study, this paper attempts to fill a literature gap on how immersive technologies—such as virtual and augmented reality documentaries—can be used as a pedagogical approach to help undergraduate students in the UAE reimagine and meaningfully learn from the past. In his seminal book Pedagogy of the Oppressed, Paulo Freire (1970) highlights the transformative ability of critical pedagogy to empower learners through critical thinking and conscientious reflection. By focusing on Palestinian oral history and cultural narratives, the One Upon a Time in Palestine XR documentary relies on immersive technologies to center and amplify marginalized voices. Students are given a mediated experience that deepens their understanding of history, while shaping collective memory.

Multimodal pedagogical practices are very important considering the diverse student body in contemporary higher education. Philippe et al. (2020) discuss the benefits of multimodality in learning environments, specifically as it relates to VR. They argue that VR is a pedagogically rich strategy, which can help students become active learners through collaborative activities. Although focusing their research on primary teachers in Australia, Tan et al. (2023) examines AR curriculum integration as a method to engage multimodal literacies in students. Using a multimodal discourse analysis framework, Herdiawan et al. (2023) explains how immersive and interactive learning is created when VR technology is used to learn about folklore narratives. The authors argue that folklore-based VR is multimodal, which helps students learn about their cultural heritage and encourages critical thinking. Further, this pedagogical structure can facilitate empathy for different cultures to build a more open-minded and tolerant society. Thus, digitally mediated pedagogical practices are multimodal and encourage learning.

Further, VR and AR also evoke emotional responses that connect users to historical content by generating empathy and fostering emotional immersion. Sara Ahmed (2015) uses affect theory to explain the relationship between emotions, language, and bodies. Emotions and affects are not individual experiences. Rather, they are socially and politically shaped by power structures inherent in organized societies. Ahmed’s (2015) affect theory is relevant to the Once Upon a Time in Palestine XR documentary as it allows students to connect in a visceral way and learn about the displacement of Palestinian communities. Further it provides the capability to internalize and recognize the experiences of marginalized voices through the immersive nature of XR technologies. This allows students to feel themselves as part of a greater, shared humanity. The XR documentary instigates critical reflection in a deep and meaningful way that fundamentally transform how students perceive and engage with the world.

Given the nature and context of the entire project, focusing on a single case study allows for an in-depth qualitative analysis of each phase of the project, which includes the design choices made, technical innovations used, impact on youth engagement, and its pedagogical value. There are nuanced insights that would not be possible to uncover if this research focused on multiple semesters of student projects. Moreover, funding (or lack thereof) is another variable that affects research outcomes when multiple cases are examined. The goal of this paper is to provide a comprehensive analysis of a single study, rather than make broad generalizations that can change depending on shifting contexts, design choices, and varying student topics. In the Once Upon a Time in Palestine XR Documentary, the case study focuses on VR, AR, AI, and 3D environments to assess how these elements reshape historical narratives and the pedagogical processes involved. In addition, the XR documentary focuses specifically on pre-1948 Palestine, which has its own robust historical and cultural context. A close examination allows the researchers to demonstrate how students engage with oral history and process it—literally and figuratively—in a contemporary and immersive format. As such, the researchers focus on the innovations, challenges, and successes of one specific project and how educational design facilitates interdisciplinary student collaborations in a multimodal format. The XR documentary bolsters a sense of ‘presence’, which varies across multiple iterations of the project. Moreover, a single case study that integrates technology with historical narratives serves as an example for educators to emulate in their own unique contexts.

The Process

The process of producing the Once Upon a Time in Palestine XR documentary involved a complex collaboration that undertook a novel approach. Initially, after conducting video interviews with village elders, the stories were subsequently researched and authenticated. Once authenticated, these narratives were reimagined into mini scripts that were loosely based on the interviews. This process involved the mass communication students going through 60 hours of oral history interview footage, which was conveyed by Palestinian village elders, to extract the best stories and edit them into mini documentaries. This process resulted in 29 mini documentaries. Ultimately, seven stories were chosen for inclusion in the final XR documentary from the 29 mini documentaries. Once the final seven stories were chosen, the students then searched for archival photographs to serve as the basis for the visual representation. Subsequently, the stories were rewritten as mini fictional narratives, which were loosely based on oral history. Throughout this process we ensured the key elements of these stories remained accurate. Meanwhile, any supplementary dialogue was included to ensure the story was engaging, conversational, and loosely connected with the selected archival photographs. Once this process was complete, the stories were workshopped with actors while industrial design students simultaneously created props that matched the objects in the archival images. The volumetric videos were then produced using Depthkit volumetric video capturing software­—Depthkit—in conjunction with Microsoft Kinect for Windows V2 Camera and the companion Kinect motion sensing device. The Depthkit software then generated a depth map that was used within Unity3D, which was made possible by using a special Depthkit plugin. Simultaneously, the student animators were working to design a 3D environment that resembled a Palestinian village and one section of the old city of Jerusalem. These environments were recreated using the archival photographs as the visual foundational reference. Concurrently, the animation students were tasked with duplicating the photographed 3D environment to create the village used in the final rendered story. The final phase involved the collaboration between the animation, mass communication, and programming students to situate the volumetric videos into the 3D reconstructed Palestinian village, as well as in the section of the old city of Jerusalem. This was achieved by using the Unity3D Gaming Engine, which is named Unity3D. The process of importing the volumetric videos and seamlessly integrating them in the 3D environment was a time consuming, painstaking, and complex process. As such, it required the programmers to work closely with both the media students and animators to ensure the videos played smoothly within the Unity3D environment.

An example of this process is demonstrated through the story concerning the 1918 famine that killed millions in the Levant. Initially, the students found an archival photograph that best represented the story (see Fig. 1). Relying on the selected archival image in conjunction with the elder’s story, the students wrote the script while integrating a reimagined dialogue between the two women in the photograph. While this process occurred, a team of industrial design students created props and costumes to best match the objects and clothes in the photograph. This collaboration allowed the students to recreate the scene in the photograph by shooting live volumetric videos against a green screen (see Fig. 2). The resulting videos incorporated all the necessary data points, which allowed them to be displayed seamlessly within the 3D reconstructed village (as illustrated in Fig. 3). The final output was an experiential VR documentary that could be played on the Oculus Go VR headset (OG). This headset allowed viewers to walk through the environment and experience the selected stories, which appeared as mini scripted volumetric videos.

Figure 1: A large 'Grain rubber.' Process demonstrated by two peasant girls, Palestine. [1922].

Figure 2: Wheat grinder volumetric video shoot.

Figure 3: Wheat grinding scene in a recreated 3D village.

In addition to the VR documentary, the students created a supplementary AR mobile application, which automatically triggered the volumetric mini stories if a phone camera pointed at any of the archival photographs. The AR application projected the corresponding volumetric video that had a Depthkit map, which enabled it to display the video in 3D format in front of the archival image (see Fig. 4).  The AR mobile application endowed the photograph with new meaning by extending the story directly in reference to, and beyond, the source photograph (see Fig. 5). The AR documentary was designed as a standalone experience for users unable to access a VR headset or for an exhibition where users, after experiencing the documentary on the OG, can explore the original archival photographs. They can use a smart device to auto trigger the volumetric videos in context to the photographs, thus reliving the experience and extending the narrative.

Figure 4: Example Depthkit volumetric video with depth map.

Figure 5: Volumetric video in AR app triggered when the app points at the original image.

Reimagining History

The following section provides a detailed account of the process involved in producing one of the seven mini stories. This will provide a comprehensive explanation of the production process and highlight the extent of undergraduate student involvement in this project. The story highlighted in this section pertains to an encounter between notable Palestinian Journalist Khalil Sakakini and one of the elders in the 1920s. The elder was interviewed by a youth who is from Jerusalem. In the interview, the elder conveyed his experience of living in Jerusalem prior to 1948 and his multiple meetings with Khalil Sakakini, one of the most prominent Arab intellectuals of early 20th century. The exchanges he described occurred in the Vagabond Café, which was in Jerusalem. The videoed interview was conducted in a village in Palestine on the outskirts of Jerusalem. The elder recounted a story that occurred while he worked in the café as a young boy. Once the mass communication students sifted through all 60 hours of raw interview footage to locate the story and subsequently consulted with the researcher to determine the value of the story. The story was selected and edited by the students.

Both the student and the primary author of this study researched the authenticity of the story and confirmed the existence of the café. After conducting rigorous research, it was confirmed that Sakakini was a patron of the café as conveyed by the elder (Tamari 2003). After authenticating the story, it was selected as one of the seven that would be produced for the XR documentary. The next task was finding visual references for the Vagabond Café in Jerusalem and scripting the mini stories based on this research, which was completed by mass communication students under direct supervision of the researcher. The process involved choosing an accurate and appropriate photograph, authenticating it, and drafting numerous iterations of the script. During this process, one of the students found a well-suited reference in Salim Tamari’s notable book, Mountain Against the Sea (Tamari 2008). In this bound collection of essays there is mention of Khalil Sakakini frequenting the café and a suitable extract, which quotes a letter from Sakakini as he laments about his days in the café:

“Tomorrow is my last day with the age of idleness, and what a magnificent period it was. Tomorrow the Vagabond Café will be deserted. Our fabulous sessions will meet no more. The Brotherhood of Amity will seek me and not find me. ‘Til today I have never left my home without spending a [substantial time] in bathing, playing, reading, writing, singing, and smoking. From today on, I will leave in black [attire], going straight to work, greeting nobody on the way” (Tamari 2003, 35).

The above extract from Sakakini’s letter was documented and corroborated by the elder’s oral interview, which was then used as the basis for the script that was rendered into a volumetric video. In the video, Kahlil Sakakini sits in the café writing the letter that was extracted from Tamari’s book. As such, this real-life event is reimagined and brought to life through the mini story that was produced. Meanwhile, the researcher worked with the 3D animators to replicate the café in 3D. Once programmed into Unity3D, the students and researcher brought the elder’s original story to life. The narrated mini story appears when users click on an empty chair in the 3D recreated café, which matches the original picture (See Fig. 6).

Figure 6: A 3D construction of Vagabond Café in Jerusalem, with/without Khalil Sakakini’s volumetric video.

Collaboration was an integral and necessary component in the process of producing the documentary. For example, the media communication and computer engineering students worked together to ensure the volumetric videos worked properly in Unity3D by using the Depthkit Unity plugin, which is a process that took months of experimenting. This was finally perfected when the videos were produced and compressed correctly so they would playback as volumetric videos within Unity3D. This involved capturing the volumetric videos using a Microsoft Kinect depth camera, which captured both video and infrared depth information and transformed them into a mesh overlay. The 2D video recorded by a 4K camera was then combined with the Depthkit mesh to create the volumetric video. This process required several adjustments as there were many challenges in capturing certain clothing that did not reflect infrared accurately. Once finalized, the videos were handed over to the programmers to integrate into the virtual environment using the Unity 3D engine and Depthkit Unity 3D plugin. This step also required tweaking to adjust the mesh texture and reposition the videos in space to prevent distortion when viewed through the OG headset. The animators worked closely with the programmers while designing the environment and setting up trigger objects to ensure proper porting into Unity 3D. Since Unity 3D proved to be resource-intensive when exported as an Android package for the OG headset, the 3D objects had to be optimized multiple times to run smoothly on the OG. The 3D visual replica of Jerusalem had to be designed to look authentic and match reference images of the old city, while also lowering the poly count to accommodate the OG's processing limitations. Another key element that enhanced a viewer’s presence and experience involved the interactive design, particularly for novice users unfamiliar with VR headsets or the OG controllers. A simple point-and-click interaction system was implemented, which involves a light arrow that is used to point to objects in the environment. When the user clicks on the object, the volumetric video is activated and plays within the space, which creates an interactive and immersive experience that allows users to navigate and engage with the stories as naturally as possible. For example, in the village, a mechanism was needed to help users move from one location to another, which was achieved by having them click on a donkey to be transported to the next area. Similarly, to move between the village and Jerusalem, a 3D replica of a 1940s bus was designed. The bus was stationed on the outskirts of the village or outside the old city gates, which allows users to travel between the two locations. The final phase of the project involved extensive testing to ensure everything was ready for submission to an international exhibition, which was produced under a very tight deadline. This process involved solving extremely complex challenges with novel technologies to ensure the documentary was complete and ready for exhibition. It is important to note the programming students continually demonstrated an extremely professional attitude while operating under tight deadlines. They solved problems with untested software and completed the task within six months. Considering the difficulties they faced, this is impressive. The project was successfully exhibited internationally at the Museum of the Future in Paris, France.

Discussion

A key aspect that underwrites this project is the creative freedom to reimagine history while producing a mini narrative, which was loosely based on real oral history yet remained authentic. The key to this ensuring authenticity throughout this process involved the research conducted to locate archival photographs that match the oral stories, the era, and the location where the originals stories occurred. Those photographs were used to construct the 3D environment where the volumetric videos were eventually activated. The premise here is that immersive technologies—VR and AR—can transform the perception of mediated experiences into non-mediated ones, which creates a more interactive relationship between the audience and content. In particular, placing narratives within a well-constructed virtual reality environment can make the narrative feel more real and plausible (Brown et al. 2003; Lombard and Ditton 1997). This is a natural result of virtual reality manifesting a presence for the audience who is situated in those historical events, despite these events happening in the past. This presence, where the audience is consuming reimagined narratives as factual representation, creates an added responsibility for the filmmaker to be as authentic as possible (Filimowicz and Tzankova 2020; Slater 2018). However, one can argue this mediated experience offers creative freedom to reinterpret historical content, which adapts it to the constraints of new immersive technologies while bridging the narrative gaps inherent in oral storytelling. As such, this often invites diverse interpretations. More importantly, using immersive media to recount the past requires the environment and narrative mediation to be engaging to cultivate a presence so the audience is captivated by and immersed in the historical content (Huang et al. 2018). The intentional reimagining of the content is intended to recount stories more effectively by adapting them to technological means. In the case of the Once Upon a Time in Palestine XR Documentary, user feedback was overwhelmingly positive, which supports the theory that presence is enhanced in virtual reality immersive experiences. This demonstrates significant value, as observed by the undergraduate students who participated in the study. Their feedback was validated through survey results, our observations, and qualitative comments such as: "I felt as if I was there," "I don’t want to leave," "I love being here," and "Thank you. With this, I feel like I have been to Jerusalem." However, some participants had negative experiences. These few select individuals found the experience to be uncomfortable or overwhelming. Such responses precipitated comments like: "I feel dizzy," "It flickers a little," "How do I go inside this door," and "I’m stuck." The user test results demonstrate that most users found the experience engaging, which cultivated a feeling of being immersed and transported into the virtual space and—in this case—into the past.

Nevertheless, it's important to recognize the limitations of the technology. Additionally, novelty effects play a significant role since users without prior experience with VR headsets often encounter difficulties, which manifests as issues with using the equipment and focusing on the narrative. The novelty may distract from fully engaging with the story, which diminishes some of the intended goals of the experience.

Conclusion

The Once Upon a Time in Palestine XR Documentary reimagines historical narratives told through the oral tradition by immersing the audience in the content. Immersive content amplifies the viewer’s experience, and engages the youth, while exposing them to important stories pertaining to pre-1948 Palestine. Thus, XR technologies demonstrate the potential to expand and stimulate the knowledge and education of youth and inspire an appreciation of their heritage in an engaging way. Using VR technologies is well-suited to reimage oral histories as factual narratives conveyed through folk stories can be enhanced when situated in a virtual environment. This is reinforced by the viewer’s experience of ‘presence’, which virtually situates the viewer in the historical period when the story occurred. When constructed effectively, this reimagined virtual environment stimulates a temporal plausibility that enhances the viewing experience and creates a stronger engagement with the historical content. Despite these reimagined narratives not being a meticulously accurate representation of factual historical events, the benefits of these reimaginations are manifold. This reimagination not only has artistic merit, but also the potential to engage and stimulate the viewer’s cognizance and engagement with the original stories. Further, our research demonstrates this is a valid method of retelling history as it is analogous to historians who use prose to convey historical facts. Thus, in our opinion, mediating factual narratives using immersive media technologies opens the door for these narratives to be reimagined with a degree of creative freedom that is suitable and feasible to available technologies.