University of Southern California Dissertations and Theses

Silent sparrows: exploration into augmented reality narrative frameworks

(USC Thesis Other)

Silent sparrows: exploration into augmented reality narrative frameworks

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Silent Sparrows Exploration into Augmented Reality Narrative Frameworks by Christy Ye A Thesis Presented to the FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree MASTER OF FINE ARTS (INTERACTIVE MEDIA) May 2023 Copyright 2023 Christy Ye D e d i c a t i o n For Yeye. ii A c k n o w l e d g e m e n t s Thank you to everyone who helped me throughout this project. I could not have done this without my advisors, Martzi Campos, Scott Fisher, , Nahil Sharkasi and the extended USC faculty. Thank you Qi Yao Xie for all the work you have done on this project. My cohort and the IMGD community, for your support. Dr. , and my family for your time and memories of “Ye Ye”. Thomas Pearce iii T a b l e o f C o n t e n t s Dedication.....................................................................................................................................ii Acknowledgements...................................................................................................................iii List of Figures..............................................................................................................................v Abstract....................................................................................................................................... vi Chapter 1: Introduction.............................................................................................................. 1 Chapter 2: Walkthrough.............................................................................................................2 Chapter 3: Thesis Question...................................................................................................... 5 Chapter 4: Objectives................................................................................................................ 6 4.1 Enhance narrative through hybrid space.................................................................6 4.2 Spatial UI for mobile AR................................................................................................6 Chapter 5: Project Scope.......................................................................................................... 7 5.1 Gameplay.......................................................................................................................... 7 5.2 Novelty of the Medium.................................................................................................. 7 Chapter 6: Technical Exploration............................................................................................ 8 6.1 Background Research................................................................................................... 8 6.2 Technical Prototype Takeaways................................................................................10 Chapter 7: Narrative Exploration........................................................................................... 12 7.1 Background Research...................................................................................................12 7.2 Qi Cheng Zhuan He.......................................................................................................13 7.3 Nodular Linear Narrative for AR.................................................................................14 Chapter 8: Interactive Installation........................................................................................ 19 8.1 Background Research..................................................................................................19 8.2 Installation Process..................................................................................................... 21 8.3 User Interface for Hybrid Spaces.............................................................................22 Chapter 9: Playtests................................................................................................................ 24 9.1 First Playtest Results..................................................................................................24 9.2 Second Playtest Results............................................................................................ 25 Chapter 10: Conclusion........................................................................................................... 27 10.1 Answering the Thesis Question............................................................................... 27 10.2 Analyzing the Objectives......................................................................................... 28 10.3 Going Forward............................................................................................................ 28 Chapter 11: Glossary................................................................................................................ 30 References................................................................................................................................. 31 iv L i s t o f F i g u r e s Figure 1: Trailer screenshot of Asobo Studios’ Fragments.................................... 8 Figure 2: Screen capture of Pokemon Go!.................................................................9 Figure 3: Graph of gameplay mechanics for Silent Sparrows............................. 10 Figure 4: Screen capture of Silent Sparrows Demo - Capturing a procedural AR object..............................................................................11 Figure 5: Painting of Da Guan Yuan by Sun Wen (1818-1904)............................. 12 Figure 6: Visual graph of a Qi Cheng Zhuan He narrative structure.................. 14 Figure 7: Visual graph of a branching narrative structure................................... 15 Figure 8: Screenshot of narrative beats for first half of Silent Sparrows’ AR experience.....................................................16 Figure 9: Screenshot of narrative beats for second half of Silent Sparrows’ AR experience...................................................... 17 Figure 10: Screenshot of AR objects during second half of Silent Sparrows..................................................................................18 Figure 11: Screenshot of physical installation for ILMxLab’s CARNE y ARENA................................................................................. 20 Figure 12: Photo of player using AR device disguised as camera in Masaki Fujihata’s BeHere, 1942.............................................................................20 Figure 13: Screenshot of IoTs proof of concept. Light changes color and record player plays when the AR device detects input................................................................................................................................ 22 v A b s t r a c t Silent Sparrows is an augmented reality installation experience that explores how AR installation spaces can enhance narratives. This project used mobile Augmented Reality (AR) and an interactive installation composed of the Internet of Things (IoTs) in order to describe a granddaughter’s journey to uncovering her grandfather’s past in China, 1980s. This paper will go through my ideation and production process in making this project. vi C h a p t e r 1 : I n t r o d u c t i o n The goal of this project was to explore the ways AR can be used to create immersive narrative experiences. Through this exploration I hoped to evaluate whether or not AR could help create alternative frameworks for linear narratives. In order to create a fully immersive experience, my project is composed of two parts: a virtual space and a physical space. The virtual space consisted of all AR elements and oral anecdotes that drive the main narrative of this project. These verbal anecdotes were collected from interviewees and are represented by visual AR objects. The physical space consisted of an interactive installation containing physical prints and archival research collected for this project. These will be physical copies of newspaper articles, vinyl album records, and the Internet of Things (IoTs). 1 C h a p t e r 2 : W a l k t h r o u g h Before entering the space, the player starts the experience at the front of the installation. The player sees a collage of research documents and handwritten notes sewn into the surface of the wall. These documents used to belong to “Yeye”, a grandfather who at one point studied abroad in England during the 1980s. Hand written journal entries surrounding these documents question why Yeye’s past has remained a secret and speculate what could have happened to him during that time. These entries are from Yeye’s granddaughter, the Narrator of this experience. This collage of papers visually connected by a web of ivy converges and leads the player across the wall. The artist’s statement of this experience is attached to this point of convergence. In it the Narratorconveys her desire to find out about Yeye’s past. The player is encouraged to read these documents and entries before entering the installation space. Once the player is finished reading the context on the wall, they are given an iPad and lead into the installation space. The interior of the installation space contains four vinyl albums, which serve as the image anchors for this experience. These albums seem to be a random assortment of Western Classical music. Each album is accompanied by a small hand written note by the Narrator. From her notes, the player realizes that Yeye bought these albums back from England to China, and although he no longer plays them, these were once his greatest treasure. These image anchors are arranged across the installation space, on the walls and furniture of the space. These image anchors will not be activated until the second half of the experience. The interior space also contains a vinyl record player, one lamp light, and one spotlight. The vinyl record and the lamp light are plugged into smart plugs in the space. As the player progresses through certain narrative scenes, the AR app will trigger IFTTT API calls that will cause the plugs to automatically turn on or o . The smart light bulbs in the lamp and the spotlight are also connected to the AR app through IFTTT. These light bulbs will change colour when the player interacts and listens to specific oral anecdotes during the experience. When the player starts the app, the spotlight will change from yellow to white, and then green, when the player interacts with the first procedural AR object. In this dimly lit room, the spotlight washes the installation in a pale, lime colour. The player follows the instructions in the app to scan the installation interior in search for these procedural AR objects. There are two types of procedural AR objects. 2 The first are plant models that spawn on any horizontal surface in the space. The player cannot interact with these objects using the AR app. Some of these AR plants have spatial bird sounds attached to them, so when the player physically approaches these objects, these sounds will trigger. As the player slowly walks around the installation space, sparrows will chirp and leaves will rustle as virtual plants and ivy grow around them. The second are procedural AR objects in the form of glowing cubes of ivy. At the center of this ivy cube is a single bird feather. These AR objects spawn on any vertical surface in the space (i.e. walls). When the player clicks a button in the app, an audio clip of the Narrator’s Dad or the Narrator’s voice will play. These audio clips are pieces of oral anecdotes collected by the narrator during her interview process. Sometimes the Dad relays a forgotten memory of Yeye before and after England. Sometimes the Narrator will express her frustrations in her inability to find the actual truth behind her family’s reluctance to remember Yeye’s past. Everytime the player uses the camera to interact with these audio bits, a screenshot of the AR object overlaid in the installation space will also be saved to the player’s in-game journal, so they can refer back to their findings throughout the experience. Some of these procedural objects contain special audio clips that highlight the moment the Narrator thinks she discovered the truth behind why her family covers up her grandfather’s past achievements. Upon collecting at least one of these special AR objects, the player will unlock the next half of the experience. All procedural AR objects and plants “growing” in the installation space will disappear. The record player will automatically turn oand the lamp and spotlights will flood the installation space with a vivid, eerie red. In a now ‘empty’ space, the player will move the iPad around the installation space until they scan the first image anchor: Twelve Poems of Emily Dickson by Aaron Compland. If the player read the note next to this album, they would know that all these songs on this album are about Death. The AR model attached to this image anchor is a bird’s nest with three feathers inside. When the player uses the app button to take a screenshot of this AR object, they will hear an oral anecdote of the Dad explaining that the real reason the grandfather doesn’t talk about his past achievements anymore is because his achievements are tied to trauma enduring the Cultural Revolution, and that remembering his past glory also meant remembering his time in a concentration camp during the revolution. 3 After listening to this oral anecdote, the following three image anchors will also be activated: Songs of Bethlehem, Swan Lake by Tchaikovsky, and an album of Beethoven’s greatest compositions. All three of these image anchors contain dead AR sparrows: one strangled with ivy vines, one drowned in a pond, and one in tree branches. As the player interacts with each of these AR objects, they are able to hear di erent stories about the Grandfather’s time in a concentration camp. After the player interacts with all four AR models, the journal button on the iPad app will animate, signaling a new item in the journal. Both the spotlight and the lamp light will change from red to yellow. In the journal, a new written entry will appear. This entry outlines how the narrator struggles to deal with this new information on the revolution. The player is then asked whether they would erase or keep all the screenshots and information they have compiled in their journal. Erasing all their information will play an audio memo where the narrator expresses her choice in protecting her family from further political harm and trauma. All journal entries and the dead sparrows attached to the image anchors will disappear. All AR plants will appear in the installation space and live sparrows will grow and appear amongst the fauna. Keeping all their information will play an audio memo where the narrator expresses her choice to protect her families’ history. The journal entries in the app and the dead sparrows attached to the installation space will remain. No AR plants or sparrows will grow in the installation space. End of experience. 4 C h a p t e r 3 : T h e s i s Q u e s t i o n To what extent can commercially available AR technologies enhance traditional linear narratives? Over the course of this project, I was able to note a variety of ways AR narratives stand apart from traditional linear narratives, specifically in terms of the technology, the narrative, and the installation space. By using all three aspects, AR is able to create a unique type of cinematic gaze – the increased gaze– one that allows players to experience narratives through motifs and emotions. Coined by Federico Biggio, the increased gaze is a term created as a response to Gene Youngblood’s Expanded Cinema. This concept shares a lot of overlap with ambiguous storytelling, a concept AR practitioners may be more familiar with. However, given the scope of this project, I will only be referencing the increased gaze. 5 C h a p t e r 4 : O b j e c t i v e s For this project, I wanted to set objectives that would help guide the design process. 4 . 1 E n h a n c e n a r r a t i v e t h r o u g h h y b r i d s p a c e This project will try to determine whether or not narratives can be enhanced by connecting digital game mechanics to physical electronics in the real world. When it comes to this objective, I am more interested in exploring the potential for alternative narrative frameworks and formal editing techniques (to parallel film terminology). As a result, I would like to acknowledge that the new and experimental nature of the technology used will not be the most ecient or immersive way of presenting alternative narratives. In short, there will be technological drawbacks due to present-day limitations in the commercially available hardware and software used in this project. Users will most likely have diculty adapting to the game mechanics and, consequently, understanding the narrative. Therefore, this story is written in a way that emphasizes emotional narrative beats over exposition, so that inevitably frustrating mechanics that muddle the story’s plot would not diminish the player’s overall enjoyment. I would like to use this chance to explore a wide variety of hybrid interactions in order to evaluate which kinds elevate or detract from the narrative. 4 . 2 S p a t i a l U I f o r m o b i l e A R Unlike mainstream interactive media narratives, commercially available AR technology can only support a much smaller amount of information on screen at a time. Is it possible to have multiple interfaces, in both the virtual AR world and in the physical installation world, work together to create a unique user experience? When it comes to this objective, I am interested in exploring the possible best practices for using multiple user interfaces across di erent mediums simultaneously. If this new format of UI is possible, then what kinds of advantages could that bring to AR narratives? 6 C h a p t e r 5 : P r o j e c t S c o p e Both AR and installation projects cover a wide range of di erent genres and media. In order to keep the focus of this project on AR and its e ect on narrative, I need to limit the scope of my project. 5 . 1 G a m e p l a y I wanted the experience to only have one type of game mechanic. From a development standpoint, mobile AR technology is still not very stable. Although it is theoretically possible to create software that can support multiple game mechanics, the current hardware does not have the computational or graphical capabilities to execute this complex code (in comparison to mainstream PC games). From a user’s standpoint, because AR is already a dicult medium to use, if I include multiple game play mechanics, the learning curve of this project would be too high. 5 . 2 N o v e l t y o f t h e M e d i u m Although AR games have been out in the market for almost a decade, the technology is still fairly new. Majority of players were unfamiliar with the medium and would need additional time and help to learn to use it. The novelty of this medium will also mean players would most likely be unfamiliar with the narrative pacing and structure. Given the scope of this project, this disconnect between the player and the medium may not be solvable and its e ect on the player’s overall experience should be taken into account. As a designer I needed to make certain design choices in order to compensate for the technology’s weakness(es). These choices would be taken into consideration when answering my thesis question at the end of this project. 7 C h a p t e r 6 : T e c h n i c a l E x p l o r a t i o n For this project I spent most of my pre-production phase creating tech prototypes. These prototypes were meant to evaluate the level of diculty in coding specific features and the stability of the code base. My only requirement was that the gameplay mechanics be something meaningful to this AR narrative experience so that I could explore the impact of AR interaction on narrative. 6 . 1 B a c k g r o u n d R e s e a r c h Within the AR space, there are two dominant forms of narrative: linear and nonlinear. F ig u r e 1 : T ra il e r s c r e e ns h o t o f A s o bo St udi os ’ F ragm e n t s For the most part, linear experiences seem to be more prevalent in headset devices like Microsoft's Hololens, where immersive games like Asobo Studios’ F r a gm en t s leverage this format to assimilate players into the game world (Microsoft). This narrative format is not unlike the majority of VR headset games (during this time), where players are allowed to embody avatars to interact with the game narrative's world and characters (Slater). 8 F ig u r e 2 : S c r e e n c a p tu r e o f Po k e m o n Go! Nonlinear experiences are more prevalent within the mobile AR space. Games like Niantic’s P o k e m o n G o ! following this format tends to leverage the player mobility’s to create geolocation-based gameplay. These games overlay story elements on specific geographic coordinates and require players to physically move to these spaces. None site-specific experiences, like the LA C MA xS na pC h a t experience, also follow a similar non-linear narrative format (LACMA). These AR experiences re-purpose public spaces using image anchors or simultaneous localization and mapping technology (SLAM) in order to custom create physical spaces that can host the AR experience gameplay. Unlike headset experiences, these mobile AR games do not actually allow users to spend a lot of time in the AR view. Instead AR is used sparingly to enhance visual or auditory immersion during specific narrative beats while the majority of the gameplay takes place through a 2D mobile interface. Mobile AR experiences are more physically taxing on the player. Having a linear narrative that unfolds in AR would require players to be physically able to hold a mobile device up to their face for long stretches of time. Rather than having a continuous, linear narrative, it would be better to have a game system that breaks up the gameplay and narrative into nodular segments in order to give players enough physical rest between narrative beats. 9 As of developing this project, the majority of commercially available mobile AR development tools provide toolsets that are designed for nonlinear, mobile experiences. These development platforms focus on rendering high fidelity, static models for non-linear experiences. So while these tools have fairly sophisticated systems for spatial awareness, they have little support for game event systems that include custom game mechanics, animation, shaders, and audio systems. Because of this, I wanted to choose a game engine that would also have an AR plugin that could support high fidelity visuals and complex game systems. After several prototypes, I decided to use Unity with a Niantic’s Lightship ARDK plugin to develop this project because of Unity’s robust support for game development and Lightship’s diverse toolset for spatial awareness. 6 . 2 T e c h n i c a l P r o t o t y p e T a k e a w a y s F ig u r e 3 : G ra ph o f g a m e pl a y m e ch ani cs f or Si le n t S parr o w s Unlike most mobile AR games, I wanted to focus on building a linear, 360 immersive narrative experience. Given the history of my narrative, I wanted to construct a straightforward narrative that maintains the fidelity of the content. Like F r a gm e n t s, I wanted to give the player enough autonomy to engage with the characters and propel themselves through the narrative, but not so much that they would re-interpret the information as they pleased. In terms of mobile AR game mechanics, I wanted to focus on using the camera as a In cinematic terms, the gaze is a way of seeing the world of the narrative, and provides cultural models that a ect the individual cognition (Biggio). explore as many navigational or explorative systems as possible. The idea was that if the player was able to roleplay as the narrator and physically move around a space, then they would be able to feel more engaged and immersed in the narrative. 10 F ig u r e 4 : S c r e e n c a p tu r e o f S ile n t S p a r r o w s De m o - Cap turi ng a pr o c e du ral AR o b j e c t Of the prototypes I had tested, the most simple and mobile mechanic was to have players take pictures of hidden AR objects. As the players move the iPad camera around a space, SLAM technology in the AR app will locate and map all horizontal and vertical planes in the space. AR objects – mainly plants or interactive audio objects – will anchor to certain planes and ‘grow’ in the space. The more the player moves around, the more accurate the plane detection data will become, thus resulting in a 360 view of a virtual garden overlaid in real space. In terms of game mechanics, I felt that photo camera inspired mechanics were the most e ective in communicating meaning and engaging the player. Moving forward, I tried to center my project around this mechanic. It also became clear that regardless of how I designed my project, the fact that it is AR will be a pain point for many users, due to the novelty of this medium in mainstream games. Therefore, I needed to spend a lot more time than estimated to create an emotionally impactful narrative in order to compensate for the technology’s complexity. 11 C h a p t e r 7 : N a r r a t i v e E x p l o r a t i o n To make this project I wanted to use existing narrative frameworks and see which ones would be best fit for the AR game system I designed. The goal of this process was to find a narrative framework that would not only compensate for AR technology’s complexity but also support a story that would blend well with the medium. 7 . 1 B a c k g r o u n d R e s e a r c h I tried to create a narrative structure that would encourage the player to use the camera-driven game mechanics. During the research process, I tried to compile historical events that were interesting enough to trigger an intense emotional response. Because of mobile AR’s nodular organization of information, the players would encounter narrative beats piece by piece. To make sure that players would not forget the narrative content if the pacing between the beats dragged out for too long, it was necessary for each beat to contain enough emotional or dramatic tension to stand on its own. This way even if players do not encounter all narrative beats in a timely manner, they would still be able to remember and contextualize the story. F ig u r e 5 : P a i n t ing o f D a G u a n Y u an b y Su n W e n ( 1 8 18 - 19 0 4 ) 12 I based the virtual imagery and soundscape of the experience around a famous mythical Chinese garden called Da Guan Yuan, from the D r eam o f the R ed C h a m b e r s (Britannica). The initial inspiration came from an archived speech/note written by my grandfather in 2010, reflecting on his travels to England and his contributions to Aeronautics during that time. In the speech, he used an allegory stemming from th e D r e a m o f th e R ed C h a m bers to describe his encounter with western culture and technology. Loosely translated, the allegory refers to a scene in the novel where Granny Liu, an uneducated commoner, enters the Da Guan Yuan, a nobel’s garden, for the first time. This allegory is a humble way of emphasizing one’s vastly inferior status when entering a brave new world. I found the imagery in this speech incredibly impactful. For a man who valued education above all, my grandfather would have probably found the astronautics work at Bristol almost fantastical. That in contrast to the hardship he faced during the Cultural Revolution helped give the narrative I was creating a more somber and tragic light. If the oral anecdotes of this narrative described the death and melancholy in my grandfather’s past, I wanted the AR visuals and spatial soundscape to create a fantasy space of life and wonder. The juxtaposition of both sides of this story would allow me to use a strong palette of emotions that could paint a story despite the convoluted historical exposition and AR’s technological complexity. In this project, I wanted to go further by linking all the pieces of discovery in AR to feed into one, overarching, linear story. It was important that I identified a formal structure to which I could then fill with story content. 7 . 2 Q i C h e n g Z h u a n H e Common western narrative frameworks like the Hero’s Journey (Campell) are largely driven by dramatic tension and resolution. The protagonist starts with a problem and then goes through several trials and tribulations in order to reach a resolution that solves said problem. When compiling the narrative content for the story, I thought it would be more appropriate to have no resolution, no way for the narrator or the player to have a satisfying conclusion to the story. Although this work is fictitious, I felt it would be too disingenuous to imply that the generational trauma from these events have truly concluded. 13 F ig u r e 6 : V is u a l g r a ph o f a Q i C h e ng Zhu an H e narr a tiv e s tru c tu r e I decided to use a Qi Cheng Zhuan He, a four act structure commonly used in traditional Chinese poetry (also known as Kishōtenketsu). This type of narrative structure does not rely on conflict and resolution to create dramatic tension. Rather it relies on plot twists in order to move the narrative forward. Stories and poems written in this style are usually open-ended, with the narrator reaching a conclusion that is distinctly di erent from where they started. In a sense, the climax of the story is the ending as the union between the narrative’s setup and the new narrative direction created by the shocking twist (Chang Gao). Qi Cheng Zhuan He focuses on the characters’ emotional struggle in adjusting to their world as their expectations of their world are suddenly thrown away. While the Hero’s Journey can also be internal, this narrative framework focuses more on a character’s change in mental state (Chang Gao). In the case of this project, this framework allows for the narrator to have a deeper reflection on her journey through this experience and focus on the internal struggles that come with researching the past (Vorontzov). By the end of the narrative, there’s no resolution or closure, but rather a change in perspective. 7 . 3 N o d u l a r L i n e a r N a r r a t i v e f o r A R The background research and narrative frameworks research so far has informed what kind of content I should use and in what order to utilize them. 14 After various di erent narrative prototypes, I concluded that it is not possible to tell a plot-based story using a non-linear narrative framework. The non-linear structure gave the audience too much freedom in interpreting the story. Information from the oral anecdotes, and the garden plants were so subtle, players began to fill in the blanks using their own conjectures. As a result, this very specific story was twisted in a variety of ways. Ultimately, I needed a way to control the way players interpreted and experienced the narrative. At this time of writing, AR by nature separates narratives into nodes across a real physical space. Each AR element, a model or a spatial audio file, exists as a virtual object attached to an area in real, physical space. This means that, visually, an AR narrative is physically scattered around a specific, physically navigable area. Using this visual feedback, players choose to engage with whichever narrative node whenever they choose. Thus, as players progress through a narrative, they choose to compile narrative nodes at their own pace and order. The question now is how I can encourage players to engage with these narrative nodes in order to guide through the narrative. F ig u r e 7 : V is u a l g r a ph o f a b r a n c hi ng nar rat iv e s t r uc tur e Borrowing some techniques from branching narrative systems, I found there were two general ways to control how players will interpret the story (Rollings). One, if the player is given a choice, to repeat important information in all these choices. Two, to have the player’s choices converge into ‘narrative chokeholds’. 15 F ig u r e 8 : S c r e e n s h o t o f n a r ra tiv e b e a ts f or fi rs t ha lf o f Silen t Spa r r o w s ’ A R e x p e r ie n c e For the first half of the experience, where the player is searching for information on the Grandfather, they are allowed to choose which pieces of information to capture from pre-determined sets provided by the game system. In order to make sure that the narrative does not become too confusing, every AR object in this half all harkens back to the narrator’s struggle in discovering more information on her Grandfather. One audio might illustrate a vague event that happened in the Grandfather’s past, another might contain the narrator’s frustration in not finding an answer to her question. In order words, regardless of which AR object the player chooses, every audio clip attached to the object will reflect the narrator’s research process and her struggle. 16 F ig u r e 9 : S c r e e n s h o t o f na r r a t iv e beat s f o r se c o nd hal f o f Si l e n t Sp arr o w s ’ AR e x p e r ie n c e Regardless of the choices the player makes in the first half, they will still encounter the same second half of the experience. From here on into the second half of the experience, the player is required to listen to all narrative beats in order to progress to the ending. While players get to choose the narrative beats in this section are anchored to image anchors. Players are also required to listen to all narrative beats during this section. While the order is up to the player, the information gathered is always consistent. 17 F ig u r e 1 0 : S c r e e n s h o t o f AR o b je c ts d u r i ng se c o nd h a l f o f Sile n t Sp arr o w s Although I had mentioned players get to choose the narrative nodes they engage with, with these limitations, there is no actual player choice given when it comes to the narrative (Upton, 28). Rather, choice is more so which AR object the player wants to focus on and where in the installation they want to direct their gaze. Maybe the player hears a sparrow chirp near an AR plant and chooses to walk to that location. Or maybe the player sees a sparrow pop out of an album and chooses to point the camera in that direction. In other words, each node is attached to a 3D motif that represents a part of the narrative. As players progress, they start to engage from one nodule detail to detail. Players need to choose narrative nodes in order to progress through the story. But the choice is not made because these nodes contain di erent or new narrative information. Instead it is because nodes are connected to visually or auditory representations that attract players to specific locations in the space. 18 C h a p t e r 8 : I n t e r a c t i v e I n s t a l l a t i o n AR is a medium that augments real space and so it needs to be paired with a custom space in order to reach its full potential. In the case of this project, the space that houses the AR experience needs to fulfill two goals. First, to contain any AR objects within the space such that the outside world and the world of the experience are visually separated. Second, to form a connection between the virtual AR interactive objects and the physical Internet of Things (IoTs) in real space. Both goals will work together to create a fully immersive environment using commercial-grade mobile AR . 8 . 1 B a c k g r o u n d R e s e a r c h New Media installations cover a very broad range of experiences and I needed to narrow down my scope of references. While very few mobile AR experiences use custom installation spaces, planetariums, VR, and hybridized new media spaces have developed some best practices for interactive installation work. Planetariums are a good example of a 360 immersive non-headset experience that utilizes lighting and audio in order to construct environments that feel like the cosmos (Elwes, 171). The combination of both the planetarium projectors and the dome environment becomes a cinematic apparatus that uses controlled lighting and shadows in order to target the human senses (Elwes, 171). 19 F ig u r e 1 1 : S c r e e ns h o t o f ph y s ic a l i ns t all a tion f o r IL M xLa b ’ s C A R N E y AR E N A VR headset experiences have also used physical installations to heighten the narrative tension during certain scenes. ILMxLab’s C A R N E y AR ENA filled an installation space with interactive heaters and sand in order to fully simulate the di erent weather conditions Mexican refugees faced when crossing the desert (ILMxLab). Instead of lighting and shadows, VR installation experiences like C AR NE y A R E N A rely on touch and temperature to enhance the narrative experience. F ig u r e 1 2 : P h o t o o f pl a y e r u s i ng AR de v ic e di s gu is e d a s ca m era i n Ma s ak i F u j i ha ta ’ s B e H e r e , 1 9 4 2 Hybridized spaces cover a broad variety of analog and digital mediums that bridge the connection between human and machine interaction (Spielmann). These hybrid spaces tend to target transcultural and technology space by blurring the divide between the traditional media and computing. This results in a situation where there is a continuous flow of di erent cultures, an ambiguous state where the player is present in both the virtual and the physical space. Experiences like B e H e r e , 1 9 4 2 have used this sense of ambiguity to incorporate the player as both a voyeur and a character in the narrative. Acting as a photographer both in the AR space and in reality, players can experience both nostalgia and discovery, reflecting a state of cultural diaspora multicultural children often experience. 20 Design elements from these three categories of new media experiences have greatly informed the installation prototyping process of my project. By integrating lights and sound that respond to the certain narrative scenes, I was able to create a space that blends past and present-day memories. 8 . 2 I n s t a l l a t i o n P r o c e s s The method to contain the AR objects using an installation space was fairly straightforward. Furniture and partitions in the space were arranged towards the edges of the space such that players would not trip over or bump into any physical objects while using the mobile AR experience. Tables and chairs were covered cloth to prevent procedural AR objects from growing and clipping between the legs. Connecting the physical installation space with AR was much harder. I wanted to find a way to change the visual appearance and soundscape of the installation in response to certain dramatic narrative beats. The overall mood of the installation needed to reflect the mood of the narrator relaying the narrative to the player. As she was designed to be equally fascinated and horrified by the story, it was important to me that the players in this installation would feel a similarly ambiguous state of conflicting emotions. 21 F ig u r e 1 3 : S c r e e ns h o t o f I o T s pr o o f o f c onc ep t . Ligh t c ha nges c ol o r an d r e c o r d p l a y e r p la y s w h e n th e AR d e v ic e d e t ec ts inpu t Of the IoT devices I tested, the Philips Hue coloured light bulbs and Kasa smart plugs seemed to work the best. Coloured light bulbs were installed in all the installation lights and programmed to change colors depending on which AR objects the player took a picture of. The Kasa smart plugs were connected to an electric vinyl record player in the installation and programmed to turn on or o depending on where the player was in the narrative. In addition to these devices, I also added several garden themed props into the installation walls and furniture pieces. These props were designed to be at most half of the total garden-themed imagery, a design that would only be completed with the AR experience. On its own the installation stands as a space that is starkly di erent from the outside world but also not di erent enough to convey its own story without the AR narrative experience. An ambiguous state of unreal and real. 8 . 3 U s e r I n t e r f a c e f o r H y b r i d S p a c e s This project was unique in that it utilized multiple layers of user interfaces at the same time. On one layer was the AR iPad interface, where the core game mechanics 22 and narrative take place. The other was the installation space, where the interactive IoTs, vinyl album records (acting as AR image anchors), and physical prints (historical archival material that started this project) take place. Players needed to constantly move between the AR interface and the installation space in order to piece together the narrative. I realized through playtests that between the AR and the installation, players felt overwhelmed by the complexity of the experience. With the narrative split between the AR experience in the iPad and in physicals in the installation space, players would only interact with one or the other and miss parts of the story. Context like why was the experience garden-themed, why were there albums in the space, why were the characters important were common frustrations. There needed to be some way for the player to navigate between both the AR and the installation interfaces. To help players navigate the installation, I split the space into visual categories to guide the players from one narrative scene to the next. Props in the installation were rearranged so that objects of similar textures– paper, plastic, cloth–would take up distinct sections in the installation. Physical objects, like the image anchors, were hung on walls and surrounded by fake plants to mark them as part of the AR experience. To help the players understand the narrative context, I added historical exposition to the installation and gameplay tutorials to the AR mobile app. Before the start of the experience, before the player uses the AR app or enters into the installation space, to help players make more informed choices during gameplay. This historical context covered the characters’ exposition, the setting, and other relevant research used in the narrative. Gameplay tutorials in the mobile app taught the player what to expect when using AR and how to play the game before beginning the AR experience. When it comes to AR experiences with multiple user interfaces, I found that players have an easier time navigating the experience if information is not spread across multiple interfaces. If the AR device contains the core gameplay and narrative, then the other supporting physical interfaces should contain the narrative exposition. Within each interface, players tend to find the information more accessible if organized through visual textures. 23 C h a p t e r 9 : P l a y t e s t s 9 . 1 F i r s t P l a y t e s t R e s u l t s The first round of playtests conducted for this project consisted of remote and in-person sessions. Players could play the installation by video calling into the mobile device or by physically visiting the installation space. These playtest rounds were also conducted before the narrative’s ending was implemented. The focus of these playtests was to evaluate the clarity and emotional impact of the narrative. Overall, players seemed to find the narrative impactful and easy to understand. Once the audio clips started playing, players were drawn to the story and were able to piece together core emotions of melancholy and sadness very easily. It was dicult for some players to understand the more contextual details of the story. These players were confused on which characters were who, who’s space the installation represented, and what the relationship between the characters were. These were aspects that I would need to add to the exposition in the installation. I also would add a tutorial page in the app before players enter the AR view to explain this context. Political censorship factored greatly in what I was allowed and comfortable in sharing. But because I did not make this fact clear, a lot of players felt that they were doing something wrong in not understanding or not having knowledge of this historical time period. I felt that more extensive disclaimers were also needed for this project. One interesting aspect of this project was the e ectiveness of the IoTs in space. Despite not having as much narrative value, the presence of these interactive devices in the space contributed greatly to the novelty of the experience and caused many players to feel attracted to the experience in spite of their frustration with the technology. On the other hand, the AR was less e ective than I expected. Although AR did also contribute greatly to the novelty of the experience, it was also the cause of most player’s frustrations in not being able to progress correctly or at all. More extensive tutorials would be added after this observation. Remote playtesting for this experience worked better than I thought it would. Although players were unable to see or hear the interactive installation, the general narrative through AR was quite clear. However, in-person playtesting did seem to generate more accurate feedback. One major issue with remote playtesting was 24 that players were reluctant to interact with the UI buttons in the app and ‘physically’ walk around the space. Because players had to ‘puppeteer’ another person who was physically in the space in order to play the experience, they were more conservative in their actions and only did the bare minimum to progress through the experience. Another interesting observation between the two was like remote playtesters tended to understand the narrative faster than in-person playtesters. Because remote players only did the bare minimum to progress through the game, they did not interact with any additional information in the physical space, and thus were less easily distracted. On the other hand, in-person playtesters were faster at learning how to play as they were more likely to tap buttons on the app and move around the installation space. However, this led them to draw connections that were not intended, and thus these players were more likely to become lost in the narrative or confused. 9 . 2 S e c o n d P l a y t e s t R e s u l t s The goal of these playtests was to evaluate the e ectiveness of the installation and the UI experience in guiding the player through this experience. When it came to the installation, most players enjoyed interacting and experiencing the inside space of the installation. The spatial arrangement of props and image anchors in the installation space seemed to contribute to the immersion of the experience. Players entering the space felt like they were physically entering a new world, and the arrangement of furniture in the space helped create a more intimate and personal feeling. The historical content arranged on the front of the installation wall helped some players by informing them why certain physical objects like the vinyl records and the record player were included in the space. For these players the content described on the wall helped explain the world and characters of the narrative. For other players, this installation wall was confusing because the layout and organization of the papers was not clear enough. For these players, they chose to only read parts of the content, and thus, felt like they were missing narrative exposition. Going forward, more visual guidance – titles, visual formatting – will be included in order to make the historical exposition process less painful. When it came to UI, most players found the UI helpful in guiding them through the mechanics and interactions in the experience. The journal view where players were 25 able to review photos they’ve taken of AR objects was helpful in reminding them where they were in the narrative. UI feedback mainly consisted of desires for higher fidelity photos (i.e. photos where the AR objects would be more visible) and better player feedback when it came to collecting photos of AR objects. Some players felt that the UI needed better visual theming and the journal or scrapbooking motifs were not strong enough. For these players, it was important that the UI visuals were more meaningful and helped support the journaling/exploratory themes of this project. 26 C h a p t e r 1 0 : C o n c l u s i o n This project was a collection of experiments that sought to evaluate to what extent AR can enhance narratives. Exploration into AR technology, AR narratives, and interactive installation has helped provide the following answers to my thesis question and objectives. 1 0 . 1 A n s w e r i n g t h e T h e s i s Q u e s t i o n Can AR enhance linear narratives? To answer my thesis question, AR as a technology does not enhance, but rather changes how players understand traditional, linear narratives. In some respect, AR technology hinders the overall narrative experience. The complexity and novelty of the technology meant that players had a more dicult time interacting with the experience and certain design choices were made just to accommodate for this learning curve. Overall clarity of the experience was diminished as some players found the technology was too frustrating. However, because of AR, a nodular narrative structure was developed, where the camera and player gazes focus in on specific details during the experience. In combination with the interactive installation, this experience created a new type of increased gaze, one that allows players to focus on specific details or motifs in the narrative (Biggio). Visuals and information in AR space are combined with the physical world, but point of view is limited to that of the viewing subject. What visuals AR materializes and where they materialize impacts where the viewer looks and what viewers engage with. More so than the film camera, the AR camera allows players to zoom in on visual representations of the narrative. Subtle information like plants and sparrows (in the case of this piece) become main characters and plot driving devices. Emphasizing details or motifs allow emotions or information tied to these objects to become more pronounced. In a way, the dramatic impact of the narrative is now felt whenever these objects appear visually or auditorily. If the motifs themselves are not interesting or meaningful (in the context of the larger narrative) then there is little AR can do to elevate the quality of the narrative. As such in order to support this kind of narrative, all other game elements should try to revolve around these motifs. Mechanics tied to AR must try to tie the AR 27 objects to the story. The physical space needs to house or connect to the AR world in some way. For these kinds of linear narrative experiences, random public spaces or procedural narrative restructures would detach from the narrative. 1 0 . 2 A n a l y z i n g t h e O b j e c t i v e s When it comes to enhancing narrative through hybrid space, the short answer is yes, it is possible. However this possibility ultimately depends on the emotional strength of narrative and user’s technology fluency. The narrative needs to contain strong emotions, not just a strong plot or premise. Emotions are the most important aspect to these narratives because while information is easily lost in the media, primal feelings are conveyed quite clearly. In most commercial AR games, physical navigation mechanics are less dependent on technology and more so on the user’s experience in interactive spaces (i.e. themed attractions, escape rooms and museums). Outside of this genre, players should have some level of AR fluency. This would help lessen the initial frustration felt when engaging with the medium. Another method would be to create an interactive installation space that would compensate for the UX faults of AR. Unlike what I initially assumed, spatial UI works best when implemented in the installation space rather than in AR. 2D UI elements in AR clutter the screen, while tutorial guides for using the app in AR are hard for players to understand and imagine, especially when they have never used AR or XR before. In contrast, installation UX and UI work slightly better as more players are fluent in themed amusement designs and/or escape room guides. Players are more likely to navigate and hunt for clues or signage in a physical space because they are used to this kind of interaction. Similarly, players are also more likely to draw conjectures based on the physical visuals, colour and textures, in a space, as most physical signs or clues are pictorial. 1 0 . 3 G o i n g F o r w a r d AR’s greatest advantage (when compared to other current mixed reality mediums) is its ability to integrate with real, physical space. Hybrid spaces can greatly contribute to the e ectiveness of AR and help lessen a lot of the player’s pain points. Even simple ways of altering the lighting or sound in the space can enhance the novelty of the experience and increase player engagement. 28 What AR lacks (at this point in time) is a clear set of UX and design principles. A lot of the technology that I have attempted to use in order detracted from the linear narrative experience. I think the greatest issue is that commercial mobile AR as a technology is a nodular way of presenting information. And linear narrative is the way most people are trained to understand stories. Unlike VR and other narrative media, AR as a medium is not linear, and that in itself means that future designers need to find ways to make the navigation from narrative beat to narrative beat easier. I should also note that at this point in time AR technology is still new for most new media audiences. As time goes on, the general public will become more familiar with this medium and this will help lessen a lot of the diculties I experienced as a designer during this project. The technology is also evolving, and it is very possible that mobile AR will not be the dominant form of AR in the future. The narrative of this project was also chosen and designed specifically for the current state of AR and hybrid installations, so a lot of the conjunctures I made during this thesis also might not hold true. All in all, Silent Sparrows was a successful exploration into AR’s narrative potential. While AR is still a developing medium, its ability to unify both the virtual and real spaces has the potential to give narratives greater emotional impact. It was interesting to see how far I could push a relatively new medium and use it in ways that enhance storytelling. I look forward to how other designers might choose to take this medium forward. 29 C h a p t e r 1 1 : G l o s s a r y T r a n s c u l tu r a l m e d i a refers to media content that transcends cultural and geographic boundaries, incorporating various cultural elements from di erent regions and appealing to audiences from di erent backgrounds. C u lt u r a l d i a s p o r a refers to the dispersal or migration of a particular culture or group of people from their original homeland or cultural center to other parts of the world. This diaspora can lead to the creation of new cultural identities and traditions as well as the preservation of existing ones. A u g m e n t ed R ea lity ( A R ) is an interactive technology that overlays digital information or virtual objects onto the real world through a device, such as a smartphone, tablet, or headset. AR enhances the user's perception of reality by combining the physical and digital worlds. Io T ( I n t e r n e t o f T h in g s ) refers to a network of physical objects that are embedded with sensors, software, and other technologies that allow them to connect and exchange data with other devices and systems over the internet. In s t a lla t i o n refers to a form of art that involves creating an immersive, three-dimensional environment or experience that engages the viewer's senses and emotions. N o n - lin e a r N a rr a t i v e refers to a type of storytelling that does not follow a linear, chronological sequence of events. Instead, the narrative may jump back and forth in time or use di erent perspectives to explore multiple storylines simultaneously. B r a n c h in g N ar r a tiv e refers to a type of narrative structure that allows the audience to make choices that a ect the outcome of the story. C in e m a t ic G a z e refers to the use of film techniques, such as framing, lighting, and camera angles, to manipulate the audience's perception and interpretation of a scene or character. C h o i c e ( S itu a t io na l G a m e D es i g n ) refers to the design of video games that o er players a range of choices that a ect the gameplay experience. These choices may be situational, meaning that they are presented to the player in response to a specific situation or context within the game. N a rr a t iv e refers to a sequence of events or experiences that are presented in a coherent and meaningful way, often with the purpose of telling a story or conveying a message. A narrative can take many forms, including written text, film, theater, or video games. A m b ig u o u s s to ry te ll in g in augmented reality (AR) involves creating uncertainty or multiple meanings, allowing users to actively engage with the AR or new media environment and construct their own narrative experience. Stories of this kind often include open-ended plotlines, or fragmented information across di erent media platforms 30 R e f e r e n c e s 1. "Hybrid culture Japanese media arts in dialogue with the West" by Yvonne Spielmann 2. Zhishu Xiao Cexiang Huabao (2019), http:/ /www.zhscxh.com/view/common/articledetail.aspx?id=5F88951E45BA B190. 3. "Expanded Cinema" by Gene Youngblood. 4. "Planetariums and the rise of spectacular science" by Alexandra Witze, Nature, vol. 551, no. 7680, 2017, pp. 291-293, https:/ /www.nature.com/articles/d41586-017-08441-9. 5. "Navigate the Heavens from the Comfort of Your Seat" by Josh Walawender, Harvard University Center for Astrophysics, 19 Apr. 2019, https:/ /www.cfa.harvard.edu/news/navigate-heavens-take-seat. 6. "Qi Cheng Zhuan Jie: The Chinese Four-Act Screenplay Structure, Part 2" by Marilyn Horowitz, Script Magazine, 17 Jul. 2014, https:/ /scriptmag.com/screenplays/qi-cheng-zhuan-jie-the-chinese-four-a ct-screenplay-structure-part-2. 7. "See Carne y Arena's Ocial Video" by ILMxLAB, 28 Feb. 2018, https:/ /www.ilmxlab.com/news/see-carne-y-arenas-ocial-video/. 8. "Be There 1942" by Japanese American National Museum, https:/ /www.janm.org/exhibits/behere1942. 9. "Dream of the Red Chamber." Encyclopedia Britannica. Encyclopædia Britannica, Inc., n.d. Web. DATE ACCESSED. https:/ /www.britannica.com/topic/Dream-of-the-Red-Chamber. 10. "The Hero with a Thousand Faces." TV Tropes, n.d. Web. DATE ACCESSED. https:/ /tvtropes.org/pmwiki/pmwiki.php/Literature/TheHeroWithAThousand Faces. 11. Fallen Angels. Directed by Wong Kar-wai, performances by Leon Lai and Michelle Reis, Criterion Collection, 2020. 12. Wang, Jianguo. "Interactive Storytelling in Games: A Survey and Research Agenda." In: Advances in Computer Entertainment Technology. Eds. 31 Rongfang Bie et al., Springer, 2020, pp. 1-14. doi: 10.1007/978-981-15-2337-3_1. 13. Rollings, Andrew and Ernest Adams. Fundamentals of Game Design. Prentice Hall, 2006. 14. BeHere/1942. The augmented reality experience of Japanese American incarceration, n.d. Web. DATE ACCESSED. <URL>. 15. Fragments. Developed by Asobo Studio, 2019. 16. Robertson, Campbell. "Canadian Military Issues Pokémon Go Warning: 'We Need You to Pokémon No!'" The New York Times, 1 Jan. 2020, https:/ /www.nytimes.com/2020/01/01/world/canada/pokemon-go-canada- military.html. 17. Sun Wen, Red Chamber 2, Ink and color on paper, ca. 1750, Wikimedia Commons, https:/ /commons.wikimedia.org/wiki/File:Sun_Wen_Red_Chamber_2.jpg. 18. Tapper, James. "Branching Narratives: Are They the Future of Theme Park Storytelling?" Blooloop, 6 Apr. 2021, https:/ /blooloop.com/theme-park/opinion/multi-branching-narrative/. 19. Debruge, Peter. "Alejandro G. Inarritu's 'Carne y Arena' Virtual Reality Installation Wows Cannes." Variety, 18 May 2017, https:/ /variety.com/2017/film/spotlight/carne-y-arena-venice-virtual-reality -cannes-1202524881/. 20.Upton, Brian. Situational Game Design. Addison-Wesley Professional, 2004. 21. "Kishotenketsu: Exploring the Four-Act Story Structure." Art of Narrative, 8 July 2020, https:/ /artofnarrative.com/2020/07/08/kishotenketsu-exploring-the-four-a ct-story-structure/. 32

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Untitled: an exploration of virtual reality as a cinematic medium

PDF

Developing a playful situational creator for mixed reality: design analysis on Neon City, a city building game

PDF

The mountain calls: an exploration of immersive storytelling through art and level design

PDF

The extension of our reality

PDF

Penrose Station: an exploration of presence, immersion, player identity, and the unsilent protagonist

PDF

Garden designing a creative experience with art and music orchestra

PDF

One little change: exploring relationship abuse in a virtual reality narrative

PDF

Past imagination, present creation and the reality of tomorrow: explore the impact of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) design

PDF

Last broadcast: making meaning out of the mundane

PDF

Virtual production for virtual reality: possibilities for immersive storytelling

PDF

Project stranger

PDF

Morana: explore healing potential of virtual reality storytelling

PDF

Grandma Green: kindness in bloom

PDF

The Darkest Plague: insights on the development of an action adventure game

PDF

Metamorphosis: Thyota's journey - embedding environmental themes into fantastical elements in games

PDF

That's not how it happened: unreliable narration through interactivity

PDF

A monster's hearth: seeing yourself through monsters

PDF

On virtual, augmented, and mixed reality for socially assistive robotics

PDF

Lake Minnewaska | uplifting a Black narrative in an underrepresented industry | exploring grief themes in a fishing video game

PDF

Hedge hug -- a narrative-driven interactive exploration on low self-esteem and social anxiety

Asset Metadata

Creator Ye, Christy (author)

Core Title Silent sparrows: exploration into augmented reality narrative frameworks

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Degree Conferral Date 2023-05

Publication Date 04/14/2023

Defense Date 04/13/2023

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag ambiguous storytelling,AR,augmented reality,expanded cinema,frameworks,installation,Interactive Media,Internet of Things,IoT,mixed reality,mobile AR,narrative,new media,OAI-PMH Harvest,physical,smart devices,transcultural,virtual,XR

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bilson, Danny (committee chair), Bolas, Mark (committee member), Fullerton, Tracy (committee member)

Creator Email christyspace1731@gmail.com,yechrist@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC113015431

Unique identifier UC113015431

Identifier etd-YeChristy-11623.pdf (filename)

Legacy Identifier etd-YeChristy-11623

Document Type Thesis

Format theses (aat)

Rights Ye, Christy

Internet Media Type application/pdf

Type texts

Source 20230414-usctheses-batch-1022 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu