University of Southern California Dissertations and Theses

Untitled: an exploration of virtual reality as a cinematic medium

(USC Thesis Other)

Untitled: an exploration of virtual reality as a cinematic medium

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Untitled An Exploration of Virtual Reality as a Cinematic Medium by Sangyup Kim A Thesis Presented to the A FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree MASTER OF FINE ARTS (INTERACTIVE MEDIA) May 2022 Copyright 2022 Sangyup Kim Acknowledgments Nothing I have done through my three-year journey at USC’s IMGD program could have been possible without the following people, and I would like to take this opportunity to thank them: My loving family who have always been there for me USC IMGD Faculty - Richard Lemarchand, Martzi Campos, Laird Malamed, Jesse Vigil, Tracy Fullerton, Andreas Kratky, Sean Bloom USC IMGD MFA Class of 2022 - Meny Menczel, Michelle Ma, Seb Baracaldo, Taylor Dinwiddie, Kathryn Yu, Abby Sherlock, Zian Zhang, Claire Hu, Olivia Peace, Yang Cao, Izzy Jones, Cloud Tian, Maddie Emery, and JP Jupiter A dear friend David Wu And last but not least, my teammates - Seleny Xie, Mike Wahba, Simon Lee, Alisina Saee-Nazari, Jerry Tang, Jackson Roberts, and Yujia Shan iii Table of Contents Acknowledgments ............................................................................................................................... iii List of Figures ......................................................................................................................................... vi Abstract ........................................................................................................................................vii Chapter 1: Introduction … … … … … … … … … … … ..........................................................................................1 1.2 The State of VR as a Cinematic Medium ................................................................. 3 1.2 Thesis Question............................................................................................................. 7 1.3 Project Overview........................................................................................................... 8 1.4 Design Goals & Player Experience Goals..................................................................8 1.5 Prior Art / Inspirations..................................................................................................9 1.6 Unique Technology & Features..................................................................................11 Chapter 2: Development........................................................................................................................11 2.1 Original Idea ...................................................................................................................11 2.2 Complications & Mitigation ........................................................................................12 Chapter 3: Production … … … … … … … … … … … … ......................................................................................14 3.1 Scope & Limitations……................................................................................................14 3.2 Controls & Mechanics .………………………………..................................................................14 3.3 Art & VFX………………………..................................................................................................15 3.4 SFX & Music ..…………………………………………………..................................................................16 3.5 Playtesting …………..........................................................................................................17 3.6 Future Plans ………..........................................................................................................18 Bibliography …………..................................................................................................................................19 iv List of Figures 1 Lumiere Brothers’ Workers Leaving the Factory .................................................. 1 2 Marketing Images from the VOID VR ……………………….................................................. 4 3 Marketing Images from SandboxVR ………………………................................................... 5 4 Marketing Images for Monster Hunter, Final Fantasy, and Evangelion XR from Universal Studios Japan ......................................................................................................... 6 5 Screenshots from Half-Life Alyx and Vader Immortal ......................................... 7 6 TeamLab Tokyo’s Immersive Installation ……………................................................... 9 7 Multiple Scenes from Marvel’s Dr. Strange ...........................................................10 8 Screenshots from the lost scenes …………………………..................................................12 9 Screenshots of the hard drive fail …………………………................................................. 13 10 Single gesture test ……………………………………………………….................................................14 11 Multiple gesture test .................................................................................................14 12 Screenshots from the work-in-progress scenes .................................................15 vi Abstract UNTITLED is a gesture-based, reclined, interactive virtual reality experience that explores virtual reality as a cinematic medium. It is an experiential canvas that aims to experiment and explore the unexplored expressionistic potential of virtual reality through a series of larger-than-life fantastical scenes. This paper will cover the process of translating the principles of cinematic grammar widely used in motion pictures and applying them to virtual reality’s immersive, interactive, experiential format. It is worth noting that the idea has gone through two technical failures that demanded adjustments in the project’s scope and some of its elements. The paper will also address the complications and the measures taken to mitigate those complications. vii Chapter 1: Introduction Since the Lumiere Brothers’ first film debuted 127 years ago, the motion picture has gone from a simple 1 min documentary featuring workers leaving the factory, shot at 16 frames per second, to the epitome of the most comprehensive art form to date. On the other hand, virtual reality is a fledgling technology that still has many hurdles to overcome before it can be a universally mainstream form of media. Figure 1: Lumiere Brothers’ Workers Leaving the Factory Since the launch of the first large-scale consumer-targeted virtual reality devices, such as the Oculus Rift and HTC Vive, there have been numerous eorts to explore the untapped potential of virtual reality as a new form of entertainment. As my background centers around film and television production, my main interest has been finding ways to apply some of the established cinematic grammar to the immersive medium of virtual reality. To do so, the term ‘cinematic’, foremost, needs to be defined. 1 Definition of ‘Cinematic’ The term ‘cinematic’ has mostly been referred to in the world of passive visual mediums. However, with the introduction of interactive mediums such as video games and virtual reality, cinematic grammar is making its way across every other visual medium. The Merriam-Webster’s dictionary defines cinematic as: 1. Of, relating to, suggestive of, or suitable for motion pictures or the filming of motion pictures 2. Filmed and presented as a motion picture Cinematic, by definition, could be used to describe any element related to motion pictures, whether it be a narrative element, the visuals, the audio, the movement of the camera, the editing, the framing, or even a brief moment of media content. What, then, is expected when we describe something to be cinematic? What lies at the tangent of all these elements that would make something cinematic? Furthermore, with interactive mediums such as video games and virtual reality, can other forms of media also be ‘cinematic?’ Kara Keeling proposed that the public has processed film grammar and language to the extent that they view the world in relation to film: “Processes of ‘cinematic perfection’ enable us to consume cinematic images wherever and however such images are found. Cinematic perception is not confined to interactions with moving-image media such as film and television. Involved in the production and reproduction of social reality itself, these perceptual and cognitive processes work to order, orchestrate, produce and reproduce social reality and sociality… Neither cinematic perceptual schemas nor cinematic matter precedes the other. Together, they constitute the cinematic, an assemblage that might also be 2 referred to as “twentieth-century reality” because we neither posit nor access “reality” except via these processes, which were perfected by film (Keeling, 2007).” In essence, ‘cinematic’ becomes a ‘feeling’ that can be applied to multiple mediums. Deborah Jaramillo states that “a cinematic movie is one that requires a theatrical viewing in order to extract every ounce of visual and aural depth from it. The term is reserved for films that reveal the exploitation of filmmaking technologies in the service of skill and creative vision (Jaramillo, 2021) In conclusion, ‘cinematic’ is not just about conveying information but delivering a compelling visual, aural, or a combined experience with a clearly defined purpose, strong intent, and elaborate aesthetics to eectively immerse the audience in your creative vision. 1.1 The State of Virtual Reality as a ‘Cinematic’ Medium 1. Location-Based Virtual Reality Experience Location-Based Virtual Reality (LBVR) is a place of business where virtual reality devices are deployed, allowing users to experience virtual reality by physically interacting with the environment in a way they cannot in their own homes. Similar to how video game arcades came before consumer-targeted consoles became prevalent or how theatrical experiences came before home entertainment became more accessible, LBVR serves not only as a stepping stone for virtual reality entertainment to become more mainstream but also as a great showcase of what this immersive format could be in the years to come. 3 The VOID Figure 2. Marketing Images from the VOID VR Utilizing popular Disney IPs such as Star Wars and Avengers, the VOID provides a multiplayer co-op experience that focuses on the cinematic experience with a light game element. The VOID also utilizes physical props and sets for 1:1 translation of the virtual and the physical elements in the content, as well as fans, smell machines, and haptic vests for the most immersive multisensory virtual reality content that I have yet experienced. It, however, lacks replayability as there is no real challenge nor a sense of agency in its game elements. For example, in the Avengers experience, when the robots attack the getaway ship, the experience will continue as planned regardless of how many robots you destroy. 4 SandBoxVR Figure 3. Marketing image from SandboxVR SandboxVR takes a dierent, perhaps a directly opposing, approach to the VOID. While it shares some of the characteristics of the VOID (e.g., use of fans and haptic vests, room-scale multiplayer co-op experience), SandboxVR focuses heavily on the game element. There is a clear objective, to survive through a wave of enemies in a confined space, and going below a certain performance threshold could end the experience within a few minutes. Therefore, there is a much heavier emphasis on the social element and a stronger sense of agency as the team must constantly communicate with each other to hold o a certain number of enemies eectively. With fewer hardware setups implemented to enhance immersion, was SandboxVR’s content less immersive than the VOID? For me, the answer is no. Both contents had big battle scenes where the players needed to use their weapons to survive. Still, the contents from SandboxVR had agency with direct consequences for failing the objective, which was conducive to the experience, whereas the VOID’s content did not. 5 Universal Studios Japan Figure 4. Marketing images for Monster Hunter, Final Fantasy, and Evangelion XR from Universal Studios Japan Converting conventional theme park rides into virtual reality-enhanced rides has been a recent trend, and Universal Studios Japan was one of the early players that have started this trend. By doing so, they were able to cut the production and maintenance costs of the rides while opening up limitless creative possibilities. There are no in-experience interactions implemented for the players for safety concerns. Still, the combination of thrill rides with the audiovisual feeds that match the movement of the rides provides an unparalleled passive cinematic virtual reality experience that genuinely makes it unique. 6 2. Home/Personal VR Figure 5. Screenshots from Half-Life Alyx and Vader Immortal As the consumer virtual reality market has been rapidly growing in its size, there have been multiple attempts at creating what could be described as a “AAA” virtual reality game that could be enjoyed at the convenience of one’s own home. Half-life Alyx (Valve, 2020) and Vader Immortal (ILMxLab, 2019) are probably the two that showcased what the future of home virtual reality entertainment holds with their stunning visuals and compelling audio. Even though they did not have the peripherals that enhanced the immersion, nor physical set and props as a virtual counterpart, the audiovisual experience was nothing short of what I have experienced from a location-based virtual reality entertainment. Additionally, the stronger emphasis on what players could do (interactions) combined with clear objectives and a fail-state provided more substantial immersion than the location-based virtual reality contents. 1.2 Thesis Question The project seeks to explore the question, ‘How can we take the concept of cinematic language and apply it to the medium of virtual reality to create an ‘immersive cinematic experience?’ 7 1.3 Project Overview ● Logline: A journey from the depths of the subconscious to the surface of reality. ● Platform: PCVR ● Genre: Interactive virtual reality experience, fantasy, virtual installation art experience ● Target Audience: Fans of immersive installations and surrealist/fantastical experiences, people who are just entering the virtual reality community to virtual reality fanatics (under the age of 13) 1.4 Design Goals & Player Experience Goals Design goals: ● Immersive ‘cinematic’ experience ● Gesture-based controls for added immersion - conducive to the experience of wielding superpowers ● Surrealist/fantastical scenes and interactions ● Reclined experience Player experience goals: ● Empowered & fulfilled - using no controllers, but hand gestures, to control the environment will provide a sensation of using supernatural abilities. Provided that we, as consumers, crave content that could satisfy our desire for whatever we had (or have) been deprived of, this experience should satiate the child-like sensibility of anyone who ever dreamed of having superhuman powers. 8 1.5 Prior Art / Inspirations Untitled derives its inspiration from an immersive installation and films and a music video with surreal visuals. Visuals & Interactions: TeamLab’s immersive installation (https://www.youtube.com/watch?v=IExpZvkjqlI&t=88s&ab_channel=teamLab) Immersive installations use multiple projectors to turn a designated space into a canvas that presents an interactive, immersive audiovisual experience. It simulates a similar feeling that one would expect to experience while being immersed in a virtual reality space without needing any head-mounted displays. The use of abstract images and visual effects mixed with a matching soundscape to invoke a certain mood/emotion in immersive installations could be applied to represent different sections of the fragmented memory/subconscious. Figure 6. TeamLab Tokyo’s Immersive Installation 9 Dr. Strange (https://www.youtube.com/watch?v=7GR-9mxLzcs&ab_channel=ClipsyBox) ● Applicable elements: ○ A mix of unique arrangements of the familiars with abstract & fantastical environments & effects to create a magical, surrealist world. ○ Use of hand gestures to control and manipulate time and the environment. Figure 7. Multiple scenes from Marvel’s Dr. Strange Music / SFX ‘Fragment II’ - Johann Johannsson (https://www.youtube.com/watch?v=0ilKM1JNrBs) ● Applicable elements: Mix of atmospheric sound, abstract sound, and music to create a dream-like atmosphere that invokes a strong emotional reaction ‘Bad Guy’ loop station cover - Henry Lau (https://www.youtube.com/watch?v=YpRywDm3nRs) ● Applicable elements: Use of recognizable and familiar sounds as part of the music - use of sounds from the ‘real world’ as part of the music (ex. Hans Zimmer’s ‘Dream is Collapsing’) 10 1.6 Technology & Unique Features ● Gesture-based controls via gesture recognition ● Fully reclined experience Chapter 2: Development 2.1 Original Idea The original concept for the project was a narrative puzzle game where the players play the role of a man who goes through a mysterious journey to investigate the identity of a severely damaged body, only to realize that the body is his own and that he is stuck between reality and death. The players would have solved a series of puzzles to learn the backstory and why the character must live on to ‘get back to reality.’ The idea had initially been developed for film, so I thought it would be interesting to try to translate content designed for a film into a first-person virtual reality experience. Unfortunately, there were two major technical failures that made it inevitable to adjust both the project’s content and scope. The following section will elaborate on the specifics of the complications and what changes were made to address those complications. 11 2.2 Complications & Mitigation I started working on prototype scenes during the winter of 2020 and made around 15 unique scenes by the summer of 2021. Figure 8 - Screenshots of the lost scenes 12 However, due to technical issues with Unity’s project file, I had lost all of the progress. After importing several assets into the project file, Unity behaved somewhat erratically. I saved the project and restarted Unity to access the project file fresh. However, everything except the main camera and the assets folder was gone when I opened the project file. I scavenged the internet to find similar cases and for a potential solution to the problem, but to no avail. In retrospect, it was a grave mistake to not use any sort of version control or have a backup drive in case of such a devastating situation, and I had to learn my lesson the hard way. Figure 9 - Screenshot of the hard drive fail Two months into restarting the project, another system failure occurred. This time, my computer randomly froze, forcing me to do a hard restart of the system. Upon restart, I noticed that my workstation (desktop PC) would not go past the POST (Power-On Self-Test) screen. Instead, it would continue ‘diagnosing the PC’ indefinitely, so I tried to use Windows Startup Repair, but the prompt suggested that the drive was corrupted. I tried connecting the drive to an external NVME SSD reader to at least retrieve the data, but the drive was simply inaccessible due to ‘data corruption.’ Finally, after exhausting my options to solve the issue, I had to format the drive and do a fresh install of all the apps, and needless to say, all the data was gone. At this point, I had already lost 1+ year's worth of progress, and with a significant loss of progress and allocated production time, I had to adjust the project’s scope accordingly. 13 As the core of the project was to explore virtual reality as a ‘cinematic’ medium, I decided to focus on the visuals, the mechanics, and the controls, while eliminating the puzzle and the narrative element of the experience. Chapter 2: Production 3.1 Scope & Limitations Scope: 3 scenes with interactions Limitations: ● Small team with 1-person department for every department ● Shorter production timeline due to multiple cases of technical failure ● Performance limitations on the hardware 3.2 Controls & Mechanics Figure 10 - Single Gesture Test Figure 11 - Multiple Gesture Test 14 Controls (Gesture): ● Hands apart ● Twist right hand Mechanics: ● Players would be in a reclining position throughout the experience 3.3 Art & VFX Figure 12 - Screenshots from the work-in-progress scenes ● Keeping a delicate balance between the identifiable and the fantastical to add realism to the surrealistic experience ● Strategic use of contrast, colors, and composition of the visual elements ● Fantastical VFX in larger-than-life scale that engulfs the players ● Animation and eects that corresponds to the interactions in a very satisfying way 15 3.4 SFX & Music Audio plays a paramount role in how the audience perceives the quality of a given experience. MPEG Group’s study on best data rate for the audio portion of the signal - 3 video clips with identical video but varying data rate audio - showed that even though the video was the same on all three clips, viewers rated the video quality in direct proportion to the audio quality (Beerends & Caluwe, 2012) Therefore, great sound is not an option but is an absolute necessity. Sound should not only reflect but enhance the experience. The goal of the SFX and music for the project has been: ● Like the visuals, the soundscape should engulf the audience and elevate the immersion. Without overdoing it, dierent types of sound should come from all spherical directions with varying properties (distance, volume, etc.) that could give the players a sense of scale and realism. ● In order to emphasize the surreal nature of the scenes, the sound should be a good mix of “realistic sound” and a “synthesized sound.” I put quotation marks around the sound types because I realize most sounds that we think are natural sounds are products of excellent foley, a reproduction of everyday sound eects that are added to films, videos and other media in post-prodution to enhance audio work. To clarify what I mean by “realistic” versus “synthesized,” I am referring to a commonly shared perception of the sound by the general public. 16 3.5 Playtesting Playtests have been held since I made my 1st prototype back in 2020. Playtest methodology: ● Tested 11 varying scenes for movement speed comfort level, VFX speed, level of engagement, and interest for the specific scenes across a varying group of audience Playtests: ● Internal playtest (thesis class and team tests) ● External playtest (mix of game non-development virtual reality consumers and first-time players) Findings: ● Most favored a scene with a good mix between the surreal and realistic over the purely fantastical or purely realistic scene ● Most favored a scene with complex layers of visual elements and strong visual contrast ● The second most positively received scene was the “Matrix eect” scene ● Fast movements created jitter that provided discomfort to almost all players, except a few advanced virtual reality users. Even without the jitter, quick movements provided discomfort to more than half of the players. ● Rotating the player during movement, most felt nauseated. Removing the rotation but keeping the speed improved the comfort level by a significant amount. ● Most players found scenes with bent laws of physics to be the most interesting (ex. slow motion, being teleported through a portal) 17 3.6 Future Plans ● Plan to finish: 1. Music for each set 2. “3D audio design” 3. UI design ● If time and budget allowed: 1. Alternative control tests - Voice, brain signal (a device like NextMind, but preferably something with more accuracy, less latency, and no rapidly blinking lights for ‘interactables’) 2. Multiplayer experience 18 Bibliography Keeling, K. (2007). The Witch’s Flight: The Cinematic, the Black Femme, and the Image of Common Sense, Duke University Press Jaramillo, Deborah L. “Rescuing Television from ‘the Cinematic’: The Perils of Dismissing Television Style.” Television Aesthetics and Style. Ed. Jason Jacobs and Steven Peacock. New York: Bloomsbury Academic, 2013. 67. Bloomsbury Collections. Web. 20 Aug. 2021. <http:/ /dx.doi.org/10.5040/9781628928327.ch-004>. The Influence of Video Quality on Perceived Audio Quality and Vice Versa, 2012, Beerends & Caluwe 19

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Penrose Station: an exploration of presence, immersion, player identity, and the unsilent protagonist

PDF

Developing a playful situational creator for mixed reality: design analysis on Neon City, a city building game

PDF

The extension of our reality

PDF

Virtual production for virtual reality: possibilities for immersive storytelling

PDF

Sentenced an interactive exploration of compassionate communication through a black lens

PDF

Shepherds - cooperating from divergent path: a design exploration

PDF

Morana: explore healing potential of virtual reality storytelling

PDF

Silent sparrows: exploration into augmented reality narrative frameworks

PDF

Wandering light: an interactive visual novel

PDF

The Darkest Plague: insights on the development of an action adventure game

PDF

The mountain calls: an exploration of immersive storytelling through art and level design

PDF

Inclusive character creator: an exploration of inclusive design for 3D character creators

PDF

Oopz-Oofs: the creation of a dog simulator game

PDF

That's not how it happened: unreliable narration through interactivity

PDF

“Mystical” VR: an experimental VR experience exploring concepts and symbolism of ancient Andean worldview based on the symbol ""La Chakana""

PDF

A game called Paako: the challenge of an autobiographical video game

PDF

Hearth and home: interior design as tool and weapon in creating an atmosphere of dread

PDF

Against reality: AI co-creation as a powerful new programming tool

PDF

Garden designing a creative experience with art and music orchestra

PDF

BeachcomberVR: inspiring exploration: concepts in the production of a virtual reality game

Asset Metadata

Creator Kim, Sangyup (author)

Core Title Untitled: an exploration of virtual reality as a cinematic medium

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Degree Conferral Date 2022-05

Publication Date 04/19/2022

Defense Date 04/18/2022

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag cinematic,cinematic virtual reality,OAI-PMH Harvest,virtual reality,virtual reality experience

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Nealen, Andy (committee chair), Bilson, Danny (committee member), Lemarchand, Richard (committee member)

Creator Email catharsis9999@gmail.com,sangyupk@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC111014557

Unique identifier UC111014557

Document Type Thesis

Format application/pdf (imt)

Rights Kim, Sangyup

Type texts

Source 20220419-usctheses-batch-929 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu