University of Southern California Dissertations and Theses

Virtual production for virtual reality: possibilities for immersive storytelling

(USC Thesis Other)

Virtual production for virtual reality: possibilities for immersive storytelling

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content VIRTUAL PRODUCTION FOR VIRTUAL REALITY
POSSIBILITIES FOR IMMERSIVE STORYTELLING
by
Nathan Fairchild
A Thesis Presented to the
FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
MASTER OF FINE ARTS
(INTERACTIVE MEDIA)
May 2024
Copyright © 2024 Nathan Fairchild

Acknowledgments
I owe a huge thanks to the wonderful group of people who were such an invaluable help on this
project: producers Akaash Tumuluri, Karolina Esqueda Rocha, and Zane Swift, technical director Jordan
Halsey, Casey Landman in the role of Flora, Christopher Bowles as the voice of AIDEN, directors of
photography Eva Wu and Max Huang, gaffers Jay Caidan, Herman Ha, and Wayne Zhang, environment
artist Miru Jun, engineer Ethan Tat, script supervisor Zain Shukr, technical consultant Shaoyu Su,
narrative consultant Andrew Kovac, production sound recordists Abe Akinsete and Jinrun Han,
production assistants Xander Beaky, Christian Cuevas, Lilleth Glimcher, and Brandon Walsh, faculty
advisors Mark Bolas, Scott Fisher, and Bruce Block, and Laird Malamed and Martzi Campos for their
continuous support through this process.
ii

Table of Contents
Acknowledgements…………………………………………………………………………………………………….……………………………ii
List of Figures……………………………………………………………………………………………………..………………………………….iv
Abstract.………………………………………………………………………………………………………………………………………….……….v
Chapter 1: Introduction………..……………………………………………………………………………………………………………….1
1.1 Contemporary VR Workflows and Their Shortcomings.……………………………………………………. 1
1.2 Virtual Production – For Virtual Reality?…………………………………………………………………………..2
1.3 Help Is On the Way…………………………………………………………………………………………………………...3
1.4 Prior ‘Hybrid’ VR Efforts………………………………………………………………………………………………….5
1.5 Project Goals…………………………………………………………………………………………………………………….9
Chapter 2: Creating Help Is On the Way – Technical Challenges………………………………………………………10
2.1 Core Compositing Approach.………………………………………………………………………………………… 10
2.2 Approaching Depth………………………………………………………………………………………………………..12
2.3 Viewing Angles and Billboarding……………………………………………………………………………………..14
2.4 Character Movement and Interaction with World …………………………………………………………….16
2.5 Lighting …………………………………………………………………………………………………………………………16
Chapter 3: Creating Help Is On the Way – Artistic & Formal Explorations……………………………………..20
3.1 Approaching the VR Camera .…………………………………………………………………………………………20
3.2 Approaching Editing ………………………………………………………………………………………………………27
3.3 Directing Audience Attention …………………………………………………………………………………………30
Chapter 4: Conclusions ………………………………………………………………………………………………………………………..34
4.1 User Testing – Did the Project Work for Audiences?...………………………………………………………34
4.2 Overall Hurdles and Successes………………………………………………………………………………………….36
4.3 Limitations of Our Approach ………………………………………………………………………………………….36
4.4 Takeaways ………………………………………………………………………………………………………………………37
References………………………………………………………………………………………………………………………………………………39
iii

List of Figures
1 Home After War: Character lighting versus world lighting……………………………………………………..………..6
2 On the Morning You Wake……………………………………………………………………………………………………………..7
3 The white treadmill……………………………………………………………………………………………………………………..12
4 Our first “four pack”...................................................................................................................................13
5 Billboarding implementation……………………………………………………………………………………………………….15
6 First lighting setup………………………………………………………………………………………………………………………17
7 Final lighting setup (red, green, blue).........................................................................................................19
8 Camera Setup 1: Alongside protagonist………………………………………………………………………………………..22
9 Camera Setup 2: Face to face………………………………………………………………………………………………………..23
10 Rejected camera setup…………………………………………………………………………………………………………………24
11 Camera Setup 3: Villain POV………………………………………………………………………………………………………25
12 Villain POV vignette…………………………………………………………………………………………………………………..26
13 First environment concept image…………………………………………………………………………………………………31
14 Final environment lighting………………………………………………………………………………………………………….32
iv

Abstract
‘Help Is On the Way’ is a project that leverages virtual production techniques to demonstrate
possibilities for live-action virtual reality (VR) storytelling. Our team has worked to combine key strengths
of live-action filmmaking (viz. the realism and intimacy of human characters and performance) with key
strengths of real-time engines (viz. possibilities for six degrees of freedom immersion, and an open-ended
potential for world-building). This project seeks to show that combining the traditionally disparate
approaches of cinematic virtual reality with real-time engine-based virtual reality is a path worth
considering for advancing the field of immersive storytelling. Artistically, attention has been given toward
the affordances that result from combining these traditionally separate workflows. Technically, particular
consideration has been given toward how to accomplish virtual production for VR on a very limited
budget, a natural outgrowth of the constraints our team faced on this project.
v

Chapter 1: Introduction
1.1 Contemporary VR Workflows and Their Shortcomings
Much of the initial inspiration for this project came from my frustrations with the current state of
virtual reality (VR) creation. Over the last ten years, there have been two dominant approaches to making
VR experiences: live-action 360/180 videos, and game engine-based VR experiences. Filmmakers have
generally pursued 360-degree live-action material, while engine-based approaches have mostly been used
for gaming, albeit with some exceptions for cinematic storytelling. But a common paradigm for
immersive filmmaking has proven elusive, due in part to the respective limitations of these two production
approaches.
360 and 180 degree videos benefit from photorealism, a relatively simple workflow, and fast
production pace. Examples of this include the works of Felix & Paul Studios and videos posted to
YouTube 360. While the photorealism and intimacy of capturing real people, places, and things is
immersive video’s primary calling card, these strengths are dampened by strong limitations in a head
mounted display (HMD)-based context. Both 360 and 180 degree formats can be either monoscopic or
stereoscopic, but neither format offers viewpoint-dependent parallax – the six degrees of freedom (6DoF)
rendering that is physiologically essential for a viewer in an HMD to get a feeling of presence in a virtual
world (Slater, 2022). In terms of production workflow, these formats also suffer from limited
post-production flexibility relative to engine-based experiences, and similarly less opportunities for
stylized world-building. However, the biggest drawback of 360 and 180 videos remains their lack of six
degrees of freedom, which alone means that these are not the ideal format for a VR headset.
Game-engine based experiences have a different set of strengths and limitations. Experiences
made in a real-time engine offer full view-dependent 3D rendering, ideal for a VR headset. Examples of
1

this approach include popular games Beat Saber and Half Life Alyx. Their 6DoF rendering allows for
room-scale experiences, and these engine experiences also offer possibilities for a huge variety of
programmatic interactivity. But these experiences carry their own limitations. To achieve high levels of
interactivity, game engine experiences generally contain completely synthetic visuals, losing the intimacy
and realism with which photography and video can present people, places, and situations. They are thus
ideal for making gaming experiences, but not for representing real people, places, and things – an
essential objective for filmmakers.
As an immersive media maker who seeks to capture the world in the manner of a filmmaker, I
have long considered what alternatives could disrupt these two entrenched paradigms and fuse real-world
capture with 6DoF rendering. With several exciting possibilities such as mobile light field video capture
still in their early days of research and not yet production ready, I have become interested in any currently
available possibilities for combining live-action video with game engines to leverage the strengths that
both have to offer. In the period of ideation leading up to this project, I found myself returning to this
interest and asking repeated questions: Are these two approaches our only options, or can we get the best of both?
Could we build an example of narrative storytelling in VR that combined the realism and intimacy of human
performance with the 6DoF immersion allowed by an engine? Could we bring together the most powerful elements
that both video and game engines offer to VR by combining the two?
1.2 Virtual Production – For Virtual Reality?
While considering how common VR workflows could be modified or hybridized, it occurred to
me that filmmakers had already taken up the practice of combining live-action human performance with
virtual, game engine-based environments. Over the last several years, filmmakers have turned to virtual
production workflows to bring the real-time graphics possibilities of 3D engines to live-action
filmmaking. This suggested to me that similar thinking could be applied to making VR experiences, and
2

led to an updated version of my initial question: Could virtual production for virtual reality combine the
strengths of 360 videos and game engine VR?
This seemed like an interesting question to explore, but I knew from the start that our version of
virtual production would have to differ from the popular examples set by big-budget studios. Beginning
with Disney’s use of virtual production on The Mandalorian and spreading across film and TV
productions globally, film studios have placed actors in front of LED volumes as backdrops to fuse the
physical and digital in front of a camera. In our case, two major variables were different. First, because
this was going to be a VR project, we would need to bring live-action performance into the game engine
rather than place an actor in front of some backdrop imagery on a stage. Second, for us, “virtual
production for VR” was also going to mean “virtual production for VR on a budget.”
This thought was soon followed by another: virtual production for VR did not have to mean
expensive LED backdrops, but could reach a near-zero budget by combining older compositing
techniques like green screen with game engines. In VR production, green screens are often used to record
full 3D volumetric capture of performers. We wanted our approach to differ from volumetric capture,
since this approach generally costs thousands of dollars per minute of footage and transforms the human
performer into a mesh, losing a great deal of realism in the process. I wondered if it would be an effective
way to both lower costs and produce higher-fidelity results to instead use regular video in combination
with a game engine. I felt this would work provided our project was a seated and stationary VR
experience. If this approach worked on a both technical and artistic level, my hope was that we could
demonstrate a set of low-budget but effective possibilities for VR storytelling and cinematic experiences.
1.3 Help Is On the Way
With this approach in mind, I set to work drafting a narrative that would take advantage of this
workflow. Fresh off of a prior project that I felt was too heavy with narrative exposition for an
3

already-overwhelmed VR viewer, I knew I wanted the dramatic setup of this piece to be immediately
graspable. Considering various simple premises, I felt the scenario of someone being followed by a
threatening but mysterious entity would be a potent starting point in VR. Shortly thereafter, a second idea
came to me, of a character dialing 911 and being greeted by an A.I. dispatcher. This felt like a good
elaboration, the second layer of premise we needed to get started. The narrative premise became:
Walking alone late at night, a woman (Flora) finds herself followed by a strange and persistent pursuer. When
Flora dials 911, her real challenge begins: explaining her plight to a fancy new A.I. 911 dispatcher. Immediately
I felt this dual narrative tension – of 1) an eerie pursuit played out visually and spatially, and 2) a
Kafkaesque phone call playing out in dialogue – provided two strong dramatic elements that could collide
with each other to build suspense. I also found the piece’s direction to resonate with an increasingly
ubiquitous contemporary feeling: a menacing sense of unease regarding the future of our social systems
and our technologies. This was a story about a person in the near-future facing an ambiguous but possibly
existential threat, crying out for help, and being met by their society with a dystopian reply.
As this idea germinated, I began listing a set of goals in terms of the project’s artistic approach.
First amongst these goals, I was committed to the idea that viewers should journey through the story
alongside Flora as the narrative unfolded. I felt that audiences being physically alongside Flora, immersed
with her in a threatening environment, probing the environment for a glimpse of this treat, moving with
her down the road with no idea of what lies ahead, and waiting for help to arrive would viscerally build
suspense, while also powerfully communicating the story’s themes and ideas. Second, I felt that all of
this would be elevated by the dramatic situation playing out in real time. Third, when thinking about
interesting ways to tell Flora’s story through editing, I started to question if it would be possible to tell the
story from multiple different points of view (POVs) – specifically whether it would be possible to alternate
between the POVs of Flora and her pursuer as the story unfolded. While this was a strange idea to attempt
in a VR context, I felt that if it worked, this could elevate the suspense even further by giving audiences
4

bits of agitating information that Flora lacked – like the old Hitchcock trick of “tell[ing] the audience
there is a bomb under the table and it will go off in five minutes” (Hitchcock, 1970).
Overall, one of the things that pushed me toward pursuing a story in the suspense/thriller genre
was the challenge of pulling off such a story in VR. A suspense thriller requires a sophisticated
manipulation of pacing, tension, and viewer attention moment by moment. Because suspense generally
requires a high degree of control, it is very hard to do in VR. I knew that working in this genre meant
intricately coordinating a variety of cinematic elements, and presumably doing interesting things with the
medium to make this work.
1.4 Prior ‘Hybrid’ VR Efforts
As my intrigue around this process developed, I surveyed prior immersive work that pursued
similar “hybrid” (video and game engine) approaches to making VR. One piece that used a similar
technical approach of putting live-action human subjects into a 6DoF engine environment was the 2018
VR documentary Home After War. To tell the story of an Iraqi family returning to their home in Fallujah
and finding it booby-trapped, the filmmakers combined a 6DoF photogrammetry environment of the
Khalaf family’s home with stereo billboarded video of the story’s protagonist Ahmaied. The project’s
director Gayatri Parameswaran described this approach in the following way:
[W]e didn’t want to stick to only 360 videos, because that would take away from the immersive aspect of
it….With the stereo [billboarded video] capture, I think one of the biggest reasons for that is we had
experimented with [volumetric capture tools] DepthKit and Kinect, but we didn’t like the look of a person’s
figure disintegrating…because it didn’t go with the look and the realness of the story. So that’s why I think
we picked stereo billboarding, despite all of the massive challenges that it brought up later. We had
challenges with placing a 2D object in a 3D space….[B]eing able to move in the space and being able to
see a person’s figure clearly and crisply, and that resolution, is what made us choose [a combination of]
photogrammetry and stereo capture. (Parameswaran, 2020)
This approach was effective for allowing viewers to both explore Ahmaied’s home and to get a feeling of
Ahmaied’s presence in the room as he tells his story. However, there were aspects of this engine-video
integration that I felt we would want to do differently in this project. This mainly had to do with the fact
5

that Ahmaied appears placed in this photogrammetry environment, but does not visually fuse with it. The
lighting on Ahmaied is flat and is not adjusted from scene to scene to match changes in the environment
lighting around him.
Figure 1 – Home After War: Character lighting versus world lighting
This piece also does not feature Ahmaied moving through the story world with the audience – in each
scene, he appears standing in a single location, but does not come with us as we move through space.
While stationary character blocking was fine for this documentary context, I knew that for our fictional
suspense piece about a character moving quickly down a dangerous road, we would have to contend with
the further challenges of moving Flora’s video continuously through our 3D environment.
Another high-quality piece that explored live-action capture in engine was the 2022 VR
documentary On the Morning You Wake (At the End of the World). The project is based on the 2018
Hawaii false missile alert, when Hawaiian residents received emergency notifications on their phones
warning them of an incoming nuclear ballistic missile attack. This piece featured more narrative edits
than Home After War and even moments of a moving camera, although this was not the main technique of
6

drawing a viewer through its world in the way I was considering. The virtual production aspects involved
compositing volumetrically captured performers into an immersive game engine environment, and
because these performers were fully 3D, the challenges of matching 2D and 3D elements were not the
same as in our project. I was interested to note that just like in Home After War, the experience was
powerful despite its absence of realistic lighting on these characters. On the Morning You Wake embraces
such stylized computer graphics (CG) for its environment that there is no expectation of characters fusing
fully with the world.
Figure 2 – On the Morning You Wake
Despite this, the piece was visually compelling and narratively powerful. This piece and Home After War
held out the possibility that I didn’t need to seamlessly fuse the lighting on our character videos with our
engine environment to make a compelling immersive story. However, thinking further about Help Is On
the Way, I doubled down on my goals to make Flora feel like a seamless element of the story world. I felt
strongly that this would help viewers feel present alongside Flora, and therefore decided I wasn’t willing
to give up lighting convincingly and making this fusion look realistic.
7

Outside of the question of combining 2D video with 3D worlds, I knew that the biggest challenge
our project would face was managing pacing and building tension in the context of a thriller piece, which
is generally a cinematic exercise of maximum control. I knew that doing this in VR would demand a
hands-on approach to character blocking, camera movement, and editing to control pace and perspective
moment by moment. For this I looked at Doug Liman’s 2016 360 video miniseries Invisible, which is
uniquely aggressive in its approach to blocking, editing, and at times camera movement. It is one of the
very few examples I have found of action filmmaking in VR. Going through the episodes of the show, I
was surprised at the pace of action and editing that Liman was able to execute without causing audience
discomfort. This was encouraging – but what the piece left me wanting was the sense that I was exploring
a world. In Invisible, viewers move through the story world via constant cuts and at times split screens,
but I never had the feeling of exploring a coherent space, which I felt a continuously moving viewer
camera would provide. When Liman moves the camera for action shots, it is brief and we generally still
enter the next room via a cut. After watching the series, I was further convinced that a more steadily
moving, walking-pace camera in an immersive film could give us the feeling of exploring the story world
and going on a journey with the characters – even if the experience is linear and we are not ourselves
controlling these large camera movements.
Over my time perusing adjacent VR filmmaking efforts, I was unable to find examples of pieces
that used the precise workflow I had in mind, or which took the same stylistic approach I wanted to
execute: transporting a viewer through an immersive story in real time while cutting between multiple
perspectives. This was a motivating recognition, because I felt this indicated I was not on well-trodden
ground and had some interesting things to learn. Despite the number of tricky items I felt our team would
face, I was spurred on by reading a quote from immersive media pioneer Michael Naimark: “Clearly, the
hybrid region between cinema and games is a place of interest. Importing camera-based imagery into
model-based game engines offers truly ‘new media’ opportunities.” (Naimark, 2018)
8

1.5 Project goals
Bringing these elements together, I felt our team had a clear set of goals for the project. Our
technical goal was to execute virtual production for VR by integrating live-action video with immersive
3D environments in Unreal Engine. From a design standpoint, I resolved that viewers would move
through the story world alongside Flora in real time to increase feelings of presence – and in the case of
our story, uneasiness and suspense. Our experience goal was accordingly for audiences to feel a presence
of suspense while experiencing a VR film. Together, all of this defined our overall thesis question: Can
we use virtual production techniques to make a cinematic VR experience that compellingly combines the strengths of
live-action filmmaking and real-time engines?
9

Chapter 2: Creating Help Is On the Way – Technical Challenges
We started work on the project facing a handful of technical questions around how to implement a
“virtual production for VR” workflow. Most of these involved the core process of compositing live-action
2D video with 3D environments for VR headset viewing. It quickly became clear that this would require
finding solutions to several compositing-related challenges: clean background removal, addressing the
depth disparity between 2D footage and a 3D environment, achieving 2D character movement and
interactivity within a 3D world, and the question of how to position a viewer to view 2D characters in a
3D environment. These were questions that had to be answered for the project to get off the ground at all.
Covering our approach to these challenges, I will address each with the following structure: 1) What
problem were we trying to solve? 2) How did we go about solving it? 3) What were the results – what did we learn?
My hope was – and remains – that in answering these questions, we could demonstrate a viable and
worthwhile workflow for filmmakers taking up the challenge of producing immersive experiences.
2.1 Core Compositing Approach
The first question we had to address in order to composite live-action footage with our Unreal
Engine environment was how to approach background removal on video footage of a moving subject. The
most common approach to background removal in live-action video is the use of green screen and chroma
keying to isolate a single color and remove it from a shot. While green screen is a common and very well
understood approach, there are aspects of this technique that are not ideal for a lower-budget setup like
ours. Primary amongst these is that when filming in a small space, the problem of green bounce (‘spill’)
onto the subject is hard to avoid. To avoid such spill requires shooting in a larger space than most people
have access to. If one does have a sufficiently large space to set up a professional-caliber green screen,
there is still the time, money, and effort required to transform this space to green.
10

Working on a very small budget and not wanting to reduce our schedule to the slots available on
our school’s green screen stage, we tried a different approach to shooting our subject. As an alternative to
chroma keying, machine learning tools for background removal have been hitting the market in large
numbers. One amongst these, called MODNet
1
, is the software we settled on for background removal.
Using a machine learning tool for background removal allowed us to shoot against a plain white wall with
no special backdrop construction, and still achieve a clean result.
The more challenging puzzle was getting footage of our character walking continuously against
this backdrop. For this we used a treadmill, but for background removal purposes, the treadmill had to be
white as well. It didn’t take much time perusing the Internet to recognize that treadmills aren’t sold with
white belts, presumably for the reason that the belt would not remain white after use. We therefore set off
on the task of painting a black treadmill white, which ended up involving more misadventures than I
expected. Our first approach to treadmill painting was to sand the treadmill body so that paint would
adhere, and then spray paint the entire machine white. This looked good and we had high spirits as we
waited for it all to dry. We had less high spirits when we turned it on and all of the paint flaked off as the
belt bent, flaking off at an even faster rate once a member of the team started walking on it. In a panic, we
had the even worse idea to cover the belt with white tape, which peeled, bunched together and clogged the
machine as soon as we turned it on. This journey of misadventures continued, including disassembling the
treadmill and trying to replace the belt with an alternate material.
Eventually, after further research we decided to purchase a new treadmill and start from scratch.
Our research clarified the workflow that Hollywood studios use to paint chroma green treadmills. This
workflow ended up requiring several layers of brand-specific paints that adhere to treadmill belts: a primer
and then the desired color. We held our breath, tried it out, and indeed the professional method worked.
Below is our result, after days of walking, jogging, stomping, jumping, and (of course) filming:
1 https://github.com/ZHKKKe/MODNet
11

Figure 3 – The white treadmill
The only maintenance this setup requires is a thin touch-up application of our top-most paint layer after
several days of intensive use. Naturally, when making use of the treadmill, we ask that all camera subjects
wear new or thoroughly cleansed footwear.
2.2 Approaching Depth
While the current trends of virtual production in film and TV integrate 2D live-action characters
with 3D environments, the output from these workflows is a flat 2D picture. Since our experience was to
be viewed in a VR headset with six degrees of freedom, we had to decide how we would approach the
depth disparity between 2D footage and 3D environments. On a prior immersive narrative project called
The Night Watcher, I worked out a method for shooting live-action footage, alpha-matting the footage,
stacking both videos into a single file, and UV-mapping the two halves of the video texture to a game
12

engine material for playback – but this method left the live-action footage flat in an otherwise 3D world.
My prior means of dealing with this was to place the live-action footage in the middle-ground to
background, in order that the lack of parallax on the character not be strongly perceived by a viewer. For
Help Is On the Way, I wondered if this core approach could be pushed further to make regular
single-camera footage semi-volumetric, but without the high price tag or visual artifacts.
With this in mind, I worked with our technical director Jordan Halsey to implement a method that
– from a single piece of video footage – could generate not only an alpha matte from MODNet, but also
depth maps and normal maps for each piece of footage. In theory, the alpha matte could stencil the outline
of a character in the engine (as before) – but then the depth map could be used to displace the tessellated
plane on which the video footage was projected, creating volumetric depth, with normal maps adding
additional detail. Each of these were laid out on a single video texture, creating what we called a “four
pack.”
Figure 4 – Our first “four pack”
13

These videos were then UV-mapped in Unreal to feed each quadrant of the video to the correct material
channel on our video plane, to produce a volumetric result.
This approach worked, with the obvious caveat that the video footage would only appear
volumetric from a single point of view. While this would be limiting in a room-scale VR context, for a
seated cinematic experience the result was viable. From a budgetary point of view, the workflow depicted
here could be achieved with nothing more than a smartphone camera, a far cry from the budgets of
studio-based fully volumetric workflows, most of which cost six figures and up. Considering our results
and near-zero budget workflow to achieve it, we settled on this approach as a key building block for the
experience.
2.3 Viewing Angles and Billboarding
This approach gave us another challenge to address: while our footage was volumetric from a
single defined viewer angle, in 6DoF VR a viewer can shift their position relative to the footage as they
desire. This raised the question of how to constrain either the viewer position or the semi-volumetric
video orientation such that the dimension and perspective of the video would not “break” for the viewer.
I faced this same challenge on The Night Watcher and addressed it by billboarding the 2D character
footage so that it would always rotate to face the VR viewer. A programmatic implementation of this can
be seen below:
14

Figure 5 – Billboarding implementation
While in extreme cases, e.g. of a viewer running across the room with a headset on (not our desired
viewing condition), a viewer will see the trick of the character’s video plane rotating as they move. But in
the seated viewing conditions that we recommend for our project, the rotational approach is not
noticeable, since it only operates on the scale of a viewer shifting back and forth in their seat.
It should be underscored that our approach here has limitations, mainly that (as suggested above)
it does not work for fully room-scale VR experiences. This approach represents volumetric capture on a
budget, with the caveat that the volumetric capture is only visible from a narrow range of viewpoints. The
illusion breaks if a viewer’s range of movement is room-scale. If the footage does not rotate to face the
viewer camera, an extreme change in viewer position reveals the lack of additional captured viewpoints of
the video subject. If using our rotational billboarding approach, extreme viewer movement causes the
subject to appear to spin around its up-axis. Therefore, we believe this approach is effective strictly at
15

integrating live-action characters with 3D engine environments for seated or standing, stationary
cinematic experiences in a headset.
2.4 Character Movement and Interaction with World
In a similar vein, this approach to character-world integration begs the question of how a
character may move and interact with the world of the story. The answer reveals further limitations to our
approach. The treadmill setup described above goes a surprising way toward allowing a character to
move through a virtual world as defined during post-production. The post-production implementation
simply requires keyframing the character’s position across a 3D environment, such that the movement of
the character appears to match the speed at which their legs are moving in the video footage.
However, while our use of the treadmill allows character movement through a virtual
environment, it does not solve the problem of the character otherwise interacting with objects in the
virtual world. In both films and games, audiences are used to seeing characters interact with their worlds
in myriad intricate ways – either via a physical performer manipulating physical objects in a physical
world captured by a camera, or a virtual character manipulating virtual objects in the virtual world of a
game. In our part-physical part-virtual production approach, we necessarily faced limited capacities for a
physical character to interact with objects in a virtual world. In this case, any objects a live-action
character interacts with must generally be interacted with physically and captured in camera, prior to
engine integration. It should be noted that this same challenge exists across all virtual production
approaches, including the six-figure approach of fully volumetric capture.
2.5 Lighting
We knew that the make-or-break element that would tie together and sell our workflow to viewers
was lighting. If we could successfully match the physical lighting on Flora with the virtual lighting of our
16

3D environment, this would viscerally fuse our character and world for audiences. If we failed to do this –
as we did at first – we felt this would break the coherence of the piece and invalidate our workflow. This
became the area where we spent most of our time, trying out multiple approaches to lighting in an effort
to make this work.
The first approach we tried was to shoot with physical lighting that matched, as evenly as
possible, our lighting in the engine. The lighting in our virtual scene consists of a chain of street lights
under which Flora passes as she moves down the road. My first inclination was therefore to set up an
overhead key light that would dolly or pan above Flora at intervals while we were shooting, and then set
up the lights in the engine to match. The setup looked like this:
Figure 6 – First lighting setup
We found this approach strongly limited our flexibility to light scenes as we wished in post. Due to the
inflexibility of the video footage that resulted, we accordingly felt this was not the right process to settle
on as a best practice for VR virtual production.
17

The next approach we tried was to shoot footage of Flora with flat lighting, and virtually re-lit this
footage later on in Unreal using the depth maps mentioned above. In terms of workflow, this was the
simplest of the approaches we tried. This meant that on set, we simply had to light our actress with static,
neutral lighting for the full duration of each shot. Then in the engine, the theory was that once we
plugged in the footage to our custom character material, the lighting might take care of itself. While this
approach was ideal in theory, we found that our depth maps were not detailed enough to produce a
realistic play light and shadow, especially across Flora’s face. We found that members of our team were
particularly sensitive to the nuances of how shadows fall across a face, and even the slightest deviations
from realistic shadows pulled audiences out of the experience. So in turn, we put this approach to rest as
well.
After neither of these initial approaches to matching physical and virtual lighting worked, it was
clear that we had to be more creative. Over the course of several energetic last-minute discussions with
our camera team, we hatched a new plan: to do classic Hollywood three-point lighting (key light, fill
light, and backlight) using red light for our key, blue for our fill, and green for our backlight. This would
allow each core component of Flora’s lighting to be dialed up or down in post simply through keyframing
the intensities of the red, green, and blue channels of her color texture. Flora’s lighting could therefore be
tuned moment by moment in the engine to have just the right strength of key light when Flora walked into
and stood beneath a street light, just the right amount of fill to produce flat lighting when Flora was
between street lights, and a stronger backlight when she exited a light.
18

Figure 7 – Final lighting setup (red, green, blue)
This was a gigantic breakthrough for us. In the virtual world, when Flora was beneath a street light the
contrast on her could be high (strong red, weak blue), as it would be beneath a strong key light. When she
was out of the light, in relative darkness with no strong key light, her contrast could be dialed lower for an
appropriately flat look. This all worked because the final output of our project was monochrome (black
and white), meaning that these adjustments did not impact the final color of the footage – simply the play
of luminance across Flora as she walked through the world.
19

Chapter 3: Creating Help Is On the Way – Artistic & Formal Explorations
The goal of all of this technical problem-solving was to get to a place where we could begin to
address the formal and creative challenges that VR filmmakers routinely face. Our goals here were
twofold: 1) to demonstrate a set of techniques that VR filmmakers could use to overcome classic design
challenges of cinematic VR, and 2) to implement these ideas in a manner that fit the needs of our story
experience.
3.1 Approaching the VR Camera
One challenge that has plagued cinematic VR material for years is having an immobile viewer,
with the resulting effect that a viewer often feels “stuck” in an immersive experience. When a viewer puts
on a VR headset, they are greeted by the physiological suggestion of an expansive immersive world, and
generally respond to this cue with a psychological desire to explore the world presented. In engine-based
VR games, this itch can be scratched – the viewer controls the VR camera frame by frame and can
generally move about as they wish. Conversely, in most cinematic VR material, the camera seldom if ever
moves. In shot 360 or 180 degree video content, the position and viewpoint of the 360 or 180 degree
camera directly matches what viewers will see later on in a headset. At the time of shooting, these
physical camera systems are almost always stationary, with human characters and action blocked to
surround the camera for the duration of a scene. This practice is so widespread that even in those rarer
cases when game engines are used to tell cinematic VR stories, creators tend to keep their virtual cameras
mostly stationary. In all of these cases, although viewers feel on the one hand that they are present in an
expansive virtual world, a stationary camera has the paradoxical effect of causing viewers to feel
claustrophobic and trapped in their position.
20

Since 360 and 180 videos lack six degrees of freedom, viewers are completely unable to shift
their viewpoint and experience novel, dynamic viewpoints of a scene. In such an experience, the visual
field presented feels most natural when a viewer holds their head perfectly still. Head rotation alone
breaks the illusion of presence in such videos, because the human head translates when it rotates rather
than rotating around a nodal point (Bolas, 2015). Therefore, even the three degrees of freedom promised
by panoramic videos is not a comfortable three degrees of freedom.
Additionally, scenes with a stationary VR camera tend to lack the dynamism that comes from
characters moving continuously through a world. These scenes feel unnatural in their blocking, since
actors or documentary subjects are generally arranged to encircle the VR camera, which is cropped out of
the frame – producing the effect that characters are encircling a phantom, invisible object.
Much of the reason for this stationary approach to camera position comes from an important goal:
avoiding motion sickness. Virtual reality has always had to be ultra-conscious of the vestibular-ocular
conflict: a feeling of illness that results from a disparity between what a VR subject sees in the headset
and what their vestibular system perceives to be their motion (or lack thereof) through space. This issue
can arise from latency in a VR system, or it can result from a VR creator visually moving their audience
through space when, e.g., that same audience is sitting stationary in a chair – a good example being any
immersive experience where a 360 camera has shot a roller coaster ride.
For this reason, VR filmmakers have rightly been cautious to implement aggressive and dynamic
camera movements that don’t align with their audience’s physical movement through space. The
limitation here is that this cautiousness most often takes the form of not moving viewers through the
virtual environment at all. This is limiting. As industry authorities such as former Oculus CTO John
Carmack have pointed out, moving a viewer through a virtual environment is not likely to cause motion
sickness if they “stand still or move sort of in a straight line, as long as there's no acceleration. The worst
things are sort of parabolic.” (Carmack, 2019) Parabolic curving movements, as well as aggressive
accelerations and decelerations, should generally be avoided. Barring these motions, we are relatively
21

safe to move the camera and take audiences on a dynamic voyage through our virtual worlds – it simply
must be done correctly.
From my experiences as both a consumer and creator of VR experiences, I knew going into this
project that the biggest thing I wanted to explore was moving the VR camera. My feeling was that
immersive virtual worlds and their visitors cry out for exploration, and that bringing audiences through
these worlds alongside story protagonists was the fastest route to demonstrating dynamic VR filmmaking.
When immersive media presents to us a world, we want to feel that we are present in this world, and we
want to voyage through it – either actively as an agent in an immersive game, or passively as an observer
in an immersive film.
For this reason, we hoped to make progress on the question of how to implement dynamic
cameras to tell stories in cinematic VR pieces. From the beginning, I knew that I wanted to have multiple
camera setups, each of which would move the viewer through the story world in a different way. The first
setup I felt was fundamental was a virtual camera that would move directly alongside Flora in parallel, as
if we were walking alongside her down the road in real time. This meant that when Flora heard something
threatening behind her and looked over her shoulder, we could do the same.
Figure 8 – Camera Setup 1: Alongside protagonist
22

This setup meant shooting Flora in profile, then placing the virtual camera next to her in the engine. I
wanted the camera to start off facing down the road, and traveling in the same direction Flora walked. We
noticed that this identified our perspective with Flora’s in the story by almost matching her viewpoint. We
also found that unsurprisingly, moving the camera in a forward direction at walking pace felt much more
natural than trucking sideways.
Our second fundamental camera setup was the most traditional to the language of film: facing
Flora head-on as she moves down the street, and dollying backward at walking pace.
Figure 9: Camera Setup 2: Face to face
This allowed us to keep the movement of the viewer and character and story going while giving viewers a
clear view of Flora’s face and emotional state as her conversation with the A.I. (AIDEN) unfolded. This
was great for moments of emotional reaction, such as the relief on Flora’s face when AIDEN first appears
to understand her situation and offer a promise of help. It also served the second purpose, which was to
give audiences a view of the road behind Flora, where we know her pursuer lurks. This encourages
23

audiences to probe this darkness, and also gives them glimpses of things that Flora doesn’t see, such as
lights behind her turning out, and eventually a more physical volume of darkness devouring the road
behind Flora.
While both of these setups were compelling for bringing us close to Flora’s character in different
ways, not all of our setups proved fruitful. We also considered a camera setup where audiences would be
behind Flora but in front of her pursuer.
Figure 10 – Rejected camera setup
Not much came from this setup that we didn’t get in a better form elsewhere. First, we didn’t get much
information or emotion from Flora when placed behind her. Second, we couldn’t see the road ahead of
Flora because she was blocking it. Third, looking over our shoulder at the possible threat creeping up
from behind was no different from the experience of doing this when we were placed alongside Flora in
Camera Setup 1. As a result, we ended up tossing this setup out.
24

There was one final setup that we tried as an experiment. This was placing viewers directly in the
villain’s POV as it pursued Flora. Unlike the prior setup, we realized that being in the POV of the villain
gave away the villain’s position and – depending on the behavior of the POV – the psychological
character of the villain, all information that was absent from the prior setup. At first we didn’t know
whether or not cutting from a third-person view alongside Flora to a first-person POV of the villain would
work in VR, but we decided the camera setup would be worth trying, adding dynamism and another
dramatic layer to the experience.
Figure 11 – Camera Setup 3: Villain POV
We knew in order for this shot to read as the villain’s POV, it would have to look completely different
from any of the other shots – so we differentiated this viewpoint by vignetting the view of the scene,
distinguishing this as a first-person POV gazing at Flora through some kind of unknown optics.
25

Figure 12 – Villain POV vignette
This setup, leveraging a narrow field of view, also allowed us to move the camera quickly and
aggressively as the villain moved in on Flora, building pacing and tension. Because of all of the
possibilities it allowed us, we were very happy when later user testing confirmed that this shot worked for
audiences.
A fourth camera setup we used was an aerial view of the story environment, from which we saw
Flora moving down the street but could also glimpse a wide view of what lay ahead. We realized this
could further heighten audience suspense by giving a glimpse of the ominous road ahead of Flora. It also
gave us a great novel angle to cut away to when wanting to cut back close Flora from a different angle.
Lastly, this gave us the idea for a narrative reveal that we wrote into the script – that AIDEN, the A.I.
dispatcher, had been watching a live feed of Flora this entire time, the whole time Flora had been busy
trying to explain her situation. This shot therefore became “AIDEN’s POV.”
In the end, we wound up with four camera setups we found to be both experientially exciting and
narratively motivated. Each served a purpose in the story in the moment it was used. For example, we
26

used the “alongside” setup for a shot where Flora approaches a fork in the road, and AIDEN urges her to
take the path she’s not familiar with. In this moment, placing a viewer alongside Flora, we can look down
each side of the fork and size up Flora’s options, just as she is doing the same. Thankfully, after following
the basic rulebook for avoiding motion sickness – no parabolic curving movements, no strong
accelerations/decelerations – we received zero reports of motion sickness amongst our dozens of user
testers. Even amongst the half-dozen testers that reported high sensitivity to motion sickness, we received
zero reports of motion sickness. I hope that at least the first two of these camera setups (facing a character
while dollying backward, moving alongside a character while dollying forward) can serve, at least in my
own practice, as fundamental types of shots for designing cinematic VR experiences.
3.2 Approaching Editing
Another of the most common criticisms leveled at cinematic VR experiences is that without
traditional cinema’s active approach to editing, cinematic VR feels slow and boring. This, in combination
with the general lack of camera movement, often contributes to the sense of a slow pace and lack of
dynamism in VR films. As with camera movement, cutting in VR is generally done sparsely to avoid
jolting audiences. To the extent that VR creators do cut – cutting is more frequent than camera movement
in cinematic VR – it is generally a cut from one environment to another, and aims to move viewers to the
next scene with minimal discomfort. In addition, VR filmmakers often try to arrange their immersive
compositions and edits so that wherever a viewer is looking in the preceding shot, the object of interest is
in the same place in the next shot. Former Google VR filmmaker Jessica Brillhart articulately describes
this approach:
Points of interest (or POIs) are elements within an experience that attract a visitor’s attention. Now virtual
reality being what it is, I can never be 100% certain that you’re going to look somewhere, but I can make
some solid bets on where you’re most likely going to look by evaluating the entirety of the experience….[I]f
I can make a solid bet on a POI…I might be inclined to do a match on attention. This would involve
identifying where a visitor’s attention lands and then cutting from that to something else I’d like for the
visitor to pay attention to. By doing something as simple as identifying visitor attention, you can start to
27

craft an edit that feels far more natural and purposeful than just cutting from whatever, whenever you feel
like it. (Brillhart, 2016)
This technique works particularly well for moving a viewer from one environment to another
without breaking the flow of an experience. However, there wasn’t much precedent for how to edit a story
like ours, which would play out entirely within a single location in real time. This, combined with the
challenge of having to nimbly manipulate pacing and tension in the thriller genre, presented a great
opportunity for a challenge: could we develop an effective approach to cutting within a single coherent
location in a dramatic VR film?
While we didn’t know exactly how we would make this work, we did have ideas about what
would not work. Many of the most classic shots we’ve come to recognize in cinema, such as cutting from
a wide angle to a close-up on a character, don’t work in VR because they produce what could be termed a
spatial jump cut. Since a VR spectator is made to feel that they are immersed in a scene, if we tried
cutting from, say, what could be termed a medium shot on Flora to a close-up on Flora, the result would
be to tug a viewer some number of meters across the scene, as if they’ve just been pushed or pulled closer
or further from the subject in front of them. This is both physiologically jarring and stylistically awkward
in VR. Cuts like these, if desired, require first cutting to a completely novel viewpoint: an editorial
“hinge” that can cut back to whatever adjacent viewpoint on the subject we wish. This could be an aerial
view of the scene, or cutting 180° from a front angle on Flora to a back view of Flora (the villain POV).
Naturally, we did both of these things.
We quickly found our editorial bottom line to be: cut when desired to define pacing, but make
sure to cut to a completely novel viewpoint. While cutting from one environment to the next in VR is
common and uncontroversial, we suggest liberalizing the conservative approach of “cut from novel
location to novel location” to the more broad “cut from novel viewpoint to novel viewpoint.” After early
tests, we found it was effective to cut from, e.g., a front angle on Flora to a villain POV shot from the
back, then cut to an aerial view of the scene, and then back to moving alongside Flora in profile. These
28

early tests also helped us to define our camera angles. As a VR filmmaker, we’ve come to believe that you
are relatively free to cut when you want – just avoid those uncomfortable spatial jump cuts for a viewer.
At the time of shooting, we didn’t yet know how the pace of the edit would play out best in a
headset. While we had some ideas on where we would cut and what angles we wanted to cut to, we
didn’t yet know how the pacing would feel in a headset. With this in mind, we chose to shoot the whole
script, from start to finish, from every single angle we were considering placing the viewer. This followed
the filmmaking paradigm of “shooting coverage,” and in the editing phase allowed us to place a cut at any
moment we wanted to, from any angle to any other angle that we deemed best at that moment. This
meant we could cut sparingly or liberally, from any shot angle to any shot angle, as desired as the piece
unfolded.
Editing this coverage, we learned a few encouraging things. First, in terms of when to place a cut,
we could pretty much place a cut whenever we wanted provided that we cut to a completely novel
viewpoint – and bonus points if this viewpoint was narratively motivated. To put this to the test, we
executed a fast-paced edit for the first direct physical confrontation Flora has with the creature pursuing
her. We started with a lengthy front-angle shot of Flora speed walking down the street as a shroud of
darkness clips the street behind her, getting ever-closer to Flora and to the viewer. We then cut to a
vignetted villain POV shot of the pursuer rapidly closing in on Flora, Flora turning, looking the villain
POV in the eye, and hurling a pineapple from her grocery bag into us. Then a cut to black, accompanied
by a loud screech. Then a cut back to walking down the street alongside Flora in the aftermath of the
confrontation, then a cut to an aerial view (AIDEN’s POV) to reveal that AIDEN has been watching Flora
this whole time. This amounted to five shots in about twenty seconds, and our user tests showed that for
most viewers, it was one of their favorite parts of the experience.
Cutting between all of these novel viewpoints also reinforced that when done with intention,
cutting in VR can be another powerful means of exploring space. Our aerial shots allowed us to get a
glimpse of what lay ahead further down the road from Flora, our frontal shots of Flora allowed us to see
29

the space behind her, and our profile shots of walking alongside Flora allowed us to journey through the
space with her. Cutting from one of these angles to another gave us a multi-perspective experience of our
3D environment, enriching its sense of dimensionality and narrative power.
Our last important takeaway was that it is important for filmmakers to have motivated reasons for
each cut that not only define the pace, but which tie to the narrative – such as choosing moments where a
POV shift would be most dramatically powerful – and cut to what the creator most wants the viewer to see
in that moment. Conceptually this is not too different from traditional filmmaking – VR practitioners just
have a few more boxes to check. A final tip for others editing in VR is to test every editorial idea in the
headset before moving on. Since this is how the audience will experience the edit, this should be done as
early and often as possible. If not, unpleasant surprises will await.
3.3 Directing Audience Attention
Perhaps the most classic challenge that immersive filmmaking has always faced is the question of
how to direct a viewer’s attention in an immersive space that lacks a traditional picture plane. As we set
to work on the project, we compiled a set of approaches to choreograph viewer attention and tell our story
in a clear manner. Our idea was that in layering these techniques together, we could weave an engaging
world that brought our story to life in an intentional way.
Our first approach to directing audience attention was to design an environment that would serve
the needs of our narrative. We intentionally laid out the environment as a single road for viewers to travel
down alongside Flora, moving further down the road as the story advanced – with no objects of interest
outside of this road to distract from the story’s progression. My initial idea in terms of lighting this
environment was to lean into the high-contrast style of film noir for its mood, but this intuition quickly
expanded. I realized that if we took high-contrast lighting to the extreme, we could create vignettes of
30

attention and action that would define where viewers could look when probing the story’s action. This
was our first concept image.
Figure 13 – First environment concept image
These pools of light would be points of interest where action played out, and outside these spatial nodes, a
viewer would see only darkness. This lighting therefore could be used much like spotlights on a stage,
illuminating areas we wanted audiences to pay attention to, and offering absence (darkness) in areas that
might otherwise distract. Below is a screenshot from the final piece.
31

Figure 14 – Final environment lighting
We found this technique to be effective at guiding audience attention. When doing user tests, we
saw that viewers were given enough visual information that no one’s attention was attracted to areas of
empty space. This effectively kept attention choreographed toward our vignettes of action. We also got
the feedback that this minimalist approach to environment design and lighting gave the world of the story
a distinct stylistic feeling that both helped viewers feel transported elsewhere, and gave the arc of the
experience a visual continuity.
Related to this environment and lighting layout, I realized that another tool we had at our disposal
was the direction of character blocking and story action. I had the thought that in our experience, if Flora
moved in a straight line down the street, and viewers spent most of their time alongside her, the
experience would always have a clear sense of story direction. Indeed, even with us flipping the viewer
camera around to all kinds of different angles, we kept all of the narrative movement of Flora and her
villain unfolding in a single inexorable direction. Our viewers can therefore remain grounded in their
sense of direction despite our perspective shifts and at times rapid editing.
32

Another technique we used to guide audience attention was spatial audio cues. This has been a
core technique of VR filmmakers since the early days of cinematic VR. On the use of cues, Brillhart
writes:
[If] you’re stuck needing a visitor’s attention to land somewhere specific, you could add cues. Cues can be
pieces of music, a sound effect, a haptic response, a color shift, an animation, etc. How intensely noticeable
these cues are is entirely up to you....For VR [cues are] particularly useful because you can plant them to
create attention spots or reinforce pre-existing POIs to strengthen the kind of experience you hope the
visitor will have. (Brillhart, 2016)
We used a variety of spatial audio cues to define our dramatic situation and guide viewers’ attention. At
the very beginning of our story, we placed spatialized audio of a slithering creature (the villain) behind
Flora and the audience, and nudged its presence closer and closer to Flora as the narrative unfolded. We
placed Flora’s dialogue directly alongside the audience as viewers moved down the road with Flora to
reinforce our sense of journeying alongside her. In addition, throughout the experience we further
reinforced a sense of how far away the villain was by spatializing the sound of streetlights buzzing and
flicking out as the villain passed beneath them.
Bringing elements like these together – environment layout, lighting, character blocking, and
spatial audio – allows immersive designers and storytellers to create a rich choreography of attention for
audiences. On this topic, one takeaway our team learned from this project is that absence – doing less –
can often be at least as powerful as adding elements for the eye and ear. Continuous user testing is key for
understanding how these elements are interacting with each other for viewers from moment to moment.
Paying attention to where the audience’s attention goes (and doesn’t go) over the course of an immersive
experience is the reality check upon which these designs can improve.
33

Chapter 4: Conclusions
4.1 User Testing – Did the Project Work for Audiences?
When the narrative was sequenced together in the engine, we had the opportunity to run several
dozen users through the experience. This yielded a big sigh of relief. As mentioned above, our most
precious stylistic element was a constantly moving active camera with multiple perspective shifts and cuts
throughout the experience. While we took precautions against motion sickness, we didn’t know for sure
how viewers would react to a full sequenced narrative in this style. Would people be confused or
disoriented by the perspective shifts? Would they feel physically uncomfortable or aggressively pushed
around?
We were very happy to find that none of our users reported motion sickness, nor did any users
report any other discomfort, disorientation, or confusion. Interestingly, two users (a professional VFX
supervisor and a professional cinematographer) wanted the villain camera movement to be even more
aggressive, with a handheld shakiness that would provide an organic realism as the creature slides across
the ground. While this handheld ‘shaky cam’ approach would not be possible in most wide field of view
VR shots, in these narrow field of view vignetted shots, this could work and is something we’ve been
exploring.
Users reported loving the POV shifts, particularly the aerial views, although two people thought it
was too far away to see anything of interest, which we’ve responded to by adjusting the camera height for
these shots. We got the encouraging note that when alongside Flora, the direction of her gaze drew the
viewer’s attention to where she was looking. While this was good to hear, we did notice that many
viewers didn’t look down both sides of the fork in the road when Flora approached it, one of our
34

important dramatic moments. The team agreed that we may need to add visual or audio cues to make
viewers look down both sides of the fork at this moment.
While no users reported being confused by the overall story, we did hear from two users that they
didn’t know for sure whether the villain POV was indeed a villain POV – and these same people were
unclear that our aerial shots represented AIDEN’s POV. Regarding the villain POV, we’ve adjusted our
sound design so viewers clearly hear the sound associated with the villain during each of the villain POV
shots. In terms of clarifying AIDEN’s POV, we have made no adjustments yet but believe that this could
be made clearer by adding a line of dialogue earlier in the piece during one of the aerial shots that
suggests that AIDEN is watching.
Some of our most encouraging overall feedback was that viewers felt completely immersed in the
world of the story. At the end of the experience, the first words out of many users’ mouths was the
exclamation that they really felt they were ‘there’ (present) in the world of the story throughout the whole
experience once the main story kicked in. The active approach to the camera made them feel they were
not bored or stuck but right there moving through this world and its actions, and despite the
multi-perspective editing that built narrative counterpoint and suspense, these viewers felt that through
most of the experience they were psychologically right alongside Flora. The caveat here is that several
users reported wanting to be physically closer to Flora while moving through the narrative with her.
While we wanted this as badly as any audience members did, our approach features the hard limit that we
can’t get closer than about a 1.5 meters away from Flora without noticing that her depth data is low
resolution – we therefore will have to keep the viewer at some distance from Flora going forward.
35

4.2 Overall Hurdles and Successes
I certainly feel we have had our share of challenges and misadventures over the course of figuring
out a workable approach to this type of production. Charting our challenges chronologically, these have
ranged from the physical (having to turn a treadmill white) to the physical-digital hybrid challenge of
achieving realistic lighting, to purely digital challenges like re-creating realistic walking in the engine. In
this final case – similar to getting keyframes on the lighting “just right,” the solution was time and effort,
nudging keyframe after keyframe over and over in the game engine. Hence, while many of our challenges
have been in the technical vein of a “how to,” after solving those issues our approach still demands a large
number of labor hours to tune physical-digital integration just right.
These challenges noted, I feel that we’ve accomplished something notable in making Help Is On
the Way. We’ve demonstrated an effective low-budget setup for shooting moving live-action characters
for background removal, which does not require an expensive green screen and large physical space.
We’ve also demonstrated an effective technical workflow for adding depth to regular, flat video footage in
a game engine. To the extent that our story requires it, we’ve successfully integrated live-action footage
with 3D virtual environments for the purpose of VR storytelling that has both the intimacy of human
performance and immersion of a 6DoF game engine-driven VR experience. Our choice to tackle a
suspense/thriller-genre piece pushed us to innovate in terms of approaching stylistic fundamentals of VR
filmmaking – freeing the viewer camera, developing a multi-perspective approach to editing, and directing
viewer attention through the spatial design of environment and lighting.
4.3 Limitations of Our Approach
Our approach also carries with it several notable limitations. The first and most obvious of these
comes from shooting traditional flat video with a single camera – this means both that we cannot place the
36

viewer camera wherever we want in post, but also that our approach is not ideal for making full
room-scale experiences. To the first item, we would point out that over the history of filmmaking,
filmmakers have always had to choose their desired angle on a subject or shoot coverage for flexibility in
post – this scenario is the same in our workflow. In terms of the audience being constrained to a relatively
stationary experience, we think that this is generally the appropriate way to view cinematic storytelling
material in a headset. Moving on to consider our environment and the overall visual fidelity of our
project, real-time 3D is still not quite at the level of cinematic fidelity that we are accustomed to with
live-action video footage. Additional limitations include that we cannot really present a character
interacting with objects in a virtual environment in very sophisticated ways – in our view, this is a
considerable and notable drawback. Lastly, as suggested above, the turnaround time using this workflow
is not great – it certainly is a more convoluted and time-consuming process than the run-and-gun ease of
simply shooting digital video, as in a 360 or 180 video approach to VR. We understand that these
limitations, particularly in terms of time expenditure, come with the territory of what it is we’ve been
trying to do.
4.4 Takeaways
While I do not feel that this is the only path forward for immersive filmmaking, I do feel that we
learned some important things for advancing the practice. First, our experience affirmed that 6DoF VR
experiences are preferable to 3DoF experiences, even for stationary seated pieces like ours – and that
making a 6DoF experience does not need to mean the absence of live-action human characters. Second, I
believe that immersive filmmakers should actively explore their world’s space with the camera – the VR
camera must be the intrepid explorer of worlds. Third, filmmakers can use editing to control pacing and
viewer perspective in a manner somewhat similar to traditional filmmaking, but would do well to define
narratively motivated perspectives and only cut to novel viewpoints, so as to avoid causing audience
37

discomfort. Lastly, VR directors can direct the attention of viewers through a counterpoint of
environment design, lighting design, character/object blocking, and spatial sound design. Together, these
elements create stylized dynamic worlds that can choreograph a user’s attention and experience in
immersive stories. I hope that as this piece is seen by more viewers, the techniques we’ve laid out here can
percolate outward to other immersive creators who want to tell dramatic narratives in VR.
38

References
Bolas, Mark. “Light Field Imaging.” Presented to the Visual Effects Society. 2015, Los Angeles, CA.
https://www.youtube.com/watch?v=ftZd6h-RaHE.
Brillhart, Jessica. “In the Blink of a Mind — Attention.” Medium, 2016,
https://medium.com/the-language-of-vr/in-the-blink-of-a-mind-attention-1fdff60fa045.
Carmack, John. “#1342: John Carmack.” The Joe Rogan Experience, Spotify, Aug. 2019.
https://open.spotify.com/episode/6kOa4mSuZcIOEPaGmDZf9r?si=niElijWjTV2rV4KZoJ4-Hw.
Hitchcock, Alfred. “Harold Lloyd Master Seminar.” Presented at the American Film Institute. 1970,
Beverly Hills, CA. https://www.afi.com/master-seminars-from-the-afi-archive/.
Home After War. Directed by Gayatri Parameswaran. NowHere. 2018.
https://www.meta.com/experiences/2900834523285203/?utm_source=www.homeafterwar.net&utm_medi
um=oculusredirect.
Invisible. Directed by Doug Liman. Condé Nast Entertainment, Jaunt. 2016.
https://www.youtube.com/watch?v=MCip8T-_Yvw&list=PLiNGxb5ChRJ_UvjgphsN8Sr4kvDLAwm28&
index=1.
Naimark, Michael. “VR / AR Fundamentals — Prologue.” Medium, 2018,
https://michaelnaimark.medium.com/vr-ar-fundamentals-prologue-b7aa3d119087. Accessed 2023.
On the Morning You Wake (At the End of the World). Directed by Arnaud Colinart, Pierre Zandrowicz,
Mike Brett, Steve Jamison. Archer’s Mark, Atlas V. 2022.
https://www.meta.com/experiences/5334662579895130/.
Parameswaran, Gayatri. “VR for Good: ‘Home After War’ Explores the Civilian Impact of IEDs in Iraq.”
The Voices of VR Podcast, Jan. 2020.
https://voicesofvr.com/878-vr-for-good-home-after-war-explores-the-civilian-impact-of-ieds-in-iraq/.
Slater, Mel, et al. “A separate reality: An update on place illusion and plausibility in virtual reality.”
Frontiers in Virtual Reality, Vol. 3, 27 June 2022, https://doi.org/10.3389/frvir.2022.914392.
39

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Untitled: an exploration of virtual reality as a cinematic medium

PDF

Iterating adaptation in Morte Arthure: a perspective on adapting literature to interactive media

PDF

Penrose Station: an exploration of presence, immersion, player identity, and the unsilent protagonist

PDF

Morana: explore healing potential of virtual reality storytelling

PDF

Virtual documentary: the virtual real and its rhetorical legitimations

PDF

Exploring space literally & figuratively in Eight Moons to Eternity

PDF

Jizo No Akumu: A psychological horror game exploration of generational trauma and mixed-race identity

PDF

The mountain calls: an exploration of immersive storytelling through art and level design

PDF

The Glitch Witch

PDF

Earth and ash: inventing (not reinventing) within interactive spaces

PDF

Installation wizard: how communication makes or breaks a game

PDF

One little change: exploring relationship abuse in a virtual reality narrative

PDF

Real fake rooms: experiments in narrative design for virtual reality

PDF

Silent sparrows: exploration into augmented reality narrative frameworks

PDF

Defectum: a journey toward embracing failure

PDF

Brinkmanship and the process of narrative design

PDF

Developing a playful situational creator for mixed reality: design analysis on Neon City, a city building game

PDF

Past imagination, present creation and the reality of tomorrow: explore the impact of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) design

PDF

Ascension: a look into context-dependent memory development in virtual worlds

PDF

TCam: Touch driven communication for virtual moviemaking

Asset Metadata

Creator Fairchild, Nathan (author)

Core Title Virtual production for virtual reality: possibilities for immersive storytelling

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Degree Conferral Date 2024-05

Publication Date 05/28/2024

Defense Date 04/17/2024

Publisher Los Angeles, California (original), University of Southern California (original), University of Southern California. Libraries (digital)

Tag immersive film,immersive filmmaking,OAI-PMH Harvest,virtual production,virtual reality,VR,XR

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bolas, Mark (committee chair), Fisher, Scott (committee member), Nealen, Andy (committee member), Russworm, TreaAndrea (committee member)

Creator Email nfairchi@usc.edu,nfairchild93@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC113968219

Unique identifier UC113968219

Identifier etd-FairchildN-13038.pdf (filename)

Legacy Identifier etd-FairchildN-13038

Document Type Thesis

Format theses (aat)

Rights Fairchild, Nathan

Internet Media Type application/pdf

Type texts

Source 20240529-usctheses-batch-1163 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu