University of Southern California Dissertations and Theses

Montage of interaction: conceptual exploration and creative practice

(USC Thesis Other)

Montage of interaction: conceptual exploration and creative practice

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Montage of Interaction: Conceptual Exploration and Creative Practice
by
Yuefeng He
A Thesis Presented to the
FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
MASTER OF FINE ARTS
(INTERACTIVE MEDIA)
May 2024
Copyright © 2024 Yuefeng He

Table of Contents
List of Figures.................................................................................................................................iii
Abstract...........................................................................................................................................iv
Chapter 1: Introduction....................................................................................................................1
Chapter 2: The Fundamental of the Montage of Interaction............................................................3
2.1 Interaction Molecule............................................................................................................ 3
2.2 Metaphors and Information Compression........................................................................... 7
Chapter 3: The Significance of the Montage of Interaction and Metaphor..................................... 9
Chapter 4: State of Affairs of the Montage of Interaction............................................................. 11
Chapter 5: Creative Practice of the Montage of Interaction.......................................................... 13
5.1 Problems Encountered....................................................................................................... 13
5.2 Our Solutions..................................................................................................................... 14
5.3 Examples of Our Experiments...........................................................................................18
Chapter 6: Conclusion....................................................................................................................20
Bibliography.................................................................................................................................. 21
ii

List of Figures
1 Screenshot of Psychedelics: Childhood playground...........................................................1
2 Screenshot of Psychedelics: A closed door........................................................................ 4
3 Screenshot of Psychedelics: An opened door..................................................................... 5
4 Screenshot of Psychedelics: Five opened doors with different camera configurations......7
5 Screenshot of Psychedelics: Animals in the forest........................................................... 16
6 Sequence as cuts of shots..................................................................................................17
7 Sequence as instruments on tracks....................................................................................17
8 Screenshots of Psychedelics: Toy airplane in the sky, car in the city, space shuttle in
the solar system.................................................................................................................18
9 Screenshot of Psychedelics: Combination Sequence........................................................19
10 Screenshot of Psychedelics: Shooting rocketlauncher, flying rocket, space shuttle.........19
iii

Abstract
This paper explores the concept of the Montage of Interaction, drawing on the general
principle of construction and assembly, metaphor, and information compression to understand
and create interactive experiences, with a primary focus on video games. Through theoretical
analysis and creative experimentation, we delve into the intricacies of assembling interactive
molecules to foster meaningful experiences among participants or players. By addressing the
challenges inherent in the interactive creative process, we propose possible solutions and the
direction for further experimentation to practicalize the Montage of Interaction. Our results
suggest an alternate framework for creators to move beyond the confines of traditional
interactive genres, enabling them to craft interactive experiences that better align with their
expressive intention. This paper implies a methodology that can open up possibilities for more
nuanced interactive expression, thereby enriching the landscape of interactive media design,
bridging the medium with many other fields of practice, and creating new applications in the
future.
iv

Chapter 1: Introduction
The project "Psychedelics" is an exploration of montage as a way to structure interaction
patterns in gameplay. Rather than focus on delivering a specific experience, our team aimed to
create a development system that allows for rapid assembly, editing, and modification of
sequences while providing an integrated approach to unify visual, audio, and interaction.
Integrating ideas of constructivism, we decided to construct all interactive objects in point cloud
forms with a massive number of cubes, which guaranteed the convenience of material selection
and visual consistency.
Figure 1: Screenshot of Psychedelics: Childhood playground
Additionally, drawing inspiration from tracks and sequencers in audio engineering, all
interactive objects and logic are deployed on different tracks, allowing for convenient and
flexible editing and modification of interactive content. In the current experience, participants
can see clusters of cubes roaming in the three-dimensional world as flocks of fish and forming
various interactive scenes. These interactive scenes guide participants to intuitively engage in a
wide range of orchestrated interactions through movement and point-and-clicks. Such a
framework will support us in quickly bringing our ideas about montage of interaction to reality,
enabling participants to playtest and make adjustments based on feedback.
1

The discussion of montage here does not solely point towards the modern editing
technique in visual media. It is better to view montage as its French roots, meaning to assemble
or to construct (Sperling et al.). This broader concept of montage can be found across various
expressive mediums, from language and hieroglyphics to poetry, musical compositions, and
films. More recently, new variants derived from the aforementioned mediums all have absorbed
the concept of assembly and construction as well, such as the widespread usage of sampling in
modern music, the editing of different materials in fan-made videos, and the re-recording of short
clips according to popular trends. As for interactive media, particularly video games, which will
be our focus in the following discussion, interaction should serve as more than merely a
sequencer linking and arranging other expressive mediums or as a source of attraction to draw
players into experiencing the content. To truly incorporate interaction into expression, it is
necessary to explore and research methods such as montage of interaction that could help
creators organize and design interactive content with clear intention.
2

Chapter 2: The Fundamental of the Montage of Interaction
2.1 Interaction Molecule
Before delving into the details of the montage of interaction concept, creators must first
determine what molecule, the basic unit of construction, creators are assembling.
Modern commercial game engines, like Unity and Unread, have already established
development processes grounded in the assembly philosophy. Developers assemble interactive
sequences with different programming logic, art, music, and other materials as components
(Unity Technologies). These materials, however, should not be considered the basic molecule or
unit for montage of interaction, just as filmmakers would not regard the props, actors/actresses,
or lighting on a film set as the basic molecules of cinematic montage. Filmmakers think of
montage in terms of "shots" because shots are the primary means by which audiences receive
information, and the combination of shots delivers meanings (Sperling et al.).
In the context of interactive video games, we propose to analyze how participants or
players absorb information from three levels: the programming level of data relationships, the
functionality level of interaction, and the metaphor level of interaction.
At the programming level, creating a certain interaction essentially means designing and
developing a data relationship (Meier, 00:02:42 - 00:03:14). When participants perform an
interaction, they are actually creating, deleting, or modifying one or a set of data inside the
program. For instance, when a player stands in front of a door in a first-person game and presses
the interaction key, they, in fact, modify an associated boolean value.
3

Figure 2: Screenshot of Psychedelics: A closed door
Obviously, if only this data is modified, the player would not perceive that an interaction has
happened. In an interactive experience, such a change in data is often processed through a series
of program logic, altering a chunk of related data. The player will feel the input and output cycle
until the final rendering or other data for the end interface gets updated. This kind of data
modification chain triggered by the player is the functionality subjectively designed and
implemented by the creator.
At the functionality level, in the example above, if the creators intend to implement a
feature that allows players to remove an obstacle, such as opening a door, they would design a
series of logic such as: after the associated boolean value is modified, the Y-axis rotation data of
the model and collider of the door will be updated, then related data in the physics engine and
rendering pipeline will alter accordingly.
4

Figure 3: Screenshot of Psychedelics: An opened door
Of course, many other ways exist to modify data to achieve the same obstacle-removal function.
For example, developers can instead modify the X-axis or Y-axis position data, the angle data on
the Z-axis, or delete the door and its collider and create some particle effect data. Nevertheless,
from the player's perspective, the same function, due to different data modifications, will be
perceived differently: "I pulled open a door, " "I slid a door, " "I lifted a door, " or "I destroyed a
door." These are the different interaction metaphors that players could experience.
At the metaphor level, the different data modification designs above take advantage of
players' common knowledge of doors and create different feelings. Some might consider such a
simple and direct example of metaphor to be a straightforward imitation of repeated actions that
could happen in real life. However, this interaction indeed utilizes players' subjective actions and
the image and changes of the door to make players recognize the meaning of "I opened a door"
from the actual process of inputting commands and modifying data. In Metaphors We Live By,
George Lakoff and Mark Johnson sharply pointed out that metaphor is not just a characteristic of
language or a rhetorical device. Metaphor permeates our lives, and "human thought processes are
largely metaphorical." Humans understand and experience one thing in terms of another through
conceptual metaphors (7).
5

To understand this subject more intuitively, let us slightly change the data modification
logic of opening a door, making it deviate from real-life experiences. Suppose when a player's
interaction modifies the associated boolean value, the program logic also modifies the camera's
height, the player's collider size, and other related data to clearly convey the result of the player
becoming taller. Isolating this interaction, players might describe what they felt happened as "I
opened a door, and then I got taller, bigger, or stronger." Players' subjective actions and the
camera height change borrow the general knowledge of how height and body size affect one's
field of view, leading players to feel taller or bigger. Likewise, as Lakoff and Johnson would
argue, in most cultures, because of physical and social basis, "up," "higher," and "bigger" are
often coherent with "good," which explains the feeling of being "stronger." Since the door
opening and the increase in height occur in the same time window, players link these two events
together, combining the metaphors. Of course, this single metaphor combination within a time
window still seems far from the rich expressive power of literary metaphors. However, we can
imagine further combining this combination with other metaphors in different time windows.
What would players feel if we alternated the metaphor of opening doors to grow taller with
another set of metaphors that make players feel like they are exploring and collecting? What if,
after repeatedly opening doors to grow taller then exploring and collecting, the height becomes
such that players can no longer pass through the opened doors? What if we provide players with
metaphors for actions like crouching and crawling, allowing them to pass through doors opened
after exploring and collecting as they grow taller? These individual metaphors of interaction
could be combined into something more complex, perhaps even possessing a more powerful
expressive quality.
6

Figure 4: Screenshots of Psychedelics: Five opened doors with different camera configurations
Thus, we consider it reasonable to view interaction combinations that provide describable
metaphors within a time window, such as growing taller when opening a door, as the basic unit
or molecule in our concept of interactive montage.
2.2 Metaphors and Information Compression
In the above deduction on metaphors and the process of players perceiving and
experiencing one thing in terms of another, upon closer consideration, it is apparent that the
complexity of 'thing' provided by creators is relatively lower than the ‘other thing' experienced
by players. In the interaction example of opening a door, creators do not need to provide a
complete process of physical contact, mechanical movement of the door, body movement, and
other necessary processes that occur in reality. Metaphors here served as a method of information
compression. Parallel to Lakoff and Johnson's examination of metaphor, computer scientists and
cognition researchers like J Gerard Wolff argued that "much of human learning, perception, and
cognition may be understood as information compression via the discovery of patterns that
match each other" (1). Metaphor closely aligns with the prevalent idea of dictionary compression
in the information compression field. When transmitting information, if we have a dictionary
containing a series of keys and their corresponding values, where the size of the keys is much
smaller than their corresponding values. A sender can transmit a series of keys and let the
receiving side decode the original complex information based on this dictionary. Similarly,
7

creators provide only a considerably simplified interaction combination as keys, and participants
use their "knowledge" as a dictionary to decode a complete and complex sensation.
The "knowledge" dictionary here can be a pre-established conventional compression
protocol, namely common sense or culture. In cinematic montage, a very typical example is the
"creative geography" discovered by Kuleshov (Long 15). If a character in the first shot stands up
and walks towards the exit of a café, and in the next shot, the character walks on the street,
audiences will understand that the character has just left the café and walked onto the street
without any confusion. The audience received the keys of standing up, walking towards the exit,
and walking on the street from these shots. Based on their common sense, they interpreted the
cause and effect of these events. Such a cut of two shots might only take a few seconds; if
conveying the spatial change was the sole purpose, filmmakers could utilize this to omit the
tedious process of the entire action.
Meanwhile, creators can append the dictionary during transmission. If the protagonist is
set to experience several misfortunes in a movie and, before each misfortune incident,
filmmakers cut a shot where oranges appear. After seeing the combination of oranges with
misfortune events a few times, the audience will add the orange as a key to misfortune into their
dictionary. When they see oranges again, they will likely anticipate that a misfortune is about to
happen because of the filmmakers' deliberate choice of how to assemble their shots.
We believe interactive creators can use the montage of interaction to achieve similar
information compression effects.
8

Chapter 3: The Significance of the Montage of Interaction and Metaphor
Suppose creators can apply montage of interaction and its accompanying idea of
conceptual metaphor in interactive design. In that case, they will obtain a method to utilize
interactive expression and convey more complex information and meanings under limited
development resources and constrained experience time.
In the current game production process, creators might start with some creative intention.
Through research, prototyping, and experimentation, they will gradually develop ideas for
mechanics, genres, themes, etc. (Lemarchand 60). By the end of this ideation process, they will
likely adopt an appropriate framework based on the selected genre and theme. For commercial
game products, the genre or theme, hence the framework, may be chosen based on market trends
before the ideation process. Regardless, the chosen framework, evolved from past viable success
and represents specific interaction combinations, becomes a dominant guide in the creative
process, and those interaction combinations will constitute a significant portion of the interaction
sequence. Such fixed interaction combinations defined by the framework limit the range of
metaphors in the creators' hands, because the interaction combination inherent in the framework
itself carries a set of conceptual metaphors.
A framework formed by mature and proven interaction combinations seems to promise
attraction through playability and game depth, meaning players are likely to remain enthusiastic
about repeatedly experiencing the variations of these interaction combinations, which is
undoubtedly vital for interactive game products as commodities. Nonetheless, if the creators'
goals extend beyond making an entertaining interactive game, and the creative intention does not
fully align with the metaphors of any existing framework, relying on a specific framework
9

during the creation process can cause a certain disjunction between the final result and the
creative intent.
Periodically, an interactive game product emerges that boasts exquisite art and sound
design, a story that seems to contain profound meaning, and a mature, common gameplay
mechanic. The player's experience often splits into two parts: playing the game and experiencing
the story. As mentioned at the beginning, interaction serves merely as a tool for linking and
arranging sequences of other expressive mediums and as a source of attraction for experiencing
the content. The contribution of interaction in expressing meaning only extends to providing a
surface-level "role-playing" imagination. This attempt to match meaning with existing
combinations of interactions directly leads to two problems: interaction, as the core of interactive
media, does not fully participate in the expressive process, and the expressive capability through
interactive media is directionally limited.
Such a creation pattern prevents interaction from being considered an applicable
expressive medium. Exploring the montage of interaction concepts aims to provide creators with
a new methodology for constructing interaction combinations that align with their creative
intentions. Moreover, the information compression offered by montage and metaphor raises the
expectation that participants can grasp the intended message from creators in less time. Although
the expectation for playtime in commercial games and the underlying business model is not the
main focus of this discussion, the emergence of interactive game works that are low-cost,
disregard playability and game depth, and can profoundly express the creator's intent in a short
amount of time, such as Nicky Case's We Become What We Behold, suggests the potential for
new directions in creation and business models.
10

Chapter 4: State of Affairs of the Montage of Interaction
Some developers and researchers have proposed ideas about montage of interaction.
However, they either aimed towards arranging visual attractors in the space and cut scenes
(Nitsche) or did not provide detailed concepts and practice processes (Kimball). Nonetheless, in
various commercial games, creators attempted to add different interactions into their games'
established framework, and many achieved remarkable outcomes. In Mario & Luigi: Bowser's
Inside Story, creators combined input-based jump dodge and item usage to its turn-based combat
interaction. In the God of War II, creators added interaction commands for attack and dodge in
their action games' cinematic sequence. In Detroit: Become Human, creators implemented quick
selections of paths and responses into its narrative-driven adventure gameplay. Creators usually
put these designs under the general umbrella of Quick-Time Events or QTE. Some may argue
that these interactive contents primarily come from creators' attempts to enhance players'
engagement or more naturally orchestrate cinematic sequences. Based on players' responses,
these designs undoubtedly effectively convey the actions or coordination of characters, anger,
impulsive and cathartic emotions, and the tension of making quick critical decisions (Gough and
Sanfilippo).
Closely inspecting many other games, people will discover numerous scenarios that are
similar but overlooked, which emerge either from the design or players' creativity. Examples
include expression and movements triggered by radial UIs, quick squatting or jumping as taunts,
and rapidly opening the inventory to equip or unequip items or gears. These interactions,
interspersed within the original set of interactions, endow interactive games with additional
expressiveness, offering players extra experiences. It is necessary to systematically explore these
11

interaction combinations and the montage philosophy behind them, as they are contributing to
the interactive experience expressively, whether intentionally or unintentionally.
Meanwhile, creators can also turn their attention to other fields where research is being
conducted on designed subjective actions and the cognition of the actors involved. In
performance art, performers and theorists seek to understand how specific processes of acting
impact the cognition and emotions of the performers, thereby validating and further developing
performance methodologies (Olenina et al. 2; Rokotnitz 471). In choreography, researchers
discovered the performance process may positively impact dancers' mental and physical state
(Jaque et al. 2-3). Some started experimenting with methods to effectively utilize this positive
impact to extend body awareness or even as a form of physical therapy (King). With growing
interest in bridging dance performances with multimedia presentations, some choreographers
also tried to draw conceptual inspiration directly from interactive media and games (Broadhurst
and Machon 43-56). In psychotherapy, there is interest in the application of regular and simple
repetitive actions, such as lateral eye movements, known as Eye Movement Desensitization and
Reprocessing therapy (EMDR). Although the underlying principles are still debated, EMDR has
been widely used in treatments for PTSD and other disorders (Valiente-Gómez et al. 1-2).
Cognitive linguistics has also long been involved in researching the function of gestures and
peoples' cognition (Cienki 2-3).
Despite the complicated exploration within these diverse fields, they cannot yet provide
interactive creators precise answers on how to construct interactive montage sequences
specifically. Nevertheless, they reveal the potential-filled future and encourage the evolution of
interactive media to look beyond internal-iteration and inspiration from other entertainment
forms.
12

Chapter 5: Creative Practice of the Montage of Interaction
5.1 Problems Encountered
First, unlike passive mediums like film and music, where audiences absorb information
without active engagement, interactive media requires active participation from participants to
extract information. A primary challenge in our interactive montage exploration is how to
quickly familiarize participants with the interaction control and guide them to complete the
interaction sequence smoothly. If we fail to fulfill this condition, confusion or even frustration
can dilute or even pollute the metaphor combinations planned by creators. Although confusion
and frustration can also be parts of the metaphor, depending on the intention, neuroscience
generally believes that active parts of the brain vary when facing different situations. The
ventromedial prefrontal cortex manages negative emotions, assesses risk and fear, and regulates
the amygdala (Motzkin et al. 276). We speculate that when participants do not know how to
perform and complete an interaction combination, this part of the brain becomes particularly
active. If aiming to convey metaphors more effectively, we hope the brain will be more engaged
in learning, thinking, and abstract understanding, which requires the hippocampus (Ackerman
22, 124; Anand and Dhikav 239) and rostrolateral prefrontal cortex (Dumontheil 58) to be more
active. Of course, the brain mechanisms are complex, and various parts function simultaneously.
We should think interaction combinations are trying to stimulate brain activity directly, but some
brain models suggest that during the process of learning and thinking, confusion can lead to a
state of cognitive disequilibrium. If the equilibrium is not restored and allows learners to engage
again, it can lead to frustration and further progress into boredom (D’Mello and Graesser 147).
Creators would only want participants to experience this in very niche circumstances. Therefore,
13

we must ensure participants grasp all interactions involved in the sequence and understand their
goals in any interaction combination.
Secondly, in practicing our idea, creators need to produce various prototypes quickly,
invite participants to playtest, and evaluate the feasibility and expressiveness of different
interaction combinations. During this process, creators must manage another subtle balance: the
unity of materials constitutes different metaphors versus the need for rapid iteration. It is
necessary for creators to select or produce visual and sonic materials that blend together.
Otherwise, participants might also feel confused or surprised when interacting with mismatched
components, which would similarly affect the interaction sequence the creators intend to
evaluate.
Lastly, it is still being determined whether we can apply standard playtest and feedback
collection methods to assess the feasibility and expressiveness of interaction combination
designs. Would questionnaires with example answers influence participants' retrospective
feelings? Complex metaphors require time to be fully digested; would an immediate feedback
survey following the playtest interfere with this process? Can feedback on metaphor
interpretation from participants with diverse common knowledge or cultural backgrounds
represent different groups accurately?
5.2 Our Solutions
We predicted and encountered the problems above when we started to put the concept of
interactive montage into practice. Therefore, we shifted the early focus of our project to solving
these issues.
14

Regarding interaction fluidity, we opted to limit the range of interactions to use in the
Psychedelics project. We focused on interaction involving participants to control their movement
and point-and-clicks interactions for three reasons:
1. These interactions are the basis of most common interactive games today, making them
very familiar so that participants with little game experience could naturally understand
how to engage.
2. As seen in many interactive games, these two interactions can morph into different forms.
Movement can quickly establish a subjective connection between the participant and the
moving character or object, and point-and-clicks combined with different visual and
sonic materials can make participants feel they are performing various actions.
3. We believe these interactions have inherent intuitiveness: "If I can move, I must move
somewhere" and "If I can click, I must click somewhere."
Additionally, we referenced the human-computer interaction design principles proposed
by Ben Shneiderman in his early years, striving to assemble our interactions around intuitive
design. In our test sequences, we associated different point-and-clicks interactions with primary
colors(red, yellow, blue), basic shapes(triangle, circle, square), and fixed audio cues. Since
interaction was our main subject, we decided to abandon most interface prompts, retaining only
cursor indicators to emphasize intuitiveness and reduce cognitive loads on participants. While
text and other symbol prompts could aid participants in comprehending the interaction sequence,
for a better understanding of how interactive montage plays the expressive role, we deemed such
a decision necessary at this exploratory stage.
To ensure both material consistency and rapid iteration, we developed a system to
construct all interactive objects and environments as point cloud clusters with a massive number
15

of cubes. Using the laser radar cameras now commonly found on smartphones, we can easily
scan point cloud data from various objects, environments, and people. These point cloud scans
can be imported into our system developed with the Unity DOTS framework, allowing
participants to interact with the entirety or a portion of the cluster in real time. Due to current
system optimization limitations and because our goal is for participants to quickly recognize
visual materials rather than pursue the point cloud visual aesthetics, we need to process scans to
reduce their density before using them in interaction sequences. With open-source software like
CloudCompare, this process is convenient and fast. CloudCompare can also convert regular
models into point cloud data. Ultimately, we can easily use all kinds of visual materials to
conduct experiments while maintaining visual unity.
Figure 5: Screenshot of Psychedelics: Animals in the forest
Additionally, at the beginning of the project, to embrace the assembly concept, we
planned to emulate the early soviet montage theorist experimenting with different combinations
of shots (Eisenstein and Leyda 48) by cutting and pasting films by managing and composing
interactive combinations as entireties similar to shots, where each switch of interactive
combinations would reshape the entire scene.
16

Figure 6: Sequence as cuts of shots
However, we gradually realized that such a pattern limited us to use continuous space as
a metaphor material, and having every interactive combination completely reshape the space did
not align with many participants' intuition and expectations. Thus, we shifted towards a mode
similar to using sequencers and tracks in music production, managing materials and interaction
logic as instruments assigned to different tracks. When transforming interactive combinations,
only materials on specific tracks need to be modified, making it possible to incorporate spatial
continuity into our design. Now, we can freely and quickly assemble interactive sequences and
modify them based on participants' feedback.
Figure 7: Sequence as instruments on tracks
Meanwhile, we still follow the conventional approach to playtest and collect feedback,
involving observation, verbal questionnaires, and digital surveys. Currently, our experiments still
focus on evaluating intuitive designs and simple short interaction sequences. Nevertheless, it is
17

foreseeable that in the future, when we need to evaluate more complex interaction sequences, we
will need to adjust our methods. In this regard, we still need more practice and exploration.
Thus, we have addressed most of the critical problems related to practicing our vision of
the montage of interaction. In the future, we will continue to refine our system and playtest
methods based on newly discovered challenges.
5.3 Examples of Our Experiments
Alongside the process of resolving issues and refining system design, we also conducted
numerous interactive experiments under various prompts, and here are some examples.
The door opening sequence, shown in Figure 5, was created under the prompt of growth
and progress, using a repetition of bigger and stronger metaphors to show growth. Additionally,
we also assembled different sequences, including one constituted by controlling a toy airplane in
the sky, driving a car in the city, and flying a space shuttle in the solar system, which aimed to
depict transitions in life stages.
Figure 8: Screenshots of Psychedelics: Toy airplane in the sky, car in the city, space shuttle in
the solar system
Noticing the above sequence lacks a sense of confirmation and the door opening
sequence lacks contexts from playtests, we constructed a new sequence with the interaction
molecule from them contemplating each other and aim to create better rhythm from repetition
with variation.
18

Figure 9: Screenshots of Psychedelics: Combination sequence
In this way we can continue improving the sequence towards our creative intention based
on playtest feedback.
Under another prompt of enhancing action, we reused the space shuttle interaction and
assembled it with the shooting rocket launcher interaction and flying rocket interaction.
Figure 10: Screenshots of Psychedelics: Shooting rocketlauncher, flying rocket, space shuttle
The intention is to magnify the action of shooting by letting the player experience the
metaphor of the process of the rocket flying towards the target, and adding the space shuttle
interaction to apply an emotional paint to the result of shooting the rocket.
The system we created also allows us to reuse interaction molecules for different purposes.
Therefore, it is possible to experiment with ideas with minimum extra work.
19

Chapter 6: Conclusion
Through Psychedelics, we devoted ourselves to theorizing the montage of interaction,
and paved the way to put it into practice. We hope methods like the montage of interaction will
help actualize an alternative framework for designing interactive media contents, which roots
from creators’ expressive intention.
At this stage, we have completed a variety of initial theoretical research on montage of
interaction and gradually experimented with several expressive ideas in the Psychedelics project.
There is still a lot of work to be done on playtest and feedback assessment. We will continue
building and refining the experimentation process. Although we have not reached the point of
providing profound findings like the Kuleshov effect in cinematic montage, our work identified
issues that need resolution in practicing montage of interaction and has presented our solutions.
We believe it is critical to establish a development environment that enables creators to rapidly
actualize ideas into minimum presentable form, easily store and organize works into interaction
molecules, and freely reuse and edit interaction molecules.
The discussion so far can serve as a starting point for more practice and application of
interactive montage. Methods like montage of interaction will likely help or inspire further
development of interactive media as a medium of expression, open new possibilities for
interactive games, and participate in a broader discussion and application regarding the general
field of designed subjective actions and the feelings of actors involved.
20

Bibliography
Ackerman, Sandra. Discovering the Brain. National Academy Press, 1992.
Anand, Kuljeet, and Vikas Dhikav. “Hippocampus in Health and Disease: An Overview.” Annals
of the Indian Academy of Neurology, vol. 15, no. 4, 2012, pp. 239–46,
https://doi.org/10.4103/0972-2327.104323.
Broadhurst, Susan, and Josephine Machon. Performance and Technology : Practices of Virtual
Embodiment and Interactivity. Houndsmills, Basingstoke, Hampshire ; New York,
Palgrave Macmillan, 2006.
Cienki, Alan. “The Study of Gesture in Cognitive Linguistics: How It Could Inform and Inspire
Other Research in Cognitive Science.” Wiley Interdisciplinary Reviews. Cognitive
Science, vol. 13, no. 6, 2022, pp. e1623-n/a, https://doi.org/10.1002/wcs.1623.
Detroit: Become Human. Quantic Dream/Sony Computer Entertainment, 2018. Sony PlayStation
4 game.
Dumontheil, Iroise. “Development of Abstract Thinking during Childhood and Adolescence: The
Role of Rostrolateral Prefrontal Cortex.” Developmental Cognitive Neuroscience, vol. 10,
no. C, 2014, pp. 57–76, https://doi.org/10.1016/j.dcn.2014.07.009.
D’Mello, Sidney, and Art Graesser. “Dynamics of Affective States during Complex Learning.”
Learning and Instruction, vol. 22, no. 2, 2012, pp. 145–57,
https://doi.org/10.1016/j.learninstruc.2011.10.001.
Eisenstein, Sergei, and Jay Leyda. Film Form: Essays in Film Theory. Harcourt, 1969.
God of War II. Santa Monica Studio/Sony Computer Entertainment, 2007. Sony PlayStation 2
game.
Gough, Hamish, and Chris Sanfilippo. “12 Games Where Quick-Time Events Were Actually
Good.” TheGamer, 2 Apr. 2022, www.thegamer.com/best-qte-quick-time-events-games/.
Accessed 27 March. 2024.
Jaque, S. Victoria, et al. “Creative Flow and Physiologic States in Dancers During Performance.”
Frontiers in Psychology, vol. 11, 2020, pp. 1000–1000,
https://doi.org/10.3389/fpsyg.2020.01000.
Kimball, Reid. The Interactive Montage. 1 May 2009,
www.gamedeveloper.com/design/the-interactive-montage.
21

King, Hetty. “Somatic Movement Dance Therapy: The Healing Art of Self-Regulation and
Co-Regulation Somatic Movement Dance Therapy: The Healing Art of Self-Regulation
and Co-Regulation Amanda Williamson. Intellect, 2023. 352 Pages; $129.95
(Hardcover): Amanda Williamson. Intellect, 2023. 352 Pages; $129.95 (Hardcover).”
Journal of Dance Education, 2023, pp. 1–1,
https://doi.org/10.1080/15290824.2023.2248838.
Lakoff, George, and Mark Johnson. Metaphors We Live By. University of Chicago Press, 1980.
Lemarchand, Richard. A Playful Production Process : For Game Designers (and Everyone). The
MIT Press, 2021.
Long, Christian B. The Imaginary Geography of Hollywood Cinema 1960-2000. Intellect, 2017.
Mario & Luigi: Bowser's Inside Story. AlphaDream/Nintendo, 2009. Nintendo DS game.
Meier, Luna. "Options for Entity Interaction - Unite Copenhagen." Youtube, uploaded by Unity,
9 Oct. 2019, www.youtube.com/watch?v=KuGRkC6wzMY.
Motzkin, Julian C., et al. “Ventromedial Prefrontal Cortex Is Critical for the Regulation of
Amygdala Activity in Humans.” Biological Psychiatry (1969), vol. 77, no. 3, 2015, pp.
276–84, https://doi.org/10.1016/j.biopsych.2014.02.014.
Nitsche, Michael. ‘Games, Montage and the First Person Point of View’. Digital Games
Research Conference 2005, Changing Views: Worlds in Play, June 16-20, 2005,
Vancouver, British Columbia, Canada, 2005,
http://www.digra.org/digital-library/publications/games-montage-and-the-first-person-po
nt-of-view/.
Olenina, Ana Hedberg, et al. “Embodied Cognition in Performance: The Impact of Michael
Chekhov’s Acting Exercises on Affect and Height Perception.” Frontiers in Psychology,
vol. 10, 2019, pp. 2277–2277, https://doi.org/10.3389/fpsyg.2019.02277.
Rokotnitz, Naomi. “Performance and Cognition: How the Performing Arts Contribute to The
Science of Mind.” Interdisciplinary Literary Studies, vol. 20, no. 4, 2018, pp. 470–85,
https://doi.org/10.5325/intelitestud.20.4.0470.
Shneiderman, Ben. “The Eight Golden Rules of Interface Design.” Umd.edu, 2016,
www.cs.umd.edu/users/ben/goldenrules.html.
Sperling, Joshua, et al. “Montage.” Routledge Encyclopedia of Modernism, May 2016,
https://doi.org/10.4324/9781135000356-remo18-1.
Unity Technologies. “Unity - Manual: Glossary.” Unity Documentation,
docs.unity3d.com/Manual/Glossary.html.
22

Valiente-Gómez, Alicia, et al. “EMDR beyond PTSD: A Systematic Literature Review.”
Frontiers in Psychology, vol. 8, 2017, pp. 1668–1668,
https://doi.org/10.3389/fpsyg.2017.01668.
We Become What We Behold. Nicky Case, 2016. Browser Game.
Wolff, J. Gerard. “Information Compression as a Unifying Principle in Human Learning,
Perception, and Cognition.” Complexity, vol. 2019, Feb. 2019, pp. 1–38,
https://doi.org/10.1155/2019/1879746.
23

Abstract (if available)

Abstract This paper explores the concept of the Montage of Interaction, drawing on the general principle of construction and assembly, metaphor, and information compression to understand and create interactive experiences, with a primary focus on video games. Through theoretical analysis and creative experimentation, we delve into the intricacies of assembling interactive molecules to foster meaningful experiences among participants or players. By addressing the challenges inherent in the interactive creative process, we propose possible solutions and the direction for further experimentation to practicalize the Montage of Interaction. Our results suggest an alternate framework for creators to move beyond the confines of traditional interactive genres, enabling them to craft interactive experiences that better align with their expressive intention. This paper implies a methodology that can open up possibilities for more nuanced interactive expression, thereby enriching the landscape of interactive media design, bridging the medium with many other fields of practice, and creating new applications in the future.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Lake Minnewaska | uplifting a Black narrative in an underrepresented industry | exploring grief themes in a fishing video game

PDF

The mountain calls: an exploration of immersive storytelling through art and level design

PDF

Hearth and home: interior design as tool and weapon in creating an atmosphere of dread

PDF

Exploring space literally & figuratively in Eight Moons to Eternity

PDF

The extension of our reality

PDF

Shepherds - cooperating from divergent path: a design exploration

PDF

Iterating adaptation in Morte Arthure: a perspective on adapting literature to interactive media

PDF

Earth and ash: inventing (not reinventing) within interactive spaces

PDF

Free Will: a video game

PDF

Cards of Heart: a cozy role-playing card game for promoting mental health

PDF

Oopz-Oofs: the creation of a dog simulator game

PDF

Garden designing a creative experience with art and music orchestra

PDF

Mechanics as metaphor: representing physical disability and aging in video game mechanics

PDF

Revisions: an exploration of metafiction and metaphors in game design

PDF

The Glitch Witch

PDF

The Cleaner: recreating censorship in video games

PDF

Jizo No Akumu: A psychological horror game exploration of generational trauma and mixed-race identity

PDF

Within a fold: an exploration of alternative controllers and childhood storytelling

PDF

Courier: Dragons Within, a video game. An exploration into the magic circle as healing circle: restorative game design for a masculine framework free from the template of domination [document]

PDF

duOS

Asset Metadata

Creator He, Yuefeng (author)

Core Title Montage of interaction: conceptual exploration and creative practice

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Degree Conferral Date 2024-05

Publication Date 05/17/2024

Defense Date 05/14/2024

Publisher Los Angeles, California (original), University of Southern California (original), University of Southern California. Libraries (digital)

Tag assembly and construction,games,interactive design,interactive media,interactive montage,OAI-PMH Harvest

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bolas, Mark (committee chair), Huntley, Jim (committee member), Lemarchand, Richard (committee member)

Creator Email yfhe15@gmail.com,yuefengh@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC113939710

Unique identifier UC113939710

Identifier etd-HeYuefeng-12916.pdf (filename)

Legacy Identifier etd-HeYuefeng-12916

Document Type Thesis

Format theses (aat)

Rights He, Yuefeng

Internet Media Type application/pdf

Type texts

Source 20240517-usctheses-batch-1151 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu