University of Southern California Dissertations and Theses

Kinesthesia: a multi-sensory gesture driven playground of the future

(USC Thesis Other)

Kinesthesia: a multi-sensory gesture driven playground of the future

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Kinesthesia
A Multi-Sensory Gesture Driven Playground of the Future
By
Keshav Prasad
Master of Fine Arts
Interactive Media & Games Division
School of Cinematic Arts
University of Southern California
May 2019

Kinesthesia Page 2 of 31
Acknowledgements
Thank you to my family for supporting and loving me unconditionally. Thank you to my
teachers for seeing things in me that I did not see in myself. Thank you to my classmates for
expanding my world view. Thank you to my team for staying by my side on the course of this
journey, even though there were times I could not see the way forward.

Kinesthesia Page 3 of 31
Abstract
Kinesthesia is an interactive visual-music installation designed for music festivals. Even
though music festivals are fully immersive experiences, rarely, if ever, do they consist of
interactive experiences. I envision a music festival of the future that uses cutting-edge interactive
technology to facilitate playing with and physically embodying music. In order to realize this
vision, I built a system with motion-sensing technology and custom software, and have installed
prototypes of Kinesthesia in multiple music festivals, or other similar contexts over the last year.
The experience places participants inside of a living, breathing audio-visual painting inspired by
Abstract-Expressionist painter Jackson Pollock. Player movement feeds into parameters of a
generative audio-visual system, which grows and evolves in response to increasingly intense
motion on part of the user, and invites players to experiment in real-time with varying degrees of
intensities of gestural movement in space. My design intention was to make players of the
experience feel spontaneous creativity, and to feel as if their body is merging into the
surrounding aesthetic environment. The aesthetic draws inspiration from the paintings and
artistic statements of Jackson Pollock. I had to solve not only how to measure players’
movement, but also how to design a system that responds to motion with aesthetically
compelling images and sounds. Over the past year, installations of Kinesthesia successfully
inspired participants to adopt a creative, self-expressive state of mind, and it increased physical
engagement with live music. Moreover, my interpretation of Abstract Expressionist aesthetics as
a design pattern provides a novel conceptual framework for exploring new paradigms of
computer-human interaction.
Keywords: Audio-Visual, Computer-Human Interaction, Embodiment, Games, Procedural Art

Kinesthesia Page 4 of 31
Table of Contents

Acknowledgements 2
Abstract 3
Table of Contents 4
Introduction 5
Survey of Prior Literature 8
Responsive Environments 9
Calligraphic Media 12
Abstraction, Immersion, and Flow 13
Shared Encounters in Public Spaces 14
The Future of Interaction Design 16
Development Process 17
Measuring Movement 18
Image Making 19
Prototype Exhibitions 20
Exhibit 1 21
Exhibit 2 24
Exhibit 3 27
Conclusion 29
Works Cited 31

Kinesthesia Page 5 of 31
Introduction
This project began with a peculiar, recurring dream. I dreamt I was a jellyfish.
Emancipated from the rigid bodily confines of my waking self, I floated in sync with the rhythm
of the ocean. Although I could not feel my body, my sensation of the water surrounding me was
heightened, as if my body had expanded and coalesced with the entire ocean. This dream
inspired me to create an interactive art project that captured the sensations and emotions I
experienced in my dream: a profound feeling of rhythmic connection to my surroundings. I
envisioned a space that used interactive technology to interpret people’s body movement, and
then responded in real-time to the movement with audio-visual stimulus. I imagined an
environment that could mirror the rhythms of the people walking around inside of it. I dabbled as
an amateur oneironaut by maintaining a journal of my dreams and the recurrent symbolic
patterns inside of them. I started to observe a common theme of expanding outside of my own
body, of melting through the boundaries of self and surroundings. On a fateful trip to an art
museum the summer before beginning the project, I experienced a particularly visceral emotional
response to an art exhibition.
I was moved and inspired after visiting the San Francisco Museum of Modern Art
(SFMOMA) exhibition on Abstract Expressionist painting the summer before beginning my
project. At the exhibition, I felt myself swimming in a vast ocean of creativity. The paintings at
SFMOMA blurred the boundaries between my subjectivity and my surrounding environment.
Simultaneously excited and humbled by the experience, my intuition lead me to further
appreciate my enthusiasm for music festivals.

Kinesthesia Page 6 of 31
Music festivals celebrate the shared social experience of live music performance. Often
held for multiple days at events across the world, music festivals have grown increasingly
popular and sophisticated over the past decades. The internet’s transformation of how songs are
discovered and distributed has radically disrupted the economics of the music industry. Artists
make less money from sales, but benefit from increased visibility. Therefore, artists entice
audiences with live shows - experiences that cannot be recreated by simply playing the artist’s
songs at home. Musicians collaborate with event design companies to build spectacular,
memorable shows that must be experienced live. Although the music festival is itself a modern
invention, live music events and communal festivals are ancient traditions. Festivals like these
are an opportunity for people to gain an intimate connection to the art, one that blurs the
boundaries between art and life, artist and audience alike. I interpreted the symbolism of my
jellyfish dream as an unconscious rumination on the relationship between the self and its
surroundings.
Jellyfish are essentially thin sacks of skin that contain water and consciousness (Berwald
9). Lacking both a spine and a central neural network, it may appear at first glance that these
creatures are primitive and unremarkable. Nevertheless, they are one of the most ancient
surviving organisms on the planet, and one of the few marine life forms that not only resist the
deleterious effects of ocean acidification, but rather thrive on it. The simplicity of the jellyfish’s
biological design is a strength, granting it tremendous adaptability, and consequently, an
impressive resilience to millions of years of radical ecological transformations (Berwald 35-37).
Modest in stature and mutable in form, jellyfish maintain close synchrony with their ocean

Kinesthesia Page 7 of 31
environment. They are not just from the ocean - they are the ocean. The jellyfish’s affinity with
its containing space reminds me of Jackson Pollock’s relationship to his paintings.
Jackson Pollock’s style of Abstract Expressionism abandoned representing nature,
mythology, and religion, and tried instead to convey his inner life. Pollock stopped painting on
an easel entirely, instead laying his canvas on the floor. He developed a technique called “drip
painting”, splattering the canvas with several different layers of acrylic paint. Upon first glance,
Pollock’s paintings appear to be haphazard and devoid of form, content, or meaning altogether.
Upon closer study, however, these paintings contain complex compositional patterns.
Painstakingly crafted one layer, one texture at a time, often over the course of several weeks,
each piece is a symphony of rhythmic paint strokes. About his process, Pollock has said:
When I am in my painting, I am not aware of what I'm doing. It is only
after a sort of 'get acquainted' period that I see what I have been about. I have no
fears about making changes, destroying the image, etc., because the painting has a
life of its own. I try to let it come through. It is only when I lose contact with the
painting that the result is a mess. Otherwise there is pure harmony, an easy give
and take, and the painting comes out well...I can control the flow of paint; there is
no accident…(Pollock 544)
Drip painting can be thought of as an attempt to synchronize the painter’s brush with the
painter’s spontaneous imagination. One observes their own emotional state, drawing on instinct,
to uncover the underlying patterns of composition onto their canvas. This method of
synchronizing with one’s subconscious inner thoughts and feelings blurs the boundary between
the artist, their thoughts, and the work that they create.

Kinesthesia Page 8 of 31
In this paper, I will discuss how my dreams and subconscious inspirations motivated a
series of design decisions and ultimately manifested an interactive installation. The first section
begins with a review of prior literature relevant to my inspiration. The second section discusses
the mixed methods research design process, which consists of data-driven user testing and
ethnography. In the third and final section, I reflect on my personal artistic journey, analyze the
data gathered from empirical study, and conclude by speculating on future possibilities for the
project.

Kinesthesia Page 9 of 31
Survey of Prior Literature
Responsive Environments
Krueger’s seminal work from the late 1970’s defines the term “Responsive Environment”
(Krueger 1).
A Responsive Environment is an empty room in which a single participant's
movements are perceived by the computer which responds through visual displays
and electronic sound. Since 1970, video projection of computer graphic images
has been used to provide the visual response. (Krueger 1)
His project Videoplace , submitted to SIGGRAPH in the 1970’s, provides an example of a
responsive environment that uses a computer-drawn representation of people to stimulate a new
type of interaction between people and their surrounding space. Participants walk into a room
with blank white walls, an overhead camera facing them, and a rear-projected screen that faces
them (Figure 1). The camera sends a video feed of the participants to a computer program, which
uses a computer vision algorithm to isolate the participants’ figures from the background and
creates a colorful, two-dimensional virtual avatar of each participant’s body to be projected onto
the screen. People can control their digital avatar, as well as create motion trails and interact with
digital objects in the background of the projection screen by using physical hand motions to
manipulate the size or position of these objects.

Kinesthesia Page 10 of 31

Figure 1
Krueger’s use of a camera as a sensor and computer-vision software to analyze raw input from
the camera was highly innovative at the time, and prefigures much of the work done by
Microsoft’s Kinect team: pose recognition, response to numbers of people, and so on.
Furthermore, he innovated in the field by defining “response as a medium”. This work provides
me with inspiration for thinking about novel interaction patterns between people and their
surrounding physical space.
Close study of Videoplace provided me with a framework for analyzing and devising the
aesthetics of action and reaction between participant and their surrounding space. The
responsiveness of the space and the relationship created between the environment and its
participants is unique to this medium. Specifically, I drew inspiration from the use of a digital
avatar to facilitate participants’ subjective identification with the projected imagery in the space.

Kinesthesia Page 11 of 31
A to-scale recreation of the participant’s body, what I refer to as a “digital silhouette”, aids in
creating a one-to-one relationship between body and space (Figure 2).
This “digital silhouette” becomes a formal element of the aesthetic composition on the
projected screen, implicating the participant’s body as an integral compositional element of the
space. Motion trails, or colorful traces of the participant’s body-movement from previous
moments in time, provide direct visual feedback that the computer system in the space is
measuring body movement. Moreover, motion trails give the participants a sense of agency over
the space.

Figure 2

Kinesthesia Page 12 of 31
Calligraphic Media
Sha Xin Wei’s paper discusses several multi-media installations created by the
Topological Media Lab at Arizona State University from the early 1990’s to the present day.
These projects all explore a specific configuration of multi-media within the larger category of
“Responsive Environments” defined by Myron Krueger’s earlier work. Wei calls these
installations examples of “Calligraphic Media” (2). Calligraphic Media are defined as:
continuous fields of video and sound or other computationally
animated materials, continuously modulated by gesture or
movement… One of my key experimental motivations is to explore
how we could make possible a compelling experience without relying
on pre-scripted, linguistically codable, narrative structure. (Wei 3)
This work provides useful conceptual models for thinking about gestures,
which I define as the movement of a human body through space. In order for a space
to be responsive to the human body, the sensors and software within the space must
be programmed with a set of rules for categorizing the meaning of various gestures.
This is a complex problem, as Wei notes, because it is programming a computer to
discern between a gesture made intentionally and one made by accident or chance is
mathematically and philosophically confounding (12).
The Topological Media Lab addresses this design problem by measuring
overall changes in the intensity of how the participant’s body is moving in space
rather than creating discrete events triggered by a specific pose or poses. The systems
they build are like musical instruments: musical instruments react continuously to the

Kinesthesia Page 13 of 31
input of the musician, and do not judge the correctness of the input. The correctness
of the input into a musical instrument is determined not by the musical instrument
itself, but rather by the musician’s and their audience’s ears. Certain series of inputs
into a piano have been codified as harmonic scales by the social agreement of
musicians and music appreciators rather than by the people who build pianos. I
applied this insight into the design of my own system. Rather than creating a set of
rules for categorizing a specific set of gestures or body motions, I decided to
experiment with rules that distinguish between intensities of movement.
Abstraction, Immersion, and Flow
Kim & Kim conduct a theoretical and historical analysis of Abstract Expressionist
paintings in order to draw parallels between the formal qualities of these paintings and of
interactive public artworks from the early 2000’s. Abstract Expressionist painters highlight the
material of the painting rather than representing real-world objects. The analysis advances a
novel interpretation, however, by arguing that Abstract Expressionist paintings represent the
process of natural phenomenon rather than the form of natural phenomenon.
“Pigment is dispersed throughout the canvas, forming various effects, and the
pigment seems to completely remove the illusive elements and combine into the
‘painting itself’ by revealing the properties of the canvas or pigment as they are…
The artists’ behaviour and physical phenomena function in a unified action.”
(Kim and Kim 5)
The paintings visualize the process of paint spreading across the canvas in response to the
force generated by the painter’s motion and by gravity. The authors draw a parallel between this

Kinesthesia Page 14 of 31
observation and the properties of interactive art works in which the player must discover the
rules of the system that the artist has programmed into the experience. Players of interactive art
have the ability to discover rules about their relationship to the space when they receive enough
feedback and indication that their actions are causing a visual response in the space in front of
them (Kim and Kim 7). This analysis provides me with a framework for unifying participants
with their surrounding space by codifying the aesthetic principles of Abstract Expressionist art
into a gesture-tracking computer program.
Participant motion trails can be resampled by a computer program to generate continuous
changes in the audio-visual systems surrounding the participants. As long as the system provides
a clear and easily understandable relationship between body movement and the changes that
propagate throughout the space, the audience’s body movement becomes the driving force
behind the audio-visual stimulus of the space. Thus, participants are unified with their
surroundings through the use of motion-sensing technology and real-time audio-visual software
design. In order to further increase the immersive aspects of the interaction, I consider the social
context surrounding the space and guiding the participants’ behavior inside of it.
Shared Encounters in Public Spaces
Through a series of public media facade interventions, Hornecker et. al identify the
importance of designing not only the media screen, but also of integrating spatial and physical
interaction design into the creation of a public media facade. This paper emphasizes the
importance of studying social context and behavior as a part of the media design process,
defining an instance of an interaction between social context, participant interaction, and public
media artifact as a “Shared Encounter” (Hornecker 1).

Kinesthesia Page 15 of 31
the interaction between two people or within a group where a sense of
performative copresence is experienced by mutual recognition of spatial or social
proximity’...They are a symbolic intervention, with the ambition to
understand urban space, not as data source but as experience itself.” (Hornecker
2-3)
The authors created an experience called the SMSlingshot, in which a cellphone integrated into a
slingshot moulding, can be pointed at media wall on a building, and a text message can be fired
onto the building. Hornecker et. al set up the SMSlingshot at many different public settings with
a variety of floorplans and social contexts, ranging from informal public spaces to private
corporate buildings. The authors use video and photos of audience participation, aerial layout
diagrams of each space, and individualized user test data in the form of exit interviews as the
three measures of how social context and media design create a specific, contextualized user
engagement (Hornecker 3-4).
This methodology of testing the same core interactive experience at a variety of locations
and using mixed methods of research to gather data on the types of interactions afforded by the
interactive media design provide me a framework for designing user tests for my own interactive
installation. I am interested in intervening at music festivals in order to transform music
appreciation into direct interaction and embodiment of the music by the participants of the
festival. In order to understand how users behave in social settings like a music festival, I can use
this framework to identify common layouts of festival-like settings, the general social
atmosphere of public settings similar to a music festival experience, and gather data from

Kinesthesia Page 16 of 31
participants of the experiment using mixed methods in order to analyze how spatial and social
context materialize in specific embodied interaction patterns.
The Future of Interaction Design
Bret Victor’s “A Brief Rant on the Future of Interaction Design” critiques contemporary
visions for interaction design. Victor surveys existing touch-screen applications for tablets and
phones as well as speculative films that imagine future touch-based interactions with holographic
augmented-reality technologies. In both cases, these paradigms abstract the interaction to a single
point on a 2-Dimensional surface, ignoring the intrinsic information and feedback provided by
the physical relationship between real-world objects and the human body.
Although there is a rich design language around gesture in the prior literature discussed
above, as Bret Victor’s essay concludes, few of these paradigms have made notable influence on
mainstream interaction design frameworks. Although we do not see it, we are perpetually
submerged in an ocean of digital information. Despite its importance in the development of
communication, gesture is subordinate in its role in communication today. Digital
communication prioritizes symbolic expression. This project provides a playful and artistic
experience that demonstrates the potential of full gestural interaction with music to a large
audience with less exposure to the theoretical discussion of gestural computer-human interaction
design.
My initial dream of the jellyfish surrounded by an ocean reminded me of the importance
of embodied experience of the space surrounding me. In creating an artistic interactive
experience that captures the feeling of this dream, I want to create a space that uses interactive
design paradigms to blur the boundary between people, space, and interface. Participants of the

Kinesthesia Page 17 of 31
experience make meaning out of the abstract experience over time as they move around the
space, slowly discovering that there are rules for how their body movements and postures are
re-interpreted into audio-visual stimulus. People can choose to make meaning through
improvisational movement through the space, creating an audio-visual symphony of motion.
They go in and out of form in a loop to demonstrate that their body is being processed by the
system.
Development Process
Unlike other art forms, interactive art requires the direct input of its audience (Kwastek 2).
My personal artistic motivations are tempered by a pragmatic design philosophy. I want to
consistently evoke a specific experiential and emotional reaction from players of my installation.
Even though I am inspired by a subjective personal interpretation of abstract paintings, the
feeling of spontaneous embodied creativity that Jackson Pollock felt while painting is highly
specific. In his artist statements, he says, “On the floor I am more at ease. I feel nearer, more a
part of the painting, since this way I can walk around it, work from the four sides and literally be
in the painting” (Pollock 42-45). Players of the installation will embody the art of the space and
feel as if they are the audio-visual stimulus itself .
In order to realize this experience goal, I built a real-time flowing system of data that
processes information from a Microsoft Kinect version 2 sensor. Data runs through a real-time
graphics programming environment called Derivative TouchDesigner. TouchDesigner processes
skeletal and infrared data from the Kinect into visual content, and then communicates via Open
Sound Control (OSC) protocol with Ableton Live. Ableton Live activates time-sequenced loops
of audio in-sync with the visual content created in TouchDesigner. The players’ movements are

Kinesthesia Page 18 of 31
then UV-Mapped onto a 3D model of an object in physical space, and projected onto the
immersive walls surrounding players. Players are engulfed in an audio-visual space generated by
their own movement. They embody this sensory experience. They animate it and it animates
them.
Measuring Movement
Initial prototypes used readings from the Kinect Version 2’s infrared sensor because of its
high resolution color and depth-sensing cameras. I decided on TouchDesigner as the
development tool because of its ability to rapidly prototype data manipulation. The Kinect
Version 2 sensor reads depth data from audience participants, converting the infrared signal into
a gray-scale image representing depth from the Kinect, and then filters this image data through a
luminance threshold ignoring pixels outside of a specified value.

Kinesthesia Page 19 of 31
This creates a real-time black/white silhouette matte of players. In other words, I am
converting infrared light readings into a matte of players’ bodies in order to create real-time
image compositions. The ability to visualize computer-generated effects implemented on Kinect
data in real-time facilitates rapid iteration across visual designs, programming, and interaction
design not only because of its intrinsically visual approach, but also because it facilitates a
unifying design language by representing motion data in visual form.
Image Making
Research by a Physics Professor at Portland State University found that Jackson
Pollock’s paintings are perfect fractal images - meaning that they have an internal mathematical
consistency that can be modelled by a computer (Taylor 120-123). I used Perlin Noise and
Similar Algorithms to create textured PBR Materials in order to recreate the texture and rhythm
of Pollock digitally (pictured below)

Kinesthesia Page 20 of 31
Another 8x1 Perlin Noise Texture generates a procedural color look-up table. Parameters
on this texture are set to discover high-contrast opposing colors that spread across the lighting
information on the PBR texture (pictured below).

Prototype Exhibitions
Interactive installations are difficult to prototype because it is uniquely difficult to reproduce the
sensory experience of being in a public space during a play test and a prototype. The sensory
affect of interacting with media at a festival is significantly different than from interacting at
home on a computer or television screen. Moreover, the proxemics and social atmosphere of a
music festival are a part of the aesthetic experience that greatly affect the user experience.
Therefore, I conducted a series of shows every few months in public settings. At each stage, I set
an intended user experience goal and series of hypothesis guiding the incremental updates to the
project. I conducted a series of smaller iterations and playtests between each exhibition in order
to refine the experience based on feedback from the prior installation. Then, before each
exhibition, I would make note of the significant changes made to the project’s aesthetic and

Kinesthesia Page 21 of 31
technological features. I then measured user interaction with the system in three distinct ways.
Observations of the interactions recorded on video, exit interviews with players of the
experience, and numerical analysis of raw system metrics. I began initially with video
observation and exit interviews in order to verify whether experience goals were generally being
met, and to gather feedback on how to add functionality based on how players engaged the
experience. Once the project had enough data and development time to reach a stable Alpha, we
introduced a system for measuring the numerical inputs of users into the experience. This
allowed us far more precise heuristic for validating or disproving any hypotheses for how players
engaged the experience.
Exhibit 1
The Getty Unshuttered in June 20, 2018. Approximately 2,000 people registered for the
event online.

Kinesthesia Page 22 of 31
I placed the Kinect and control software far behind the wall. The projector mounted far
above the crowd. Hiding the technology created a sense of magic. The audience had no idea
how, but suddenly, they were on-screen, larger than life, part of a glorious tableau of color,
motion and form.
Informal exit interviews and footage of the participants interacting with the exhibition
were the primary methods of design research for this exhibition. Playtesters understood that their
body appeared on the canvas, they moved around and particularly appreciated motion trails and
other visual feedback for movement. Look 1 (pictured below) created a block background
disrupted and displaced by player movement. The faster the players move, the more
displacement the background receives. This form of visual response from the system for player
movement was easy for people to understand and created a feedback loop of increasing
experimentation with body movement.

Kinesthesia Page 23 of 31
In Look 2 (pictured below) player’s bodies displaced bubbles appearing in the
foreground. This encouraged players to reach out for the bubbles to try and burst them. The use
of a foreground color to encourage an action has been noted as a successful strategy for
encouraging movement around the space.

Another technique for encouraging movement is the use of particle systems that receive
player’s velocity as an input. The faster players move, the farther the particles are knocked away
from the players’ bodies. This technique is more effective than the previous look of simply
displacing bubbles because the interaction system provides room for increasing levels of

Kinesthesia Page 24 of 31
participation by encouraging faster movement speeds in order to knock the particles farther
away.

Exhibit 2
A series of three user tests conducted over one month with students from the University
of Southern California. I A-B tested interactivity, the type of digital avatar, and control over
music through gesture. I also had to create a way to simulate the installation experience for
personal use. It is impractical to test new spatial layouts, interaction paradigms, and audio-visual
aesthetics at only public installations. This would make it very difficult to experiment in between
installations. Therefore, I thought about an installation design that could be recreated with a
personal projector and portable gear. This approach allowed me to create an easy to transport
prototype that I can set up at many locations, with an install set up time of less than 10 minutes
in length. Despite its small scale and fast construction time, the prototype must make players

Kinesthesia Page 25 of 31
feel like they are submerged in imagery and sound. This is achieved with a folding, four-panel
silk screen, where individual panels are 4’x2’ in size (pictured below).

The four-panel display surrounds players from three sides, creating a feeling of
envelopment into the projected imagery of the space. Although I plan to create a final
installation of 20’x20’, I lack the storage space to set up such a large installation for regular
testing, so testing user responses to a small-scale prototype allows for frequent and low-cost
prototyping of an otherwise massive installation with a complex set up and tear down process. I
built a to-scale model of the 4-panel silk screen, and then created a UV coordinate map of the
screen that converts a rectangular video file into the shape of the prototype installation structure.
This image can then be configured for projection mapping by placing a virtual camera inside of
TouchDesigner, with its distance from the 3-D model of the miniature structure corresponding to
the real-world projector placement (pictured below).

Kinesthesia Page 26 of 31
Over the course of tests with 30 users over a period of two months, play testers generally
commented that they understand that their body movement corresponds to changes in music and
sound. If users do not see a silhouette, they find it very difficult to map what they are doing to
the feedback they are seeing. There is also a certain amount of dysmorphia created by attempts at
high-fidelity representation of players. For example, if I analyze their skeletal data to generate an
avatar, people fixate on the imperfections in the representation of their skeleton. An abstracted
avatar of players reduces the confusion and awkwardness of representational shortcomings. The
final design presented players as flat, single color tone silhouettes, each with a unique color,
outline and motion trail. The individualized silhouette design simultaneously anonymizes and
individualizes each player’s avatar. Their avatar is a two-dimensional abstraction of their form,
but its visual style maintains personalized effects.
One of the most difficult design problems to solve is striking a balance between giving
players a sense of agency while consistently maintaining a cohesive aesthetic. For example,

Kinesthesia Page 27 of 31
initially I designed the music to slow to a crawl and the instrumental layers of the DJ set to mute
if players are not moving. This choice assumed that increasing the music’s tempo and number of
playing instruments in response to increasing motion would incentivize players to move rapidly
and spontaneously. During formal playtests, many users did not understand how to control the
piece and feared they were not playing it properly. Diminishing the music so suddenly not only
confused players, because of the sudden and drastic changes, but also inhibited their willingness
to move.
Exhibit 3
In the third iteration on Kinesthesia, the only source of input into the system will be the
overall speed at which players move inside of the space. The faster that players move, the more
change will be created on each of the audio-visual systems that surround them .The new approach
is to analyze each player figure’s pixel count per frame. Velocity can also be accessed by
lowering image resolution, caching multiple frames in memory, performing a difference
composite, and then setting switch case events for events where velocity is above a threshold
(pictured below).

Kinesthesia Page 28 of 31

An audience of several hundred USC students from all backgrounds as well as a select
group of industry sponsors will be able to experience this iteration on March 30, 2019 at USC
SpringFest, a music festival annually conducted by the University of Southern California. In
Collaboration with the USC Iovine & Young Academy of Art, Design, and the Business of
Innovation, I will install a 20’x20’ implementation of my work so far. The installation will be on
display inside a 10,000 square foot musical Playground, inviting festival-goers to explore the
space and connect with music through new multi-sensory experiences (concept rendering
pictured below).

Kinesthesia Page 29 of 31
Conclusion
My dreams of jellyfish have manifested into an interactive audio-visual installation
intended to accompany and augment music festivals. Music festivals are events that submerge
audience’s minds into a world of aesthetic stimulation. Like jellyfish in the ocean, people dance
around to the rhythm of their environment, and the only separation between individual and
environment is each individual’s subjective, embodied perception of itself and its surroundings.
Kinesthesia’s audio-visual aesthetic references Abstract Expressionist painter Jackson Pollock in
order to transform music festival participants’ embodied perceptions of their surroundings.
Pollock’s drip paintings do not attempt to represent scenes from the natural world, nor do they
depict scenes from mythology or religion. Instead, Pollock’s canvas is a physical record of his
creativity in motion, spanning three dimensions of space, a fourth dimension of time, and finally,
coalescing into a two-dimensional, flat image. His paintings are portals into his imagination. The
painterly aspiration to manifest a tangible artifact of abstract creativity provides a valuable
conceptual framework for understanding how to design responsive environments
Responsive environments operationalize computation in order to mediate the relationship
between spaces and the people inside of them. Unlike the dominant personal computing model of
mouse, keyboard, and screen, interactive digital art installations often explore unprecedented
configurations of digital and physical objects (Kwastek 1-9). In doing so, they provide a unique
method for exploring innovative computer-human interaction design paradigms. To cutting-edge
design researchers such as Bret Victor, computers have the potential not only to enhance human
communication and productivity, but also to expand human consciousness itself. Bret Victor’s
work quotes the visionary ruminations of computer science pioneer Richard Hamming:

Kinesthesia Page 30 of 31
Just as there are odors that dogs can smell and we cannot, as well as sounds that
dogs can hear and we cannot, so too there are wavelengths of light we cannot see
and flavors we cannot taste. Why then, given our brains wired the way they are,
does the remark ‘Perhaps there are thoughts we cannot think,’ surprise you?
Evolution, so far, may possibly have blocked us from being able to think in some
directions. There could be unthinkable thoughts. (Hamming)
I consider Abstract Expressionist paintings tangible examples of thoughts that were
previously unthinkable. Painters encode their inner worlds onto a canvas with abstract colors and
forms. Just as mathematics provides a method for visualizing objective thought and reason,
Abstract Expressionist paintings provide a method for visualizing subjective creativity and
emotion. Kinesthesia references the visual language of Abstract Expressionism in order to
transport its players into an ocean of subjective creative-expression. Data from an infrared light
and a camera is processed by computer vision technology in order to encode players’ movements
through space over time into a two-dimensional signal. My custom-built software then
re-interprets this signal into a visual vocabulary inspired by Jackson Pollock. Finally,
projection-mapping and whole-body interactions make the realm of imagination visible,
physical, and visceral. Players of the experience swim inside the residue of their physical
responses to live music. The only boundaries between body, movement, and space are the
subjective perceptions of one’s own body in the space; but, just like jellyfish in the ocean,
participant’s bodies are merely pieces of the ocean experiencing themselves subjectively. Fluid
motions of self-expression spread waves of sound and image across the space, unifying one and
all in a dance of spontaneous creativity.

Kinesthesia Page 31 of 31
Works Cited

Berwald, Juli. Spineless: the Science of Jellyfish and the Art of Growing a Backbone. Black Inc.,
2018.
Fischer, Patrick Tobias, and Eva Hornecker. “Urban HCI: Spatial Considerations in the Design
of Shared Encounters for Media Facades.” Proceedings of the 2012 ACM Annual Conference
on Human Factors in Computing Systems - CHI '12, 2012 , doi:10.1145/2207676.2207719.
Kwastek, Katja. Aesthetics of Interaction in Digital Art. The MIT Press , 2015.
Kyoung-nam Kim & TaeYong Kim (2012) Utilization of material-focused
paintings in interactive art through the analysis of immersive elements, Digital Creativity ,
23:3-4, 278-290, DOI: 10.1080/14626268.2012.719239
“Pollock Three Statements 1944-1951.” Theories of Modern Art. A Source Book by Artists and
Critics. Herschel B. Chipp. Contributions by Peter Selz and Joshua C. Taylor, by Herschel
Browning. Chipp et al., University of California Press , 1969, pp. 546–550.
Sha, Xin Wei. Poiesis and Enchantment in Topological Matter. MIT Press , 2013.
Taylor, Richard. “Fractal Expressionism—Where Art Meets Science.” Art and Complexity ,
2003, pp. 117–144., doi:10.1016/b978-044450944-4/50012-8.
Victor, Bret. “A Brief Rant on the Future of Interaction Design.” Worry Dream , 8 Nov. 2011,
worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/.

Abstract (if available)

Abstract Kinesthesia is an interactive visual-music installation designed for music festivals. Even though music festivals are fully immersive experiences, rarely, if ever, do they consist of interactive experiences. I envision a music festival of the future that uses cutting-edge interactive technology to facilitate playing with and physically embodying music. In order to realize this vision, I built a system with motion-sensing technology and custom software, and have installed prototypes of Kinesthesia in multiple music festivals, or other similar contexts over the last year. The experience places participants inside of a living, breathing audio-visual painting inspired by Abstract-Expressionist painter Jackson Pollock. Player movement feeds into parameters of a generative audio-visual system, which grows and evolves in response to increasingly intense motion on part of the user, and invites players to experiment in real-time with varying degrees of intensities of gestural movement in space. My design intention was to make players of the experience feel spontaneous creativity, and to feel as if their body is merging into the surrounding aesthetic environment. The aesthetic draws inspiration from the paintings and artistic statements of Jackson Pollock. I had to solve not only how to measure players’ movement, but also how to design a system that responds to motion with aesthetically compelling images and sounds. Over the past year, installations of Kinesthesia successfully inspired participants to adopt a creative, self-expressive state of mind, and it increased physical engagement with live music. Moreover, my interpretation of Abstract Expressionist aesthetics as a design pattern provides a novel conceptual framework for exploring new paradigms of computer-human interaction.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

The future of games and health: towards responsible interaction design

PDF

Ascension: an analysis of game design for speech recognition system usage and spatialized audio for virtual reality

PDF

BeachcomberVR: inspiring exploration: concepts in the production of a virtual reality game

PDF

Last Martelé: a virtual reality approach to analogic violin performance

PDF

Morana: explore healing potential of virtual reality storytelling

PDF

Wetware: designing for a contemporary dilemma

PDF

Cognitive science, belief, and interactive media as a distribution platform for experiences that enhance human development

PDF

Palimpsest: shifting the culture of computing

PDF

Exit

PDF

Light at the End of the Tunnels: level design and its relationship to a spectrum of fear

PDF

Ascension: a look into context-dependent memory development in virtual worlds

PDF

The Toymaker’s Bequest: a defense of narrative‐centric game design

PDF

One little change: exploring relationship abuse in a virtual reality narrative

PDF

The real-time editing system for virtual production and animated interactive project in real-time rendering game engine

PDF

duOS

PDF

The Palimpsest project: producing a cultural shift to enable a systematic shift

PDF

Creatively driven process: using a virtual production workflow to achieve a creative vision

PDF

Epidemic vectors: an action game about mosquito-transmitted disease prevention

PDF

The Death Mask: a study in interactive mystery

PDF

Stepstone Island

Asset Metadata

Creator Prasad, Keshav (author)

Core Title Kinesthesia: a multi-sensory gesture driven playground of the future

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Publication Date 04/28/2019

Defense Date 05/08/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag audio-visual,computer-human interaction,embodiment,Games,OAI-PMH Harvest,procedural art

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Wixon, Dennis (committee chair), Fullerton, Tracy (committee member), Watson, Jeff (committee member)

Creator Email keshavpr@usc.edu,kprasad.1221@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-151485

Unique identifier UC11660426

Identifier etd-PrasadKesh-7302.pdf (filename),usctheses-c89-151485 (legacy record id)

Legacy Identifier etd-PrasadKesh-7302.pdf

Dmrecord 151485

Document Type Thesis

Format application/pdf (imt)

Rights Prasad, Keshav

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA