University of Southern California Dissertations and Theses

Toward a theory of gesture design

(USC Thesis Other)

Toward a theory of gesture design

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content
TOWARD A THEORY OF GESTURE DESIGN
 
 
by

Diane Tucker

A Thesis Presented to the
FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree of
MASTER OF FINE ARTS
(INTERACTIVE MEDIA)

May 2011

Copyright 2011 Diane Tucker
ii
Dedication

Without the wisdom and kindness of an awful lot of people, Mother Nature would
not have become the verdant home for play and thought that it has.
Indeed, I might well have disappeared from USC before I’d put a single seed on a
screen had it not been for the dogged support of Scott Fisher, the IMD Faculty, and
Program Coordinator Adrienne Capirchio. I remain enormously humbled by the loyalty
and commitment they showed me.
Chris Swain’s support was valuable, for without it, the project Mother Nature would
not have found a home as part of USC’s Advanced Game Project. I also owe great thanks
to Chris for his aid in addressing design and production challenges.
It would have been tough for me to undertake a project that demanded so much
software engineering – let alone so much innovation -- without Thai Phan. Were I a fairy
godmother, capable of bestowing Project Leads a blessing in human form, I’d give them a
Thai: hard-working, humble, of perfect disposition to everyone even in the roughest of
times.
Indeed, I owe the whole team of Mother Nature an enormous and grateful thanks
for their willingness to undertake a project that presented such a novel set of challenges and
for having triumphed over so many of them. That team includes: Software Engineers:
Brandon Booth, Bryan Duran, Jorge Garrido, Yixin Geng, Peter Landwehr, David
Mazzocco, Hiral Patel; Concept Artist: Suni Paek; Animators: Joy Han, Kat Bakonyi,
Janet Y. Lee, Max Rubin; Designers: Jimmy Gorham and Sarah Scialli; Composer:
iii
Simone Scazzochio. I owe Jimmy Gorham and David Turpin special thanks for having
maintained such steadfast confidence throughout.
Tracy Fullerton’s feedback and critiques improved the quality of the player
experience and game play of Mother Nature immeasurably. As much as I worked to
anticipate Tracy’s critiques as I prepared my designs and play-tests in my three years at
USC, there was literally no time at which I presented work to Tracy that she did not
provide a brilliant comment that reoriented my work or thought in an important way. Her
unrelenting attention to and insight about player experience meant that I could launch
work in a new form of play – gestural -- confident that she wouldn’t let me miss something
central about player experience.
I owe enormous thanks to Andreas Kratky for having held such confidence in my
abilities to build, read, think and write and for having urged me on when I flagged.
I’ve been the beneficiary of Laird Malamed’s commitment to helping others. He
provided me very, very helpful feedback throughout this project in spite of an enormously
demanding schedule.
Many thanks to Evan Suma for his help in designing and conducting the study on
gesture and metaphor.
While I’ve never relied on the kindness of strangers, I’d be a miserable mess without
the help, support, love and patience of my family and friends: Marcia, Meredith, Irene and
Julian Tucker; Corinne Stone; Chuck Weinstock , Willann Stone, Katherine Kurs,
Edward Robinson and Naoko Takahatake. They help me to know, every day, how
enormously lucky I am to have them.
iv
Thanks to Microsoft, who willingness to give USC students early exposure to Natal
prompted me to start thinking and reading about gesture when their plans for it seemed to
many like pipedreams.
Finally, there is one person whose help has proven so foundational both to Mother
Nature: The Practice and Mother Nature: The Theory that I can scarcely imagine them
without him. I can’t adequately articulate my appreciation for the year I spent learning
from, working with, and being aided by Mark Bolas. I can only hope he determines that
the time and effort invested was worthwhile.
v
Table of Contents
Dedication........................................................................................................................... ii

List of Figures .....................................................................................................................vi

Abstract..............................................................................................................................vii

Introduction......................................................................................................................... 1

Mother Nature: The Theory................................................................................................. 3
Embodied Cognition: Its Meaning, Scope and Power.................................................... 3
Metaphor......................................................................................................................... 7
Metaphor and Embodiment............................................................................................ 8
Metaphor, Emotion, Mother Nature and Prior Art....................................................... 10

Mother Nature: The Practice.............................................................................................. 13
Mother Nature: The Game ............................................................................................ 13

Mother Nature: Theory and Practice.................................................................................. 18

Mother Nature: Testing the Theory and Practice............................................................... 21

Conclusion......................................................................................................................... 24

Bibliography ...................................................................................................................... 25

vi
List of Figures

Figure 1: Middle Lasso Gesture ……………………………………………………….. 14

Figure 2: High Lasso Gesture …………………………………………………………. 15

Figure 3: Rain Gesture ………………………………………………………………… 15

Figure 4: Grow Gesture ……………………………………………………………….. 16

Figure 5: Aiming Seed …………………………………………………………………. 21

Figure 6: Bloom Up ……………………………………………………………………. 22

Figure 7: Bloom Down ………………………………………………………………….23
vii
Abstract

The enormous transformation in how humans engage with technologies – providing direct
access through touch or gesture, without any mediating controller – has just reached
mainstream computing, games and home theaters, with the recent releases of the Kinect
and the WAVI Xtion. This change has opened up huge new opportunities for the design
of games, interactive experiences and applications. This paper presents the evidence of the
connection between the body and perceptions, emotions, and mental states; the powerful,
extensive, and surprising ways those connections are manifest; and the unexpected and very
potent role that metaphor plays. This paper then presents how that evidence points to a way
of employing the emotional and cognitive armature attached to human movement as a
means of developing emotionally compelling gestural game-play.

1
Toward a Theory of Gesture Design

Introduction
The enormous transformation underway in how humans engage with technologies
like game systems and computers – providing direct access through touch or gesture
without any mediating controller – has just reached mainstream computing, games and
home theaters, with the recent releases of the Kinect and the WAVI Xtion, which brings
gesture-control to the PC. This change alters players’ and computer users’ experience in
fundamental ways and has opened up huge new opportunities in and for the design of
games, experiences and applications.
As these changes and opportunities arise, they bring with them innumerable
questions – questions that, if investigated, promise to bring game-, experience-, and user-
interface- (UI) designers closer to understanding how they might construct their designs to
exploit the new controllers’ affordances and improve players’ and users’ experience.
Intensely curious as to what affordances gestural interaction might provide and
seeing little information immediately available, I decided to undertake an investigation. Of
particular interest was whether gesture might be linked to emotion, such that one might
design a gestural vocabulary as a means of developing and fostering emotionally compelling
game-play. That search revealed a vast amount of evidence showing: the body’s connection
to perceptions, emotions, mental states; the powerful, extensive, and surprising ways those
connections are manifest; and the unexpected and very potent role that metaphor plays. I
exploited those insights in the design and development of a gestural game, Mother Nature,
2
and indeed found that incorporating the body did appear to have a positive, powerful effect
on players’ experience.
There is no evidence to suggest that employing the emotional and cognitive
armature attached to human gesture and movement is a design strategy game designers
have considered employing. If one were to conduct a thought experiment and to imagine --
based on how gestural games have been designed thus far -- how designers would design a
gestural gardening game, a very concrete picture would come to mind: a game in which a
player would start by thrusting her arms to the left and then over her left shoulder, as
though using a shovel. Next, she would wave her arm right and left a bit in front of her to
water the shoveled spot with a hose, and then squeeze some sort of atomizer or depress the
button on the top of a spray can to distribute a pesticide and protect a seedling from
predatory insects.
That kind of design generates gestures that are relatively easy to learn and
remember; and it is perfect for some kinds of play experiences – experiences in which the
game world largely replicates the real world and, thus, where videogame play resembles
real-world play. Yet, the designs that employ gesture so literally fail to exploit a huge
number of the affordances that gestural play might provide. It also bespeaks a narrow or
flat view and deployment of gesture – one in which gesture is merely a source of data/input
to the computer system but not a potential source of enriching chemical, emotional, and
psychological experience in gesturing players – experience that could radically enhance and
deepen their play experiences.
3
Mother Nature was deliberately designed to exploit the affordances that scholarly
research reveals inhere in gesture to give the player a richer play experience, by employing
more and different pathways in the brain, than that provided by games that employ either
traditional button or standard, more literal gestural controls. In what follows, the reader
will find descriptions of the method by which Mother Nature’s gestural vocabulary was
designed, the motivation for that design, and the method by which I intend to test the
effectiveness of that design strategy in the coming weeks. I have opted both to provide this
extended description and to investigate whether or not that design strategy did, in fact,
improve the play experience to see whether this gestural design method could or should
extend beyond this project. Indeed, I wonder whether it might serve as a model method for
designing a gestural vocabulary that develops and fosters emotionally compelling game-
play. Call it a modest stab at developing a Unified Theory of Gestural Vocabulary
Design…
***
Mother Nature: The Theory

Embodied Cognition: Its Meaning, Scope and Power
Mother Nature is a game that brings the fundamental components of nature into
one place and uses the players’ own body to make the verbs of nature – grow, hunt, eat,
dodge, repel, smell – into the tools of play. Mother Nature’s objects and those objects’
interactions are based on the logics of the real world. This makes the player a literal mover
and shaker – a god both participating in and affecting the processes of nature.
4
Mother Nature’s gestural design is ultimately founded in interrelationships between
the body, its actions, and cognition now held as normative in contemporary theories of
cognition. Research from a vast collection of fields, including psychology, cognitive science,
robotics, linguistics and philosophy has increasingly concluded that the body is the basis for
much – by some accounts perhaps all – of the social, emotional and conceptual processing
that humans do – even the most abstract conceptual thinking. These accounts of
“embodied” or “grounded” cognition suggest that, as a person has an experience, the brain
does three things: (a) captures all the information about that experience from the person’s
motor (e.g. movement, proprioception), perceptual (e.g. visual, auditory) and introspective
(e.g. affect, mental state) systems; then (b) integrates that information into a multimodal
representation; and then (c) stores that representation in her memory.
1

Later, when she needs to represent to herself some part of that experience, a person’s
brain reactivates all of those previously stored multimodal representations, simulating the
perceptions, actions and introspections the brain experienced before. As far as Mother
Nature’s design is concerned, most suggestive about this model is that all of that
multimodal information is bundled together into one mass and that a single trigger
activates all the material (sensory, perceptual, motor, emotional) stored in that one
multimodal representation.

1
PM Niedenthal, LW Barsalou, P Winkielman, S Krauth-Gruber, and F Ric,
‘Embodiment in Attitudes, Social Perception and Emotion’ in Personality and Social
Psychology Review Vol. 9, No. 3, (2005 ) pp. 184-211; LW Barsalou, ‘Grounded
Cognition’ in Annual Review of Psychology 2008 Vol. 59, pp. 617-45.
5
Many people will be familiar with some of embodied cognition‘s effects. The
experience of feeling transported to a time and place in the distant past as a result of
hearing a song, smelling an odor, or tasting a flavor one associates with the past, is a
consequence of embodied cognition. Other effects become clear through research – studies
that reveal how entwined perception, action and introspective states are such that the
triggering of one prompts the expression and experience of another. For example, several
researchers have shown that priming the elderly stereotype prompts people to walk and to
complete lexical decision tasks more slowly than those in whom the stereotype had not
been primed.
2
Study participants asked to recall an unethical deed were more likely to take
antiseptic wipes and complete word tasks with cleansing-related words than those who
recalled an ethical action.
3
Participants in a group of studies looking at the effects of
holding heavier versus lighter clipboards showed that those who held a heavier clipboard:
(1) judged higher the value of foreign currencies, (2) judged more important the
employment of fair decision-making practices, (3) gave more consistent judgments on test
questions (revealing that they had given them deeper thought), (4) were less tolerant of
weak (compared to strong) arguments, and (5) judged prospective job candidates as better
overall and more serious about performing the job in quality – than those participants who

2
JA Bargh, M Chen, and L Burrows, ‘Automaticity of social behavior: Direct effects of
trait construct and stereotype activation on action’ in Journal of Personality and Social
Psychology, Vol. 71 (1996) pp. 230-244. cited in Niedenthal, Barsalou 2005, p. 184
3
CB Zhong and K Liljenquist ‘Washing Away Your Sins: Threatened Morality and
Physical Cleansing’ in Science Vol. 313, (8 September 2006) pp. 1451-1452.
6
held a lighter clipboard.
4
Researchers found that study participants engaging in “mental
time travel” leaned forward when imagining the future and leaned backward when thinking
about the past.
5
Three experiments showed that after participants do something charitable
or after they write about themselves doing something charitable or harmful to another,
participants hold a five-pound weight longer than they did before.
6
In other studies,
participants who remembered and wrote down a regret, sealed the inscribed paper in an
envelope, and handed the paper to the questioner felt less negative about that event than
participants who returned the paper to the questioner without first enclosing it in an
envelope.
7
An experiment showed that people who received good news while seated with
slumped heads and shoulders reported feeling less proud and being in a worse mood than
those who received good news while seated with their head and shoulders held high.
8

4
NB Jostmann, D Lakens, and T Schubert, ‘Weight as an Embodiment of Importance’ in
Psychological Science Vol. 20 No. 9 pp. 1169-1174. J Ackerman, C Nocera, and JA Bargh,
‘Incidental Haptic Sensations Influence Social Judgments and Decisions’ in Science Vol. 328
(25 June 2010) pp. 1712-1715.
5
L Miles, L Nind, and CN Macrae, ‘Moving Through Time’ in Psychological Science Vol.
21, No. 2, pp. 222-223.
6
K Gray, ‘Moral Transformation: Good and Evil Turn the Weak into the Mighty’ in
Social Psychological and Personality Science 2010 Vol. 1, pp. 253-258.
7
X Li, L Wei, and D Soman, ‘Sealing the Emotions Genie: The Effects of Physical
Enclosure on Psychological Enclosure’ in Psychological Science Vol. 21, No. 8, pp. 1047-
1050.
8
S Stepper, and F Strack, ‘Proprioceptive Determinants of Emotional and Nonemotional
Feelings’ in Journal of Personality and Social Psychology Vol. 64, No. 2 (February 1993) pp.
211-220. Cited in PM Niedenthal, ‘Embodying Emotion’ in Science Vol. 316 (18 May
2007) pp. 1002-1005.
7
Metaphor
While perceptual, motor and introspective data together compose the multimodal
representations the brain stores, that data does not all hold equal weight there. Organizing
the material in the brain are metaphors. George Lakoff explains that though they are
typically expressed through and understood as language, metaphors are ultimately
independent of language. They are mental structures that are ultimately physical – more
precisely, neural. He explains that our references to prices rising or someone being at the
peak of health descend not from prices or health going literally upward but from our
understanding things like prices and health in terms of verticality, where an increase is
marked by a higher vertical position. Similarly, references to people’s warmth or coldness
allude not to their literal temperature but to how affectionate they are. Conceptual
metaphors like these that shape so much of our language, Lakoff explains, are ultimately
products of the quotidian experiences beginning in childhood, where accumulating more
things on a table or more water in a glass caused them to pile up – creating an association
between verticality and quantity -- and where being held and being given affection caused
one to have a literal, physical experience of warmth – creating the association between
affection and warmth. Because ongoing experiences of verticality and quantity, and those of
warmth and affection, are so frequently activated simultaneously, in time they become,
literally, neurally connected, in keeping with neuroscientists’ expression “Neurons that fire
together wire together.” With each use, chemical receptors for neurotransmitters increase
8
at the neural synapses, causing the neural circuit to grow ever stronger. That “recruited”
neural circuit, Lakoff explains, constitutes the metaphor.
9

Metaphor and Embodiment
In the research on embodiment cited above, one can see the ubiquity – the
pervasiveness – of participants’ physical experience of metaphor. Consistent with the
affection-is-warmth metaphor and its implied correlate, two experiments showed that social
exclusion literally felt cold to test participants. After remembering an experience of social
exclusion, test participants judged the room colder than did participants who recalled an
experience of inclusion. Participants socially excluded in a second study reported greater
interest in warming food and drink than participants given an experience of inclusion.
10

Consistent with the light-is-good and dark-is-bad metaphors, study participants in a room
in which the lights were slightly dimmed cheated with more frequency and collected more
undeserved money than participants in a well-lit room. In a second experiment, in a money
distribution task, participants wearing sunglasses showed much more self-interest than did
those performing the same task in clear lenses.
11

9
G Lakoff, The Political Mind: Why You Can’t Understand 21-st Century Politics with an
18-th Century Brain (New York: Viking, 2008) pp. 82-84; G Lakoff and M Johnson,
Metaphors We Live By (Chicago: University of Chicago Press, 1980) p. 15
10
CB Zhong, and GJ Leonardelli, ‘Cold And Lonely: Does Social Exclusion Literally Feel
Cold?’ in Psychological Science Vol. 19, No.9, pp. 838-842.
11
 CB Zhong, V Bohns, and F Gino, ‘Good Lamps are the Best Police: Darkness Increases
Dishonesty and Self-Interested Behavior’ in Psychological Science Vol. 21, No. 3, pp. 311-
314. It is worth noting that what affected participants in the second study was participants’
experience of darkness; the researchers point out that “they generalize their experience to
others, expecting that others will conversely have difficulty perceiving or seeing them.”
9
Indeed, all of the cited embodiment studies are evidence of the power, pervasiveness,
and ubiquity of metaphors. The clipboard experiments show the strength of the weighty-is-
serious metaphor. Studies pointing to the meaningfulness of the virtue-is-clean metaphor
are not just those showing the increased use of antiseptic wipes by those who recall doing an
immoral deed but also studies that showed the use of clean smells promoting virtuous
behavior.
12
The leaning-with-time experiments are reflective of the (foreseeable) future-
events-are-up (and ahead) metaphor, while those that show emotional closure coming with
closing regrets into an envelope bespeak the metaphor that states-are-containers.
13

In addition to showing the frequency and breadth of our experience of metaphors,
these studies suggest that metaphors serve a vital function: providing people an orientation
to the physical events or experiences they undergo. Sometimes – e.g. affection is warmth --
that orientation is on an emotional scale, at others – e.g. light-is-good – a moral one, and
still others, an aesthetic one. Metaphor’s work in orienting people to the events they
experience or the movements they make suggest that employing metaphor as the basis for
designing gestural play would be a way of layering the perceptual, emotional and
psychological sensation already stored in the brain onto the experience the game play
provides.
In some ways, the proposition of deploying metaphor toward design is unsurprising.
After all, metaphor has long been vitally important to user interface (UI) designers –

12
 K Liljenquist, CB Zhong, and AD Galinsky, ‘The Smell of Virtue: Clean Scents
Promote Reciprocity and Charity’ in Psychological Science Vol. 21, No. 3, pp. 381-3833.
13
Lakoff 1980 pp. 16 and 30.
10
standing as among their most reliable tools for providing computer-users intelligible,
usable access to the invisible, enormously complex operations that computers perform.
While metaphor’s application to the field of human computer interfaces (HCI) is usually
focused on the cognitive benefits metaphors typically provide -- grounding a novel or
complex process within one that appears familiar – there is an emotional component; they
can reduce significantly the anxiety users can experience as a result of their obligation to
succeed with a technology that feels alien.
14

Metaphor, Emotion, Mother Nature and Prior Art
Notwithstanding that application of metaphor to players’ emotion, the method of
applying metaphor to emotion that Mother Nature proposes is different. Whereas UI- and
HCI- designers typically and primarily exploit metaphor’s cognitive components, proposed
here is employing metaphor’s emotional and physiological armature to supplement the
emotional return that successful game play provides.
This approach is very new; as a result, finding prior art proved enormously difficult.
Multiple searches of the game- and HCI- literature revealed no suggestions that either
game- or HCI- designers are working or have worked to exploit the emotions bound to
metaphor-enriched gestures. A question that did engage a group of researchers was
whether body movement’s contributing to game control did or did not increase players’

14
AF Blackwell, ‘The Reification of Metaphor as a Design Tool ‘ in ACM Transactions on
Computer-Human Interaction Vol. 13, No. 4, (December 2006) pp. 490-530; DC Neale,
and JM Carroll, ‘The Role of Metaphors in User Interface Design’ M Helander, TK
Landauer, and P Prabhu, (eds) Handbook of Human-Computer Interaction (Elsevier Science,
1997) p. 441-462
11
immersion (in playing Guitar Hero using the Guitar Hero guitar). The authors conclude
not only that movement increased immersion but that the more movement, the more
immersion – perhaps because the form of movement fostered fantasy that might have
enhanced immersion.
15
A second study investigating whether body movement (playing via
Donkey Konga Bongos) increased immersion and/or social interaction determined that
body movement increased both, compared to play using a standard controller.
16
The most
forensic or analytical treatments that game magazines have given to the processes and
problems associated with Kinect design and development have focused almost exclusively
on designing and developing successful user interfaces, not on methods for designing
emotionally rich gestural vocabularies or on compelling gesturally-controlled game-play.
17

15
 N Bianchi-Berthouze, WW Kim, and D Patel, ‘Does Body Movement Engage You
More in Digital Game Play? And Why?’ in Proceeds ACII 2007 (Springer, 2007) pp. 102-
113. Indeed, that narrative fantasy has physiological outcomes is suggested by the Gray
study cited above, where both players who behave heroically and those who behaved poorly
were capable of holding a weight for a longer period than test subjects who had not been
primed.
16
SLindley, J LeCouteur, N Bianchi-Berthouze, ‘Stirring Up Experience through
Movement in Game Play: Effects on Engagement and Social Behaviour’ CHI 2008
Proceedings pp. 511-514.
17
J Nash, ‘The Player Becomes The Producer: Natural User Interface Design Impact’ in
Gamasutra 24 February 2011, viewed on 21 March 2011,
http://www.gamasutra.com/view/feature/6296/the_player_becomes_the_producer_.php; C
Nutt, ‘MIGS 2010: Harmonix’s Solution for Kinect UI Design’ in Gamasutra 8
November 2010, viewed on 21 March 2011,
http://www.gamasutra.com/view/news/31414/MIGS_2010_Harmonixs_Solution_For_Ki
nect_UI_Design_.php; J Nielsen, ‘Kinect Gestural UI: First Impressions’ Gamasutra 11
January 2011, viewed on 20 March 2011,
http://www.gamasutra.com/view/feature/6253/kinect_gestural_ui_first_.php
12
Among those who have tried to develop a model for designing optimal gestural
vocabularies, finding ways of adding layers of positive emotional experience was not a goal.
The bulk of this scholarly work preceded the release of the Kinect and concentrated on
employing hands – not a whole body – as the controller. The researchers’ goals were
designing a vocabulary that was both intuitive and comfortable for, and easily learned by,
the gesturer while remaining legible to – or capable of being read or recognized accurately
by – the gesture-reading technology. All worked to determine how they could develop
vocabularies whose gestures could be learned as quickly, efficiently, and with as little
frustration as possible. For these researchers, emotion was perceived only as a potential
negative – a problem to be avoided – not an opportunity or mechanism for improving
players’ or users’ experience.
HCI literature has shown no interest in using gesture toward producing emotion.
Indeed, writers suggest that the notion of employing emotion at all toward improving user
experience is a fairly new one for the field. Scott Brave and Clifford Nass write that,
typically, considerations of emotional engagement have been deliberately and systematically
omitted from HCI design processes; the thinking has been that “users must discard their
emotional selves to work efficiently and rationally with computers, the quintessentially
unemotional artifact.”
18
The bulk of HCI literature has regarded usability, efficiency and
ease-of-use as goals; there has been no notion that creating an emotional experience, apart
from avoiding frustration, might be an end worth pursuing. Encouraging belief that change

18
 S Brave and C Nass, ‘Emotion in Human-Computer Interaction’ in A Sears and J Jacko
(eds) The Human- Computer Interaction Handbook: Fundamentals, Evolving Technologies,
and Emerging Applications (New York: Lawrence Erlbaum Associates, 2008) p. 78
13
might be on its way are publication of case studies that explain both that positive emotional
experience makes people more creative and efficient, as well as studies showing that
interfaces can provide people that positive experience. Additionally enabling change is the
refining of tests that reveal users’ emotional responses more effectively than previous
tests.
19
Yet, whatever increased interest HCI designers might take in emotion, there is no
sign that they will look to the emotion affixed to gesture to provide it.
***
Mother Nature: The Practice

Mother Nature: The Game
Mother Nature is a game/interactive experience that makes the fundamental
components of life in the natural world -- the beauty, the generativity, the conflict and the
destruction -- into the tools of play. In Mother Nature, the body is the controller; the
objects it controls and the interactions it influences are modeled on the objects and
interactions of the real world.
A person intending to build a gardening game in which the player’s experience is
intended to center on generating life ought to include: (a) a way of bringing seed into the
world; (b) a way of bringing nourishing rain into the world; and (c) a way of turning that
potential for growth into real, meaningful growth. In nature, much has a dual nature: what
is helpful in some circumstances and in some degrees can be harmful at other times or in a

19
A Agarwal and A Meyer, ‘Beyond Usability: Evaluating Emotional Response as an
Integral Part of the User Experience’ CHI 2009 Case Studies New Usability Metrics and
Methods pp. 2919 – 2930.
14
non-optimal dose. That the same element can be helpful at times but harmful at other
times means that person engaged with that system will need to employ strategy to succeed
in it.
Mother Nature has been designed to include those processes and to be controlled by
gesture. Mother Nature’s gestures work as forms of indirect control. A lasso gesture
generates wind; where the player performs that gesture determines what that wind does to
or brings into the game world. The lasso gesture performed in the middle of the body (with
the hand at or below the shoulder) brings a seed into the world.
20

Figure 1: Middle Lasso Gesture
The lasso gesture performed with the hand to the right or left above the head brings a
cloud into the world.

20
Because still photography cannot capture motion, the white “ribbon” on the screen (that
shows the history of where the arm has been) better represents the gesture that prompted
the seed movement than the placement of player’s arm in this photograph.
15

Figure 2: High Lasso Gesture
while the lowest lasso gesture twists long grass into a nest and distributes smells along the
ground. The drumming gesture transforms three dark clouds (minimum) into rain.

Figure 3: Rain Gesture
16
A raising of the hands, from the sides of the body to the level of the brows, transforms the
potential for growth into realized growth.

Figure 4: Grow Gesture
If the player gives the seed enough - but not too much – rain, the player can employ
the growth gesture to make the seed and seedling grow. The player must take care,
however, not to overwater or underwater seeds and seedlings (seed(ling)s). Overwatering –
indicated by the seed(ling)’s becoming covered with spots -- will cause the seed(ling) to
wash away. Under-watering -- indicated by the seed(ling)’s lightening in color– cause
seed(ling)s to dry up and disappear.
After the player succeeds at growing a flower, the game introduces a second
component of strategic play: the goal of attracting a spider into the world. On one hand,
the spider is very helpful: it repels the grasshoppers that would otherwise consume the
flower the player has worked so hard to produce. Yet, it also repels the bees the player needs
17
to pollinate the flower to transform the flower into a fruit. Thus, the player needs to work
strategically. She must both bring a seed into the game world close enough to a flower to
cause a spider to build web between them, and she must plant other seeds at a distance that
permits them to enjoy spiders’ grasshopper-protection while avoiding spiders’ repulsion of
bees. If the player succeeds, a bee-pollinated flower will turn into a fruit, bringing into the
game world a bird that both consumes the fruit and drops a new seed. If the player moves
slowly, the seed will drop to the ground and will be snatched by a mouse that has emerged
from a hole in the foreground. If the player employs the middle lasso gesture quickly
enough, she will both loft the falling seed into the air and cause that falling seed to break
into a collection of smaller seeds that will land and generate tufts of grass. The player uses
the middle lasso gesture to distribute the seeds the grass generates to grow grass that
stretches across the game world. The player is then be challenged to create a nest for the
ground-bees – something she can achieve by using the lowest lasso gesture to “spin” a nest
for the mouse from the tall grasses, prompting the mouse to abandon the hole it was using
before and enabling the ground-bees to build a hive in the hole. The increase in the bee
population its new, local ground-nest permits brings a sizable increase in the number and
kinds of flower blooms. Yet, that increase in pollination also brings an increase in the honey
the ground-bees produce – one that, in time, prompts the mouse to raid the bees’ hive.
That raid brings a serious decline in the flower blooms – a decline the player may stop by
employing the lowest lasso gesture to distribute the mouse’s smell to its predators – a group
that includes a snake that uses its tongue to smell.
18
Mother Nature includes tutorials that introduce players to gestures, prepare them to
play and gives them the room to experiment with gesture.
21
In the first tutorial, designed to
teach players how the middle lasso gesture transports and “aims” seeds, players transport a
seed from one side of the game world to the other, then aim seeds at the butterflies that
appear, randomly, in the holes of trees. In the second tutorial, players learn how to use the
highest lasso gesture to “write” with clouds, or – if performed successfully --to gather them
into a mass that darkens. In the third tutorial, players learn the drumming gesture that first
brings rain onto a turtle and -- if maintained – prompts the turtle to retreat into its shell.
The fourth tutorial, the player learns how to raise a bird’s nest from the game world’s
ground to its sky without dumping and shattering the eggs inside. In the fifth, the player
learns how to use the lowest lasso gesture to create the funnel of air that scatters dirt and
reveals an acorn that had been underground – then wafts the smell of that acorn to a
squirrel, who comes to enjoy it.
***
Mother Nature: Theory and Practice
The design of Mother Nature ‘s gestural vocabulary was influenced by the literature
on embodiment like that described above.
22
One study tested participants’ feelings about

21
 Play-tests made evident that people were so excited by seeing themselves manifest in
game world that they really wanted chance to experiment with movement and gestural play
but wanted that experimentation to come without penalty. 
22
Because gestural play is so new and unfamiliar, Mother Nature’s play-testers proved to be
kin of the play-testers described in both the academic and applied (game) literature in
requiring time and practice to remember and employ properly the gestures within the
game’s vocabulary. To aid players’ memory, I employed the tactic that these writers also
found helpful: designing gestural controls that are reasonably intuitive. At the same time,
19
neutral Chinese ideographs placed in front of them while performing two different gestures
– the flexing of muscles achieved when one pushes up against the bottom of a table toward
body and the extending of muscles achieved when one pushes down against top of the table.
After factoring in participants’ views of the ideographs before the test, researchers
determined that participants preferred the ideographs viewed while flexing muscles toward
them to the ideographs viewed while extending their arms away from them.
23
This research
suggested quite a bit: first, that both gestures altered participants’ views of neutral object;
second, that the two gestures’ effects were systematic; and third, that flexion consistently
created more positive feelings than extension. In another experiment, participants were
divided into two groups – one instructed to push a lever away from their bodies and the
second instructed to pull a lever toward their bodies. When presented with images that
typically provoke positive or negative responses, those in the group pushing the lever away
responded more quickly to negative images than positive ones, while participants in the
group pulling the lever toward them bodies responded more quickly to the positive images
than the negative images.
24
Like the previous study, this study suggests that the body
associates a pulling gesture with the positive and a pushing gesture with the negative.

determining that designers’ and players’ understanding of intuitive was the same took lots
and lots of play-tests, as was true of these other projects as well.

23
JT Cacioppo, JR Priester, and GG Berntson, ‘Rudimentary Determinants of Attitudes
II: Arm Flexion and Extension have Differential Effects on Attitudes’ in Journal of
Personality and Social Psychology Vol. 65 (1993), No. 1, p. 5-17.
24
KL Duckworth, JA Bargh, M Garcia, and S Chaiken ‘The Automatic Evaluation of
Novel Stimuli’ in Psychological Science Vol. 13, No. 6 (November 2002) p. 513-519 cited
in PM Niedenthal, ‘Embodying Emotion’ in Science Vol. 316, (18 May 2007) p. 1002-
1005.
20
A great deal of research on the relationship between affect and vertical position has
also been done. One study showed that, among words flashed on a computer screen,
positive words (e.g. ethical) were evaluated more quickly when they were high on the screen
rather than low on the screen, and negative words (e.g. crude) were evaluated more quickly
when they were low on the screen than when they were high on a screen. In a second study
(by the same researchers) in which participants were first primed by a positive or negative
word at the center of the screen, participants primed with a positive word more quickly
identified letters in the top position than the bottom position; and participants primed with
a negative word more quickly identified letters in the bottom position than the top
position.
All this research informed my design of Mother Nature’s gestural controls, in hope
that, by using gestures that generated a positive emotional change, players who succeeded
in play might experience a “one-two pleasure punch” – the first stemming from succeeding
in game play and the second from having employed a gesture itself associated with pleasure
to create that success. Because the flexion gesture layered a positive affect on a neutral
object and because the lever tests suggested the existence of a link between pulling and the
positive, flexion and pull toward the body were made into the core of Mother Nature’s lasso
gestures – one experienced most vividly in the middle lasso gesture.
25
Research that linked
the higher spaces with the positive and lower spaces with the negative suggested that

25
Interestingly, a loop-hole in the gesture coding made the lasso gesture associated with
extension (pushing/circling the hand away from the body in a clockwise direction) as
effective a means of bringing a seed into the world as the counter clockwise lasso gesture
(associated with flexion). Yet, though players frequently lassoed incorrectly, they never
opted to lasso clockwise (away from the body).
21
player’s transforming something from low to high might give them a pleasurable
experience that exceeded the pleasure derived simply from a player’s being successful in
making things grow. Thus, the gesture of raising the arms from low to high was made into
the game’s grow gesture.

Mother Nature: Testing the Theory and Practice
To see whether or not the emotion linked to the grow gesture enhanced the emotion
stemming from successful game play in the way I hoped it would, with the help of Evan
Suma, I have designed a research study that I expect to complete in approaching weeks. In
that study I will try to tease apart which gesture(s) people enjoy because it provides them an
experience of efficacy and which gesture(s) people enjoy because gesturing permits them to
benefit from having activated a metaphor that is itself associated with positive affect.
I anticipate the study to be organized roughly as follows: participants will be divided
into four groups. Members of all groups will first employ the lasso-low gesture to aim
seeds at a tree stump whose top is at the center of the game screen.

Figure 5: Aiming Seed
22
One-quarter of participants will perform the grow gesture, and that performance will
prompt a flower to grow upward and bloom. One quarter will perform the grow gesture,
and that gesture will prompt a vine to grow downward and a flower to bloom low on that
vine. One quarter will perform an inversion of the grow gesture, employing energy to push
her arms down from a space in front of her brow to the two sides of her body; that
performance will prompt a flower to grow upward and bloom. A fourth group will
perform the inverted grow gesture, which will prompt a vine to grow down and a flower to
grow low on that vine.

Figure 6: Bloom Up
23

Figure 7: Bloom Down
For this study participants will complete 4 different kinds of tests. Before and after
play, participants will complete a PANAS (Positive and Negative Affect Schedule) to
gauge emotional response to play. Throughout play, participants will wear sensors that
measure their galvanic skin response, a psychophysical measurement that corresponds to
arousal. Afterward, participants will complete a word completion test to determine
whether the grow gesture activates the affects associated with the orientational metaphors
up (positive) and down (negative). In addition, they will also complete a qualitative test by
which they can provide researchers more detailed feedback about what components of play
(e.g. successful play versus the gestural play) created the most emotionally compelling play
experience. I hope that this study will prove that employing a gesture associated -- via
metaphor -- with positive affect increased players’ experience of pleasure beyond that
achieved through successful game play alone.
***
24
Conclusion
The body’s links with the emotional, cognitive, and sensory materials of the brain
make the ascent of gesture-reading technology not only a means of developing new games
and interactive experiences but games and interactive experiences that engage players both
differently and more deeply. I believe that, as a consequence both of the body’s links to the
emotional and sensory data in the brain and metaphor’s great power and pervasiveness in
layering color or value onto otherwise neutral experience, gestures that uphold metaphors
might be more powerful than gestures that do not. I hope to have evidence that supports
that theory in the weeks to come.
25
Bibliography

Ackerman, J, Nocera, C, and Bargh, JA ‘Incidental Haptic Sensations Influence Social
Judgments and Decisions’. Science, Vol. 328, 25 June 2010, pp. 1712-1715.
Agarwal, A and Meyer, A ‘Beyond Usability: Evaluating Emotional Response as an
Integral Part of the User Experience’. CHI 2009 Case Studies New Usability Metrics and
Methods pp. 2919-2930.

Bargh, J.A, Chen, M., and Burrows, L. ‘Automaticity of Social Behavior: Direct Effects
of Trait Construct and Stereotype Activation on Action’. Journal of Personality and Social
Psychology, Vol. 71, (1996) pp. 230-244.

Barsalou, LW ‘Grounded Cognition’ Annual Review of Psychology Vol. 59 (2008) pp. 617-
45.

Bianchi-Berthouze, N, Kim, WW, and Patel, D ‘Does Body Movement Engage You
More in Digital Game Play? And Why?’ Proceeds ACII 2007 Springer (2007) pp. 102-113.

Blackwell, AF ‘The Reification of Metaphor as a Design Tool’. ACM Transactions on
Computer-Human Interaction Vol. 13, No. 4, (December 2006) pp. 490-530.

Brave, S and Nass, C ‘Emotion in Human-Computer Interaction’. Sears, A and Jacko, J
(eds) The Human- Computer Interaction Handbook: Fundamentals, Evolving Technologies,
and Emerging Applications (New York: Lawrence Erlbaum Associates, 2008) pp. 77-92.

Cacioppo, JT, Priester, JR, and Berntson, GG ‘Rudimentary Determinants of Attitudes II:
Arm Flexion and Extension have Differential Effects on Attitudes’. Journal of Personality
and Social Psychology Vol. 65, (1993), No. 1, pp. 5-17.

Duckworth, KL, Bargh, JA, Garcia, M, Chaiken, S. ‘The Automatic Evaluation of Novel
Stimuli’. Psychological Science Vol. 13, No 6, (November 2002), pp. 513 – 519.

Gray, K ‘Moral Transformation: Good and Evil Turn the Weak into the Mighty’. Social
Psychological and Personality Science Vol. 1, (2010), pp. 253-258.

Jostmann, NB, Lakens, D, and Schubert, T. ‘Weight as an Embodiment of Importance’.
Psychological Science Vol. 20, No. 9, pp. 1169-1174.

Lakoff, G and Johnson, M Metaphors We Live By (Chicago: University of Chicago Press,
1980)

26
Lakoff, G The Political Mind: Why You Can’t Understand 21-st Century Politics with an
18-th Century Brain (New York: Viking, 2008)

Li, X, Wei, L, and Soman, D ‘Sealing the Emotions Genie: The Effects of Physical
Enclosure on Psychological Enclosure’. Psychological Science Vol. 21, No. 8, pp. 1047-1050.

Liljenquist, K, Zhong, CB, and Galinsky, AD ‘The Smell of Virtue: Clean Scents
Promote Reciprocity and Charity’. Psychological Science Vol. 21, No. 3, pp. 381-383.

Lindley, S, LeCouteur, J, Bianchi-Berthouze, N ‘Stirring Up Experience through
Movement in Game Play: Effects on Engagement and Social Behavior’ CHI 2008
Proceedings pp. 511-514

Miles, L, Nind, L, and Macrae, CN ‘Moving Through Time’ Psychological Science Vol. 21,
No. 2, pp. 222-223.

Nash, J ‘The Player Becomes The Producer: Natural User Interface Design Impact’
Gamasutra 24 February 2011, viewed on 21 March 2011,
http://www.gamasutra.com/view/feature/6296/the_player_becomes_the_producer_.php

Neale, DC and Carroll, JM ‘The Role of Metaphors in User Interface Design’. Helander,
M, Landauer, T.K. and Prabhu, P. (eds) Handbook of Human-Computer Interaction
(Elsevier Science, 1997) pp. 441-462.
Nielsen, J ‘Kinect Gestural UI: First Impressions’ Gamasutra 11 January 2011, viewed on
20 March 2011,
http://www.gamasutra.com/view/feature/6253/kinect_gestural_ui_first_.php
Niedenthal, PM, Barsalou, L, Winkielman, P, Krauth-Gruber, S, and Ric, F
‘Embodiment in Attitudes, Social Perception and Emotion’. Personality and Social
Psychology Review Vol. 9, No. 3, (2005) pp. 184-211.

Nutt, C ‘MIGS 2010: Harmonix’s Solution for Kinect UI Design’ Gamasutra 8
November 2010, viewed on 21 March 2011,
http://www.gamasutra.com/view/news/31414/MIGS_2010_Harmonixs_Solution_For_Ki
nect_UI_Design_.php

Stepper, S. and Strack, F. Stepper ‘Proprioceptive Determinants of Emotional and
Nonemotional Feelings’. Journal of Personality and Social Psychology Vol. 64, No. 2
(February 1993) pp. 211-220. cited in Niedenthal, PM ‘Embodying Emotion’ Science Vol.
316 (18 May 2007) pp. 1002-1005.

Zhong, CB, and Liljenquist, K ‘Washing Away Your Sins: Threatened Morality and
Physical Cleansing’ Science Vol. 313 (8 September 2006) pp. 1451-1452.

27
Zhong, Chen-Bo, and Leonardelli, Geoffrey J. “Cold And Lonely: Does Social Exclusion
Literally Feel Cold?” Psychological Science Vol 19 No.9 838-842

Zhong, Chen-Bo, Bohns, Vanessa, and Gino, Francesca “Good Lamps are the Best Police:
Darkness Increases Dishonesty and Self-Interested Behavior” Psychological Science Vol. 21
(3) p 311-314.

Abstract (if available)

Abstract The enormous transformation in how humans engage with technologies – providing direct access through touch or gesture, without any mediating controller – has just reached mainstream computing, games and home theaters, with the recent releases of the Kinect and the WAVI Xtion. This change has opened up huge new opportunities for the design of games, interactive experiences and applications. This paper presents the evidence of the connection between the body and perceptions, emotions, and mental states

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Combiform: a console for the new communal casual game genre

PDF

Maum: exploring immersive gameplay with emerging user interface devices

PDF

Garden designing a creative experience with art and music orchestra

PDF

Come with Me: a cooperative game focusing on player emotion

PDF

SomeDay: designing a game about different thought processes

PDF

Teaching interactive production

PDF

Alma: designing compassion for healthcare workers through interactive play

PDF

Chaos

PDF

There You Are: an exploration of storytelling methods using in video games

PDF

Ascension: an analysis of game design for speech recognition system usage and spatialized audio for virtual reality

PDF

Working through death and grief with a video game: the design and development of Where the sea meets the sky

PDF

Seeds: role-play, medical drama, and ethical exploration

PDF

Last broadcast: making meaning out of the mundane

PDF

Courier: Dragons Within, a video game. An exploration into the magic circle as healing circle: restorative game design for a masculine framework free from the template of domination [document]

PDF

Wetware: designing for a contemporary dilemma

PDF

Hedge hug -- a narrative-driven interactive exploration on low self-esteem and social anxiety

PDF

Free Will: a video game

PDF

The future of games and health: towards responsible interaction design

PDF

That's not how it happened: unreliable narration through interactivity

PDF

Nahui Ollin, “in search of the divine force”: a phyical immersive interactive dance experience

Asset Metadata

Creator Tucker, Diane (author)

Core Title Toward a theory of gesture design

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Publication Date 05/04/2011

Defense Date 05/02/2011

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag design,emotion in games,game design,gestural design,gestural vocabulary,Gesture,gesture design,human centered computing,human computer interaction,metaphor,OAI-PMH Harvest,user interface design

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bolas, Mark (committee chair), Fullerton, Tracy (committee member), Kratky, Andreas (committee member), Malamed, Laird (committee member)

Creator Email diane.tucker@gmail.com,dmtucker@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m3891

Unique identifier UC1192559

Identifier etd-tucker-4587 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-469656 (legacy record id),usctheses-m3891 (legacy record id)

Legacy Identifier etd-tucker-4587.pdf

Dmrecord 469656

Document Type Thesis

Rights Tucker, Diane

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu