1
Toward a Theory of Gesture Design
Introduction
The enormous transformation underway in how humans engage with technologies
like game systems and computers – providing direct access through touch or gesture
without any mediating controller – has just reached mainstream computing, games and
home theaters, with the recent releases of the Kinect and the WAVI Xtion, which brings
gesture-control to the PC. This change alters players’ and computer users’ experience in
fundamental ways and has opened up huge new opportunities in and for the design of
games, experiences and applications.
As these changes and opportunities arise, they bring with them innumerable
questions – questions that, if investigated, promise to bring game-, experience-, and user-interface-
(UI) designers closer to understanding how they might construct their designs to
exploit the new controllers’ affordances and improve players’ and users’ experience.
Intensely curious as to what affordances gestural interaction might provide and
seeing little information immediately available, I decided to undertake an investigation. Of
particular interest was whether gesture might be linked to emotion, such that one might
design a gestural vocabulary as a means of developing and fostering emotionally compelling
game-play. That search revealed a vast amount of evidence showing: the body’s connection
to perceptions, emotions, mental states; the powerful, extensive, and surprising ways those
connections are manifest; and the unexpected and very potent role that metaphor plays. I
exploited those insights in the design and development of a gestural game, Mother Nature,