University of Southern California Dissertations and Theses

Seymour Deeply: exploring stereoscopic 3D as a storytelling tool in interactive media

(USC Thesis Other)

Seymour Deeply: exploring stereoscopic 3D as a storytelling tool in interactive media

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content SEYMOUR DEEPLY: EXPLORING STEREOSCOPIC 3D AS A STORYTELLING TOOL IN INTERACTIVE MEDIA by Michael Annetta A Thesis Presented to the FACULTY OF THE USC SCHOOL OF CINEMATIC ARTS UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree MASTER OF FINE ARTS (INTERACTIVE MEDIA) May 2012 Copyright 2012 Michael Annetta ii Epigraph We are all in the gutter, but some of us are looking at the stars. --Oscar Wilde, Lady Windermere’s Fan iii Dedication For his unending belief that furthering one’s education is the best way to further oneself, I dedicate this master’s thesis to my late father, Michael Annetta Sr. and for her tireless efforts to make sure her children had the opportunity to further their education, to my mother, Marie Annetta. iv Acknowledgements It is a gross fallacy to assume that a single individual, without support, wholly creates any modern academic work, let alone any interactive artwork. I acknowledge the following individuals and groups who have been instrumental in their support in helping me create this thesis paper and the accompanying project. My first thanks must go to my family, my parents, sister Rosemary and brother Leonard, who have always been supportive of my work and my studies. Thanks especially to my nieces and nephews (Derek, Samantha, Joseph and Gianna) who help keep their Uncle Michael thinking so young. My invaluable thesis advisement committee: Perry Hoberman, Scott Fisher, and Joseph Garlington for their guidance and support throughout the entire thesis process. Special thanks especially to Perry for his groundbreaking work with the Unity3D engine and stereoscopy. The IMD and SCA Faculty and staff, especially Anne Balsamo, Steve Anderson, Mark Bolas, Jeremy Gibson, Andreas Kratky, Laird Malamed, Tracy Fullerton, Adrienne Capirchio, Eric Furie and Dan Weishaupt. The Fox Corporation for their generous fellowship, which helped fund this project. The USC Lambda LGBT Alumni Association, whose inaugural NOLGSTP (National Organization of Gay and Lesbian Scientists and Technical Professionals) Scholarship in Innovation also helped fund this project. v My entire project team: Producer: Konstantin Brahznik; Designers: Jimmy Gorham, Nathan Burba, Patrick Meeghan; Software Engineers: Madhulika (Ayesha) Mazumdar, Talgat Duisenov, Andres Ramirez, Ino Mantaring, Winters Lu, James Chang, Jz Chen, Ran Zhai, Chao Zhang, Ryo Chibo; Animators and Modelers: Willie Hoffman, Elyse Kelly, Borja Gonzalez; Composer and Sound Designer: Matt Soule, Concept and Comic Artists: Miss Tak and Lizzie Nichols. Extra special thanks to Konstantin Brahznik whose enthusiasm and help with this project have been unparalleled. My supportive classmates, particularly Erin Reynolds, Andy Uehara, Juli Griffo, Jacob Boyle, Greg Nishikawa and TigerBrooke Fifer. Interactive Media Division Alumni who have blazed the trail: Andrew Sacher, Diana Hughes, Mike Rossmassler, Nahil Sharkasi, Sean Bouchard, Elizabeth Swensen, Logan Olson, Daniel Ponce, Bill Graner, and Diane Tucker. Professor Tara McPherson for suggesting that I apply to the University of Southern California’s Interactive Media Division in the first place and who has been exceptionally supportive throughout my time here. School of Cinematic Arts alumnus Brian Peterson for embracing me and indoctrinating me into the larger Trojan Family. F. Scott Fitzgerald for daring to suggest that there are no second acts in American lives (and allowing me to prove him wrong). vi Finally and most importantly, I could never, ever have gotten through this often grueling and transformative three-year process without my rock and soul, my partner and love, Kevin Fabian. vii Table of Contents Epigraph.................................................................................................................................... ii Dedication................................................................................................................................iii Acknowledgements.................................................................................................................. iv List of Figures........................................................................................................................viii Abbreviations........................................................................................................................... ix Abstract..................................................................................................................................... x Preface...................................................................................................................................... xi Introduction............................................................................................................................... 1 Depth as a Visual Storytelling Tool.......................................................................................... 6 Stereoscopic 3D as a Visual Storytelling Tool ......................................................................... 8 Stereoscopic 3D and Gesture.................................................................................................. 10 Project Concept and Development Process ............................................................................ 13 Early Prototypes.................................................................................................................. 13 Current Prototype................................................................................................................ 24 Transmedia Extension......................................................................................................... 29 Use of COTS Technologies as a Constraint ....................................................................... 32 Concluding Remarks and Suggestions for Further Exploration ............................................. 33 Glossary .................................................................................................................................. 39 Bibliography ........................................................................................................................... 44 Appendix................................................................................................................................. 46 The Tale of Seymour Deeply: Complete Comic Book script............................................. 46 viii List of Figures Figure 1: The Stereo Volume.................................................................................................... 8 Figure 2: Screenshot of early prototype, “Drop of Colour”.................................................... 18 Figure 3: Winteract 2011 Seymour Deeply setup and playspace (Player: Kevin Fabian) ...... 23 Figure 4: Player gesture to put on virtual stereoscopic binoculars (Player: Scott Hardie)..... 27 Figure 5: Screenshot of partially completed constellation puzzle in Level 1......................... 28 Figure 6: Rough sketch of page one of Seymour Deeply comic............................................. 31 Figure 7: Traditional Counterpart Play Zones ........................................................................ 36 Figure 8: Future Continuous Play Zone.................................................................................. 36 ix Abbreviations Although most readers of this paper will be familiar with the majority of the terminology used herein, the abbreviations below are used throughout and are listed here to assist the unfamiliar reader with the concepts presented. 2D (in reference to games or animation): A rendering style that has no inherent volumetric depth, unless painted and/or textured to imply such depth. 2.5D (in reference to games or animation): A rendering style that has inherent volumetric depth since it uses 3D modeling but is not presented with stereoscopic depth. COTS: Consumer Off The Shelf NPC: Non-Player Character. POV: Point of View s3D: stereoscopic 3D x Abstract The promise of a “virtual reality” in which users may immerse themselves into other worlds and other realities has been little more than that, a promise, for the past thirty or so years. While numerous projects have been produced in research settings or theme park environments, modern consumer-grade products allow us to bring the experience home. Until recently, little work has been done to combine the existing hardware tools of immersive and interactive media into a home setting where the average person, not a researcher, can explore these fantastical imaginary worlds. Seymour Deeply is, at its core, an immersive interactive storybook. The project uses commercial/consumer off the shelf (COTS) stereoscopic (3D Television) and gestural control (Microsoft Kinect 1.0) products to construct an immersive playspace that is not site-specific. It takes the work of virtual reality out of the lab and brings it to the living room. Its goal is to prove to those who doubt the power of stereoscopic 3D, who we term “stereoskeptics”, that adding stereodepth to an interactive experience increases the immersion for the viewer/player and can add narrative meaning. The entire narrative of Seymour Deeply is explored as a transmedia experience, with the backstory told in comic book form and the major portion of the narrative told as a playable interactive experience. Keywords: Interactive Media, Games, Narrative, Stereoscopic, Stereoskeptic, 3D, Gestural Interface, Kinect, COTS, Transmedia, LGBT xi Preface In a 1923 interview 1 , Harold Lloyd, world-famous star of silent films and early stereoscopy enthusiast, was asked about 3D. He responded: I believe that the man who invents a means of producing a perfect stereo motion picture will have accomplished the greatest achievement since the first motion picture . . . today, the motion pictures projected on the most perfect screen are lacking in solidity and relief. If the characters could only be made to stand out as they do in stereopticon pictures, and still retain the action of motion pictures of today, I think the ultimate would be reached by the cinema. Almost a century later, we are poised to realize Lloyd’s dream. With modern digital imaging, we can create better stereoscopic content with mathematically exact parallaxes and convergences, precisely synched cameras, and virtual cameras that do not have any of the physical limitations of live-action cameras. Stereoscopic vision is how we naturally see the world. We sense depth and volume because of our two forward-facing eyes. Stereoscopy is an active perception rather than just the spectacle that it has so often been presented as. Meant to Be Seen (MTBS) 2 , who claim to be the world’s first and only stereoscopic 3D certification and advocacy group, have conducted surveys of their members to determine their driving interest in stereoscopic 3D. 1 http://www.haroldlloyd.com/cms/index.php?option=com_content&view=article&id=136:3d &catid=44:articles&Itemid=174 2 http://www.mtbs3d.com/ xii The consensus of the members is that stereoscopic 3D technology offers superior immersion, more visual beauty, and is an effective tool to help tell interesting stories. 3 On the other hand, stereoscopic 3D has had its detractors. These “stereoskeptics” doubt the usefulness or even necessity of s3D as an imaging or storytelling tool. Primary among these disbelievers is film critic Roger Ebert, who detailed the various reasons he dislikes 3D in his now-infamous “Why I Hate 3-D (And You Should Too)” 4 . In this editorial, Ebert lists nine reasons for hating 3D (specifically 3D films). He added a tenth reason six months later when he printed a letter from Academy Award-winning editor Walter Murch on his Chicago Sun-Times Blog under the title “Why 3D Doesn’t Work and Never Will. Case Closed.” 5 Ebert admittedly makes some solid points about the economics of modern 3D when he notes that movie theater owners are sometimes coerced into investing in new digital projectors and that these costs are then passed down to the consumer in the form of a “premium” surcharge to the ticket prices of 3D movies (although Ebert avoids the 3 Schneider, Neil. “Breaking the Stereoscopic 3D Paradox.” http://www.siggraph.org/. Association for Computing Machinery’s Special Interest Group on Computer Graphics and Interactive Techniques, Aug. 2008. Web. 15 Feb. 2012. <http://www.siggraph.org/publications/newsletter/volume-42-number-3/breaking-the- stereoscopic-3d-paradox> 4 Ebert, Roger. “Why I Hate 3-D (And You Should Too).” Editorial. Newsweek 9 May 2010. The Daily Beast. Newsweek/Daily Beast, 09 May 2010. Web. 20 Mar. 2012. <http://www.thedailybeast.com/newsweek/2010/04/30/why-i-hate-3-d-and-you-should- too.html>. 5 Ebert, Roger. “Why 3D Doesn’t Work and Never Will. Case Closed.” Web log post. Roger Ebert’s Journal. Chicago Sun-Times, 23 Jan. 2011. Web. 20 Mar. 2012. <http://blogs.suntimes.com/ebert/2011/01/post_4.html>. xiii common stereoskeptic jeer that “3D” is actually an abbreviation for “3 Dollars”). However, his statements that 3D is a “waste of a dimension,” that it “adds nothing to the experience” and that he “cannot imagine a serious drama in 3D” are opinions, not valid arguments, based on what has been done with s3D in the past and how traditional 2D movies are shot. Ebert leaves no room for the imagination that, like color and sound before it, s3D is a tool that can be used skillfully to add to a motion picture experience and immerse an audience. Indeed, his (and Murch’s) arguments that s3D can cause headaches and that the issues of dim projection and stereo convergence vs. focus are at the core of those headaches are undermined by the fact that not all 3D presentations cause problems in all audiences. Incorrectly balanced sound can also cause headaches. Rapid editing can cause discomfort. The simple fact is that any improperly used tool can be used to bludgeon an audience and s3D is a tool that has a long history of misuse and shock use. Blogger and stereographer Clyde DeSouza submitted an excellent rebuttal to these arguments 6 where he rightfully focuses on the evolution of motion pictures and the fact that modern s3D digital production is significantly different from that of the 1950s or even the 1980s. He concedes that the tools and methods of content creation have changed and our expectations of content have similarly changed. Further, DeSouza takes his metaphor of evolution beyond the medium and places it on the audience. He argues that younger eyes 6 DeSouza, Clyde. “‘This Isn’t Your Grandfather’s 3D’. A Response to the Murch / Ebert Critique.” RealVision.ae Knowledge Base and Stereoscopic 3D Blog. Real Vision FZ LLC, 25 Jan. 2011. Web. 20 Mar. 2012. <http://realvision.ae/blog/2011/01/this-isnt-your- grandfathers-3d-a-response-to-the-murch-ebert-critique/> xiv adjust better to s3D than older ones (Ebert is 69 and Murch is 68) and calls out the obvious— that the audience of tomorrow is young today and today’s audiences are adjusting to, accepting and expecting 3D presentations. Our audiences (regardless of medium) are evolving. We should expect to do the same. As Lloyd noted nearly 90 years ago, after motion picture sound and color, stereoscopic 3D is the next major barrier. Today, the definition of “motion picture” has expanded to include any medium with a moving image: film, animation, television, interactive media (like games) and mobile content. 1 Introduction The use of s3D in interactive media is a relatively new phenomenon, at least in the consumer market and especially when compared to its use in other media. Additionally, narrative and storytelling in interactive media is also considered relatively young in its development. Therefore, perhaps the best way to initially explore the uses of s3D as a storytelling tool is to look to other moving image media. We must ask ourselves, “Traditionally, what advantages does stereoscopic 3D give a storyteller over other potential storytelling tools?” As mentioned above, the common response is that s3D gives the audience superior “immersion” in a story. What could that possibly mean? Audiences have been “immersed” in stories in all sorts of media for years. How could 3D possibly make a difference? Immersion is defined by Webster’s dictionary as “absorbing involvement”. In other mediums – novels, theater, film – we have seen audiences become absorbed, involved in the characters and plots of the story to the point where an individual audience member would call themselves “immersed.” This is emotional involvement. The audience in these cases is not being transported to another place or time in any real space (except in their minds). There is a passive absorption of the story (if we can stretch the point to say that reading is passive in this sense). The audience’s absorption of the story is through the storyteller’s skill in his/her use of characterization and depictions of situations that makes the audience sympathize with a character and be pulled into that character’s story. 2 This absorption can be increased with the used of 3D. In a March 2009 Time Magazine article 7 , Josh Quittner interviewed James Cameron about his then still-unreleased film Avatar and its fictional setting, Pandora. Months before the film’s official release, Quittner was able to screen a few minutes of the film. He notes: The scenes were so startling and absorbing that the following morning, I had the peculiar sensation of wanting to return there, as if Pandora were real. Cameron wasn’t surprised. One theory, he says, is that 3-D viewing ‘is so close to a real experience that it actually triggers memory creation in a way that 2-D viewing doesn’t.’ His own theory is that stereoscopic viewing uses more neurons. That’s possible. The potential biological implications aside (studies on the brain activity while viewing 3D are out of the scope of this paper), it is telling that Quittner expressed his reaction to a 3D experience in a way that suggests his immersion was as close to a physical experience as one could get from a projected image. In a 2012 interview 8 regarding his Academy Award nominated film, Hugo, director Martin Scorsese was asked by interviewer Larry Mantle if using 3D to shoot the film was a “one-off” for the director or if Scorsese felt that the technology opened up possibilities for other films. Scorsese responded: “There’s no doubt it’s opened up a whole new world for me. I happen to really be a 3D fanatic. I have been a fan of 3D since the old Viewmasters that I used to have when I was a child. I’m not saying this lightly. If, and when, the equipment becomes more flexible, and its in the process of doing that right 7 Quittner, Josh. “Are 3-D Movies Ready for Their Closeup?” Time 19 Mar. 2009. Web. 21 Feb. 2012. <http://www.time.com/time/magazine/article/0,9171,1886541,00.html>. 8 Scorsese, Martin. “The Wind-up Magic of Martin Scorsese’s “Hugo”.” Interview by Larry Mantle. Air Talk. National Public Radio. KPCC, Pasadena, CA, 8 Feb. 2012. Radio. 3 now. If and when, and its pretty soon, they’re working on losing the glasses, and the economics come into line, I wouldn’t mind shooting everything in 3D in the future. I mean, look around us. For most people, we hear sound, we see color and we see depth. So why not? It doesn’t have to be a gimmick. You don’t have to throw things at the camera. But it does something for the actors. It really does. It makes them more accessible to the audience somehow. It immerses you more into their story.” That same month, Mantle interviewed director Wim Wenders 9 about his Oscar-nominated non-fiction dance documentary, Pina. Wenders responded to the use of s3D as a thrill or shock element rather than as a story telling tool for audience immersion: Yes, I’d like to make a . . . tell a story in 3D, and of course that is relatively new territory, because mostly 3D has been used for action films and, of course, animation, very beautifully so. To tell an intimate family drama in 3D would still be a challenge and I’m really hooked on this technology and it’s a fantastic new language and we have to use it in other ways than just roller coaster rides. . . . And I think this new technology, this new medium, it’s actually a whole new language, is ideal especially in the realm of the documentary. You can take people into other people’s lives like never before. I mean, of course we saw 3D show up first on distant planets, so to speak, but I think it’s the ideal medium to take people to our own planet. Stereoscopic 3D environments give us an opportunity to virtually immerse an audience in a space, if not completely physically. By increasing the awareness of depth and more closely matching the sense of depth that we experience in the physical world, s3D displays move the POV of the player into a virtual space while the content of the experience can emotionally connect the player with that space. Combined, these two vectors can create more immersive experiences for an audience/player. 9 Wenders, Wim. “Wim Wenders’s “Pina”.” Interview by Larry Mantle. Air Talk. National Public Radio. KPCC, Pasadena, CA, 14 Feb. 2012. Radio. 4 This relationship shifts slightly for interactive media. Here the player is the audience. The passive act of absorbing a story now becomes an active act of living in the story. Regardless of whether or not a game lies within the “role playing” genre or not, almost all games engage the player in some kind of role-playing (even if the role is simply “puzzle solver”). The player takes on the role of a character to engage with the game. Because of this and because of games’ reliance on virtual worlds, it’s been said that interactive media are also much more immersive media. Here, however, the immersion is often less an emotional one than a simulated physical one, though by placing the player in the story, the emotional connection is often higher than in more passive media. The narrative content of the experience draws an audience in to the story. Stereoscopic presentation can place them into the space with the characters. How this differs for interactive media goes beyond the heightened narrative immersion of audience as player. With the technology available with interactive media, the s3D can be heightened as well. First and foremost among this interactive technology is headtracking and camera movement. For years, virtual reality research labs have used head mounted displays or motion capture cameras to precisely track where the viewer/player is looking at any one time. With this data, the stereoscopic display can be re-rendered in real time to adjust the perspective of the objects in the virtual world to match that viewpoint of the audience. This differs quite a bit from the stereoscopic presentations that we as the general public may be familiar with in a movie theatre. 5 Even though the presentation in a movie theatre is ostensibly for a large audience, the perspective of the camera dictates that only a single set of eyes is looking at the screen at any one time. This is true of all traditional media where the placement of the camera is dependent on the director. The camera location is decided in production and “fixed” in the medium in post-production. Luckily, our brains are very elastic and forgiving and we cognitively accept the shared view of the narrative world from the director’s POV. But the perspective of the world does not change for each individual. The control of camera position in interactive media is removed from the director (in most cases) and is in the control of the viewer/audience/player. As such we can move the camera around the scene and get a sense of depth through motion parallax, when the game is not presented in true s3D. Secondly, in traditional media the parallax and convergence settings are also designed for a single set of eyes (again, the director’s). This explains why there is often a “sweet spot” in a 3D theatre where the stereodepth looks better than anywhere else (and why sitting to the extreme sides in a 3D theatre is often problematic for an audience member). For s3D to work best, the viewer’s eyes need to be accurately aligned to the image. Stereo3D works because it tricks our brains into converging on flat images the way we would as if they were fully volumetric. With s3D prints or s3D films, those convergence points are set in post- production and are now in a fixed medium. Again, our brains allow for some elasticity here when we are not precisely in the viewing sweet spot and we still accept the stereodepth cues, but those cues are not quite as accurate as they would be in the physical world (it is this disparity that sometimes leads stereoskeptics to complain about headaches; their brains are 6 working harder to compensate). However, interactive media, by its very definition, is not fixed. It is dynamic and responsive. Not only does the headtracking/camera tracking ability of interactive media allow the designer to adjust the perspective of the virtual world, it also allows the designer to dynamically adjust the convergence every frame, always ensuring that the stereodepth is based on the viewer’s POV in the moment. This dynamic 3D can only be achieved with interactive technology. Finally, because these adjustments are happening in real time, the conscious awareness of 3D begins to fade for the viewer. The 3D is not there to call attention to itself. Quite the opposite, it now helps replicate the natural depth that we see in real life. This is a key point in the use of s3D. Since the stereoscopy is no longer the core of the experience, its deemphasized use allows the audience/player to again concentrate on what has always been the main purpose of immersive experiences: the story. Depth as a Visual Storytelling Tool Since the rise of visual storytelling, depth has always been a desired part of the toolset. In his book, The Visual Story, Bruce Block identifies seven main depth cues that can be used for effective visual storytelling. These are, in no particular order: Size Difference, Perspective, Atmospheric Diffusion, Movement (of camera or object), Shape Change, Tonal Separation, and Overlap/Occlusion. Their use in media as tools follows the rules of affinity and contrast (the purposeful placement of similar or different depth cues next to one another in a visual sequence) to modulate the emotional content of a given scene. In the second edition (2008) there is a cursory reference to one other depth cue, stereoscopic depth. Block 7 insists that it can only be used in conjunction with the other depth cues, but I disagree. Just as the other depth cues can stand alone and be used to convey a sense of space and emotion, so too can stereoscopic depth be manipulated by the storyteller to emphasize (or deemphasize, as the case may be) the emotional content in a scene. Of course, this is all very fine for traditional linear media like film, television, sequential art and (to some extent) theatre where the director has absolute control over the mise-en-scène and the point of view of the audience. However, in a September 16, 2010 lecture at the University of Southern California, I challenged Mr. Block about the use of depths cues as storytelling tools in interactive media, where the “audience” (as player) controls the camera. His initial answer was “you can’t [use them]” but then followed that statement up by discussing the choice of the style of the game (e.g., 2D vs. 2.5D), the texturing of models, and the agency or lack thereof that the player is given in a level to move around and look at the environment (thereby creating motion parallax). From a design perspective, these choices put constraints on a creator and some of them, like the choice of rendering the game as a 2D or 2.5D, are generally controlled by the capabilities of the game engine. Stereoscopic depth, however, is a tool that can be used and manipulated regardless of the other depth cues in a scene and it can be implemented on 2D as well as 2.5D gaming engines. 8 Stereoscopic 3D as a Visual Storytelling Tool To adequately discuss how stereoscopic 3D can be used like other depth cues as an emotional storytelling tool I should first offer, for the unfamiliar, a quick primer on how stereo 3D works and some of the terminology used in working with s3D. Stereoscopy works because each eye is sent different perspectives of the same image. The parallax between those two perspectives determines where we perceive an object in 3D space. The stereo volume effectively has three components: the screen plane, positive parallax space and negative parallax space. Figure 1: The Stereo Volume At the screen plane there is zero parallax and both perspectives are synchronized. Objects with zero parallax appear to be right at the screen plane. It is like the window into the world. It is how standard 2D media is viewed. Objects or characters with positive 9 parallax will appear behind or “into” the screen. These objects appear more distant from us physically. Conversely, objects or characters with negative parallax will appear in front of the screen or in the audience’s space. This is true for all stereo3D media. However, the distance that we perceive from an object/character can also be mapped emotionally. Positive parallax space (defined by Lenny Lipton as “screen space”; see Glossary) can give a sense of openness and grandeur. It can make the viewer/player feel as if the world beyond the screen is endless and deep. Objects or characters in screen space might also be considered to be at a safe distance from the viewer/player if they are perceived to be dangerous or a threat. Additionally, they can also be considered emotionally distant and/or unavailable. Negative parallax space (defined by Lenny Lipton as “theater space” or “audience space”; see Glossary) can imply containment or confinement for the viewer/player, but objects or characters in audience space can also be perceived to be emotionally closer and available. Objects in audience space are often used for shock value as they can be used to be an attack or invasion on the audience or when an object quickly moves into theater space for surprise. These uses as figurative and emotional depth cues are still being explored, certainly in cinema and barely at all in interactive media. The potential is tremendous, however, especially with interactive media where the stereo depth can be controlled dynamically and independently of the camera’s position. 10 I have attempted to incorporate some of these depth cues in my thesis project, Seymour Deeply. The challenge here is similar to the challenge that a film director, production designer or film stereographer would find: how to use 3D as an emotional depth cue as one would any other depth cue without necessarily calling attention to the cue itself. As I mentioned in the Introduction, s3D’s use in media up to this point in time has mostly been a very conscious use – the audience is acutely aware that the scene they are viewing is in s3D. This may be even more pronounced in interactive media where there has traditionally been less use of s3D, so it is still considered somewhat of a novelty to an audience/player. The direction that the majority of proponents of s3D (in all media) would like to head is a use of s3D that becomes invisible, or at least deemphasized, to the audience. This is how other visual storytelling tools (depth cues, color, line direction, etc.) currently operate. Like a film director, the designer must then be conscious of the use of 3D while making it appear seamless to the audience/player. I discuss the challenges I faced in this area in the section “Project Concept and Development Process“ below and explain why even though this challenge was my primary design prompt, it became less important as the project progressed. Stereoscopic 3D and Gesture Upon deciding that my thesis project would explore the uses of stereoscopy in interactive media, I made an early decision to incorporate gestural interface in the project. In my mind at least, and especially for interactive experiences, I find the use of stereoscopy and gesture to be inexorably linked. 11 Consider for a moment a presentation in a large 3D theatre like the ones in theme parks, where there is an extreme use of negative parallax to send objects into the faces of audience members. Simply observe any audience, especially a younger audience, as that object comes off the screen plane and into the theater space. The majority of audience members make an instinctual motion to reach out into space and touch that object. After all, to the observer, it is floating there right in front of their faces. We want to be able to touch the things that appear in our personal space, even if that virtual object cannot give us the same haptic feedback of a physical object (although the current research being done to provide a sense of that feedback is fascinating, it is also outside the scope of this thesis). What that simple motion in the theater tells us is that stereoscopic 3D and gestural input are married in a fundamentally human way. With gestural controls, we can capture that motion in a home setting. This is just one of the reasons for the decision to use the Microsoft Kinect for my thesis project. Another reason to use the Kinect is that the technology offers more than just gestural input to help with stereo3D. Since Kinect has the ability to sense depth and track skeletal movements, Kinect allows for head tracking in addition to gestural input. As mentioned above, not only does this give us, as designers, the ability to dynamically adjust convergence and parallax and to simulate holography by adjusting the perspective of virtual objects according to the position of the player, the overall effect reduces the conscious emphasis on 3D as the core of the experience, so the player focuses again on the mechanic and the story. 12 Although I go into more depth in the discussion of early prototypes below, my overall philosophy toward gestures in a stereoscopic space was that they should be as natural as possible. I did not want to create a “vocabulary” of gestures that would then need to be taught to the player in order to have her interact with the system. Any non-narratively driven gestures would have been antithetical to the immersive experience I was attempting to create. Indeed, I do not think this is necessary when dealing with interactive stereo3D. If a player perceives that an object is near her because she sees it in negative parallax space it is completely instinctual that any action made toward that object would affect it (or that it would affect the player in some way). If there is a perception that an object of desire is out of reach (because it is deep in positive parallax space) a player will act to move toward it. Of course, there are technical limitations to this line of thinking. The Kinect 1.0 that we are using in the project is an amazing piece of consumer grade hardware, but it is limited in its fidelity and its response time. As I discovered through the development of the project, certain movements in s3D space are too granular for the Kinect to parse. The tracking also does not do well with static or “held” gestures. Although I am confident that future technology will allow for much greater tracking resolution than the Kinect 1.0 and that this will allow for more finite control in s3D space, we turned these current technical limitations into story elements and adjusted our narrative to fit the types of gestures that the Kinect handles best – smooth, fluid movements. Although the ideal would be to have story dictate the appropriate gesture, here the available gesture dictated changes to story. 13 Project Concept and Development Process Early Prototypes When I first approached the thesis project, I began with a simple research question: “Can Stereoscopic 3D be absolutely integral and necessary in the exploration of a narrative in an interactive project?” What brought me to this question was my interest in 3D as a means of immersing the player in a story. As discussed above, there is evidence of 3D being used as a storytelling tool in other media to immerse an audience and potential to use it to emotionally connect an audience to story, but its use in an interactive project was less defined. Additionally, I wanted to look at some of the immersion techniques already used in interactive media (primarily in the fields of virtual reality) and see how those same principles could extend from the laboratory or the theme park to the home setting, using current COTS (Consumer Off-The-Shelf) products. The first conceptual iteration of this project was broken into three components: the use of 3D in a game-like setting, the use of 3D in an educational setting and the use of 3D in a social commentary setting. In my initial research, I discovered that 3D in games has been used as a spatial mechanic in puzzle games like Super Hypercube 10 and The Depths to Which I Sink 11 and in educational settings such as virtual frog dissections like Tactus Technologies’ 10 Polytron/Kokoromi. 2008. Super Hypercube. Windows. Independent. Game. 11 Big Pants. 2008. The Depths to Which I Sink. Windows. Independent. Game. 14 V-Frog 2.0 12 . These were all practical applications of the technology. What these seemed to lack, however, was the influence of story and narrative. It is true that s3D has been implemented as an “add-on” feature for some PC and console games (especially for the Playstation 3) like Call of Duty and Assassin’s Creed: Revelations, but this use of s3D is effectively an “instant” conversion from a game that is designed to be played on a flat screen to a stereoscopic screen. This relies on the fact that modern PC games are rendered in 2.5D using DirectX. A stereoscopic 3D driver can take the 3D modeling information that is sent through DirectX and extrapolate a left and right camera view to create a stereoscopic 3D image. However, this is akin to a film designed and shot in 2D converted to a 3D print. There has been no special consideration for the use of s3D in the design of these games so the stereo3D is relegated to little more than a marketing gimmick. What’s worse, often the conversion creates rendering problems, a list of which is maintained by Meant to Be Seen under the umbrella of “Game Grade 3D.” 13 s3D’s use in a social commentary setting was more driven by my personal politics than anything else, but even there, the idea of using 3D to immerse an audience in another time, place or body than they inhabit in reality has been covered in the public sector in theme park settings like Disney’s Captain Eo or Honey, I Shrunk the Audience or Universal Studios’ King Kong 360 3D. For my second conceptual iteration, then, I decided to combine 12 V-Frog. Computer software. http://www.tactustech.com/vfrog/. Vers. 2.0. Tactus Technologies. Web. 28 Feb. 2012. 13 “MTBS GameGrade 3D.” www.mtbs3d.com. Meant to Be Seen. Web. 20 Mar. 2012. <http://www.mtbs3d.com/gg3d/> 15 my initial three vectors into one narrative, hoping to immerse the audience into embodying a character in a way that other media cannot. This second iteration is where the formation of story began to enter into the project and here is where I can caution future researchers and creators in this field. Practically any respectable media creator, regardless of his/her field, will tell you that when you are dealing with a narrative project, story comes first. Story, story, story. This mantra has led to the success of many animated films, notably the oeuvre of Pixar. As Pixar CCO John Lasseter explained to PBS’s Charlie Rose in a 2011 interview 14 , “One of the things that I’ve always, that Steve [Jobs] and I have always loved is the new technology. It is so inspiring to us and we keep pushing it, but what we never forget is the fundamental basics of what a movie is and that is the story and the characters. And so when you look ahead to where its going, its really the technology, as we develop it at Pixar, is always driven by the needs of the story.” In a traditional setting, I would have written my narrative and then looked at what technology, indeed what medium, could best be used to tell that story to an audience. In this particular case, the cart was before the horse and story came second. This is not to discount the work done in building the story and the worlds for Seymour Deeply. On the contrary, the story grew and changed based on what was and was not possible with the technology at hand. In this, the project is not unlike the anecdotal description of the way Steven Spielberg’s 14 “John Lasseter, Director and the Chief Creative Officer at Pixar and Walt Disney Animation Studios.” Interview by Charlie Rose. http://www.charlierose.com. 2 Dec. 2011. Web. 15 Mar. 2012. <http://www.charlierose.com/view/interview/12024> 16 blockbuster 1975 film Jaws was altered in editing when the special effects of the shark proved to be unrealistic. 15 In looking at “story” I knew that the familiar is often a comfort to an audience experiencing new storytelling techniques and technology. One only look at the critical response to James Cameron’s Avatar, which undoubtedly pushed boundaries in both the fields of stereoscopic 3D and motion capture animation, but still relied on a familiar story to introduce these innovations. Wired magazine’s review put it simply when saying that the premise “essentially retools Dances With Wolves’ ‘exploitive white man goes native’ scenario.” 16 To this end, I sought to retell the well-known English fable Jack and the Beanstalk. It is a simple, oft-told tale that is familiar in Western culture. More importantly in an exploration of stereoscopic 3D, Jack and the Beanstalk deals with two issues that are illuminated through stereodepth; great distance (between the earth and the land of the giants) and scale (between the humans, Jack, his mother and the villagers, and the Giants). Now that I had a story that I wanted to work with, I had to make it personal and, again looking back at my initial design challenge, see how I could use stereoscopic 3D to not only help tell the story and emotionally connect them player, but to also be integral to the game play itself. 15 “Jaws: Did You Know?” IMDb. IMDb.com. Web. 03 Mar. 2012. <http://www.imdb.com/title/tt0073195/trivia?tab=tr&item=tr0776899>. 16 Hart, Hugh. “Review: Powerful Avatar Stuns the Eye, Seduces the Heart.” Wired.com. Conde Nast Digital, 17 Dec. 2009. Web. 27 Feb. 2012. <http://www.wired.com/underwire/2009/12/avatar-review/>. 17 At its core, Jack and the Beanstalk is a classic Campbellian hero’s journey as well as a coming-of-age/sexual awakening tale. 17 Jack is a young man who must cross the threshold into another realm, which changes him before he can return home. As with any adaptation, alterations to the story to update and modernize it were required. I wanted to spin the traditional story in two ways: I wanted to make the tale a gay love story and I wanted the protagonist (i.e., the player character) to be a Giant rather than Jack. Suddenly the character of Seymour Deeply came alive. He was living in the land of the Giants, but he longed for adventure elsewhere. This led to the development of the first digital prototype, which I created in Adobe Flash. There was no stereo3D implemented and we were still looking at several options for using the Kinect with the Unity3D engine. I wanted to test basic player interactions. At this point in the process, Seymour was a painter, an artistic soul. I wanted the story to reference other depth cues and began with the use of color theory to portray depth. Color theory dictates that cool and dark colors will recede into the viewing plane while warm and light colors will advance out of the viewing plane. In this early prototype, Seymour painted in a watercolor style on the clouds and when a sufficient saturation point was reached, the cool and dark colors would recede in stereo depth and the warm light colors would advance. The player would be prompted through this level as to where to paint and with what colors. Through this mechanic, the story would be revealed as the player would communicate to 17 Bettelheim, Bruno. “Jack and the Beanstalk.” The Uses of Enchantment: The Meaning and Importance of Fairy Tales. New York: Knopf, 1976. 183-93. Print. 18 Jack through these sky paintings. Since the player was looking down through the clouds, the stereodepth would open “holes” over which objects could be positioned and dropped through, advancing the player to the next level. At this juncture, the project was (rather pretentiously) titled “Drop of Colour”. Figure 2: Screenshot of early prototype, “Drop of Colour” This intellectual and, in retrospect, highly academic, exercise proved utterly disastrous when playtested. Not only was it difficult to educate non-artists on the basics of color theory, the connection between color and depth was not clear and the resulting painting mechanic simply was not fun. On top of that, with the exception of the cloud layer, there was no evidence of the Jack and the Beanstalk fable at all. This led to a reevaluation of what the mechanic could be and what would be a fun exploration of 3D space. 19 It became clear that my thinking had, literally, been flat. I realized that I had been approaching the problem like a theatre director who is accustomed to directing on a proscenium stage and then must shift to arena/in-the-round staging. I have often seen plays in arena staging where it is apparent that the director has only ever viewed his/her staging from one vantage point in the audience. The staging is obviously, and worse, uninterestingly flat. Painting is essentially a two-dimensional art form. I needed to think volumetrically, which shifted the focus of the mechanic from painting to sculpting. This shift was fundamental in the process of the design of the project and cannot be emphasized enough. When dealing with creation of 3D content, the creator must remain vigilant to think more like a sculptor and less like a painter. In cinema training programs the presentation of the frame has been of paramount consideration for years. The layout of the picture in terms of its composition has been key. Indeed our very vocabulary for discussing media is embedded with “flat” terminology: moving image, imagination. I realized that I needed to break this mind set in order to properly create a stereoscopic cinematic narrative experience. Our second digital test moved us from Flash (a 2D platform) to Unity3D (a 2.5 platform). We were able to implement stereo3D using an early version of Perry Hoberman’s Stereoskopix package for Unity. Since some members of my team were using Macintosh computers and some were using PCs we needed a cross-platform tool for our Kinect implementation. We were also able to find this tool in ZigFu (available at ZigFu.com), which was still in beta testing at the time, but has since been released to the public as a 20 development kit. In this test, we used ZigFu’s handtracking to control particle effect game objects in a stereoscopic space. Playtesters responded positively to the fluid movements that they could make with their hands being tracked by the Kinect. As a technology test, this was successful. Although there was no gameplay per se, the feel of the game was starting to come together. We were still stuck on thinking in terms of “painting” but the paint was now particles and the “canvas” was no longer a flat plane. Engaging my design team more, we started to brainstorm on what mechanics were more sculptural and less painterly. We hoped to use the hand-tracking example to create shapes from procedurally generated 3D meshes, but this became a technical challenge (not an insurmountable challenge, but one that confounded our design and engineering team at the time). We realized that the focus of the mechanic needed to be the manipulation of objects in 3D space. It also needed to be fun. From this we came up with the idea that a Giant in the clouds would be close to the stars. He could manipulate the stars into sculptures that, to those on the ground, would simply seem like constellations. This shifted the story considerably, but for the better, as the entire story of the game began to be fleshed out. Suddenly Seymour needed to have a narrative connection to the stars. Instead of being an artist (amateur painter) he became a scientist (amateur astronomer) and therefore explored various viewing techniques like stereoscopy and binocular vision. This also helped set up narrative conflict between Seymour and his father, who, like so many parents, wants control his son’s destiny. Seymour’s motivation for moving the stars would be to send a message to Jack in the best way he knew how, by effectively writing across the 21 sky (an idea we kept from when Seymour was still a painter). The first level of the game now started to reveal itself. Seymour could create constellations that would be meaningful in the context of the story. As a creator, I could limit the agency that the player has in creation of the constellation to only those shapes that had narrative meaning. The use of s3D would be necessary to know where in 3D space each star had to go. Since we had proven that we could track hands in 3D space, we shifted the mechanic to a connect-the-dots mechanic and had the players grab stars from one part of the stereovolume and move them to another. Once there, they would “snap” into place, forming constellations that would be evocative of the story. These constellation puzzles would be based on 3D meshes, so when they were complete, they would reveal their volumetric nature. This is also a twist on how we usually perceive constellations, which are based on stars that are millions of miles apart, but are perceived as two-dimensional drawings here on Earth. Jack would see these constellations from the Belowlands (his world) and know that they were from Seymour. We would move the story along by having Jack respond to Seymour. Initially this was going to be by having Jack send a series of balloons to Abovia (the Giant’s land), which Seymour would have to pop in order to reveal the messages inside. Production time constraints forced us to abandon this idea, but it could be reincorporated in a possible future iteration of the project. Instead, the character of Colin, a bird who is a confidante to Seymour, was created to facilitate the conversation between the two main characters. A positive response from Jack 22 after the first level would spur Seymour to gather more magic beans in order to grow a new beanstalk. This prompted the second level, where Seymour must harvest beans and transform them into “magic” beans. Here the s3D would be necessary to game play because it could help separate the various beanstalks from one another since their color and texture would be similar and would otherwise appear flat in a 2D presentation. Now that we knew a new beanstalk would be grown, the final level would naturally have Seymour pursuing his objective to reunite with Jack and have him travel down the newly grown beanstalk. Here, again, the game mechanic influenced the story. How does one climb down a giant beanstalk? Climbing and crawling seemed like possibilities, but in 3D space did not seem to be particularly interesting mechanics to explore. This is where we decided to have a little fun with 3D. If the beanstalk was a large spiral and we were looking “down” it, how would that appear in s3D? It would be a long, twisting, spinning tunnel that more than likely would cause a vertigo effect in the viewer. It would also be fun. How would a player travel thought this tunnel? We took a page from Disney’s Tarzan and realized that the player could “skate/snowboard” down the length of the stalk. In order to make this believable in context, the size and scale of the stalk would have to be much larger than we originally anticipated. This had repercussions for the rest of the story. As these size and scale differences between Jack’s world and Seymour’s world had to be defined, the backstory grew. In order to present details like these, certain world building decisions had to be made and, more importantly, had to be presented to the player. I discuss the ramifications of this in the section “Transmedia Extension“ below. 23 This newly structure narrative formed the basis for the next major prototype, which was completed in time for the Interactive Media Division’s Winter 2011 show, Winteract. We presented the project with programmer art (even the particle effect hands were replaced with literal hand 3D models) and one constellation puzzle (which was a simple cube) projected on a large silver screen using polarized 3D glasses. Headtracking was implemented so the virtual world would move around in response to the player’s movements in the real world. Calibration for the Kinect, however, still had to be accomplished on a scene-by-scene basis. Figure 3: Winteract 2011 Seymour Deeply setup and playspace (Player: Kevin Fabian) The response to this prototype was that the gameplay was interesting, but the limitations of the Kinect became evident. Having a player find and select an object in stereospace proved too challenging (especially for non-gamers). The fidelity of the Kinect 24 tracking is not granular enough to track slow, precise movements (which we were asking the player to do as she grabbed a star object from one location and brought it to another as that seemed to be a natural, intuitive gesture). Players felt constrained by the precise movements. Players who had used the earlier handtracking test remarked on how they missed the fluidity and freedom that those gestures offered. Targeting was also challenging because we offered no clues for the player as to where the “proper” placement of the star should be, relying on their discovery of the constellation through exploration. Although this discovery was fun for players once it occurred, the challenge wasn’t worth the reward. Additionally, bugs revealed that we had given the player a finite number of resources (stars) to work with, but no way of replenishing those resources if they were used or lost. Finally, and most importantly, although the game was in stereo 3D and players could navigate the 3D space, the connection to story had been completely lost and, through that, any possibility of having an emotional connection through 3D. Current Prototype The winter break was used to reevaluate the project and what could be done in the second semester. It was clear that the focus of the first semester was squarely on the technical challenges of the project. Any emphasis on story or the use of stereoscopy to tell the story was reduced in order to overcome the hurdles of combining Kinect hardware with the Unity3D engine and implementing stereo3D on top of that. Part of the reevaluation for the second semester required me to question my original intent of the project – to use stereoscopy in an emotionally meaningful way – to see if that intent was still possible given 25 even more technical challenges to come. In addition to this reevaluation period, I knew that the development team would be changing as both our programmer and Kinect engineer were moving on to other projects at the end of Fall semester. In light of this, my producer and I actively recruited new members for the project from the computer science department’s mobile and console gaming class. We were thrilled to more than double the size of the development team. With a larger team, we were able to properly judge the scope of the project for the Spring semester as well as the entire project itself. I considered all the previous builds to be works of exploration and decided to use the knowledge gained from them to rebuild the project from the ground up. This decision also coincided with new versions of both Unity3D and ZigFu, so we could have the new development team build with improved tools. However, the new development team needed to be brought up to speed on the project and also needed to be introduced to the tools we were using. Since the majority of the new programmers and engineers were being “borrowed” from the mobile and console gaming class, there were certain expectations for the project that would both satisfy the needs of the project as well as the educational needs of the students from the class. A decision was made at this point to make the project as polished of a playable experience as possible by the end of the Spring semester. While it would still incorporate both Kinect gestural control and stereoscopy, the emphasis of the immersion of the project moved away from stereoscopic immersion and toward full body immersion. While I still believe that stereoscopic immersion can enhance the emotional content of the story and increase the emotional immersion for the player, I have not yet met 26 that goal with this project. Inserting narrative elements at all into the game play have proved to be more than enough of a challenge. For the final Spring prototype, we are planning three levels after an initial title screen in which the player will have her primary calibration with the Kinect. Once calibrated, the player is presented with a short animated cutscene done in “Ken Burns”-style (animated camera over still images) that briefly details the backstory of where Seymour is emotionally and where we are in the narrative. In Abovia, when a new head of the family ascends, he or she is required to prove themselves to the clan by pursuing their heart’s desire. Colin, the bird who acts as Seymour’s guide, tells him to fulfill his desire to be an astronomer and ignite the sky with stars. The game transitions into the first playable level, which is based off of the previous prototype in its goal. The player is still attempting to reveal a constellation by placing stars at the proper vertices, but now instead of grabbing a star and moving it in space to its position, players shoot fireballs out into position. Still in first person perspective, Seymour’s/the player’s hands are represented on screen as glowing balls of stardust. Even though the player will be wearing 3D glasses at this point, the scene is initially presented with little to no stereodepth. Looking out into the night sky are hundreds of stars. Some stars are white, while others are simply un-ignited balls of gas and spacedust. They vary in size. The player is prompted to rotate her hands to generate more stardust. When she is ready, the player aims in a direction of the night sky and puts her hands together. This ignites the two balls of stardust into a new star and fires it off into the distance. This 27 eliminates the limited resource issue that arose in the previous prototype since the player now becomes her own resource generator. Without stereodepth, though, the player will not be able to see where she needs to send stars. Colin prompts Seymour to “put on” his special binocular telescopes. The player does this through a gesture of raising her cupped hands to her eyes. Figure 4: Player gesture to put on virtual stereoscopic binoculars (Player: Scott Hardie) This gesture enables the stereodepth in the scene and the player can now see where the stars are in depth and in relation to one another. This is another way of not only showing 28 the differences between 2D and s3D gameplay, it also uses s3D as an embedded narrative device that also assists in gameplay. By establishing, in the previous cutscene, that Seymour’s desire is to reach the stars, the constellations are now no longer in the audience/player space in negative parallax, but placed deep in positive screen space, farther away from the player than when she was viewing the scene in 2D. This increase in perceived distance will hopefully emphasize the desire to reach something unattainable, but playtests still need to be conducted to see if this holds true beyond the theoretical. Figure 5: Screenshot of partially completed constellation puzzle in Level 1 As originally planned, each constellation is a rendering of an object or person that is descriptive of what is occurring in Seymour’s subconscious. The player must complete at least three constellations before progressing. The player can complete the constellation puzzles in any order and can even work on more than one at a time. As another way of attempting to incorporate s3D as a storytelling component, when each constellation puzzle is 29 complete it moves toward the player in negative space, so it becomes virtually close to the player. Again, this attempted connection to stereodepth and character desire still needs to be adequately playtested to see if the anticipated transference of emotion (unattainable goal in positive screen space; attainted goal in negative player space) will resonate with players. What is ultimately uncovered in the story by the end of this level, is that the completed constellations reveal that Seymour’s heart’s desire was not so much to be an astronomer, but to be with Jack. As this is still a work-in-progress, the final two levels as well as transitional cut scenes are being created, but it is still unsure at this moment if they will be complete by the end of the Spring semester given the time and technical constraints that are still on the team. These levels and cut scenes are being designed based on the original plan described above in the section “Early Prototypes“. Transmedia Extension It became apparent during preparations for Winteract that there was perhaps too much story to tell in Seymour Deeply. That is, in order to adequately set up the game play in the interactive portion we would need to tell the entire story of Jack and the Beanstalk, with the modifications we had made, either in newly developed levels or through a long, perhaps potentially clumsy, cut-scene animation before the game play began. Neither seemed to be a desirable solution. With the former, we would have to decide the character point of view of the player for the tale. With the latter, we would have to compress the story so as to not bore 30 the player or, worse, make them feel as if they had no agency in jumping in and starting to play. Arguably, this is, and always has been, Jack’s tale. He initiates the travel from the Belowlands to Abovia and his is the story that has, initially, the most action. It would make sense to have the player embody the character of Jack and play as him. However, this was at odds with my desire to tell the story from Seymour’s point of view. In the first part of the narrative, Seymour is largely inactive or, at best, reactive. Game play is about the doing; the mechanic; the activity. A passive character is a poor choice for a player game character. Seymour does not become active, in a narrative sense, until after the first beanstalk is destroyed and his father is killed. Additionally, we had already decided that it is Seymour, in his exploration of optics and astronomy, who is the character most narratively tied to stereoscopy and 3D. A solution arose. My increasing interest in transmedia storytelling presented a unique opportunity to extend the story across other mediums and to expand the project as a whole into a transmedia experience. Although this increase in the project’s scope is perhaps responsible for pulling my focus away from a concentration on interactive stereo3D , I felt that the project overall would benefit more from having the story extended. This would also allow me to play with the capabilities of s3D in a less dynamic medium to better compare its use across media. We made the decision to tell the backstory of Seymour Deeply (what we generally know as the Jack and the Beanstalk fable) in comic book form. This gave me the ability to tell a sweeping story with several necessary visual effects. It also allowed me to 31 create and maintain a sense of style across the two media. Additionally, there is quite a precedent of prior art in the way of comic books being used as transmedia extensions of game worlds, going back as far as Atari’s Swordquest series in 1983 through Team Meat’s Super Meat Boy in 2010. The comic book is short form (only eight pages). It tells the familiar tale of Jack and the Beanstalk from a third person narrator point of view. The narrator is Colin, the magical bird friend of Seymour. Colin naturally has the ability to transcend the two worlds, even when the protagonists cannot, so he seemed a perfect device to tell the story of two star- crossed lovers. Figure 6: Rough sketch of page one of Seymour Deeply comic 32 For those interested in the full story, the original comic book script is attached to this paper as an appendix and the full comic (still in production as of the writing of this paper) is available online as a webcomic at www.seymourdeeply.com. Use of COTS Technologies as a Constraint As another point in my overall design process, I need to discuss my choice to constrain my project technology to consumer products (a choice I admittedly had to forgo for the public presentation at Winteract). Too often we have seen interesting interactive projects that have the distinct feel of “tech demo” about them. These demos are frequently the result of massively funded R&D projects. However, I was inspired by the research of Johnny Chung Lee 18 , whose work at Carnegie Mellon and Google focuses on low-cost solutions to problems that traditionally would have required substantially larger financial resources. His “found object” approach to design was not only appealing from a monetary standpoint; it also stresses the growing ubiquity of this technology whose public availability, until recently, has been limited. With Seymour Deeply, I wanted to move the concentration away from the technology and back on to story. To emphasize this focus, I used readily available consumer products to tell the story, thus a consumer-level Microsoft Kinect and a consumer-level 3D TV to view it on. By not relying on specific sensors or unique one-of-a-kind hardware, this constraint 18 Lee, Johnny C. “Johnny Chung Lee: Projects.” Johnny Chung Lee. Web. 21 Mar. 2012. <http://johnnylee.net/projects/> 33 actually freed me to imagine an immersive experience that heretofore might only be limited to site-specific engagement (like research labs or theme parks). Audiences have enjoyed these experiences in their unique settings for years, but since the technology is now readily available to consumers in the home, I had to question why content creators were not focusing more on bringing the amusement park home. Why not create an experience that immersed a viewer in their home the way they can be in an amusement park? The challenge is to scale down the experience to what is possible when the content creator does not also control the technology and still make it enjoyable and engaging. That is what I have hoped to do with Seymour Deeply. Concluding Remarks and Suggestions for Further Exploration Although explored in Seymour Deeply, the use of stereoscopy to increase player immersion and emotional connection remains inconclusive. If players are “absorbed in their involvement” in a stereoscopic experience, how can we gauge what portion of that absorption is attributable to the stereo and what portion is attributable to other factors like player agency, narrative connection and full-body gestural immersion? It may be impossible to ever separate out these various factors that engage a player, but does that mean we should ignore the potential of stereo3D in games? Not at all. 34 As Richard LeMarchand discussed in his 2012 GDC talk 19 , “Attention, Not Immersion: Making Your Games Better with Psychology and Playtesting, the Uncharted Way” perhaps framing player involvement as “immersion” is faulty and that player “attention” is more accurate. Stereoscopic 3D is an excellent aesthetic tool for getting player attention, especially when combined with motion. The human eye is drawn toward movement. This fact is used in all media to direct the audience’s eyes to an important aspect of the image and through this can reveal narrative elements. In traditional 2D media, this movement is always on the X-Y axis. Even when depth is implied through diagonal movement, the action is still in the X-Y plane. With s3D, movement can also occur in the Z- axis and this is where attention can be grabbed rather easily. If a virtual object moves forward or backward in stereospace, the eye will be drawn to it. Granted, in Lemarchand’s view this would be attention garnered through the use of aesthetics, which as he rightly points out can grab attention immediately but cannot hold it for long periods. But by setting a player in a full stereo3D environment, her vigilance of attention must increase because there is now a third dimension to consider. This increased player vigilance and engagement would make an excellent topic of further study. Also, this consideration of the third dimension should extend to creators of all media who work with stereo3D and wish to engage their audiences. As I discovered in the course of my work with Seymour Deeply, modern media steeped in the cinematic tradition is based 19 Alexander, Leigh. “Forget ‘immersion’ -- Player Attention Is What Matters, Says Lemarchand.” Gamasutra. 7 Mar. 2012. Web. 27 Mar. 2012. <http://www.gamasutra.com/view/news/164914/gdc_2012_forget_immersion__.php> 35 on painting – a flat, two-dimensional representation of a three-dimensional world. Any creator working in 3D needs to be mindful that the media they are creating is akin to sculpture and imagine their work in volumetric space rather than framed, screen space. Interactive media creators and many digital animators are generally familiar with this already since our work is often constructed in virtual 3D worlds, but with the addition of s3D we need to remain aware that the space in which we are working now extends beyond the flat plane of the screen in two directions and work to fill the space, not just the frame. Finally, in reviewing my design philosophy of Seymour Deeply, another reason to combine stereoscopy and gesture was revealed late in the course of my research and development of the project. It is this reason that I feel may be most relevant for future researchers and further development, so I felt it necessary to mention it here, fresh and unformed as it may be. It hinges on redefining the play space in which we work and how this can only be achieved through a marriage of stereoscopic display and gestural input. Unlike sports or board games, digital games in the home are ones played in complete virtual space. This space has traditionally been separated into two distinct play zones – the one in the screen and the one “on the couch”. Usually these two zones are separated by several feet between player and the player’s counterpart in the virtual world, with the screen plane acting as a border for one space and the controller acting as a border for the other space (and a no-man’s-land in between of several real square feet in which no play occurs). Even when gestural control is introduced to the equation, the no play zone still exists because current tracking hardware does not have the refinement to work at close range. 36 Figure 7: Traditional Counterpart Play Zones Combining stereo and gesture, however, can fill in that no-man’s-land between these two spaces and allow it to become an area of play. This new volume exists partially in the virtual world (screen plane and positive space) and partially in the real world (screen plane and negative space), mixing the two realities in the living room where the screen becomes less a border between real and virtual than it is a bridge between real and virtual. Figure 8: Future Continuous Play Zone 37 As designers using this new technology in a consumer/home arena, we need to be aware of the mixed-reality setting in which we could be working. That is to say, for every interactive experience, we need to imagine this volumetric space for play. The screen is merely the portal through which the experience is perceived, but we need to be mindful that for the player, the objects in the world exist both behind and in front of that screen. By dynamically changing the size and location of the stereovolume for each player, we can design experiences where, for example, a virtual baseball might be pitched in positive stereospace from a mound that appears, from the player’s perspective, to be sixty or so feet away, but when the ball approaches the player, it crosses the screen plane at, say two feet away. The player would gesture to “swing” at the ball in real space, in a living room/den, and perceive that the ball is right there in the living room, in front of her. When she hits the ball back into the outfield, she’ll see a tremendous sense of depth in positive parallax, equivalent to the force with which she swung her virtual bat. In this imagined scenario, the volume of play exists simultaneously for the player across the virtual and the real. The game is no longer solely on the screen or solely in the living room. The stereoscopy is not a key component in the sense that it is being used as a formal element of gameplay (as a mechanic, a resource or a constraint), but because the player’s conscious awareness of it is reduced or eliminated, she simply exists, immersed in a playspace. We are clearly not yet at this ideal continuous play space in a home setting, although analogues for this kind of space exist in research labs in the form of stereoscopic caves and 38 multi-screen stereoscopic environments. Technical challenges still need to be overcome to bring this home, like the aforementioned tracking issue, near-field stereoscopy (where objects in extreme negative parallax may cause eye pain from over convergence), and occlusion issues (where a virtual object projected on the screen may be occluded by a real object like the player’s hand). Additionally, with the display acting less like a border and more like a portal, physical harm could come to the player if she got too close and hits the screen. As areas of further exploration, however, these kinds of experiences are precisely where we, as designers working with stereo3D should be looking. The convergence of media seems inevitable. In the coming years, it will be more difficult to define what is “television” from what is “film” from what is “interactive media”. Regardless of the label put on this new media, how we consume them is also bound to converge. Headtracking may enter into the cinema as it already has into the home. Gestural controls will enter into the corporate world as they have entered into the entertainment and play worlds. 3D displays, whether they be active or passive glasses stereoscopic, autostereoscopic, or holographic in nature are bound to become more prevalent in every day life. The content that is created for these displays must also adapt and evolve with the times. Content creators must maintain a vigilance to conceive our projects in three dimensions and not just two and to use all three dimensions to fully tell our stories. 39 Glossary This glossary is an abridged version of Lenny Lipton’s fantastic glossary of stereoscopic terms, published on his blog on March 16, 2009. 20 There are few, if any, comprehensive glossaries that cover the basis of stereoscopy and stereography as well and so it is borrowed here for the convenience of the reader. The humorous tone of the glossary is courtesy of Mr. Lipton and appears in the original. Autostereoscopic. Sometimes called “auto-stereo,” which can be confused with a car sound system. Binocular. Two eyes. The term “binocular stereopsis” (two-eyed solid seeing) is used in some psychology books for the depth sense more simply described as stereopsis. Conversion. Also known as synthesis or by the trade name Dimensionalization. A process by which a planar image is turned into a stereoscopic image. Convergence. The inward rotation of the eyes, in the horizontal direction, producing fusion. The more general term is vergence, which includes inward and outward rotation. The term has also been used, confusingly, to describe the movement of left and right image fields or the rotation (toe-in) of camera heads. Corresponding points. The image points of the left and right fields referring to the same point on the object. The distance between the corresponding points on the projection screen is defined as parallax. Also known as conjugate or homologous points. 20 http://lennylipton.wordpress.com/2009/03/16/glossary 40 Crosstalk. Incomplete isolation of the left and right image channels so that one leaks (leakage) or bleeds into the other. Looks like a double exposure. Crosstalk is a physical entity and can be objectively measured, whereas ghosting is a subjective term. Depth range. A term that applies to stereoscopic images created with cameras. The limits are defined as the range of distances in camera space from the background point, producing maximum acceptable positive parallax, to the foreground point, producing maximum acceptable negative parallax. See parallax budget. Disparity. The distance between conjugate points on overlaid retinae, sometimes called retinal disparity. The corresponding term for the display screen is parallax. Floating windows. Invented by Raymond and Nigel Spottiswoode, this is the use of printed vertical bands to create a surround to supplant the physical screen surround. The result is a so-called virtual window that is floating in space, to eliminate the screen edge cue conflicts and to extend the parallax budget of the projected image. Ghosting. The perception of crosstalk is called ghosting. HIT. Horizontal image translation. The horizontal shifting of the two image fields to change the value of the parallax of corresponding points. The term “convergence” has been confusingly used to denote this concept. Interaxial distance. Also interaxial separation. The distance between camera lenses’ axes. See t. Interocular distance. See t. 41 Interpupillary distance. Also interpupillary or interocular separation. The distance between the eyes’ axes. See t. Negative Parallax. See Theater space. Parallax. The distance between conjugate points. It may be measured with a ruler or, given the distance of an observer from the screen, in terms of angular measure. In the latter case the parallax angle directly provides information about disparity. Parallax Budget. The range of parallax values, from maximum negative to maximum positive, that is within an acceptable range for comfortable viewing. Planar. Flat. Two-dimensional. A planar image is one contained in a two- dimensional space, but not necessarily one which appears flat. It may have all the depth cues except stereopsis. Plano-Stereoscopic. A stereoscopic projected image that is made up of two planar images. Positive Parallax. See Screen space. Screen space. The region appearing to be within the screen or behind the surface of the screen. Images with positive parallax will appear to be in screen space. The boundary between screen and theater space is the plane of the screen and has zero parallax. See Theater space. Stereo. Short for stereoscopic. If you are trying to learn about multi-channel sound, you are in the wrong place. 42 Stereoplexing. Stereoscopic multiplexing. A means to incorporate information for the left and right perspective views into a single information channel without expansion of the bandwidth. Stereopsis. The binocular depth sense – literally, “solid seeing.” Stereoscope. A device for viewing plano-stereoscopic images. It is usually an optical device with twin viewing systems. Stereoscopy. The art and science of creating images with the depth sense stereopsis. Surround. The vertical and horizontal edges immediately adjacent to the screen. t. In stereoscopy, t is used to denote the distance between the eyes, called the interpupillary or interocular distance. tc is used to denote the distance between stereoscopic camera heads’ lens axes and is called the interaxial. Theater space. The region appearing to be in front of the screen or out into the audience. Can also be called audience space. Images with negative parallax will appear to be in theater space. The boundary between screen and theater space is the plane of the screen and has zero parallax. See Screen space. Window. The stereo window corresponds to the screen surround unless floating windows are used. ZPS. Zero parallax setting, or the means used to control screen parallax to place an object in the plane of the screen. ZPS may be controlled by HIT, or toe-in. We can refer to the plane of zero parallax, or the point of zero parallax (PZP) so achieved. Prior terminology says that left and right images are “converged” when in the plane of the screen. That term 43 should be avoided because it may be confused with the convergence of the eyes, and because the word implies rotation of camera heads. Such rotation produces geometric distortion and may be expedient in camera rigs, but is unforgivable in a CG virtual camera rig. 44 Bibliography Alexander, Leigh. “Forget ‘immersion’ -- Player Attention Is What Matters, Says Lemarchand.” Gamasutra. 7 Mar. 2012. Web. 27 Mar. 2012. <http://www.gamasutra.com/view/news/164914/gdc_2012_forget_immersion__.php>. Bettelheim, Bruno. “Jack and the Beanstalk.” The Uses of Enchantment: The Meaning and Importance of Fairy Tales. New York: Knopf, 1976. 183-93. Print. Block, Bruce A. The Visual Story: Creating the Visual Structure of Film, TV and Digital Media. Amsterdam: Focal/Elsevier, 2008. Print. DeSouza, Clyde. “‘This Isn’t Your Grandfather’s 3D’. A Response to the Murch / Ebert Critique.” RealVision.ae Knowledge Base and Stereoscopic 3D Blog. Real Vision FZ LLC, 25 Jan. 2011. Web. 20 Mar. 2012. <http://realvision.ae/blog/2011/01/this-isnt-your- grandfathers-3d-a-response-to-the-murch-ebert-critique/>. Ebert, Roger. “Why 3D Doesn’t Work and Never Will. Case Closed.” Web log post. Roger Ebert’s Journal. Chicago Sun-Times, 23 Jan. 2011. Web. 20 Mar. 2012. <http://blogs.suntimes.com/ebert/2011/01/post_4.html>. Ebert, Roger. “Why I Hate 3-D (And You Should Too).” Editorial. Newsweek 9 May 2010. The Daily Beast. Newsweek/Daily Beast, 09 May 2010. Web. 20 Mar. 2012. <http://www.thedailybeast.com/newsweek/2010/04/30/why-i-hate-3-d-and-you-should- too.html>. Fullerton, Tracy, Christopher Swain, and Steven Hoffman. Game Design Workshop: A Playcentric Approach to Creating Innovative Games. Amsterdam: Elsevier Morgan Kaufmann, 2008. Print. Hart, Hugh. “Review: Powerful Avatar Stuns the Eye, Seduces the Heart.” Wired.com. Conde Nast Digital, 17 Dec. 2009. Web. 27 Feb. 2012. <http://www.wired.com/underwire/2009/12/avatar-review/>. “Jaws: Did You Know?” IMDb. IMDb.com. Web. 03 Mar. 2012. <http://www.imdb.com/title/tt0073195/trivia?tab=tr&item=tr0776899>. “John Lasseter, Director and the Chief Creative Officer at Pixar and Walt Disney Animation Studios.” Interview by Charlie Rose. http://www.charlierose.com. 2 Dec. 2011. Web. 15 Mar. 2012. <http://www.charlierose.com/view/interview/12024>. Lee, Johnny C. “Johnny Chung Lee: Projects.” Johnny Chung Lee. Web. 21 Mar. 2012. <http://johnnylee.net/projects/>. 45 Lipton, Lenny. “Glossary.” Lenny Lipton. 16 Mar. 2009. Web. 02 Jan. 2012. <http://lennylipton.wordpress.com/2009/03/16/glossary>. Mendiburu, Bernard. 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Amsterdam: Focal/Elsevier, 2009. Print. “MTBS GameGrade 3D.” Www.mtbs3d.com. Meant to Be Seen. Web. 20 Mar. 2012. <http://www.mtbs3d.com/gg3d/>. Quittner, Josh. “Are 3-D Movies Ready for Their Closeup?” Time 19 Mar. 2009. Web. 21 Feb. 2012. <http://www.time.com/time/magazine/article/0,9171,1886541,00.html>. Schneider, Neil. “Breaking the Stereoscopic 3D Paradox.” http://www.siggraph.org/. Association for Computing Machinery’s Special Interest Group on Computer Graphics and Interactive Techniques, Aug. 2008. Web. 15 Feb. 2012. <http://www.siggraph.org/publications/newsletter/volume-42-number-3/breaking-the- stereoscopic-3d-paradox>. Scorsese, Martin. “The Wind-up Magic of Martin Scorsese’s “Hugo”.” Interview by Larry Mantle. Air Talk. National Public Radio. KPCC, Pasadena, CA, 8 Feb. 2012. Radio. “Top 3-D Myths.” 3dfilmpf.org. 3-D Film Preservation Fund. Web. 22 Feb. 2012. <http://www.3dfilmpf.org/info-top-10-3D-myths.html>. “Understanding Stereoscopic 3D Gaming: What Is Stereoscopic 3D Gaming?” The S-3D Gaming Alliance. Web. 19 Mar. 2012. <http://www.s3dga.com/what-is-s3d.html>. V-Frog. Computer software. http://www.tactustech.com/vfrog/. Vers. 2.0. Tactus Technologies. Web. 28 Feb. 2012. Wenders, Wim. “Wim Wenders’s “Pina”.” Interview by Larry Mantle. Air Talk. National Public Radio. KPCC, Pasadena, CA, 14 Feb. 2012. Radio. Zone, Ray. Stereoscopic Cinema & The Origins of 3-D Film, 1838-1952. Lexington, KY: University of Kentucky, 2007. Print. 46 Appendix The Tale of Seymour Deeply: Complete Comic Book script The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.1 The Tale of Seymour Deeply One-Shot Comic Issue By Michael Annetta Draft 2.2 - 03/01/2012 © 2012 Michael Annetta All Rights Reserved The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 1 - INSIDE FRONT COVER (In a text box covering the top half of the page.) The denizens of Abovia had long ago severed any and all relationship with the people of The Belowlands (who they called ‘Groundlings’ if they were respectful and simply, Belowlanders if they were not). Years ago there was an incident that connected their two worlds. A short, shocking incident that ended tragically for two families. Since that time, the people of The Belowlands have looked up with suspicion and trepidation. The residents of Abovia have enacted strict statutes to make sure that nothing like The Day the Ground Invaded could ever happen again. And yet . . . And yet, these things tend to have a way of repeating themselves. (The title and Credits will cover the bottom half of the page.) THE TALE OF SEYMOUR DEEPLY (Credits) Writer/Layouts: Michael Annetta Artist: Miss Tak Letterer: Michael Annetta Editor: TBD The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 2 - PAGE 1 PANEL 1 [If a separate title and credit page is not possible, this panel contains both the title of the book and a small credits box]. Jack’s house is shown from a distance, at sundown. The house is less-than-modest, “quaint” would be a generous description. A small window can be seen toward the back of the house and, possibly, silhouettes of Jack and his mother, lit from behind by perhaps a single candle. The silhouette of a large bird floats by. NARRATOR For Jack’s mother, it seemed this was the last straw. PANEL 2 Mid shot of Jack’s mother as she’s screaming at Jack, clearly upset, while Jack hangs his head, not so much in shame, but more in resignation. This is hardly the first time that he’s been in this scene and it certainly won’t be the last. NARRATOR Beans were a poor exchange for any animal, she bellowed, let alone one as large as a cow. And she was a good cow. The bovine, not Jack’s mother. PANEL 3 Close up of Jack from Mom’s POV. Jack is pleading with his mother that the beans really are magic and that he believes they’re the true path of escape out of their lives. He’s quite attractive. Youthful, but rugged. He’s had to grow up and represent the “man of the house” even though he’s a little uncomfortable with that role. NARRATOR Jack struggled to convince her of the true value of the deal, that these beans held magical powers. Under her breath, she cursed at herself for giving birth to such a handsome, yet guileless, young man. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 3 - PANEL 4 Mid-shot from another angle of Jack and his mother. Mom defiantly has her hand on her hip while she is holding out her other hand, with the beans in it, almost directly in Jack’s face. It’s the same kind of action that a dog owner does when she pushes the dog’s face into it’s own shit so it knows it’s done something wrong. NARRATOR Deep down, Jack knew arguing was fruitless. Even when she was wrong, his mother was right. She was always right. PANEL 5 Exterior of the back of the house. Jack’s mother’s hand is seen tossing the beans out the window. The beans, about five of them, approach the viewer in perspective, as they are coming at us and breaking the very barriers of the page. One of them looks like it might be tossed onto the next page. NARRATOR Still he was a little shocked by the force of her anger as she flung the beans out the back window to be swallowed by the night. PAGE 2 PANEL 1 And the forwardmost bean from the previous page indeed is landing here, at the bottom of the page in the bottom of Panel 1. From there, the beanstalk grows toward us, in perspective, so that the top of the page is the top of the clouds and where the beanstalk is largest and closest to us. Towards the bottom of the stalk, close to the ground, we see Jack slowly making his way up. We also see the bird from page 1, flying up and around the beanstalk. We’ll learn later that this is Colin. NARRATOR However, as these things tend to happen, overnight the beans proved their magic and grew together into a The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 4 - giant beanstalk which stretched to the sky. These stalks often prove irresistible for curious youths eager to get away from home in search of adventure. PANEL 2 The cloud “ground layer” and the Deeply Farms estate. It should look less like a fairy tale castle and more like an upper middle class McMansion. Think any overgrown home in Glendale, California. There are no trees, but there are several beanstalks which grow both up from the clouds and down from higher strata clouds, like a forest of stalagmites and stalactites. There is a back door to the house and several windows. One of which is a bedroom window with a silhouette (Seymour’s) in it and it looks like it’s using binoculars to look up in the sky. Peeking up through the clouds is the top of the beanstalk. NARRATOR Jack had heard the old fables, but never fully believed that in the clouds was a land whose inhabitants called “Abovia”. PANEL 3 Mid-Shot of Seymour through his bedroom window. It’s the first time we see him. He’s dark, tall, handsome but boyish. There’s a light in his eyes that counters the darkness of his complexion. He’s holding the binoculars (which look much more complex and homemade than regular binoculars) down around his chest, having just brought them down from his face. He’s looking out over the back yard of his estate and sees Jack. NARRATOR Jack moved cautiously up the stalk now. Things here were larger than he was. He didn’t quite know if he’d encounter foe or friend. PANEL 4 Close up of Seymour. He’s smiling. Hell, he’s smitten. He’s looking down and to the left, so his eye line seems to match Jack’s in Panel 5 The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 5 - NARRATOR Seymour barely noticed the stalk popping out of the clouds. What caught his eye was the athletic stranger climbing up it. PANEL 5 Close up of Jack. He’s also smiling and seems a bit relieved. He doesn’t feel threatened, he feels welcome. Jack’s looking up and to the right, so his eyeliner seems to match Seymour’s in Panel 4 NARRATOR Jack stopped short at the top of the stalk and looked up, transfixed at the oversized house. When the tall, handsome youth smiled at him from the window, though, his concerns dropped like stones back down to his home below. PAGE 3 PANEL 1 Seymour is kneeling on one knee next to Jack to talk and it’s the best way for them to be eye-to-eye. Geese are walking around in the grass in foreground of the panel. They’re . . . Golden. Colin, the bird, flies overhead, observing. NARRATOR The introductions were intense, but brief. Seymour had never seen a Groundling before and Jack had thought the tales of the land of Abovia were fiction. Their mutual curiosity made them fast friends. PANEL 2 Stacked on top of Panel 3, this is really the top half of one picture, divided by the gutter. This half shows Seymour’s head and shoulders. His arm is wrapped around Jack’s shoulders as he leads him inside the estate. The two of them are clearly in an animated conversation. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 6 - NARRATOR Seymour didn’t want to incur the wrath of his father or his mother (especially his mother) for associating with a Belowlander, so he invited Jack inside, where they could continue to talk in private. PANEL 3 Stacked underneath Panel 2, this is the bottom half of one picture, divided by the gutter. This half shows Jack’s back. Seymour’s arm is wrapped around Jack’s shoulders. Jack’s face is glancing backwards as he talks to Seymour (to see where he came from), but he’s happy he’s moving forward. NARRATOR Jack felt no fear. In fact, he was . . . comfortable. For the first time in his life, he felt like he belonged in the right place. PANEL 4 A pair of over-the-shoulder shots of both Jack and Seymour as they’re lying on Seymour’s bed, talking. In the upper panel, we see Jack sitting cross-legged on the bed, looking at Seymour (we see Seymour’s head and shoulders in the lower right corner of the shot). Seymour is in the lower panel, lying down with his head in his hand, looking at Jack (we see Jack’s head and shoulders in the lower left corner of the shot). They’re smiling, talking, laughing, but their eyes are deep in the other’s eyes. This is the BEST. FIRST. DATE. EVER. It’s clear they’re both smitten. NARRATOR They talked for what seemed like hours. Jack was in awe of all of the luxuries and riches of this land above. PANEL 5 As above. Seymour’s Panel. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 7 - NARRATOR Seymour was curious about life below and beyond. PANEL 6 Wider shot of Seymour’s bedroom as they both face out the window. It’s ornate, but fun. A mobile of a solar system (but maybe not ours) is hanging in the corner. On a shelf are several pairs of binoculars and telescopes, all seemingly handcrafted. Seymour is clearly a scientist and explorer. Pointing out the window, Seymour is showing Jack how to use a pair of homemade binoculars to see the stars. NARRATOR Seymour confessed of his dream to be an astronomer and touch the stars. And his father’s insistence that he follow the family line and become yet another gold farmer. (Then in a caption below) NARRATOR Jack mentioned that he and his mother were simply too poor to dream anymore. They just had to get by and Jack couldn’t think of his own desires until his mother was taken care of. PAGE 4 PANEL 1 Seymour and Jack still at the window. We can see Seymour’s face, though, and see his concern for Jack. In the background, through the window, we can see all the golden geese flocking on the estate grounds. NARRATOR Seymour’s heart sank at the thought of this. He took Jack out to back of the estate where the geese roamed freely. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 8 - PANEL 2 Close up on a young, golden gosling. (We’re outdoors now and a little time has passed). PANEL 3 Seymour hands Jack a gosling small enough for him to hold. SEYMOUR This will supply your mother with enough gold for two lifetimes. PANEL 4 Seymour and Jack smile at each other. JACK Thank you. PANEL 5 Close up on their hands. They gently come together. PANEL 6 In a deep one-point perspective shot, from the archway that leads from the estate, the sound of Stanley Deeply, Seymour’s father, can be heard echoing, echoing, echoing. We see Stanley in silhouette, partially obscured by shadow. There’s a sense that he might be wearing an eye patch. The sound of his footsteps is loud, threatening, menacing. PANEL 7 Seymour’s head whips around at the sound. He’s alarmed. SEYMOUR My father! You need to take her and flee. (Continuing) PANEL 8 The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 9 - Seymour turns to Jack. His concern for Jack is palpable. Jack is scared shitless. SEYMOUR Go! Now! PAGE 5 IN THE GUTTER (beanstalk drawn running down the length of the page) Jack is seen escaping down the stalk (with the gosling). Each horizontal row of panels shows Jack’s progress down the stalk. We can see Colin, the bird, following Jack at a distance to keep an eye on him. PANEL 1 Stanley, Seymour’s father, confronts him. Seymour’s head hangs, not in guilt, but in supplication. We can see Stanley much better now and he clearly has an eye patch over his left eye, but it’s a classy eye patch, from a surgery. He doesn’t look like a pirate. STANLEY What have you done?! SEYMOUR What do you mean? PANEL 2 Seymour is responding to his father, but not looking at him. He’s trying to look down the stalk to see how far Jack’s gotten. SEYMOUR I came outside to see where I could set up a telescope for tonight. PANEL 3 The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 10 - Stanley is suspicious of where Seymour is looking. He is a man who runs his household like his own little dictatorship. STANLEY Don’t be cute. I smelled one. A Belowlander. Here. On our grounds! Our family estate! PANEL 4 STANLEY What would the Council do to us if they found out you were harboring one of them? You know the law forbids contact with those Below. What would your mother do? (then) STANLEY Did you lure a Belowlander here with your . . your eye gadgets? PANEL 5 SEYMOUR What? No! I ... I ... they’re not “gadgets,” they’re scientific instruments! I want to travel to the observatory to study the stars. STANLEY Seymour, we’ve been over this. We are a long line of gold farmers; you shall be a gold farmer and protect our land from vermin like that. PANEL 6 Full-frontal close up. Stanley is scary and intimidating. You see that he’s not the father of this piece. He’s the villain. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 11 - STANLEY Now get inside and wash. You reek of him. PAGE 6 PANEL 1 Jack’s mother in the kitchen observing the gosling. NARRATOR Jack’s reunion fared not much better . . . JACK’S MOTHER A single golden goose? Jack, you need to go back. You need to get more, and quickly, before the Villagers find out another stalk has grown. PANEL 2 Jack sits in a kitchen chair. Exhausted. Wistful. Jack’s mother has no time to waste (or, it seems, much concern for her son’s health or well-being) JACK’S MOTHER We have to act fast! There are rules against traveling to the Above. The authorities may be here soon. Here. Take your father’s blade. You may need it to fight those monsters! She’s thrusting it into his hands. It’s a simple short sword. To someone Seymour’s size, it might seem only like a dagger. PANEL 3 Jack’s mother practically pushes him out the back door and up the beanstalk, which he begins to climb. JACK Mother, I told you, they’re not “monsters.” They’re beautiful. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 12 - JACK’S MOTHER Enough. Go! Hurry back! But grab as much as you can carry! PANEL 4 No image, just a thin text box with perhaps a simplified version of Colin’s head to indicate that he has been the narrator all along. NARRATOR(COLIN) I flew ahead to inform Seymour of Jack’s return. PANEL 5 It’s still night time in Abovia when Jack pokes his head out of the “cloud ground” and there, crouching down to greet him, is Seymour. Seymour has one of his pair of binoculars hanging around his neck. PANEL 6 Seymour extends his hand to Jack. PANEL 7 Seymour pulls Jack into him. They embrace. SEYMOUR (whispers) Nice sword. JACK (whispers back) I missed you too. PAGE 7 PANEL 1 Banner panel across the top of the page. Seymour cups Jack’s head in his massive hand and they kiss. PANEL 2 The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 13 - Jack pulls away, slightly embarrassed. JACK I didn’t know you knew . . . PANEL 3 SEYMOUR I know. And I’m glad. PANEL 4 Jack smiles. PANEL 5 They start toward the Estate. Jack is understandably exhausted. JACK My mother sent me back for more geese. She’s never had anything and now she says she needs everything. SEYMOUR So we’ll bring her some. I want to come with you. I can’t be myself here. PANEL 6 They stop and they kiss again (reverse shot of the one above). PANEL 7 Behind them, through a glowing window on a balcony of the estate, Seymour’s mother, Shirley, sees both young men. She tosses open the curtains (intentionally dramatic, Shirley doesn’t do things lightly) SHIRLEY (gasps) Seymour! The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 14 - PAGE 8 PANEL 1 Seymour’s face turns serious and scared. He hands Jack his binoculars. Jack hands him back the short sword (which in Seymour’s hands looks like a dagger). Behind them, Stanley is rushing from the house. SEYMOUR Go! I’ll find a way to follow. I promise. JACK (Handing him the sword-as-dagger) Protect yourself. PANEL 2 Stanley verbally confronts Seymour. STANLEY What have you done?! We will never be forgiven. Never! PANEL 3 Ignoring Seymour, Stanley races past him and follows Jack down the stalk. STANLEY I’ll get him! PANEL 4 Jack’s POV from about halfway down the stalk. He is clutching the binoculars tightly. Villagers below (looking like dots) are gathering. They carry saws, pitchforks and torches and what looks like a catapult. Some are already sawing at the trunk PANEL 5 Stanley, furious, rushes down the stalk after Jack and, since he’s larger, he’s gaining on him. The Tale of Seymour Deeply Author: Michael Annetta Draft No. 2.2 - 15 - PANEL 6 Jack is almost at the ground when he looks up with the binoculars and sees Seymour reaching down through the skies. PANEL 7 Jack pauses and is about to go back up, when the stalk shudders and starts snapping. PANEL 8 The panel is dark, with a gradient black vignette feel, as if we were fading to black. The silhouette of the stalk coming down is seen and we just hear a loud, devastating “THUD!” INSIDE BACK COVER Simple picture of Colin. COLIN Seymour’s tale doesn’t end here. Follow me, Colin, to the interactive portion of our story where you will help guide Seymour’s destiny. (In a text box covering the lower half of the page.) The interactive immersive story of Seymour Deeply will be publicly presented at the University of Southern California’s School of Cinematic Arts as part of Other Worlds, a showcase of interactive thesis projects. The showcase is open to the public from May 5 through May 11, 2012. For more information, visit interactive.usc.edu/thesis2012.

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Nahui Ollin, “in search of the divine force”: a phyical immersive interactive dance experience

PDF

Shayd: the pursuit of magic, illusion, and interactive worlds

PDF

Morana: explore healing potential of virtual reality storytelling

PDF

Grayline: Creating shared narrative experience through interactive storytelling

PDF

Untitled: an exploration of virtual reality as a cinematic medium

PDF

N|TONE: an interactive exploration of a poetic space through voice

PDF

Combiform: a console for the new communal casual game genre

PDF

One little change: exploring relationship abuse in a virtual reality narrative

PDF

The return: a case study in narrative interaction design

PDF

Inclusive character creator: an exploration of inclusive design for 3D character creators

PDF

American braves - total aesthetics: an opinion piece on game design for academics

PDF

The moonlighters: a narrative listening approach to videogame storytelling

PDF

Virtual production for virtual reality: possibilities for immersive storytelling

PDF

The voice in the garden: an experiment in combining narrative and voice input for interaction design

PDF

Everything all the time: anthological storytelling

PDF

Type Set : Exploring the effects of making kinetic typography interactive

PDF

Somatic montage: supra-dimensional composition in cinema and the arts

PDF

Ascension: a look into context-dependent memory development in virtual worlds

PDF

Silent sparrows: exploration into augmented reality narrative frameworks

PDF

Timension

Asset Metadata

Creator Annetta, Michael (author)

Core Title Seymour Deeply: exploring stereoscopic 3D as a storytelling tool in interactive media

School School of Cinematic Arts

Degree Master of Fine Arts

Degree Program Interactive Media

Publication Date 05/01/2012

Defense Date 03/26/2012

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag 3D,COTS,Games,gestural interface,Interactive Media,kinect,LGBT,narrative,OAI-PMH Harvest,stereoscopic,stereoskeptic,Transmedia

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Hoberman, Perry (committee chair), Fisher, Scott S. (committee member), Garlington, Joseph O. (committee member)

Creator Email annetta@usc.edu,mannetta@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-19862

Unique identifier UC11289216

Identifier usctheses-c3-19862 (legacy record id)

Legacy Identifier etd-AnnettaMic-695.pdf

Dmrecord 19862

Document Type Thesis

Rights Annetta, Michael

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA