University of Southern California Dissertations and Theses

Scalable peer-to-peer streaming for interactive applications

(USC Thesis Other)

Scalable peer-to-peer streaming for interactive applications

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content SCALABLE PEER-TO-PEER STREAMING FOR INTERACTIVE APPLICATIONS by Leslie Shihua Liu A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Ful¯llment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) August 2007 Copyright 2007 Leslie Shihua Liu Dedication This dissertation is dedicated to my wonderful parents, Hanqi Liu and Qingming Chen, whose unconditional support and endless love will always be remembered. ii Table of Contents Dedication ii List of Tables vi List of Figures vii Abstract ix Chapter One: Introduction 1 1.1 Motivation Of Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Highlight Of Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Contributions Of Reported Research . . . . . . . . . . . . . . . . . . . . . 7 1.4 Outline Of The Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Chapter Two: Background And Related Work 9 2.1 Internet Telephony(VoIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 P2P Protocols. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 P2P Implementations And Standards. . . . . . . . . . . . . . . . . . . . . 11 2.4 Audio Streaming For Networked Virtual Environment . . . . . . . . . . . 13 Chapter Three: ACTIVE Streaming Protocol 17 3.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.1 Statement Of The Problem To Solve . . . . . . . . . . . . . . . . . 19 3.1.2 Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.1.3 Introduction Of Credit Point System . . . . . . . . . . . . . . . . . 22 3.1.4 Statement Of Solution . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2 Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.1 Credit Point System . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3 Platform Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.1 Join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.2 Leave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.3 Error Detection And Recovery . . . . . . . . . . . . . . . . . . . . 31 3.4 User Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.1 User Classi¯cation . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.2 Floor Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 iii 3.5 Optimization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.6 Streaming Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.7 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.7.1 Control Overhead Analysis . . . . . . . . . . . . . . . . . . . . . . 39 3.7.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.7.3 Simulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.7.4 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.7.5 Experimental Scenario . . . . . . . . . . . . . . . . . . . . . . . . . 46 Chapter Four: DSM Audio Streaming Protocol 49 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.2.1 Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.2 Virtual World Model . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.2.3 Audio Streaming Scenarios . . . . . . . . . . . . . . . . . . . . . . 62 4.2.4 Problem De¯nition . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.2.5 Heuristic Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.3 The Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3.1 Mesh Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.3.1.1 Join Mesh : . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3.1.2 Mesh Maintenance : . . . . . . . . . . . . . . . . . . . . . 70 4.3.1.3 Leave Mesh . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.2 Backbone Tree Protocol . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.2.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.2.2 Backbone Tree Maintenance . . . . . . . . . . . . . . . . 73 4.3.3 Audio Stream Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.3.3.1 Stream Setup . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.3.3.2 Stream Termination . . . . . . . . . . . . . . . . . . . . . 79 4.3.4 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4.1 Simulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.4.3 Performance In Stream Tree. . . . . . . . . . . . . . . . . . . . . . 87 4.4.4 Performance In Backbone Tree . . . . . . . . . . . . . . . . . . . . 93 4.4.5 Bandwidth Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.5 Example Application Using DSM . . . . . . . . . . . . . . . . . . . . . . . 95 Chapter Five: Applications 98 5.1 AudioPeer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.1.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.1.1.1 Website . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.1.1.2 Client Software . . . . . . . . . . . . . . . . . . . . . . . . 101 5.1.1.3 Rendezvous Point Server . . . . . . . . . . . . . . . . . . 103 5.1.1.4 Administration Control Interface . . . . . . . . . . . . . . 104 5.1.2 Audio Mixing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 iv 5.1.3 Implementation Of ACTIVE Protocol . . . . . . . . . . . . . . . . 106 5.2 MStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.2.1 Application Overview . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.2.2 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3 PartyPeer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.3.1 Game Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.3.2.1 Dynamic zoning . . . . . . . . . . . . . . . . . . . . . . . 116 5.3.2.2 Visibility prediction . . . . . . . . . . . . . . . . . . . . . 116 5.3.3 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.3.4 P2P Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.3.5 Immersive Sound Design . . . . . . . . . . . . . . . . . . . . . . . . 119 5.4 LiveRH: A Tele-rehabilitation System . . . . . . . . . . . . . . . . . . . . 120 Chapter Six: Conclusion 124 6.1 Summary Of The Dissertation. . . . . . . . . . . . . . . . . . . . . . . . . 124 6.2 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.2.1 Video Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.2.2 P2P Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 References 128 v List of Tables 1.1 ACTIVE vs. DSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1 List of terms used in this paper and their respective de¯nitions. . . . . . . 20 4.1 List of terms used in this paper and their respective de¯nitions. . . . . . . 58 4.2 Audio Streaming Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.3 Simulation Parameters List . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1 Audio types supported. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.2 Statistic of User Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.3 User Feedback of AudioPeer . . . . . . . . . . . . . . . . . . . . . . . . . . 109 vi List of Figures 3.1 Example: MST vs. ACTIVE . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2 Processing Delay vs. Total Delay . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Application-layer Processing Delay . . . . . . . . . . . . . . . . . . . . . . 28 3.4 Automatic Mode Switch Using R . . . . . . . . . . . . . . . . . . . . . . . 34 3.5 Example: D A and D V vs. P . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.6 Network Topology (400 Users) . . . . . . . . . . . . . . . . . . . . . . . . 44 3.7 Delay performance of ACTIVE for active users . . . . . . . . . . . . . . . 45 3.8 Delay performance of ACTIVE for all users . . . . . . . . . . . . . . . . . 45 3.9 Example: Large Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.10 Example: Medium Group . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.11 Example: Small Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.1 Mapping of Virtual Game Space to Overlay Peer-to-Peer Architecture . . 51 4.2 Sample MST Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3 A corner of the Simulated Internet with 6000 routers . . . . . . . . . . . . 82 4.4 Distribution of RDP spt for streaming group size = 6 . . . . . . . . . . . . 84 4.5 Statistic of RDP spt distribution for streaming group size = 6 and 20 . . . 88 4.6 One simulation iteration (250 users) . . . . . . . . . . . . . . . . . . . . . 90 vii 4.7 RDP MST distribution for di®erent group sizes . . . . . . . . . . . . . . . . 92 4.8 Average RDP MST for di®erent group sizes . . . . . . . . . . . . . . . . . 94 4.9 DSM vs. StaticMesh in RDP MST . . . . . . . . . . . . . . . . . . . . . . . 95 4.10 Distribution of RDP MST in DSM for di®erent group sizes . . . . . . . . . 96 4.11 RDP spt distribution for di®erent degree limit (k=4,8,12) . . . . . . . . . . 97 5.1 AudioPeer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.2 AudioPeer O±cial Website . . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.3 AudioPeer Client Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.4 Administration Control Interface(ACI) . . . . . . . . . . . . . . . . . . . . 105 5.5 AudioPeer state diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.6 MStream System Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 109 5.7 MStream Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.8 Screen shot of PartyPeer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.9 Game Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.10 Visibility prediction for maximum speed = 5m/s . . . . . . . . . . . . . . 115 5.11 P2P streaming using introducer server and relay server . . . . . . . . . . . 117 5.12 Input Device: Phantom 6DOF . . . . . . . . . . . . . . . . . . . . . . . . 121 5.13 Screen shot from one excercise . . . . . . . . . . . . . . . . . . . . . . . . 121 5.14 LiveRH Clinic Trial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 viii Abstract Peer-to-peer (P2P) streaming is emerging as a viable communications paradigm. Re- cent research has focused on building e±cient and optimal overlay streaming platforms at the application level. Peer-to-Peer systems, either fully distributed or hybrid, are well known for features such as high scalability and low cost to deploy. Popular applications such as Skype and Bittorrent can serve tens of thousands of concurrent users with only a few centralized servers. However, it is still a challenge job to design a P2P protocol to support large interactive streaming group where both low latency and high scalability are vital. We identi¯ed two important issues that could lead to a successful design for interactive streaming applications. First, many existing designs that aim to build a low- latency streaming platform often make the unreasonable assumption that the processing time involved at each node is zero. However in a real implementation, these delays can add up to a signi¯cant amount of time after just a few overlay hops and make interactive applications di±cult. Second, scant attention has been paid to the fact that even the number of participants in a group could be large, each user may only actively interact with a subset of them at any given time. We term this sub-group of users who are ac- tively interacting with each other inside this sub-group, an active group. Depending on the streaming scenarios, there can be one or more than one active groups in an appli- cation. Some applications have only one active group at any given time, for example, ix audio conferencing or class session for remote education. Some applications usually have more than one active groups, for example, a BBQ party hosted in a networked virtual environment where participants can wandering freely in the virtual space and chat with anybodytheymet. Inthisdissertation, wewillpresenttwopeer-to-peerbasedplatforms, namely ACTIVE 1 and DSM 2 , for low latency streaming. ACTIVE platform is designed to provide low latency streaming to applications with only one active group at any given time. DSM can cluster users into multiple active groups based on the proximity of their virtual coordinates. The performance of ACTIVE and DSM platform have been evaluated in simulations and application prototypes have been developed based on these designs. Those simula- tions and application prototypes are also described in this dissertation. 1 Adaptive Clustering Technology for Interactive Virtual Environments 2 Dynamic Streaming Mesh x Chapter 1 Introduction 1.1 Motivation Of Research The ever growing capacity of computing power, storage and network bandwidth of per- sonal computer (PC) has moved many tasks that before can only be performed by super computersoradvancedserverstoregularmachinesthatavailabletoendconsumers. This trend of shifting of processing capability has brought us a new era of distributed comput- ing. As a result, in the past few years, many distributed systems, especially peer-to-peer systems that utilize the resources from the participating members, have been designed and implemented to take the advantage of this new service model. Skype, which is a voice communication system using P2P architecture, is a good example of how powerful and scalable a P2P system could be. We predict that in the near future, more and more traditional service that relies on centralized service provider will be migrated to newly designed distributed architecture. 1 Furthermore, some pioneer engineers are, instead of re-deploy existing services to dis- tributed system, working on creation of new services that take advantages of the full capacities of the distributed systems. ManyofthenewstreamingapplicationsbuiltonP2Parchitecturerequireslowlatency amongusers, e.g., massivelymultiplayeronlinegames(MMOG),audioconferencing, bat- tle¯eld monitoring and emergency team cooperations etc. We term this type of system Interactive Application, where there is certain degree of interaction among users and low latency among them is usually a critical performance concern. Interactive streaming applications are, till today, mostly the sounding domain for centralized designs. One prominent reason, if not the most dominate one, is that centralized system usually pro- vides lower end-to-end delay among users if compared to their distributed counterparts. Some decentralized systems sacri¯ce the scalability for low latency. For example, iChat builds a full mesh network among all participants to provide low latency audio streaming in a decentralized fashion, but the number of participants is limited to twelve. InthelastfewyearsmanyP2Pprotocolshavebeendesignedandimplemented.(e.g.[25, 19, 58, 7, 8, 14, 40, 54, 2]). To the best of our knowledge, none of these protocols and systems are designed to address the performance issues of low-latency streaming for in- teractive multimedia applications in a large group. For example, Skype [2], the ¯rst and popular VoIP system using P2P architecture, even scale well to support millions of concurrent users online, divides users into many small groups. While small groups suit well in many scenarios such as telephony and small project discussion, it fall short 2 in functions for scenarios such as online education, MMOG, communication among sev- eral emergency response teams, large scale online conferencing, which usually have large number of participants. The lack of research and implementations to address the demands for low latency streaminginlargescaleinteractiveapplicationshasmotivateustoadesignaninnovative P2P streaming protocol that provides scalable interactive streaming service. 1.2 Highlight Of Research Our goal is to design an innovative scalable Peer-to-Peer streaming platform for inter- active virtual environments. The platform should be able to support massive number of concurrent users and at the same time, maintaining low latency streaming among those users. We further elaborate the performance requirements as follows: ² Scalability: Thenumberofusersinthesamestreaminggroupcangrowtoaslarge asthousands,andthesystemshouldscalewellintermsofnetworkbandwidthusage, control overhead and computing load at each node. ² Low-latency: The end-to-end delay among any of two users that are interacting with each other are below a certain performance threshold so that the interactivity of the application is maintained. We soon realized that in a peer-to-peer based design, scalability and low-latency are somehow two con°icting optimization goals. For example, a good way to lower the network delay between node i to all other nodes is constructing a direct connection from i to all other nodes. However, this approach is not scalable since the bandwidth cost at node i is proportional to the number of all other nodes online. In order to bring together these two requirements in a P2P-based architecture, additional information is needed. 3 Before getting into the details of di®erent streaming scenarios and the additional information needed for each of them, we will go over two issues that have not been seriously discussed before and to great extent shaped the way we design our algorithms: ² Overlay processing delay: Most existing overlay P2P solutions ignore the pro- cessing delay introduced at each intermediate node when they optimize the delay in those systems. However, in an overlay network the application-layer processing timeisusuallymuchlargerthantheprocessingtimeinthephysicalnetworkrouters. Since the processing delay is introduced at each intermediate node when propagat- ing data through the overlay path, these relay latencies can add up to a signi¯cant amountafterjustfewoverlayhops. Forexample, wemeasuredtheprocessingdelay at each overlay node (Fig. 3.3) in an application called AudioPeer [59] for distance online education and we found that the average processing delay at each node is approximately 30ms. Compared to the two to ten milliseconds physical network latency between the hosts we tested, this contributes a signi¯cant amount to the overall end-to-end delay. The average end-to-end delay in traditional P2P systems quickly exceeds the threshold for interactive scenarios when the group size grows. This explains why full-connected (i.e., mesh) designs dominate P2P systems which require low-latency performance. However, full-connected designs su®er from cer- tain limitations. Since everybody is connected to everybody else, the system is not scalable. A lot of network bandwidth is also wasted on transporting duplicate data. The mesh based streaming architecture may be capable of providing service to group sizes of tens, but not hundreds or thousands of participants. 4 ² Activegroups: EventhoughthetotalnumberofusersinP2Pgroupsisoftenlarge, wehavefollowingobservations: 1)Foreachuser,heorhercanonlyactivelyinteract with a small number of other users at any given time,thus form an Active Group; 2) Users in each active group are associated to others in the same group based on some social facts, for example, a common topic of conversation, location closeness or common interests. For example, during an online discussion, the number of speakers at any given time is usually less than four, otherwise the conversation will become incomprehensible. Although the speakers will shift role to listeners during the discussion, the total number of speakers that are actively talking to each other are usually much less than the total number of participants in a large discussion group. An other example is, even the total number of players in a MMOG games could be in the thousands, each player only actively interact with a limited number of other players within his/her sense of present in the virtual world. We call the area in the virtual world that is of interest to each player, the Area of Interests (AOI) of that player. For a player in a MMOG game, the active users of the game are the other players inside his/her AOI. We term the group of users that are actively interact with each other an Active Group. The number of active groups in a system is determined by the nature of the application. For example, there is usually only one active group for an audio discussion, but it is natural to ¯nd many active groups in a MMOG game. The key concept of our research is to identify the activegroupsandreducetheend-to-enddelayamongusersinsideeachactivegroup. Since the total number of users inside an active group is usually much less than the totally number of users in a large scale online system, we can take this opportunity 5 Table 1.1: ACTIVE vs. DSM Active Groups Topology Cluster Information ACTIVE One Tree User activity (active or pas- sive) DSM Multiple Mesh + Tree Virtual coordinates of users to optimize the P2P structure for better end-to-end delay among these relatively smaller groups. To the best of our knowledge, no existing P2P architecture has been designed to take advantage of this opportunity. We designed two algorithms for di®erent streaming scenarios based on the number of active groups and information available in the application. ACTIVE, our ¯rst protocol, is a tree-based topology designed to quickly cluster active users when there is only one active group at any time. DSM, which stands for Dynamic Streaming Mesh, is a mesh- based approach to cluster users in multiple active groups automatically depending on their virtual coordinates reading in the virtual environments. ACTIVE protocol focuses on applications with only one active group, e.g., an online audio chat room. The details are discussed in chapter 3. In Chapter 4, we discuss the DSM protocol which supports multiple active groups by dynamically clustering users by their virtual locations. We will show that DSM is an feasible platform for interactive applications with virtual environment such as MMOG. We call our location-adaptive approach Dynamic Streaming Mesh (DSM). ACTIVE is designed for large group of low latencystreamingwherethereisonlyonetopicamongallusersatanygivenmoment,also ACTIVE does not require virtual location as an input to optimize the performance, but need the activity indicator from the application to optimize the system. DSM clusters users based on their virtual locations in the networked virtual environment and build 6 many small streaming groups inside the whole overlay network. The DSM constantly update the mesh link to re°ect the changes of virtual location in the virtual world. More details of DSM can be found in Chapter. 4. We summarize the di®erences between these two platforms in Table 1.1. 1.3 Contributions Of Reported Research Astheresultof¯veyearsresearchanddevelopmente®orts,ourACTIVEandDSMproto- colsareproventobescalableP2Pbasedplatformsforlargescaleinteractiveapplications. In the following we highlight some of the major contributions of the conducted research: ² A scalable peer-to-peer based streaming protocol is designed and implemented to accommodate massive number of concurrent users. i The streaming platform is scalable to very large number of concurrent users at the same time, e.g., thousands of concurrent users. ii The distributed system runs in a peer-to-peer based protocol so the task of a centralized server is minimized. iii The protocol is robust enough to handle unexpected disconnection of peers from the network and automatically re-heal the streaming platform. iv A patent(SN: 11/504,536)is ¯led to the U.S. Trade and Patent O±ce on the innovative low-latency peer-to-peer streaming system. ² Added overlay processing delay into the performance model so that the proposed protocol can provide more realistic results. ² Designed the optimization algorithm so that the system can automatically recon- struct the overlay streaming network to reduce the delay among active users. 7 ² Formalized the problem and compared the performance to optimal solution by sim- ulation. ² Evaluated the design and conclude the feasibility of using P2P based architecture for interactive streaming applications. ² Developed several streaming applications based on ACTIVE and DSM protocol. Theseapplicationsincludeaudiochatroom,MMOGgame,tele-rehabiliationsystem for stroke patient and location based music streaming system. ² ConductedsurveystogaugeboththeusefulnessofACTIVEandDSMprotocoland the user feedback of experience using ACTIVE and DSM based applications. 1.4 Outline Of The Dissertation The remainder of this dissertation is organized as follows. In Chapter 2 we review the related work and some background information of this research. Chapter 3 describes the problem that ACTIVE protocol is designed to solve and then discusses the details of ACTIVE architecture, with some speci¯c details related to applications with only one active group. Chapter 4 extends ACTIVE protocol to applications with multiple active groups, such as MMOG, and emphasis on the challenges and the solutions for these type of applications. In Chapter 5 we present some applications that use ACTIVE and DSM protocol as the streaming platform. Finally, Chapter 6 concludes this dissertation. 8 Chapter 2 Background And Related Work 2.1 Internet Telephony(VoIP) Internet telephony, also known as VoIP, is probably the most popular type of interactive streaming applications. Researchers and engineers from distributed systems community are trying to standardize the P2P protocols for VoIP applications. For example in [47], authors detailed the components needed for using P2P architecture and presented an overall design that uses standard protocols such as SIP [49] and SDP [48] to construct an inter-connectable service architecture so that multiple service providers can link their services together and customers from di®erent service providers can share the network resources using the same standards. However Skype [2], the ¯rst successful P2P internet telephony service, uses totally proprietaryprotocolthatwon'ttalktootherVoIPservices. In [9], authorstriestoreveal the details of how Skype works by observation of network tra±c send and received by Skype clients. This analysis is not very easy and accurate since Skype protocol is closed and and the communication packets are encrypted. Based on the information collected 9 by this work, we still can get into some details of Skype protocol design, which leads us totheconclusionthatSkype, evenisveryscalable in terms oftotalnumber of concurrent users from many small sessions online, is not designed to support large communication group due to the complicated voice forwarding scheme and method of merging audio among a group. 2.2 P2P Protocols Many P2P architectures (e.g., Narada [25], Yoid [19], HMTP [58], NICE [7], OMNI [8], Scribe [14], CAN-Multicast [40], Zigzag [54] , Skype [2] and oStream [18]) have been proposed or adapted for streaming media services. However, due to the long end-to-end delay in overlay networks, these designs either make the impractical assumption that the processing delay on relay nodes is ignorable and hence failed to provide low latency service for real applications, or manage to provide low-latency streaming by using a full- connected architecture that is not very scalable. Narada [25] is a mesh-based approach for many-to-many streaming. It constructs a tree whenever a sender wants to transmit a media stream to receivers. Due to the heavy control overhead, Narada does not scale well to large P2P groups. NICE [7] is designedtosupportalargestreamingreceiversetanditsmulti-layereddesignreducesthe control overhead. A similar design is proposed in Banerjee et al. [8] , which additionally tries to optimize the overall end-to-end delay among all streaming receivers. Zigzag [54] optimizes the performance of NICE by constraining the control overhead to a constant value. In these three designs, all peers are considered to have the same delay requirement 10 and optimization is performed to reduce the delay among all peers, not among active users as in ACTIVE. The failure to distinguish between active and passive users makes it problematic for them to achieve low latency performance in an multi-hop architecture design. Scribe[14]andCAN-Multicast[40]arebasedonDistributedHashTables(DHT), which are used to generate node identi¯ers and then create a multicast tree. In these approaches, the multicast path between nodes is determined by the current topology of the multicast members, hence the delay between users cannot be reduced dynamically, as in ACTIVE, by optimizing the tree connections.oStream [18] is designed to utilize the strong bu®ering capacities on the multicast overlay nodes and thus reduce the network bandwidth requirement for on-demand media distribution. Due to the delay introduced by the bu®ering at intermediate nodes, oStream is not suitable for an interactive live streaming environment such as an audio chat room. 2.3 P2P Implementations And Standards ThefeasibilityofusingP2Parchitecturetosupportlarge-scalelivestreamingapplications with dynamic application end-points is addressed in [50], where the authors concluded that large scale multimedia streaming is viable because of the current network capacity and research of P2P to address the dynamics and e±ciency on the overlay streaming architectures. This results can be extended to interactive streaming applications since the resources required for an interactive streaming is comparable to a non-interactive scheme. Besides the traditional research focus where P2P systems build applications like ¯le sharing anddigital media delivery, in the past few years, manydesigns havealso been 11 proposed to use peer-to-peer topologies for massive multiuser online games (MMOG). For example, in [29] the authors proposed an application level protocol using Scribe [14] as the streaming platform. Also in [51] and [5] the authors described a system design that potentially can support up to millions of concurrent players. The design of these proposalsaremostlyfocusedonbuildingascalablestreamingsystem,scantattentionhas been made to interactive domain where delay is crucial. ThelagofdeploymentofIPv6andfastgrowingnumberofInternetusershasmadethe IPaddressesademandingresourceformanyinternetserviceprovider. ThusNATdevices, whichisusedtobridgetheconnectionofpublicInternettointernalprivatenetworkusing a shared public IP address, have become a popular solution for end consumers. One of theproblemscomingwiththeNATdevicesforP2Pnetworkishowtoreachaservicethat is running behind a NAT device which usually blocks the incoming tra±c or connection that is initialed from outside. NAT device solution works ¯ne for a client and server scenario where users behind NAT devices are usually just working as clients. But for P2Pservicetogrow, everyusermustalsoservesasserverto otherusers. Somestandards have been drafted to solve the NAT device problem for P2P networks. For example, STUN [44] is a protocol to tell what is the NAT and ¯reware connectivity for a machine and allows entities behind a NAT device to ¯rst discover the presence of a NAT and the type of NAT, and then to learn the addresses bindings allocated by the NAT. ICE [43] is a protocol to use P2P NAT traversal for SIP protocol. 12 2.4 Audio Streaming For Networked Virtual Environment Much research has been done to build a application level game infrastructure for MMOG ( [57, 33, 4, 56, 4, 21, 16, 11, 24, 23]). Some of them are server based design, e.g., [4] described a server cluster based archi- tecture for MMOG. The design is to separate the game world into many di®erent regions and then assign a server to each of the game zone. server failure and hot-spot issues are alsodiscussedinthepaper. Thefeaturesofthisdesignaredynamicregionassignmentin- stead of a ¯xed static division. A communication mechanism allowing players in di®erent server to interact with each other is also proposed. More designs are proposed using distributed architecture.In [24], authors proposed a design using Voronoi diagram to manage the nodes in the virtual environment. But because the complexity to calculate the Voronoi diagram on the °y, this approach is not feasible for large scale applications with NVE when users constantly move in the virtual world and the Voronoi diagram need to be constantly re-calculated.The followup paper of [23]adaptedthedesigntosupport3DobjectmodeldatausingtheVoronoi-basedP2P streaming network. Again it did not address the issue of how the system works when every user in constantly changing locations in the virtual world. Another P2P based design is described in [56], where the system divides the game space into ¯x-sized sub space and assigned a node called responsible node for each sub-space to be in charge of deliver the message and manage the sub-space. We claim that this design, even though might be suitable for low-bandwidth messaging type such as game control packets, is not suitableforbandwidthhungrystreamsuchasaudioandvideostreaminginMMOGsince 13 all tra±c message will go through the responsible nodes and can easily overload them. When the responsible node is overloaded, a balance tree is construct on the °y to share the load. However, it is not clear how this "just-in-time" type algorithm works in a real dynamic environment. The responsible node is chosen by a centralized server called the lobby server, which is a single point of failure. We also expect there will be a lot of tra±c goestothelobbyserverwhentheresponsenodesleavescurrentsub-spaceordisconnected from the game. Distributed Hash Table(DHT) has been a popular choice for P2P-based system and both research done in [21, 30] extend DHT to MMOG domain. [30] proposed a P2P based design of MMG which is based on the fact that players in a game exhibit temporal and spatial locality i.e. a player resides in a small portion of the game map and all that important to them is what happens in the surrounding region and not the entire map. Hence the game map can be broken up into separate regions, with each region being managed by a peer called coordinator.Players in a region form a multicast group where alltheinformationissentfromplayerstoacentralpeerinthegroup(thecoordinator). It is the responsibility of the coordinator to disseminate all the information to the peers in the group. Thus, the messaging follows a single writer multiple reader policy.In the P2P scenario, the central server still maintains information such as payment information and characterinformationofpeople. Thegamestate(objects,playersintheregion)arepassed onto peers through the coordinators. However, since this design is based on Pastry, a DHT based design, the delay among players is not dynamically optimized based on their virtual locations and as shown in our simulated result, unlike to outperform a design which does update the topology based on location. [21] is another design based on 14 Pastry (a DHT based design). Again the path of delivery doesn't dynamically optimize according to the movement of the avatar in the game. Hybrid approaches [57, 16, 11] gain more and more attention recently. An interesting designthatcombinesbothstructuredandun-structuredP2Ptopologyisproposedin [57] to support massively online interactive application. This approach is very promising to build the next generation pure P2P based MMOG game architecture. Unfortunately, the author focused their research on game message delivery, instead of interactive streaming asweareinterestedinthispaper,itneedsomemoreresearchandinvestigationtoevaluate the performance of streaming scenarios. Another hybrid design proposed in [16, 11] tries to put both distributed and centralized components in the same system. The author expect that by doing this, the proposed design can combine the technical advantage of both centralized and distributed design in one system. Again the location-adaptive scheme is not considered in this design but we expect that a modi¯cation of the existing system should be able to include the location-adaptive features. The idea of location-adaptive is based on a concept that in a networked virtual envi- ronment, users will more likely to interact with other users with certain distance range measured in virtual coordinates. This concept is well studied in the research topic of Area of Interest (AoI) [42, 57]. Once the AoI is de¯ned, the users in the same AoI are expected to be clustered to reduce delay and °ooding of control message. A spatial data mining algorithm called CLARANS [37] is cited by many papers in data clustering ¯eld and paper [27] reviewed the performance of many existing data clustering algorithms. Much research, e.g., [33],has been conducted to utilize overlay network for streaming applications. As surveyed in [55], a reduced graph approach did not decrease the quality 15 of the distribution tree in any particular way. Also a fully connected mesh graph with too many edges signi¯cantly increases the execution time required by graph algorithms. This paper also mentioned that the current game design usually concerns less with the degree limit because the current network bandwidth is enough to support the game con- trol information. But author also pointed out that with more and more MMOG games consider to integrate audio and video streams into their service grid, bandwidth at the access link will soon post a degree limit while constructing an overlay multicast tree. Applicationlevelaudiostreamingdesignhasbeenverysuccessfulinthepastfewyears with some mature commercial system available using P2P based design, e.g., paper [20] described a mixing and streaming algorithm for voice application. Several P2P overlay multicast designs are evaluated in [15]. To the best of our knowledge, our design is one of the few pioneer research that built to build a location adaptive mesh based streaming network that enables 3D audio streaming. 16 Chapter 3 ACTIVE Streaming Protocol In this chapter we ¯rst discuss the design issues that need to be considered for a scalable streaming network. Then we discuss the innovative P2P based design that address those issues and ¯nally we evaluate the performance of our design by comparing to the optimal solution used by many other overlay streaming architectures. Our goal is to design an general streaming platform that can be used to accommodate millions of concurrent users in the same virtual world and interactively communicate with each other. TheexpandingcapabilitiesoftheInternettohandledigitalmediastreamsisenabling new applications and transforming existing applications in many areas. An important design concern for these new streaming applications is the scalability of the system. We want to have a service that can scale well to cope with the growing number of users and provide steadfast performance without too many constraints a centralized server. This is not only to boost the service capacity, but also to provide a more reliable system. 17 Lowlatency,asmentionedearlier,isaveryimportantqualityforinteractivestreaming applications such as MMOG games and virtual conferencing. Without a reasonable end- to-end delay, many of these applications will become uncomfortable to use. Security is another major concerns that challenges the P2P system designers. In a P2Psystem, theservicereliesonthosemachinesthatnotinfullcontrolunderthesystem administrators or authorities, information transmitted through those intermediate node could be changed or stored without permission, bogus data could be injected to the network and cause tra±c jam, even imposter clients may get access to the service if the authentication part is not well de¯ned. There are several other issues such as a feasible business model, a fair reputation system that encourage people to contribute more resources to the shared community, a smart algorithm to appropriately utilize the resource from a peer so that other more critical tasks running on the peer are not hindered. However, not all these issues are within the research scope of this report and some of them are left to be future research topics. In this report, we will mainly focus on two major challenges that are crucial for interactive streaming applications: low-latency and scalability. We will soon discuss how our design, ACTIVE [31, 33], fuses these two features under one umbrella. The ¯rst stage of our research e®orts is to design and build a P2P based reliable streaming platform for applications with single active group. In the following sections of this report, we will ¯rst report our approaches to address these two challenges in this type of applications. We assume that, regardless of the number of concurrent users in the system, there is a shared topic of common interest among all users. Examples of this 18 type of applications are: Online tutorial session, remote rehabilitation exercises, virtual concert, online conference etc. In Chapter 4 we will discuss how we extend the ACTIVE protocol to applications with multiple active groups. 3.1 Problem Statement In this section, we will discuss the problem we are trying to solve in ACTIVE protocol and details of our proposed solution to this problem. In the middle we will introduce a mathematical presentation called Credit Point System (Section 3.1.3) in order to discuss our design, . De¯nitions of some terms used are listed on Table 3.1. 3.1.1 Statement Of The Problem To Solve Recall that in Section 1.2, we mentioned that ACTIVE protocol builds a tree topology to provide low latency streaming among active users. In other words, the objective of ACTIVE is to minimize the average delay among active users, D A , in a given multicast group V. This problem can be formally stated as degree-bound minimum spanning tree (MST) problem: For a given graph G and user set V, ¯nd a tree T0, where T0µG, so that the degree constraint is satis¯ed at each node and P i;j2A jD i;j j is minimized. Beforewepresentourproposedsolutiontothisproblem,we¯rstintroducethenetwork model we will use during our discussion plus a simple introduction to the Credit Point (CP) system so that the de¯nition of our solution can be understand. 19 Term De¯nition Units V Setofallusersoncurrentstreaminggroup,jVjisthetotalnumber of all users A Set of all active users on current streaming group, jAj is the total number of active users G Application level complete graph containing all nodes in V E Set of edges that connect all nodes in V, E =V £V jE i;j j The physical network delay between host i and host j N A 0 Maximum number of active users allowed P The application-layer processing delay. Also called relay delay. ms T Current streaming tree that connects all nodes in V R System optimization frequency control parameter Hz O m System wide control overhead for operation M I i Idle time at host i sec K Degree limit of T, maximum number of children allowed for each node in V F System °oor control parameter D i;j Overlay data delivery path from host i to host j jD i;j j Overlay end-to-end delay from host i to host j ms D A System wide average end-to-end delay among all active nodes in A ms D A (i) Average end-to-end delay from host i to all other active users ms D V System wide average end-to-end delay among all nodes in V ms D V (i) Average end-to-end delay from host i to all other host in V ms HOP i;j Number of intermediate relay nodes between host i and host j CP i Credit Point assigned to node i CPt System wide CP threshold that allows a user to switch to active mode RDP-AU Ratio of average overlay delay for active nodes compared to MST tree result RDP-ALL Ratio of average overlay delay for all nodes compared to MST tree result Table 3.1: List of terms used in this paper and their respective de¯nitions. 3.1.2 Network Model The physical network consists of routers connected via links, and end-hosts that are connected to these routers through access connections. The delay between end-hosts and their accessing router is smaller than the average delay between routers. This models both the characteristics of a intra-domain and inter-domain network. Subsequently, the overlay network is built from end-hosts and on top of the physical network topology. The overlay network can be modelled as a complete directed graph, 20 denoted by G = (V;E). V denotes the set of all end hosts and E = V £V refers to the set of connections. E i;j denotes the directed edge from host i to host j, while jE i;j j represents the physical network delay between host i and j. We assume symmetric links, i.e.,jE i;j j=jE j;i j. The multicast tree T is a subset of G and represents the P2P topology. The data delivery path D i;j consists of all the intermediate nodes on T from host i to host j and all the edges connecting them. jD i;j j represents the overlay end-to-end delay between host i and host j on T. It is calculated as shown in Equation 3.1, where P k denotes the processing delay at each of the intermediate node K on the path from host i to host j. jD i;j j= j X k=i P k + X E m;n 2D i;j jE m;n j (3.1) For a distributed application, the processing delay P k accounts for a large component of the overall delay. In turn, the processing delay is determined by a lot of factors, e.g., streaming type, compression algorithm, computing power of the peer machine, etc. We model the processing delay at a audio stream mixing node that use the GSM.610 audio codec in Equation 3.2. P k =T recv buffer +T decode buffer +T mix processing +T encode buffer +T send buffer (3.2) We can see that in a peer-to-peer network, P k is usually not the same because of the varying computing power at each node. The decode/encode bu®er delay is de¯ned by the compression algorithm and the receive/send bu®er is de¯ned by the application's 21 network module. For example, there is a 34ms processing delay on average in one of our test nodes illustrated in Fig. 3.3. 3.1.3 Introduction Of Credit Point System This is just a simple introduction to some de¯nitions of the Credit Point (CP) system, which is used by ACTIVE protocol in various places. Details of Credit Point System are discussed on Section 3.2.1. In an ACTIVE-constructed streaming tree T with degree limit K , each node i is assigned a °oat value ranged from zero to one based on following formula: CP root = 1 CP Child = CP Parent =K ; K¸2 (3.3) The basic function of CP system is to serve as a distributed and universal governing mechanism to maintain the tree structure among all users. When there is an unexpected disconnection of the peer, CP system can also help to re-construct the tree without a global knowledge of all other users. ACTIVEsystemdistinguishesactiveusersfrompassiveusers. Dependsonthenumber of active groups allowed in the system, Credit Point serves slightly di®erent functions. In a system with single active group, only a subset of the total participants can become active users (and join the active group), so CP system is also used as a °oor control mechanism where a passive user i need to compare its local CP point value CP i to a 22 system wide threshold CPt before it switchs to active user. If CP i ¸ CPt the switch is done immediately, otherwise an optimization process is triggered (Section 3.5). 3.1.4 Statement Of Solution Finding an optimal solution for degree-bound minimum spanning tree is NP-hard [12], thereforeweproposeaheuristicsolutionwhichcanbecomputedinafastanddistributed fashion. The basic idea is as follows: Since the number of active users is relatively small even in a large group, forming a cluster among active users in which every active user is directlyconnectedtoanotheractiveuserwille®ectivelyreducethenumberofintermediate nodes, thus reducing the processing delay introduced by these nodes. This solution is formallystated asfollows, wherejHOP i;j jdenotesthetotalnumberofintermediaterelay nodes on the path from host i to host j: Solutiontosolvedegree-boundminimumspanningtreeproblem: For8i;j2 A; HOP0Ã max(jHOP i;j j). Find a subtree T0 (T0µ T, A½ T0), so that the following condition is satis¯ed: 8 > > > > > > < > > > > > > : HOP0·log k 1 CPt 2 ¡1 8i2A,9j :jHOP i;j j=0; j2A;j6=i The above algorithm is designed to be e±cient and minimize the control overhead by avoiding an exhaustive search for an optimal result. An e±cient solutions is desirable for the following reasons. First, while a complex heuristic algorithm may generate better 23 resultsinastatictopology,itmustrestartthecomputeprocesswheneverthereisatopol- ogy change. P2P systems are highly dynamic environments where the overlay topology is frequently changing, and a complex algorithm may need a long time to reach its result. Second, thenoveldesignofdistinguishingactiveusersfrompassiveusersmakesACTIVE able to establish very low-latency service among active users, thus reducing the need to compute an optimal result. In Section 3.7 we show that ACTIVE can even generate better performance among active users compared with the MST solution. Note that even though the goal of the optimization process is to reduce the delay among active users, in our heuristic solution we do not need to measure the delay among active users in order to achieve our performance goal, thus reducing the control overhead in our system. 3.2 Design Overview Our ACTIVE protocol is capable of providing interactive streaming services to large groups. ACTIVE contributes the following new ideas to P2P architectures. First, it distinguishes active users from passive users so that an intelligent optimization can be performed on this subgroup instead of the whole group. Second, it dynamically adapts the P2P structure to maintain delay service quality for active users. Finally, it uses a dynamic °oor control mechanism to allow a growing number of active users grows as the size of the multicast group increases. The °oor control constraint can be adjusted on the °y so that the system dynamically allow more or less active participants. By 24 dynamically optimizing the P2P structure, ACTIVE can signi¯cantly reduce the end-to- end delay among active users and at the same time, provide streaming service to very large multicast groups. In ACTIVE, virtually all users are provided with the low-latency service that before was only possible in a centralized or full-connected approach. ACTIVE protocol is built on top of following four system components: platform con- struction module, user activity module, performance optimization module and streaming policy module. The platform construction module builds the overlay peer-to-peer based streaming network and takes care of the error recovery when there is a node fail or leave. User activity module identify active users based on users' behavior. The optimization algorithm is run at each peer and the underlying streaming network is reconstructed to increase the system performance such as average end-to-end delay in an active group. Streaming policy module governs the packet handling rules at each streaming nodes in- side ACTIVE network. By putting all these components together, the design provides a scalable solution for massive interactive streaming applications. Distinguishingactiveusersiscrucialtoprovidinglow-latencystreamingservice. Exist- ingsystems are building their overlaytopologies to closely match an MST tree. However, as illustrated in Fig. 3.1(a)-(c), a MST tree linking all multicast nodes cannot guaran- tee the minimum delay among active users, to whom low latency is critical. Fig. 3.1(b) showsthatwhentheapplicationlayerprocessingdelayis30msateachnode, theaverage overlay delay among active nodes is 123.33 ms, while for the same nodes in Fig. 3.1(c), the delay among active users in an ACTIVE generated tree is only 83.33 ms. WhentheapplicationprocessingdelayP isaddedtothenetworkingdelay,theaverage latency in overlay P2P systems quickly rises above the threshold for a smooth interactive 25 C F E D B A 60 10 30 40 30 50 30 20 50 Physical Network for Node A to F a. 40 30 20 A B F C D E 10 30 b. MST: W = 130 Average Delay For B, C, F = 123.33 ms , P=30ms B C A D F E 30 80 80 30 70 c. ACTIVE: W = 290 Average Delay For B, C, F = 83.33 ms, P=30ms Active Nodes Active Nodes Figure 3.1: Example: MST vs. ACTIVE experience when the group size increases (Fig. 3.2). This occurs even if an optimal tree is built with the MST algorithm. The application processing delay P is determined by various factors, e.g., the operating system, the CPU speed, and the available memory. Fig.3.3showstheprocessingdelayattwooftheintermediatenodesofanaudioconference application using ACTIVE. 3.2.1 Credit Point System We have introduced some basic de¯nition of Credit Point System on Section 3.1.3. Now we will discuss in more details how the Credit Point (CP) system is used to control the total number of active users in the system with a single active group and how ACTIVE optimizes the tree. The CP system provides some key features of ACTIVE and works as follows. 26 0 200 400 600 800 1000 1200 1400 0 50 100 150 200 250 300 Average Delay (ms) Applicatoin-layer Delay: P (ms) 36 Users, 4 Active Users ACTIVE: Active Users MST: Active Users Figure 3.2: Processing Delay vs. Total Delay Each node is assigned a CP value based on Eq. 3.3. Once a node i is assigned its CP i , it will keep the value until there is a topology change. As mentioned earlier, if there isonlyoneactivegroup,thenACTIVEdistinguishesthenodesinV asactiveusersAand passive users V ¡A. A user can try to switch from active to passive status, or vice versa at the time they want. The request of status switching can be triggered automatically by activity detection or explicitly by user input, e.g., press a button to show interest to become an active speakeretc. Details of useractivitymodel are discussedin Section 3.4. However, thesystemcontrolsthenumberof"recognized"activeuserswithasystemwide threshold value CPt. A node has enough CP i , which is the credit point value at current node, will be recognized by the system as a active user. The formula for this recognition presented on Eq. 3.4. CP i ¸CPt ; i2V (3.4) 27 0 50 100 150 200 250 0 20 40 60 80 100 Relay Processing Delay (milli-sec) Time (sec) avg = 34ms at peer 1 avg = 54ms at peer 2 Figure 3.3: Application-layer Processing Delay If Equation 3.4 is satis¯ed, node i can switch from passive to active mode and rec- ognized by the system as active users. We term those user requested to switch to active modeandsatis¯esEq.3.4recognized active users. ACTIVEprotocolcontrolsthenumber of recognize active users, and optimize the delay among those users. Since these recog- nized nodes are usually only small fraction of the whole participants nodes, Equation 3.4 is also called the Floor Control Equation. In this report, unless specially mentioned, we use the term recognized active users and active users interchangeably. In ACTIVE, there is no constraint for switching from active to passive mode. The total number of active nodesjAj is bound by CPt and degree K. If we denote N A 0 as the 28 maximum number of active nodes allowed, then the following equation illustrates how N A 0 is constrained by CPt and K: jAj·N A 0= K¡CPt CPt£(K¡1) ; K¸2; 0<CPt<1 (3.5) The degree limit K is usually stable for a multicast con¯guration, but the threshold CPt should change according to the size of the group. If we denote F as the dynamic °oor control parameter with a value ranging from 0 to 1, ACTIVE calculates CPt based on following equation: CPt= F log k jVj ; 0<F <1 (3.6) By substituting Equation 3.6 into Equation 3.5, we arrive at an equation to calculate N A 0 from V directly: N A 0 = K£log k jVj¡F F£(K¡1) . This shows that the maximum number of active users N A 0 is a logarithmic function of the total number of usersjVj. CP points are also used to construct a correct streaming tree, this topic is discussed in details on Section 3.3.3. The complexity of the Credit Point System is low and, since the active user set A is usually only a small subset of V, ACTIVE is able to achieve close to optimal performance in most cases (described in Section 3.7). The design of ACTIVE is comprised of four components: a)Platform construction module,b)Useractivitymodule,c)Optimizationalgorithmandd)streampolicymodule. We will now describe these four components in turn. Please refer to Table 3.1 for those frequently used terms and their de¯nitions. 29 3.3 Platform Construction 3.3.1 Join During the join process, ACTIVE uses a heuristic Shortest Path Tree(SPT) algorithm (Algorithm 1) to construct the tree. A new node starts the join process by contacting a rendez-vous point (RP) node. Such a bootstrapping technique is widely used in peer-to- peerapplications. TheRPnodecouldbeadedicated,well-knownserviceorjustanynode that participates currently in the P2P multicast group, as long as the new node knows the RP's network address. After joining the P2P system, a node runs independently of the RP node. A new node obtains a list of candidate parents L during the join process. The new node ¯nds its nearest neighbor by sending out request messages simultaneously to all candidate nodes in list L. A candidate node will immediately reply once it receives the request message. The new node recodes L in the order in which it receives the response messages and the ¯rst responding node C in L will be considered the nearest node. Node C will be chosen as the candidate parent and a setup process immediately follows. If an Algorithm 1 JOIN Require: RP is online 1: LÃ candidate nodes from RP 2: while L6=; do 3: CÃ nearest node in L 4: if C is ok to join then 5: setup connection with C 6: parentÃC 7: break while loop 8: else 9: LÃ add new candidate nodes referred by C 10: remove C from L 11: end if 12: end while 30 Algorithm 2 LEAVE 1: inform all neighbor nodes that N is leaving 2: if N is the core then 3: C0Ã nearest neighbor node 4: setup C0 as the new core 5: inform all other neighbor nodes to set C0 as parent 6: else 7: if N has child node then 8: N0Ã N's parent 9: inform all child nodes to set N0 as parent 10: end if 11: end if 12: disconnect from the service error is causing a setup failure (e.g., C exceeds its degree limit or suddenly goes o²ine), the new node will remove this candidate from L, choose the next node C0 at the head of list L and starts the above process again. 3.3.2 Leave A leaving node must perform a few steps, as shown in Algorithm 2, to ensure that after its departure the multicast tree T is loop-free and all nodes are reachable through T. Failure to ¯nish these steps is considered an error in ACTIVE and its impact is discussed in Section 3.3.3. The only exception occurs when the current node has no child nodes. In this case, the node can simply disconnect from the service. 3.3.3 Error Detection And Recovery In a tree-based architecture there are two types of errors in the topology: loops and tree splits. ACTIVE uses di®erent mechanisms to detect these two errors. Atreesplitcanbeeasilydetectedatthenodeswhichlostconnectionstotheirparents or children. To avoid redundant message exchanges, we use an asymmetric scheme: a 31 lost connection to the parent is considered an error at the children nodes but a lost connection to a child node is not an error at the parent node. This rule makes it a child node's responsibility to repair the tree and ¯nd a new parent when the connection is broken. Detecting a connection error is simple in ACTIVE because it utilizes the TCP protocol in order to stream to nodes behind NAT devices, and TCP will automatically detect and report disconnection errors. The process of ¯nding a new parent is di®erent from the join process and can be done without help from the RP server. As mentioned in Section 3.3.1, each node saves a list of usable candidate parent nodes L during the join procedure. When a node loses the connection to its parent, L will be used to ¯nd a new parent, following the process described in Algorithm 3. Note that the rejoin process is di®erent from the join process. First, when initially joining, a node tries to ¯nd the closest node in L, while with rejoin a node is attached to the ¯rst usable node in L. Second, therejoinprocessisindependentoftheRPserver, whichisrequiredasastarting pointintheinitial joinprocess. Thesedi®erencesaredesigned tomaketherejoinprocess in ACTIVE fast and scalable. Whenever a node i changes its parent, we consider this to be a topology change in the multicast tree. Node i will send out update messages to its direct child nodes, which will update their local information and in turn forward the update message to their child nodes. If this change at node i forms a loop, the update message from node i will eventually be forwarded back to itself, and a loop error is detected. Once node i detects a loop, it will disconnect from the parent and start the rejoin process. Note that in the rejoin process, the new parent must have an equal or bigger CP value than CP i . This 32 Algorithm 3 REJOIN 1: LÃ the candidate node list built in JOIN process 2: while L6=; do 3: CÃ ¯rst node in L that has bigger or equal CP 4: if C is ok to join then 5: setup connection with C 6: parentÃC 7: break while loop 8: else 9: LÃ add new candidate nodes referred by C 10: remove C from L 11: end if 12: end while willavoidnodeitoformanotherloopbyjoiningthenodesinthesubtree rootedatitself, where every node has a smaller CP value. Note that ACTIVE is an event-driven protocol and there is no message °ooding at any time during tree construction, tree maintenance or tree optimization. 3.4 User Activity 3.4.1 User Classi¯cation We classify users as active or passive mode. The detection of these two modes can be done either automatically by the system or by explicit input from users. Here we only discuss how ACTIVE automatically classify active and passive users. ACTIVE protocol relies on user action monitoring module, such as silence detection or movement detection etc., to provide with user activity data. Then ACTIVE protocol tracks the the idle time I P that no user activity is detected. As shown in Fig. 3.4, the system optimization frequency control parameter R determines the idle time for a user to switch from active to passive mode and vice versa. 33 Figure 3.4: Automatic Mode Switch Using R 3.4.2 Floor Control A °oor control mechanism is of practical importance for large scale streaming systems with a common topic (single active group) because too many active users may saturate system resources or degrade the overall streaming quality. For example, if too many people are talking simultaneously in an audio chat room, the conversation will become incomprehensible. ACTIVE implements a dynamic °oor control function (Eq. 3.5) to control the total number of active users inside each active group. The maximum number of active users N A 0 gradually increases along with the size of the groupjVj. In Equation 3.6, F is called the °oor control parameter. A system administrator can dynamically change F to lower 34 or raise the threshold CPt, and thus control the total number of active users in the system. We reformat Equation 3.5 as follows: N A 0= K£log k jVj¡F F £(K¡1) (3.7) If the system requires that all users can be active at the same time,e.g., in a ¯rst person shotting MMOG game, the °oor control mechanism can be conveniently disabled by setting F to the value of F0 calculated in Equation 3.8. It is not hard to prove that in this case Equation 3.4 is always satis¯ed. F0= K£log k jVj jVj£(K¡1) +1 (3.8) 3.5 Optimization Algorithm Forasystemwithsingleactivegroup,thedelayamongactiveusersisgraduallydecreased by executing a tree optimization algorithm 1 . The optimization function is run at each node and triggered by the users' requests to become active. For example, in an audio conferencing application, speakers are active users, and passive users can become active either manually, e.g., by pressing a button, or automatically when there is a voice input detected. The local node will determine whether the request can be granted based on Equation 3.4. If the request can not be granted, the optimization process is triggered. 1 Please note that the optimization process for an application with multiple active groups is discussed in DSM at Chapter 4 35 The clustering is achieved by moving the active nodes gradually towards the root of the tree until Equation 3.4 is satis¯ed. This move is accomplished by exchanging an active child node with its non-active parent or another higher-level node (Algorithm 4). It is worth mentioning that even though the complete optimization may take as long as a few seconds, each step only requires a few milliseconds to setup the new streaming connections. The loss of data during these steps is so small that it does not a®ect the playbackquality. Infact, intheaudioconferencingapplicationthatrunsontheACTIVE protocol, we cannot detect any audible glitches during the optimization. The optimization algorithm ¯rst observes the idle time I P at the parent node to see if it has been idle for long enough. The system optimization frequency control parameter R determines how frequently a reconstruction can be performed on a parent node. If I P · R ¡1 , the optimization is canceled. For example, in a system with R = 0:1, a node would need to continuously remain idle for 10 seconds before another node can exchange position with it. As shown in Algorithm 4, the rest of the optimization procedure is performed in two phases. In the ¯rst phase, the active node i gradually moves towards the root until its parent is also an active node. In the second phase, ACTIVE condenses the cluster of active users further if necessary. If the optimization function returns without being able to ¯nd a better position for node i, it means there are currently too many active users in this streaming group, and node i will be denied to switch to active status. 36 Algorithm 4 OPTIMIZE 1: AGAIN: 2: P Ã parent of local host i 3: I P Ã idle time at P 4: if I P ·R ¡1 then 5: return 6: end if 7: //First Phase 8: while P is not active do 9: if P is core then 10: ask P to set local host i as parent 11: setup local host i as core 12: else 13: P0Ã the parent of P 14: ask P to set local host i as its parent 15: ask P0 to set local host as new child 16: P ÃP0 17: setup connection with P0 18: end if 19: end while 20: 21: //Second Phase 22: if CP i ¸CPt then 23: send update message to all children nodes 24: return 25: else 26: L0Ã all passive node immediate connected to A 27: remove all host j from L0 if CP j ·CP i or CP j =1 28: if L0=; then 29: send update message to all children nodes 30: return 31: else 32: CÃ ¯rst node in L0 33: P0Ã the parent of C 34: ask C to set P as its parent 35: ask P0 to set local host as new child 36: P ÃP0 37: setup connection with P0 38: end if 39: end if 40: goto AGAIN: 37 3.6 Streaming Policy Inthissection,wewilldiscussthestreamingpolicyforACTIVEprotocolinasystemwith single active group. As we discussed before, ACTIVE protocol builds a shared multicast tree using a numbering system called CP. If all participants (users) of the system share a common topic of interest (TOI), then the user wants to join the same active group when itswitchesfrompassivetoactiveroleinthesession. Sincethetopicofinterestiscommon among all users, there are two features for the streaming policy design: ² Universal Delivery: the packets carrying information of the topic of interest (TOI) need to be delivered to all participants, regardless their status is active or not. ² Channel Mixing: certain type of stream, e.g., audio streams, can be mixed into a single channel since all the recipients subscribed to that channel will be interested on retrieving all inputs from di®erent origins. The streaming policy is determined by the nature of the applications. Following stream policy is designed for applications with single active group: Stream packets, no matter who is the sender, are targeted to all other users in the same streaming session. Text packets are forwarded at each intermediate node to other nodesdirectlyconnectedtothatnode. Audiopacketsaremergedateachintermediatenode and the merged stream is forwarded in the same way. Please note that before forwarding, the original audio signal is removed from the mixed stream to avoid the original sender receive its one audio packet back. In the next chapter, we will discuss that above two features doesn't stand any more in an application with multiple topic of interests(TOI). 38 3.7 Performance Evaluation We evaluated our ACTIVE design with simulations in an NS-2 environment [1]. The ACTIVE code used in the simulation is the same as the one used in the audio chat room application running on Windows. Only minor changes were made to compile the code in a Linux environment. A module to collect the actual delay among nodes was also added. In the simulation we compared ACTIVE's performance to trees generated by Prim'sminimumspanningtree algorithm[17] withthesamephysicalnetworktopologies. ThegeneratedMSTtreesaretheoptimalsolutionforallexistingtree-baseddesigns(e.g., [25, 19, 58, 7]) which assume no application-layer processing delay at each node. Note that these MST trees are not the optimal solutions for ACTIVE. 3.7.1 Control Overhead Analysis We de¯ne the control overhead O for an operation M as the total number of messages received at all involved end-hosts multiplied by the frequency of this operation. Since °oor control is performed along with tree optimization, there is no °oor control overhead added to the ACTIVE system. Many existing P2P protocols depend on a refreshment based mechanism to maintain their service. For example, in the NICE protocol, each node i needs to send out a HeartBeat message periodically to all other nodes in the same cluster. This process continues to consume network bandwidth as long as the node is in the system. If we 39 denote the refreshment period as h, and N0 as the average number of nodes receiving the refreshment message, the control overhead O r for all refreshment based protocols is: O r = jVj£N0 h (3.9) ACTIVE uses the CP system to distribute the computation for tree maintenance and optimization. BecauseACTIVEisanevent-drivensystem,intheworstcasemessagesare required to be sent to all nodes in V. If the operation happens with the same frequency as in the refreshment-based approach, the control overhead in ACTIVE is: O A = jVj h (3.10) Equation 3.10 shows that given the same event frequency, ACTIVE achieves a much smaller control overhead compared with refreshment based designs. 3.7.2 Performance Metrics Inourperformanceevaluation,theMSTtreeisusedastheperformancebaseline. Herewe introducetwotermstodescribetheperformance: RDP-AUandRDP-ALL.RDPdenotes the relative delay penalty, which is the ratio of the average overlay delay in ACTIVE to the average overlay delay in MST. RDP-AU is the RDP calculated only for all active nodes and RDP-ALL is calculated for all multicast nodes. Smaller RDP values indicate better performance and an ACTIVE tree outperforms the MST tree if its RDP-AU is smaller than 1. Note that RDP-ALL is never less than 1. 40 In the NS2 environment, each node i can easily calculate the overlay delay to another node j by comparing the global time-stamp of a packet received from node j to the current system global time. The average overlay delay at node i to all active nodes can be calculated as D A (i) = P j2A jD i;j j jAj¡1 or D V (i) = P j2V jD i;j j jVj¡1 to all nodes in V . All nodes report their D V (i) and D A (i) to the RP server, where the overall average delay is calculated by using the following equations: D A = P i2A jD A (i)j jAj ; i2A (3.11) D V = P i2V jD V (i)j jVj ; i2V (3.12) To measure the performance of ACTIVE, we need to measure the actual delay in our performance evaluation experiments. An delay monitoring module is added to ACTIVE to collect such data. Each active node sends out update messages to other nodes to calculate the overlay delay when overlay topology changes. In the NS-2 environment, there exists a global system timer that every node can access. This makes the delay measurementseasyandaccurate. Whenanode isendsoutanupdatemessage, itstamps the message packet with its sending time. The update packet is forwarded at each node that receives it, after adding the processing delay P. Each node, e.g., host j, upon receiving an update message, can simply subtract the sending time from current system time and obtainjD i;j j, the overlay delay between host i and host j. As each node knows 41 if its remote nodes are active or passive by using the information contained in the update messages, it can calculate D A (i) and D V (i) as follows. D A (i) = P j2A jD i;j j jAj¡1 ; i6=j (3.13) D V (i) = P j2V jD i;j j jVj¡1 ; i6=j (3.14) Each node will send its D A (i) and D V (i) to RP server node, where the system wide D A and D V are calculated: D A = P i2A jD A (i)j jAj ; i2A (3.15) D V = P i2V jD V (i)j jVj ; i2V (3.16) For a given network topology,jE i;j j is ¯xed between any host i and host j. Also, for a given multicast tree T, the hop distance between any two hosts is also a constant. When we take a snapshot of the system topology at a particular time, D A and D V are linear function of P. Our experimental results show this relationship in Fig 3.5. 42 0 200 400 600 800 1000 1200 1400 0 50 100 150 200 250 300 Average Delay (ms) Application-layer Processing Delay: P (ms) 36 Users, 4 Active Users Active Users All Users Figure 3.5: Example: D A and D V vs. P 3.7.3 Simulation Setup Therouter-levelphysicalnetworkisgeneratedaccordingtotheTransit-Stubgraphmodel, usingtheGeorgiaTechInternetworkTopologyModels(GT-ITM).Delaybetweenrouters is randomly distributed from 5 to 35 ms. Each end-host node is randomly attached to one of the routers with an access delay of 0.5 ms. Fig. 3.6 shows one of the topologies we generated with 400 participants and 20 active users. In addition, based on the measure- ment we conducted with the audio application (Fig. ??.(d)), we used the average value P = 30 ms as the processing delay at each node to simplify the simulation. However, using this speci¯c value does not a®ect the ¯nal conclusion of our simulation results. 3.7.4 Simulation Results ACTIVE achieved dramatic performance improvements over other existing systems in our simulation. We ran 2000 iterations of the simulation with uniformly distributed user groups ranging in size from 4 to 400 and active user groups ranging in size from 4 to 20. Fig. 3.7 shows that compared with the MST solution, ACTIVE achieves a smaller 43 Figure 3.6: Network Topology (400 Users) delay among active users in 97% of the 2000 simulations we conducted. If we consider RDP-AU = 1:5 as an evaluation threshold for good overlay delay performance, then in an astounding 99.95% of all cases ACTIVE delivered good performance. For reference purposes, we also calculated the average delay among all users. As illustrated in Fig. 3.8, ACTIVE delivered good performance in 72.75% of all cases. It is interesting to see that in 1.90% of the cases, ACTIVE can, surprisingly, provide better performance than the MST tree. After investigating the cause, we found that these counter-intuitive results are not incorrect. For all leaf nodes, which will not perform any forwarding job, the application processing delay will not be counted for the overall delay. This means that the more leaf nodes are in the overlay tree, the less overall delay we can achieve. The MST algorithm only guarantees to generate a tree with minimum tree weight, not the minimum number of leaf nodes. Hence, in some rare cases, ACTIVE can outperform MST. 44 0 20 40 60 80 100 120 0.01 0.1 1 10 Number of Interations RDP-AU of ACTIVE RDP-AU: ACTIVE Figure 3.7: Delay performance of ACTIVE for active users 0 50 100 150 200 250 300 350 400 0.1 1 10 Number of Interations RDP-ALL of ACTIVE RDP-ALL: ACTIVE Figure 3.8: Delay performance of ACTIVE for all users 45 3.7.5 Experimental Scenario To show how tree optimization in ACTIVE reduces the delay among active users over time, we divided our simulation iterations into three phases: A, B and C. In phase A, the ACTIVE protocol did not identify any active users, and no optimization was performed; in phase B, active users started to emerge at random times, and optimization was dynamically invoked to optimize the tree. In phase C, all active users were identi¯ed and optimization completed, hence the multicast tree became stable. We randomly picked one iteration for small, medium and large group respectively from the 2000 simulations as examples to illustrate how ACTIVE optimizes the delay performanceamongactiveusersoverthetime(Figs3.9,3.10and3.11). Inallthesethree iterations, ACTIVE has a smaller application level delay D A than the MST when the simulation ¯nishes phase B and reaches phase C. It is very important to recognize that by having a close performance compared to MST, as shown in Fig. 3.9 and Fig. 3.11, ACTIVE out-performs other existing algorithms which consider MST as the optimal solution. Also illustrated in these ¯gures, the average overall end-to-end delay had not been signi¯cantly changed during the optimization process. Our experimental results show that while D A is reduced by 50% or more, D V remains almost the same in most cases, or is even reduced in some cases. It is also worth noting that while ACTIVE has been deployed for practical use, the MST algorithm is computationally too complex to be used in any real-time system. The computation complexity of Prim's algorithm is O(jEj+jVjlogjVj). 46 0 200 400 600 800 1000 0 20 40 60 80 100 120 140 Average Delay (ms) System Running Time (ms) 400 Users, 20 Active Users ACTIVE: Activer Users ACTIVE: All Users MST: Active Users Phase A B c Figure 3.9: Example: Large Group 0 50 100 150 200 250 300 350 400 450 500 0 20 40 60 80 100 120 140 Average Delay (ms) System Running Time (ms) 36 Users, 4 Active Users ACTIVE: Active Users ACTIVE: All Users MST: Active Users Phase A B C Figure 3.10: Example: Medium Group 47 0 50 100 150 200 20 40 60 80 100 120 140 Average Delay (ms) System Running Time (ms) 4 Users, 4 Active Users ACTIVE: Active Users ACTIVE: All Users MST: Active Users Phase A B C Figure 3.11: Example: Small Group 48 Chapter 4 DSM Audio Streaming Protocol Inthischapter,wewilldescribeourdesignofDSM,ameshbaseddesignthatdynamically adapts the overly mesh network based on the location of a node in the virtual world. By clustering nodes using locations online, multiple streaming groups are created in side the whole group of users online. DSM is designed to be a scalable streaming platform for large scale interactive applicaitons with virtual world, a good example of this type of applications is Massively Multiplayer Online Games (MMOG). Massively multiplayer online games (MMOG) have become a widely successful online business with revenue in the billions of dollars (2005), { a number that is expected to keep growing in the years to come. The dominant design for current MMOG games is, not surprisingly, the client-server paradigm. While the client-server model works well for current games where each game server needs to handle only a few thousands of concurrentusers, theconstraintsofsuchacentralizedapproachalsolimitthegrowthand deployment of MMOG games. Recently a number of research e®orts have investigated the application of peer-to-peer (P2P) architectures to the design of MMOG games [29, 14, 51, 5]. However, to the best of our knowledge no truly successful implementation of 49 an MMOG game built on a P2P-based design exists, despite the tremendous amount of research investigating P2P architectures. It is interesting to note that { in contrast to the lack of successful applications using P2ParchitecturesforMMOGgamedesign{P2Ptechnologyhasbeensuccessfullyapplied to ¯le sharing and VoIP applications. We believe that this is due to the fact that there aresomekeycharacteristicsthatdistinguishMMOG games from applicationssuchas¯le sharingandVoIP.MMOGgamesaredesignedtosupportamassivenumberofconcurrent players (the number could be in the millions) in a networked virtual environment (NVE). Eachplayer is virtually connected to every other player in this shared world. Because playersarefrequentlychangingtheirgamestate(e.g.,viewangle,pose,etc.),suchchanges need to be propagated to every other player who is a®ected by these updates as soon as possible to avoid possible inconsistencies. Since the message exchange among players is accomplished by sending out a continuous stream of message packets, we can view the tra±c model in an MMOG game as time-sensitive, many-to-many multicasting. In contrast, witha P2Pbased ¯lesharing network, eventhoughthe sizeof eachdownloaded ¯le can be large, the delay is usually not a critical issue. While the end-to-end delay between the speakers in a VoIP application is a critical performance metric, the number of participants in each conversation is usually far smaller than what we expect in a popularMMOGgamesession. WeconcludethatanMMOGapplicationcombinescritical requirements from both ¯le sharing and VoIP applications. Interest management is a popular method used in large scale client-server game de- signs. The basic idea is to organize players into sub-groups of similar interests and exchange most information within the same sub-group, instead of sending data out to all 50 Figure 4.1: Mapping of Virtual Game Space to Overlay Peer-to-Peer Architecture otherplayers. Theplayersinsidethesameinterestgroup,togetherwithallvirtualobjects within a certain range, are called the area of interest (AOI) for this interest group. A numberofrecentstudieshavetriedtoextendtheAOIconcepttoP2Pbasedgamedesign. However, the resulting solutions have so far failed to provide the game industry with a practical platform for implementations. Some of the reasons that have caused a shortage of implementations can be found in the lack of manpower in the research community to buildarealgame. Sometimesunrealisticdesignassumptionswillhinderthetransferfrom theory into practice. For example, there is no game design paper addressing the issue of network address translation (NAT) devices and the e®ect of such devices on the design. NAT devices nowadays are commonly used by home users to connect to their ISPs using cable or DSL modems. It is di±cult to imagine any P2P based game design that can be usedforarealgameimplementationwithoutsolvingtheNATdeviceconnectionproblem. 51 4.1 Introduction InthepastfewyearstheInternethasbecomeanimportantplatformforlarge-scaleinter- active applications such as Massively Multiple-player Online Games (MMOG). Most of these applications create a networked virtual environment (NVE) where users can move their avatars in the shared virtual world and interact with each other. In this study we will focus our discussion on MMOG-type applications where users share a virtual world and the end-to-end delay between participants is usually an important measure of per- formance. While many successful systems exist providing users with di®erent types of virtual world settings and experiences, almost every commercial application chooses a server-client architecture as the underlying topology design. The reasoning behind this phenomenon can be easily justi¯ed from multiple viewpoints including security, perfor- mance, business model, etc. Compared with a distributed design, a centralized system has many advantages in the early stages of the natural growth of a MMOG-type appli- cation. However when the number of concurrent online users reaches millions instead of thousands, some inherent limitations of a centralized design, such as network tra±c concentration around servers, become more and more apparent. It has been reported that a popular MMOG game in North America has over thirty thousand players on- line on average and because of its centralized design, the game company has to limit the maximum number of players in each game zone to around two hundred. The re- search community, foreseeing this problem, seeks solutions to build a scalable, secure and highperformancecommunicationplatformtosupportlarge-scaleinteractiveapplications. Many distributed systems have been proposed in the past few years, but to the best of 52 our knowledge, none of them have been deployed for any large scale commercial MMOG so far. It is worth mentioning that Skype, a very successful P2P based VoIP system, serves millions of users online on average. But in contrast to the massive number of online users overall, the number of participants in each conversation is usually very small (typically two). This makes Skype's performance optimization problem very di®erent from a MMOG-type application, which we will discuss in details in this paper. At this stage,itisstilldi±culttopredictwhenthe¯rstsuccessfulP2PbasedMMOGwillemerge in the market, but it is very unlikely that a pure P2P based design will become practical for serious commercialized MMOG type systems in the near future. In this study we will discuss our design of a P2P based scalable streaming system to provide 3D spatial audio services to MMOG-type applications. In a typical MMOG system, the game state is shared among all players through a game auditor, usually a dedicatedserverprovidedbythegamecompany. Wedesignoursystemtobeindependent of the game control architecture so that it can be integrated into any existing or future MMOG games, whether they choose a centralized or distributed design. Our design reduces the game state information required form the game auditor to basic information such as the virtual coordinates of a local player. In order to make the game scalable enough to support tens of thousands of concurrent players, game designer usually divide thegameworldintodiscretegamezonesanddesignthegameinawaythatscattersusers across these game zones. With this design, we assume that although the total number of online users is potentially large, the number within a particular game zone is usually limited. 53 InaMMOG-typeapplicationsuserscommandtheiravatarstomovearoundtheshared virtual world. The area of interest (AoI) of a user is usually de¯ned as the total space around the user in this shared virtual world where interaction, either between users or between the user and the world, is likely to happen. Much research (e.g. [57, 42]) has focused on AoI to explore the temporal and spatial locality for interaction among users in a large-scale system. The fundamental conjecture can be stated as follows: users are morelikelytointeractwitheachotherwhentheirvirtuallocationsarecloseinthevirtual world. We would like to extend this observation to streaming services in MMOG type applications, namely that users who share their AoIs are also more likely to stream from each other. Existing AoI management designs for MMOGs usually take advantage of the locality of interest of users. Di®erenttypesofdataareexchangedanddeliveredinaMMOGandeachtypeofdata usually has its own bandwidth requirement, delay constraint, etc. For example: game state update messages from a user are usually wrapped in very small discrete packets. Thosepacketsneedtobedeliveredtotheotherparties, eithertheserverorothera®ected users, in a very short amount of time (otherwise other users will notice glitches in the game). Usually the packet is designed to carry enough information to be self-describing and independent of other packets so that the e®ect of losing of a packet is minimized. Technologies such as dead-reckoning can be used on this type of message to \predict" the future. Lastly, the bandwidth consumption of this type of message is relatively low. Live audio or video streams are another type of data users exchange in a MMOG. Those data are usually delivered in a steam of packets for which the sequence is important, and packets are usually dependent on other packets with adjacent sequence numbers. This 54 type of data usually consumes more bandwidth and it is very hard to predict the content ofpacketsinthefuture(therearelossconcealmenttechniqueswhichinterpolatelostaudio and video packets). Depending on the scenario of streaming the delay requirement could be high, e.g., in a voice conversation to coordinate an attack on the opposite team, or relativelylow,e.g.,listeningtoamusicstream. Inthispaper,wewillfocusourdiscussion on designing a scalable platform for streaming media data. Thedesignofastreamingplatformcanalsovarydependingonthestreamingscenario and the data distribution plan, e.g., one-to-many or many-to-many [55]. Without loss of generality, in this paper we will further limit our discussion to low latency location- adaptive audio streaming services in MMOG type applications. Our goal is to design a P2P based Interactive Audio Streaming Platform with following features: ² ImmersiveAudio: Audiostreamsfromdi®erentsourcesarerenderedaccordingto the spatial and directional relationship between receiver and senders in the virtual world to create an immersive audio experience. For example: the voice from a user with her avatar positioned in the back of the current user is rendered in a way that it sounds like a voice from behind the user in the real world. ² Low Latency: The end-to-end delay from the source (sender) to the receiver needs to be minimized so that meaningful interaction can happen between them. The delay performance of an overlay multicast tree is determined by many factors, e.g., link quality, topology con¯guration, bandwidth capacity, etc. We will show in the next section that the problem we are considering to optimize is actually NP-complete [45]. 55 ² Scalable: In general, stream data consumes more bandwidth compared with other types of game tra±c. Our P2P based design distributes the streaming load to all participating nodes to enable game tra±c better access to the central server. Consequently, our distributed design is not only scalable in itself, but it also makes the existing MMOG type applications more scalable. EvenaftermanyP2Pbaseddesignshavebeenproposedinthepastfewyearsforlarge scale applications, there are still many challenges left that make a pure P2P based design impractical for commercial use. One problem of using a P2P based design for an MMOG is that the game state is not under the control of the game publisher and this make it easier for hackers to modify the game state for their own bene¯ts. This unaccountability makes it hard for a P2P based MMOG, even though technically practical, to become commercially successful. The research motivation behind our design is not to replace existing centralized architectures that have a proven track-record to support massive numbers of concurrent users. Instead we are proposing a P2P based streaming system that can be easily integrated into existing systems. We hope that by separating the bandwidth-hungry audio and video streams from the critical system control channel, we can keep the system as accountable as a centralized design, but at the same time, as scalable as a distributed system. Almost all existing 3D audio processing requires that the audio from di®erent sources is kept separate until rendered. There are algorithms to re-generate a 3D immersive audio from only one or two audio input channels, but the results are arti¯cial and cannot reproducetheoriginalspatialcharacteristics. Becausethenetworkcapacityforeachnode 56 in the P2P network to receive and deliver data is limited and heterogenous, a challenging problem is how to fully utilize the limited and heterogenous network resources to deliver many separate audio streams and at the same time, reduce the overlay network delay. In this paper, we present our P2P based audio streaming platform called Dynamic Streaming Mesh (DSM). In our design, we ¯rst build a mesh network among all par- ticipating users; then we build an overlay multicast tree on top of this mesh to deliver multiple audio streams. Our mesh approach is similar to designs proposed in [52, 26], but we augment the design with several important features that signi¯cantly increase the performance. The additional features we brought into our design include the following. DSM dynamically optimizes the mesh tree when users change their locations in the vir- tual world, i.e., we optimize the streaming performance based on AoI; DSM is designed tobecomplementarytoexistingandfuturegamecontrolmechanismandtheinformation requiredfromthegamecontrolgridisminimized{thismakesDSMaveryeasyadd-onto existinggames; DSMisthe¯rstP2P-baseddesigntargetedtodeliver3Daudiostreaming in MMOGs; DSM models the network capacity with a degree limit to realistically re°ect the in-coming and out-going bandwidth of the access link { this makes DSM a practical design for current overlay streaming. We organize the rest of the paper as follows: Section 2 generalizes the problem we are going to discuss and the basic network models we are going to use in our paper. The hypothesis we raised during the discussion and a heuristic solution to the problem is presented at the end of the this section. Section 3 introduces our proposed streaming systems called Dynamic Streaming Mesh (DSM) and discusses in details how the system workstodeliveradistributed3Daudiostreamingservicetousers. Section4evaluatesthe 57 Term De¯nition Units G Application level complete graph containing all nodes, G=(V;E) V Set of all users online,jVj is the total number of all users E Set of edges that connect all nodes in V, E =V £V A s Overlay audio streaming tree rooted at s, A s =(V a s ;E a s ) V a s Set of nodes receiving audio stream from s, V a s µV and E a s µE E a s The collection of all overlay links in streaming tree rooted at s d (i;j) The direct unicast network delay between host i and host j ms [g] (i;j) End-to-end overlay delay from host i to host j alone the overlay link in a overlay network g ms M Overlay mesh network,M=(V;E m ) E m Collection of all links in overlay mesh network M and E m µE B Overlay backbone tree ,B=(V;E b ) E b Collection of all links in overlay backbone tree B and E b µE N Total number of users online, N =jVj K i Degree limit at node i in simulation D ¯ i The Euclidean distance from node i to pointh0;0i in the virtual world D j i The Euclidean distance between node i and j in the virtual world N m i Neighbor nodes of node i in the DSM mesh network p b i Parents node of node i in the backbone tree P i Neighbor nodes of node i in mesh network with smaller D ¯ value than D ¯ i R Set of current root nodes in backbone tree Table 4.1: List of terms used in this paper and their respective de¯nitions. performance of our proposed system by comparing it to some exiting leading solutions. Section 5 describes a pioneering application we built called GameTalk which uses DSM torenderingaP2Pbased3Daudiochatenvironments. Section6summarizessomeofthe related work we surveyed and Section 7 wraps up the paper with conclusion and possible future research directions. 4.2 Problem Statement In this paper, we are proposing our location-adaptive audio streaming design called Dy- namic Streaming Mesh(DSM). As we mentioned before DSM is an P2P based overlay 58 multicast system, which usually carries its own characters comparing to server-based design or IP layer multicast: ² Link Stress:As data are delivered over unicast tunnels from one end system to another end system, redundant tra±c and prolonged end-to-end latency are in- evitable.This lead to link stress. ² Limitedtopologyinformation: Unlikenetworkrouters,endsystemshaveeither limited or no underlying topology information,which is the key to building e±cient overlays. ² Capacity constraint and heterogeneity: End systems often have limited pro- cessingpowerandavailablebandwidth,andthisimposesaconstraintonthebranch- ingdegree(orfan-out)inthedeliverystructure. Theproblemisfurthercomplicated by the heterogeneous capabilities of these systems ² Robustness: End systems are more susceptible to problems like system failure. In addition, an ALM session is highly dynamic as it is the end systems which may join/leave the group at will, that directly form the overlay. From the system point of view, it is a many-to-many streaming problem, as every player in the virtual world could be the source of the audio stream and these audio streams need to be delivered, with a time limit, to all receivers. Since there is no server to relay the stream and the network capacity of the source node is limited, when the number of the receivers is too big, a mulicast tree need to be constructed to relay the audiostreamfromsourcetoallreceivers. Themulticasttreeneedtobeorganizedsothat 59 0 1 20 2 25 3 20 4 22 5 32 6 12 7 37 8 29 9 43 14 10 7 11 38 13 28 12 19 16 16 32 5 7 14 20 15 8 9 20 78 17 43 13 18 8 19 20 37 Figure 4.2: Sample MST Tree while forwarding the stream from one node to the other, the degree constraint at each node is not violated. Before get into the details of the design, we ¯rst discuss several issues that related to our design. 4.2.1 Network Model The physical network consists of routers connected by links. The end systems are con- nected to this network at di®erent points through access links. The overlay network is modeled as a complete graph G = (V;E), where V is the set ofverticesandE =V£V isthesetofedges. Eachvertexin V representsanendsystem. An edge,hi;ji in E corresponds to the unicast path from i to j in the physical topology. 60 The delay of edge hi;ji, denoted as d (i;j) , is the end-to-end delay from i to j via the physical topology. Consider a session for which the rate for a single audio stream is T units per second. We ¯rst assume that the capacity of any incoming or outgoing access link is no less than T. Based on the results from Internet measurement [10], we further assume that the links in the core networks are over-provisioned, thus the bottleneck is at the access link. Hence, given the out-going access capability of a node i as c out i and in-coming access capacity as c in i , we can calculate is out-degree bound K out i and in-degree bound K in i as bc out i =Tc andbc in i =Tc respectively. 4.2.2 Virtual World Model We model the virtual world as a 2D place and each player uses a hx;yi coordinate to representthelocationintheworld. Thedistancebetweentwoplayersisde¯nedasthethe Euclideandistancebetweenthetwopoints. Thissimpli¯cationofthevirtualworldmodel is just to simplify the calculation without losing generality. Since our design minimizes the dependency on the virtual world model, our algorithm can be easily extended to spaces with 3D coordinate readings, e.g., in the format ofhx;y;zi. Alsoourvirtualworldisalimitedspaceshapedasasquare. Ifwede¯nethemaximum reading of coordinates in this world as X MAX and Y MAX respectively, the coordinates readingsofnorth-westcorneroftheworldish0;0i,thenorth-eastcornerishX MAX ;0iand southeast corner and south west corner arehX MAX ;Y MAX i andh0;Y MAX i respectively. 61 Table 4.2: Audio Streaming Scenarios One-to-One One-to-Many Many-to-Many Immersive(3D) Sound Rendering required N.A. N.A. Ambient sound Immersive(3D) Sound Rendering Not required VoIP Lectures, An- nouncements, Music Concerts Muliti-party Con- ferences 4.2.3 Audio Streaming Scenarios There are several audio streaming scenarios in a multi-user networked virtual environ- ment. Depending on the number of senders and receivers at the same time, streaming scenarios can be grouped into one-to-one, one-to-many and many-to-many. One-to-one is usually considered to be a private chat and the delay between the two conversation parties need to be low to keep the interaction smooth. One-to-many streaming usually happensduringanannouncementtypescenario, e.g, aspeakergivingaspeechorasinger performs a solo a group of audiences. Many-to-many streaming scenarios are usually happening during intensive collaborations where everybody has interest to interact with everybody else, e.g., voice streaming among a group which is mounting an attack on the opposite team in a multi-player ¯rst person shooting game. Ambient sound is another example of many-to-many streaming. Streaming scenarios can also be categorized de- pending on if a 3D immersive rending is desired. For example, a VoIP conference usually doesn't want the voice to be rendered with spatial features since listener want the voice message being rendered as clear as possible with minimum distortion. Ambient sound, on the contrary, has to be rendered with spatial information. For example, players in the ¯rst person shooting game can quickly tell the direction of enemy based on the sound of 62 sound of bullets coming from the opposite party. Table 4.2 lists the streaming scenarios in a networked virtual environment. DSM is designed to provide location-based audio streaming,which could be one-to- one, one-to-many or many-to-many. Since many-to-many is the most demanding of the three, we will focus our discussion on many-to-many scenarios. DSM can also support 3D audio streaming. But since the 3D audio requirements are usually every application speci¯c,DSMisdesignedtobea3D-Audioenabledstreamingplatformanditisuptothe application to decide whether and how to use this feature. There are many technologies existing to render 3D audio in an immersive virtual environment [38]. One signi¯cant additional requirement posed by 3D audio rendering engine is that the original streams must be keep separate all the way until before insertion into the mixing module of the rendering engine.This means that the streaming system cannot mix the stream in the middle without losing the spatial features of those streams that get merged together. From a network point of view, separate streams means more bandwidth consumption in the streaming platform and thus DSM is carefully designed to consider the bandwidth requirements of those 3D streams and to accommodate as many distinguished streams as possible by exploring the residual bandwidth capacity at each end host. There are many rendering engines available today. For example, AudiolLib from CreativeLabisanopensource3Drenderingenginewhichcanmixstreamsfrommultiple sources and render 360 immersive sound using two audio channels. But the mixing and rendering algorithm is out of the scope of the research topic of this paper and we will focus on how to build a location-adaptive ,scalable and low latency streaming platform that could be used for both 3D and none-3D streaming scenarios. 63 4.2.4 Problem De¯nition Let us discuss in detail the many-to-many streaming scenario outlined above. First we start with a special case of a many-to-many streaming scenario in which at any given moment, only one node is sending a stream to other nodes. If we denote the total number of nodes with N in the overlay network and each overlay node i is associated with a degree limit K i (2 · K i < N ¡1) then the optimum solution in the many-to- many scenario (N-to-N in this case) can be described as ¯nding a spanning tree A with minimum maximum delay in A such that degree constraint K i is satis¯ed at each node i. This problem can be formally de¯ned as a minimum-diameter, degree-bounded spanning tree (MDDBST): Minimum-diameter, degree-bounded spanning tree(MDDBST) Given an undirected complete graph G = (V,E), a degree bound K i for each vertex i2 V and a delay d (e) for each edge e2 E. Find a spanning tree A of G such that for each i2 A, degree of i satis¯es d A (i)· K i and the diameter of A, which is the sum of delays along the longest simple path in A, is minimized. For an interactive application, there is usually a delay upper bound to be satis¯ed. ThisadditionalrequirementaddsadiameterboundLtotheMDDBSTproblem. Itwas provedin[45]thatthecomplexityofadiameter-boundedMDDBSTisNP-completefor degrees 2·K <N¡1. In addition, related research reported in [41, 22] proves that the minimum diameter, minimum spanning tree and minimum maximum degree, minimum spanning tree are bothNP-complete problems. 64 Ingeneral streamingscenarios, therewillbe more thanjustonenodesendingstreams across the overlay tree A. We extend the above special case to the general case by allowing multiple nodes to send multiple streams to other nodes at the same time. We assumethatthestartingtimeofthosestreamsarediscreteandformatimesequence T = t 0 ;t 1 ;:::;t n . We call this instance the Multiple-stream, discrete-start, diameter- bounded MDDBST problem, also referred to as n-MDDBST for simplicity. It is straight-forward to prove that n-MDDBST is also NP-complete. Given a candidate solution and the start time sequence T we can verify in polynomial time if the solution satis¯es the diameter and degree constraints. Then we can stack the current streaming trees on top of each other in the order of the start sequence in T and verify in polynomial time whether the degree constraints are satis¯ed. 4.2.5 Heuristic Solution We developed our heuristic solution to n-MDDBST by using a mesh-based approach similar to MeshTree algorithms [52]. But we augmented their algorithms by dynamically adapt the overlay mesh to re°ect the location relationship between nodes in the virtual world. Our design, which is called DSM, clusters nodes in the overlay mesh if they are alsoclosetoeachotherinthevirtualworld. Thisisbasedontheideathatinanetworked virtual environment, usersare morelikelyto interactwith eachother when theyareclose and can "see" each other in the virtual settings. ForallnodesV online,wede¯nethecompletegraphthatconnectsthemasG=(V;E). DSM builds three types of subgraph of G: a mesh network and a backbone Tree among all nodes, a streaming Tree for each of the nodes that is sending streams to its receivers. 65 Mesh network is de¯ned as M = (V;E m ) where E m is the collection of all links in overlay mesh network. Backbone tree is de¯ned asB=(V;E b ) whereE b is thecollection of all links in the backbone tree. Audio Streaming Tree rooted at node s is denoted as A s = (V a s ;E a s ) where V a s is all the vertices that receiving stream from s and E a s is the streaming overlay network connecting s to every nodes in V a s . More details of DSM design is discussed in the next chapter. 4.3 The Design In a MMOG the virtual world is usually divided into many smaller areas inside which share similar geometric features or game scenarios and these areas are also called game zones. Dividing the whole world into many smaller game zones is a successful method thatgamedesignersuseswidelytobalancetheloadofagameinvolvingamassivenumber of players online. Even though the interaction between players from di®erent game zones is possible, it is unlikely that an immersive audio streaming function is needed for them since they are literally in di®erent parts of the world. So DSM is designed to support streaming groups of the size of number of players per game zone, which is usually in the range of hundreds even for a MMOG with tens of thousands players online. The streaming platform design of DSM is composed of three subcomponents: ² MeshNetwork: Ameshnetworkisbuiltusingoverlaylinksconnectingnodestoeach other. Eachnodeiinthemeshnetworkwillkeepasetof"neighbor"nodes,denoted as N m i . Node i will send out periodical information to every node in N m i to update their information, such as current coordinates. The unique design of DSM is, this 66 mesh network is updated dynamically based on the coordinates of the nodes in the virtual world, instead of based on the physical network topology. From the systempointofview,themeshnetworkisamessageexchangeframeworktoprovide topology information and candidate nodes for streaming operations. ² Backbone Tree: A backbone tree is constructed using the topology and node infor- mation provided by the mesh network. This backbone tree is constructed using our heuristic solution to build a minimum spanning tree (MST) across all nodes so that thedegreelimitaeachnodeisrespectedandthemaximumend-to-enddelayismin- imized. The backbone tree is updated when the mesh network is changed because of the movement of the nodes in the virtual world. A node can reach another node by following the links in the backbone tree. Because it is loop free and spans to all nodes, backbone tree is used as a global messaging platform to exchange messages among nodes. ² Audio Stream Tree: In order to render an immersive audio environment, audio streamsfromdi®erentoriginsneedtobekeptseparateuntiltheyreachtheendhost where the sound packets are processed. For each node s in the mesh network who hasasetofreceivers,anaudiostreamtreeisconstructedtoconnectstoallreceivers onthe°ythroughoverlaylinks. Itiseasytoseetherewillbemultipleaudiostream treesexistinginparallelandDSMneedtoadjusttheresourceusagetoaccommodate streamsonthosetrees. Aaudiostreamtree,eventhoughisconstructedbyusingthe backbone tree as a reference, will use links not in the backbone tree but existing in the mesh network to construct the delivery path in the tree, if the algorithm 67 determines that using those links will either help to reduce the delay or avoid violating the bandwidth limit on the nodes in the backbone tree. To summarize the relationship of these three subcomponents: the mesh network provides the basic overlay network topology and a distributed message exchange platform to construct and maintain the streaming path, links in the mesh network are picked as candidates to ¯x network problems when there is a node failure; A backbone tree is constructed by picking links from the mesh network and spanned crosstoallnodes,thebackbonetreeiscontinuouslyimprovedtominimizethemaxi- mumend-to-enddelayamongnodesinthetree,italsoupdatesthetreeperiodically toclusternodes"close"toeachotheras"neighbors"inthetree; Byusingtheback- bonetreeasareference,anindividualstreamtreeisconstructedbyeachnodesand allitsreceivers,therecouldbemultipledi®erentstreamtreespassingrunthrougha node and the network capacity limit is honored by the tree constructing algorithm. 4.3.1 Mesh Protocol Themeshnetworkisconstructedbycreatinglinksbetweennodes. Twonodeswithalink between them will consider each other as neighbor nodes. The set of neighbor nodes of node i is denoted as N m i . Refreshment messages are exchanged between neighbor nodes periodically to update information such as the status of the node, delay between them and the coordinates etc. 68 4.3.1.1 Join Mesh : A node must ¯rst join to the mesh network before doing any other streaming work. A join is started by submiting the JOIN request to a well-known register server called Rendezvous Point (RP). The RP, after receiving the request, will reply with a list of candidate nodes that might be already in the mesh network by looking into its local depositoryoftheonlinenodelist. Thesizeofthelistis¯xedandthecontentisrandomized to balance the initial join load. It is not guaranteed that the nodes referred in the list are actually online so the JOIN might fail if no node online responds within certain time. In this case, the node will timeout the old join process and try to get a new list of candidate nodes again from the RP server. The list of candidate nodes can also provided manually by other means. Algorithm 5 JoinMesh Require: RP is online 1: LÃ candidate nodes from RP 2: N m Ã neighbor nodesÃ; 3: KÃ degree limit of current node 4: if L6=; then 5: for all c2L do 6: send join request to c 7: if c accept join then 8: N m ÃN m [c 9: if jN m j¸K then 10: exit 11: end if 12: else if c is online but decline join then 13: L0Ã new candidate nodes referred by c 14: LÃL[L0 15: else 16: remove c from L 17: end if 18: end for 19: end if 69 4.3.1.2 Mesh Maintenance : The maintenance of the mesh is actually a two part job: ² Mesh Connectivity: DSMtriestoputalltheonlinenodesinaconnectedcomponent by 1) connecting each node to at least three neighbors (as far as the participants number is bigger than three); 2) periodically sending a refreshment message by the node with the smallest nodeid ( a unique id number assigned by the system to each node)toallothernodes. Ifanodedidnotreceivethismessageforacertainperiodof time, then there might be a separated group, in this case, an algorithm is triggered to merge the two groups. ² Update the Connections: As we mentioned before, one unique feature of our DSM designisthattheoverlaynetworkisupdatedaccordingtothevirtualcoordinatesof the nodes by maximize the number of neighbor nodes that actually are "neighbors" in the virtual world. In the case that there is no such coordinates exist or provided, our algorithm will also run as ¯ne with the update module disabled. As shown in Algorithm 6, the mesh update operation is triggered when there is information of a new node c received at local node. The local node i will check its current neighbors and ¯nd the a node t with largest Euclidean distance in the virtual world. If node t satisfy certain criteria, e.g., link (t;i) is not the only link between t and the mesh network and D c i < D t i , the link (t;i) is removed and new link (c;i) is inserted to the mesh network. If t fails to meet the removal criteria, anothernodeispickedandtheupdateroutingisranagain. Algorithm6and7have the detailed procedures of mesh maintenance. 70 The mesh maintenance algorithm can be described in details in following routine: Algorithm 6 UpdateMesh 1: N m Ã neighbor nodes 2: cÃ new node 3: iÃ current node 4: KÃ degree limit at current node 5: if jN m j<K then 6: N m ÃN m [c 7: else 8: P Ãfk2N m jD ¯ k ·D ¯ i g 9: tÃfk2N m jmaxD k i g 10: if t2P andjPj=1 then 11: tÃfk2N m ;k = 2P jmaxD k i g 12: if D c i <D t i then 13: remove t from N m 14: N m ÃN m [c 15: end if 16: else if (D ¯ c <D ¯ t andjPj<1) or D c i <D t i then 17: remove t from N m 18: N m ÃN m [c 19: end if 20: end if Algorithm 6 is called with a new candidate node c as input. Usually this node c is received from the network through information exchange. But in the case if local node i ¯nds out that every mesh neighbor node k satis¯es equation D ¯ k ¸D ¯ i and i is not root, a node c satis¯es equation D ¯ c < D ¯ i is selected from RP and mesh update function is called. Algorithm 7 lists the steps in CheckMesh routine. Algorithm 7 CheckMesh 1: N m Ã neighbor nodes in mesh network 2: iÃ current node 3: RÃ current root 4: P Ãfc2N m jD ¯ c ·D ¯ i g 5: if P =; and i6=R then 6: LÃ random nodes from RP 7: cÃfc2LjD ¯ c =min k2L D ¯ k g 8: if D ¯ c <D ¯ i then 9: UpdateMesh(c) 10: end if 11: end if 71 4.3.1.3 Leave Mesh Just notify the above backbone tree and stream tree to ¯nd an alternative path and then disconnect from the mesh. 4.3.2 Backbone Tree Protocol The backbone tree is built using a distributed algorithm and spans across all users. The treeedgesareperiodicallymodi¯edusingtheincrementalalgorithmtore°ectthechanges in the links of the mesh tree. As the underneath mesh links are updated to re°ect the geometric location of users in the virtual world, the backbone tree is also constructed using the virtual-location aware algorithm. Backbone tree has two major functions. First, it is a loosely maintained loop-free message distribution network to °ood message to all users. Second, the links in the backbone tree is used as a reference to construct a low-latency streaming tree from a source node s to all its stream receivers. Since backbone tree is loosely maintained, it is possible that the tree is broken because of heavy update or node failure. 4.3.2.1 Initialization For two nodes i;j, we denote d (i;j) as the unicast overlay delay between i and j and D ¯ i , D ¯ j are the Euclidean distance from between node i andj to the originh0;0i respectively in the virtual game space. After a user connects to the mesh, it will pick a node j2N m i as parent node p b i and another node t as backup parent node p b i 0 . The selection criteria for a parent candidate node c is that node c has smaller distance to the origin h0;0i in the virtual world, hence D ¯ c < D ¯ i . The reasoning behind this condition is that if every 72 node select a node with smaller D ¯ as the parent, the result network is guaranteed to be loop-free. Among all the parent candidate nodes, the node j with the smallest delay d (i;j is selected as the parent. This is a heuristic shortest path tree (HSPT) solution that reported to show good delay performance. Following is the details of parent selection function: p b i ´fj2N m i jd (i;j) = min k2N m i ;D ¯ i ¸D ¯ k d (i;k) g (4.1) Similarly we select a backup parent p b i 0 using following equation: p b i 0´fj2N m i ;j6=p b i jd (i;j) = min k2N m i ;D ¯ i ¸D ¯ k d (i;k) g (4.2) The degree constraint K j at node j is respected when constructing the mesh links, so when we pick one of the mesh links to connect node i and j, the degree limit will not be violated in the backbone tree. 4.3.2.2 Backbone Tree Maintenance ² LinkUpdate: BackbonetreemaintenanceroutingwillperiodicallyexecuteAlgorithm8 to ¯nd the new parent node p b i and backup parent p b i 0 using Equation 4:1. ² Root Node: As we discussed before, the backbone tree is maintained to be loop free when every node connects to a parent node with smaller D ¯ value, so the root node R is by de¯nition the node with smallest D ¯ ( Equation 4.3). But if there 73 Algorithm 8 UpdateBackbone 1: N m Ã neighbor nodes in mesh network 2: iÃ current node 3: RÃ current root 4: P Ãfc2N m jD ¯ c ·D ¯ i g 5: if P 6=; then 6: if p b i = 2P then 7: tÃft2P jd (i;t) =min k2P d (i;k) g 8: p b i Ãt 9: end if 10: remove p b i from P 11: tÃft2P jd (i;t) =min k2P d (i;k) g 12: p b i 0Ãt 13: else if i6=R then 14: LÃ root nodes from RP 15: tÃft2LjD ¯ t =min k2L D ¯ k g 16: if D ¯ i <D ¯ t then 17: RÃi 18: for all j2L do 19: notify j of new root 20: end for 21: else 22: cÃfc2V jD ¯ c <D ¯ i g 23: UpdateMesh(i;c) 24: end if 25: end if 74 are two root nodes, the backbone tree is broken. If there is a tie between two root nodes, the one with smaller nodeid is selected as the new root,otherwise the one with smaller D ¯ is chosen as the new root. R´fi2N j D ¯ i =min k2N D ¯ k g (4.3) Since the backbone tree is loosely maintained and it could be broken into a forest, the root node is responsible to send out root announcement message periodically to all nodes through the backbone tree links. If the tree is divided into a forest, then either there will be multiple nodes declaring themselves to be the root or nodes in a subtree without a root node will stop receiving root announcement message and will ¯nally time out. In the case with multiple roots, the root nodes will resolve the dispute by information provided by RP, that is, the node r with the minimum D ¯ is kept as the root and other roots were provided with a list of nodes from RP to re-establish the mesh network. ² Tree Improvement: The backbone links are chosen from the links in the underlying mesh network. However, the mesh network algorithm is really designed to be very localized and the number of neighbor nodes in mesh network is very limited. Even thoughthisdesigncanavoid°oodingmessagestoallnodesonline,itlacksamethod toexplorenewnodesandlinksthatcanimprovetheoverlayperformance. Backbone treeisdesignedtoprovidea"global"loop-freemessageexchangechannelthatallows a node to explore "new" nodes online. 75 Considering the size of all nodes in the session, it is clearly not a good idea to °ush out messages to all other nodes periodically. Research done in the past shown that withproperparametertuning,°oodingcanalsobeane±cientwaytomakemessage announcement [6]. We designed our probability-based message forwarding mecha- nism so that refreshment messages from a node, given enough time, can eventually reach all other nodes. By introducing a drop probability ¹ i for messages from node i, an intermediate node j will randomly drop the message without forwarding it to other neighbor nodes in the backbone tree. This design voids a complete °ooding inmessageforwarding. Algorithm9describestheroutineofprobabilityforwarding mechanism. Algorithm 9 MessageForward 1: iÃ current node 2: N b i Ã neighbor nodes in backbone tree 3: sÃ node receives the message 4: LÃN b i ©s 5: ¹ i Ã probability to drop the message, ¹ i 2[0;1] 6: kÃ rand() 7: if k >¹ i then 8: for all d2L do 9: forward message to d 10: end for 11: end if 4.3.3 Audio Stream Tree A key distinguished feature for 3D audio streaming, comparing to other audio streaming, is that audio streams need to be kept separate until reaching the listener where the audio is mixed and rendered. Let emphasize again here that the audio scenarios we are talking about here is 3D interactive audio streaming. Because of the limitation of human brain's 76 abilitytoprocessparallelinformation, thenumberofindividualstreamsthataparticular receiver can receive and process is limited for interactive 3D audio streaming. Let'sexamtheproblemfurtherindetails. Foranystream,thereisasendersandaset of receiver ¨. From the view of the sender s, there are two extreme cases: j¨j=1, where it is a one-to-one streaming or j¨j =jVj, where all nodes online are receivers. Example for the ¯rst case can be a private chat between two friends; a speech given by a famous player in a virtual online gathering is a good example for the later case. While both cases are ordinary scenarios in an MMOG, for the later case it is very unlikely that the out-going capacity K out s at node s is big enough to support all nodes V when the group sizejVj is large. In such scenario an overlay streaming tree rooted at node s is required. On the other hand, form the view of a receiver r, it can stream simultaneously from a set of senders S. We want to argue here that in an MMOG, the candidate nodes for node r toselectisinlargedeterminedbytherangeofdistancesinthevirtualworld. Theprocess to select members of S for node r can be either automatic or manual. The game can de¯ne a stream policy that automatically stream audio from nodes within certain range from node r; user controlling r can also manually pick up other users that they see in the world to stream from. The size of S, denoted by jSj, is bounded by the in-coming network bandwidth limit K in r at node r. 4.3.3.1 Stream Setup Given a source node s and its receivers ¨, a new node r start the process of setup stream by sending out the request of stream to source node s. It is possible that there is no 77 direct link in E b between r and s, in this case, the request is forwarded by nodes in the backbone tree (Algorithm 10). Algorithm 10 SendStreamRequest 1: iÃ current local node 2: N m Ã neighbor nodes of i in mesh network 3: sÃ source node 4: oÃ the neighbor node of i that forwarded message from s 5: if s = 2N m then 6: send stream request to o 7: else 8: send stream request to s 9: end if Upon receiving a stream request, node s will ¯rst check to see if it has the stream in local stream list, either directly or forwarded. Then node s will check if the out- going degree limit is violated if add one more stream. If degree limit is reached, s will build a refer list with all descendent nodes in stream tree and send a refer message to r (Algorithm 11). Algorithm 11 OnStreamRequest 1: rÃ new receiver node 2: sÃ source node of the stream 3: tÃ local node 4: N a s Ã neighbor nodes in audio stream tree from source s 5: K out t Ã out-going degree limit at local node t 6: kÃ number of distinguish out-going stream at t 7: if k <K out t then 8: send OK message to r 9: N a s Ã N a s [r 10: kÃk+1 11: else 12: REFERÃN a s 13: send REFER message to r 14: end if Finally, inAlgorithm12, wedescribethestepstotakewhenreceiveastreamrespond from s. 78 Algorithm 12 OnStreamRespond 1: rÃ local node (receiver) 2: sÃ source node of the stream 3: tÃ node send respond message 4: N m Ã neighbor nodes in mesh network 5: N a s Ã neighbor nodes in audio stream tree from source s 6: kÃ number of distinguish in-coming stream at local node r 7: mÃ respond message 8: if m is OK message then 9: N a s ÃN a s [t 10: kÃk+1 11: else if m is REFER message then 12: LÃ nodes referred in m 13: for all p2L do 14: send stream request to p for stream from s 15: end for 16: end if 17: //if mesh need to be updated 18: L0ÃL\N m 19: if L0=; then 20: cÃfcjd (c;r) =min j2L d(j;r)g 21: UpdateMesh(c) 22: end if 4.3.3.2 Stream Termination As the stream capacity K out i and K in i is limited at node i, we need to closely monitoring the usage of this resource and free the resource when it is not longer needed. The design of policy to terminate the stream is highly application dependent and we will leave the design details to the game developers. But similar to the stream setup process, stream termination can be an automatic process, manual input from user or the combination of these two. When a node i in the multicast tree A s rooted at node s leave the tree, either vol- untarily or involuntarily, the termination of stream S i will a®ect the descendant nodes of i. The streaming tree need to be recon¯gured to continue the streaming service to reach all descendant nodes in of i in A s . Node i announces its departure by sending out 79 leave message to all neighbor nodes in N a s . The immediate descendants nodes of i, when receivingtheleavemessage, willreestablishthestreamfromapre-selectedbackupparent p a i 0. If the backup parent failed to accept the stream request, a rejoin process will start to re-establish the stream from the source node s. Since the steps to rejoin the stream tree is very similar to Algorithm 10, we will skip the details of it. Audio stream tree is very sensitive to changes and errors in the streaming topology. Every time the tree is recon¯gured, either because of the departure of a node or to optimize the stream topology, some audio packets will be lost during that process. The consequence of such loss is very subjective and depends on many other facts and thus the side e®ect could range from unnoticeable skips to missing of some key words in a sentence. Even the policy of when and how to start and terminate an audio stream is designed by thegame developer, we suggest thatthose requestfor stream recon¯guration shall be minimized. 4.3.4 Error Handling As a distributed system running complicated algorithms, errors could happen in many di®erent scenarios that caused by the highly dynamic nature of the system. We carefully designed our algorithms to reduce the chance of error, at the same time, we run "detect- and-¯x" type of routine at each node to continuously monitor possible errors in the system. The three major components of the overlay system, namely mesh network, backbone tree and streaming tree, show di®erent characteristics when it comes to error-tolerance 80 and type of errors. So we will discuss them in a bit more details in the following para- graphs: ² Mesh Network is the most fault-tolerant part of the tree. As it is just a mesh connecting a node to its neighbor nodes, a sudden failure of a neighbor nodes is not likely to create any problem to the mesh network. The only serious problem for the mesh network is when the nodes in the mesh network are separated into several subgroups and disconnected from each other. This error is detected by routine in Algorithm 7, which constantly looking for nodes in the mesh network that has smaller D ¯ value, in the case that the local node hasn't found one so far. ² BackboneTreespansacrossallnodesandsinceitisatreestructure,wheneverything is right, it provides a loop-free overlay message delivering platform connecting any twonodesinthetree. Butalso duetothefactthat, at anygiventime, thereis only one overlay path from one node to another a failure of any node will lead to broken of path in the tree and triggers error handling procedure. We pre-select a node as backup parent (Algorithm 4:2) to reduce the time to ¯x the error when the parent fails. ² Stream Tree is very sensitive to errors because useful audio packets are streamed in the tree continuously. Errors in the stream tree can be easily detected. For all the audio packets sent from a source node s, the unique node id of s is stored in the header of the packets. And since the interval that a receiver r receives audio packets, usually in the range of milliseconds, is much higher than the refreshment rateinthemeshnetworkandbackbonetree,errorsinthedeliverpathofstreamtree 81 Table 4.3: Simulation Parameters List Name Symbol Values simulated Total Users N 10, 50, 100, 150, 200, 250 Degree Limit K 4, 8, 12, random Stream Group A s 6, 20 can be quickly detected by receivers. By utilizing the pre-selected backup parent in A s , a node can switch to the backup parents once error is detected. 4.4 Performance Evaluation We simulated our algorithms proposed in the DSM design and the result of performance is discussed in this section. We chose to use an Internet topology generator called Inet [28] to generate our physical network topology, which is reported to be able to generate more realistic Internet-like topology than other leading topology generators such as GT- ITM [13], BRITE [36] etc. Figure 4.3: A corner of the Simulated Internet with 6000 routers 82 4.4.1 Simulation Setup We ¯rst use Inet to generate 6000 router level nodes and the links connecting them as the physical network. A corner of the generated physical topology is shown in Fig. 4.3. Then we randomly pick N routers using a probability function with uniform distribution and attach one end host to each of these routers. These end hosts, together with the underneath routers connecting them, form a fully connected overlay network. One end host represents an overlay node in our discussion. We simulated our algorithm with N of value 10, 50, 100, 150, 200 and 250. These values are chosen to show the number of users online ranging from a "quiet" server to a popular server of current MMOG game system. For each end host i, a degree limit K i is chosen to re°ect the bandwidth capacity at this node. The simulation parameters used is also shown in Table. 4.3. Weranthesimulation1000iterationsforeachofaboveN values. Atthebeginningof each simulation, the avatar representing each overlay node was put randomly in a virtual world with area in size of 10000£ 10000. A di®erent rand seed was chosen for each iteration to simulate di®erent scenarios. During each simulation iteration, there were 100 simulationtimestepswhereDSMalgorithmswereexecuted, soforeachcon¯guration, we collect 100000 simulation results. Within each simulation time step, every overlay node changes its location in the virtual world by moving one step of distance 100. We use the popular random walk algorithm to decide the direction of the next step move for each node. As part of the future research, we plan to use algorithms proposed in paper [53] to simulate the player movement trace inside a FPS MMOG game. At the end of each simulationtimestep, performanceresultswerecalculatedandloggedforlaterevaluation. 83 1 1.2 1.4 1.6 1.8 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP(size=6) StaticMesh DSM 1 1.5 2 2.5 10 0 10 1 10 2 10 3 10 4 10 5 Distribution of RDP(size=6) Number of iterations RDP SPT StaticMesh DSM (a) 10 Users 1 2 3 4 5 6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP(size=6) StaticMesh DSM 1 2 3 4 5 6 10 0 10 1 10 2 10 3 10 4 10 5 Distribution of RDP(size=6) Number of iterations RDP SPT StaticMesh DSM (b) 100 Users 1 1.5 2 2.5 3 3.5 4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP(size=6) StaticMesh DSM 1 2 3 4 5 6 7 8 9 10 0 10 1 10 2 10 3 10 4 10 5 Distribution of RDP(size=6) Number of iterations RDP SPT StaticMesh DSM (c) 250 Users Figure 4.4: Distribution of RDP spt for streaming group size = 6 84 4.4.2 Performance Metrics To better evaluate the performance of DSM, we also simulated the performance of a mesh-based overlay network called MeshTree [52], which claims to out-perform many ex- iting overlay streaming designs. Unlike DSM, MeshTree algorithms don't dynamically update the mesh network based on the virtual location of a node and the overlay archi- tecture will eventually become stable and static if the members of the overlay network are stable. We refer to this type of ¯xed-goal mesh based designs as StaticMesh in our discussion. We will use the performance result of MeshTree to represent the performance of StaticMesh designs since those algorithms are relatively similar to each other. Two results are calculated to evaluate the delay performance. One is the total delay alone the overlay edges that connecting all nodes. It can be viewed as a general index to evaluatehowmuchend-to-enddelayonenodewillexperiencebyaveragewheninteracting with other nodes. The minimum spanning tree(MST) connecting all member nodes, referred also as MST for simplicity, is the optimal solution for this metric. We want to emphasize here that the total delay sum, in most cases, doesn't give the actual delay performance at a particular node, but rather a overall performance index in general case givencurrentnetworktopology. Inordertoevaluatethedelayperformanceataparticular node, we also calculate the average end-to-end delay inside the stream group. In a many- to-many stream group, every node is the source of a stream and the receiver of other streams at the same time. A stream from a node s is delivered to other receivers alone the overlay links and an overlay streaming tree is formed and rooted at the source node s. We denote V a s as all the nodes in a streaming tree rooted as s. The average delay 85 alone the overlay streaming paths from s to all receiver nodes is calculated and used to measure the delay performance at node s. In this end-to-end delay case, the shortest path tree(SPT) from node s to all other nodes in V a s in a complete graph connecting all nodes provides the optimal solution. Relative Delay Penalty (RDP) is used widely as the metric to measure delay perfor- mance in overlay streaming design. If we denote d (i;j) as the unicast delay between node i and j , [g] (s;j) as end-to-end delay from s to j alone the links in the overlay network g, then we de¯ne two measurements, RDP spt and RDP MST respectively as follows: RDP spt (g)= P j2V a s [g] (s;j) P j2V a s d (s;j) (4.4) If we denote E b as the collection of all overlay links in the backbone tree generated by DSM algorithm that connecting all nodes online (including but not only limit to those nodes in the audio stream tree A s ), and MST(G) as the minimum spanning tree connecting all nodes V online, we de¯ne the RDP MST in DSM as RDP MST (DSM); RDP MST (DSM)= P fd (i;j) j8(i;j)2E b g P fd (i;j) j8(i;j)2MST(G)g (4.5) 86 For performance comparison, we also de¯ne the RDP MST in the StaticMesh type overlay network M0 as RDP MST (M0), which can be calculated using the following equa- tion: RDP MST (M0)= P fd (i;j) j8(i;j)2MST(M0)g P fd (i;j) j8(i;j)2MST(G)g (4.6) 4.4.3 Performance In Stream Tree As we discussed before, our design tries to optimize the delay performance among nodes that are close to each other in the virtual environment. Unlike RDP mst , which provides aninsighttotheoverallperformanceamongallnodes, RDP spt isdecidedbywhichsource node to stream and who receives this stream. In our simulation, we log the stream tree performancebymonitoringtheRDP spt ofaparticularnodeateachsimulationtimestep. The detailed procedure is as follows: we ¯rst randomly pick up a node in the overlay network as the source s as the stream tree root, at the end of each simulation step, after each node online have made changes to it location, we chose Y nodes that are closest to s in terms of virtual distance. We then calculate the RDP spt from s to these Y nodes. The result value shows what is the delay s is expected to receive when s starts the interactive streaming with these neighbor nodes. We ran 1000 iteration of the simulation with 100 simulation steps inside each iteration. We simulate with Y equal to 6 and 20, for the purpose to show the performance in both sparsely populated and crowded virtual space. 87 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 DSM/StaticMesh Percentage Cumulative Distribution of DSM vs. StaticMesh (size=6) 10 Users 50 Users 100 Users 150 Users 250 Users 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 DSM/StaticMesh Percentage Cumulative Distribution of DSM vs. StaticMesh (size=20) 50 Users 100 Users 150 Users 250 Users (a) DSM vs. StaticMesh 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP (size=6) 10 Users 50 Users 100 Users 150 Users 250 Users 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP (size=20) 50 Users 100 Users 150 Users 250 Users (b) CDF Comparisonh 0 50 100 150 200 250 1 1.5 2 2.5 Cumulative Distribution (CDF) of RDP(size=6) Average SDP SPT Total Users StaticMesh DSM 50 100 150 200 250 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 Cumulative Distribution (CDF) of RDP(size=20) Average SDP SPT Total Users StaticMesh DSM (c) Average RDP Figure 4.5: Statistic of RDP spt distribution for streaming group size = 6 and 20 88 Let¯rstlookatthecaseswhen Y =6. Werunour simulationwithtotalusernumber of 10, 50, 100, 150, 200, 250. We calculated the RDP spt at each simulation step and because of the space limit, only some of the the results are shown in Fig. 4.4. Fig.4.4(a)showsthedistributionofRDP spt comparingtoStaticMeshwhentotalusers online is 10. We can see that when Y =6, DSM pronounces the same delay performance as the optimal solution (RDP spt =1) in approximately 80% of simulation cases while for StaticMesh, this number is about 25%. When the size of total users online increases,the performance for both DSM and StaticMesh decreases. For example, when the total user online is 250, DSM almost never generates the same topology as the optimal solution, howeverwecanalso¯ndthatinabout60%simulationcases,DSMdoesgeneratetopology with RDP · 1:5, which is usually used as the threshold to measure "very good" RDP results. Butregardlessoftheusernumber, DSMoutperformStaticMeshinallcomparing groups. This result clearly shows the value of DSM design: when using DSM to built the online stream network, if a user tries to interact with other users close to him, DSM will more likely to provide lower latency among these users comparing to stream over other overlay streaming network links. One interesting thing to investigate is the overall delay performance between DSM can StaticMesh. For this purpose, we introduce RDP c , which is calculated by RDP c = RDP MST (DSM)=RDP MST (M0). RDP c is calculated at each of the 100 0 000 simulation steps for all six group size con¯gurations. At a simulation step with RDP c < 1, DSM outperforms StaticMesh. The distribution of those 100 0 000 results for user group size of 10, 50, 100,150, 200 and 250, combined by stream group size of 6 and 20 are shown in Fig.4.5(a). Itisveryeasytoseethatforallgroupsizes, DSMoutperformsStaticMeshin 89 0 10 20 30 40 50 60 70 80 90 100 200 300 400 500 600 700 800 900 1000 Average Delay within Stream Nodes (size=6) Delay(ms) Simulation Time Ticks SPT StaticMesh DSM Figure 4.6: One simulation iteration (250 users) more than 90% of our simulated steps. More interestingly we can ¯nd the distributions of the RDP c are very similar for simulation cases with group size 50, 100, 150, 200 and 250. The distribution of RDP c with group size 10 shows that when the total number of users are small and close to the stream group size, the performance of StaticMesh is closer to DSM than those in other cases. Fig. 4.5(b) compares the distribution RDP spt in DSM with di®erent group sizes. The result shows a trend that the bigger the user group size, the worse the performance for DSM. We also calculated the average RDP spt of all simulated cases for both DSM and StaticMeshwithdi®erentgroupsizesandtheresultsareshowninFig.4.5(c). Datashown in Fig. 4.5(c) echoes above statement by showing the average delay inside stream groups increases alone with the size of total user numbers. However, Fig. 4.5(c) also shows that the growth rate of DSM is much lower comparing to StaticMesh and thus DSM is more 90 scalable. Theresultforstreamgroupsizeof6and20areshownandtheratiorelationship between DSM and StaticMesh is very similar with each other. Convergence is very important for a distributed algorithm. In Fig. 4.6 we show the performance result of each of the 100 simulation steps in this iteration. As we mentioned before, at each simulation step, 6 nodes that is closest to the stream source s is chosen andtheRDP spt iscalculatedforthisstreamgroup. Sinceeverynodesaremoverandomly at each time tick, the members of the stream group of s is also dynamically changing. For comparison purpose, Fig. 4.6 also shows the result at each step for the delay result of the same stream group nodes calculated by shortest path tree(SPT) algorithm and StaticMesh algorithm. We can see that at the beginning of the simulation, when nodes started to join the mesh,thedelayofDSMisveryhighevencomparingtoStaticMesh. Thisisbecausewhen a node initially join the mesh, DSM algorithm only randomly pick some candidates for joinpurpose,butthesenodesarenotguaranteedtobeclosetothenewnodeinthevirtual world. However, we can see that after running for a few time steps, DSM quickly adapt to the location relation ship among nodes in the virtual world and the average delay in the stream group is quickly reduced. We want to emphasize that during this adaptation process, the nodes in the stream group are constantly moving according to random walk algorithm but results show that DSM's dynamic mesh algorithm always quickly adapt to the movement of nodes and maintains good performance close to the optimal solution provided by SPT algorithm. On the other hand, the StaticMesh algorithms fails adapt to the dynamics of the streaming group and the delay is not reduced accordingly, and shows some random features. 91 1 1.5 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP MST Percentage Cumulative Distribution (CDF) of RDP StaticMesh DSM 1 1.5 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP MST Percentage Cumulative Distribution (CDF) of RDP StaticMesh DSM (a) 10 Users (b) 100 Users 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP MST Percentage Cumulative Distribution (CDF) of RDP StaticMesh DSM (c) 250 Users Figure 4.7: RDP MST distribution for di®erent group sizes 92 4.4.4 Performance In Backbone Tree The performance of the mesh that DSM built is measured by the RDP DSM MST among all online nodes. This value is independent of the streaming group size we picked in the previous section. We calculate the RDP DSM MST based on Eq. 4.5. To compare the performance, wealsocalculatetheRDP MST (StaticMesh)forStaticMeshusingequation in Eq. 4.6. Fig. 4.7 shows the delay performance in streaming group size =6 for total user as 10, 100 and 250. We can tell from the ¯gure that in our simulation, the StaticMesh always outperform DSM in terms of overlay delay among all nodes online. It is not very surprising since DSM and StaticMesh have di®erent performance goals. SDM is built to reduce the delay with nodes close to each other in the virtual environment. Fig. 4.8 shows the average SDP MST for DSM and StaticMesh in di®erent group sizes. We can see that even DSM is not built to optimize the delay among all nodes, when the group size is small, e.g., 50 to 150, in our simulation DSM still achieves "good" performance with RDP MST < 1:5. We can also ¯nd out that the growth rate of delay in DSM is in band with the growth rate in StaticMesh, which means when the group size grows, the performance ratio between DSM and StaticMesh remains stable. This ¯nding is also shown in Fig. 4.9, which plots the ratio of RDP MST of DSM to StaticMesh. In Fig. 4.9, with the only exception when the stream size, in this case 6, is close to the total number of users online, all other distributions are very similar to each other with di®erent group sizes. 93 0 50 100 150 200 250 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Cumulative Distribution (CDF) of RDP MST Average RDP MST Total Users StaticMesh DSM Figure 4.8: Average RDP MST for di®erent group sizes Fig. 4.10 shows how the delay performance in stream group changes when the group size increases. It is a common sense that the bigger the group size is and the worse the performance could be. As shown in Fig. 4.10, when the group size is 50 approximately 90% of the simulation cases in DSM pronounces "good" results (RDP spt <1:5). We can see that when the group size increases to 250, only about 35% simulation cases in DSM did the same job. 4.4.5 Bandwidth Impact Inourprevioussimulation,thedegreelimit,whichrepresentthebandwidthateachnode, is randomly set to value between 4 and 12. It is very desirable to know if we kept other parameters ¯xed, e.g, the number of total nodes online, how the degree limit will impact the performance. 94 0.5 1 1.5 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP c Percentage Cumulative Distribution (CDF) of DSM vs. StaticMesh 10 Users 50 Users 100 Users 150 Users 250 Users Figure 4.9: DSM vs. StaticMesh in RDP MST Werananothernewsetofsimulationwith1000iterationsforgroupsizeof10,50,100, 150, 200 and 250 respectively and for each group, we simulate the degree limit ¯xed to 4, 8, 12. The stream group size is set to 20 in this simulation set. As shown in Fig. 4.11, for the ¯xed user number, the bigger the degree limit, the better the performance in terms of average delay. The results are as expected. 4.5 Example Application Using DSM We are working on implementing a pioneer MMOG game called PartyPeer to provide an immersive audio streaming experience to multiple players in a social type virtual environment setting. DSM is used as the core streaming engine of the game and, for simplicity, the game control information is also transported in the mesh network. The ¯rst phase of PartyPeer development is completed already and we will discuss the details of current status and design of features of Partypeer in Section 5.3. 95 1 1.2 1.4 1.6 1.8 2 2.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP MST Percentage Cumulative Distribution (CDF) of RDP 10 Users 50 Users 100 Users 150 Users 250 Users Figure 4.10: Distribution of RDP MST in DSM for di®erent group sizes 96 1 1.5 2 2.5 3 3.5 4 4.5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP (50 Users) K=4 K=8 K=12 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP (150 Users) K=4 K=8 K=12 (a) 50 Users (b) 150 Users 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RDP SPT Percentage Cumulative Distribution (CDF) of RDP (250 Users) K=4 K=8 K=12 (c) 250 Users Figure 4.11: RDP spt distribution for di®erent degree limit (k=4,8,12) 97 Chapter 5 Applications In this chapter, we will present some applications that utilize ACTIVE protocol as part of their streaming engine or communication protocol. The success of these applications show the feasibility and great implementation potential of our research e®orts. Some of them are still work in progress and some of them are in ¯nal release stage as commercial level applications. 5.1 AudioPeer Acommercial levelaudio chatroom thatis built on the ACTIVEtechnology is presented here ¯rst. The reason we chose a multiparty audio chat room to test our ACTIVE protocol is: multi-user audio conference is very demanding in terms of both delay and scalability. Previousresearchhasconcludedthataroundtriptime(RTT)latencyofmore than 200ms will make a voice conversation di±cult. This amount of delay can easily be exceeded when using traditional multi-hop P2P architecture. Also, our audio system is used in distance education classroom, which each room could concurrently be used by hundreds of students from all around the country. To the best of our knowledge, current 98 popular audio systems, such as iChat and Skype, cannot support audio groups of this size. Educational tools that utilize the Internet to reach o®-campus students are becoming morepopularandmanyeducationalinstitutionsareexploringtheiruse. Herewedescribe a multiuser audio chat system called AudioPeer [59]. AudioPeer is built on the ACTIVE protocol, it is designed to foster collaboration and interactive learning between students, teaching assistants and instructors. The AudioPeer system provides an interactive plat- form where groups of students can discuss assignments, teaching assistants can conduct lab sessions and professors can answer questions from students during lectures. The de- centralized nature of the AudioPeer system avoids bottlenecks and allows it to scale to large groups of participants. A multiuser audio chat system involves numerous technical challenges that need to be addressed to build such an application. The number of participants in a chat session may be several dozens, with each student needing to hear and possibly talk to any other person in the session. Additionally, the end-to-end audio latency needs to be kept suf- ¯ciently low such that natural interaction is possible. Hence, we aim for our system to be scalable, practical (e.g., work with di®erent types of network connections), integrat- able with other distance education components, and extensible with new features (e.g., speaker recognition). In the following subsections, we will describe the system archi- tecture of AudioPeer, its multiple components and how it integrates with the existing distance education infrastructure called DEN [3] at our university. 99 5.1.1 System Architecture AudioPeer is a P2P based audio chat system applicable to, for example, distance edu- cation. As illustrated in Fig. 5.1, users are connected to each other in a decentralized fashion. Every user is regarded as a connecting node in the streaming topology and all nodes are connected to form a shared-multicast tree. Voice streams are merged at each node to maintain a ¯xed bandwidth requirement regardless of the number of users in the system. As most P2P designs do, AudioPeer also contains centralized components, namely the website and the rendezvous point server. However, these centralized parts serve principally as the bootstrap point and do not a®ect the scalability of the overall streaming. Figure 5.1: AudioPeer Architecture 5.1.1.1 Website AudioPeer is currently designed as a web-based system. This enables us to deploy and upgrade the system in a fast and easy way. Also, since most of DEN's functions are also web-based,thisapproachmakesAudioPeeranintegratedcomponentoftheDENwebsite. 100 Fig. 5.2 shows a screen shot of the current AudioPeer homepage. It provides simple but detailed step-by-step instructions on how to setup the AudioPeer plug-in for new users. On the right side of the homepage, we list the currently available chat sessions. Return users can simply click on these links to join an on-going session. This website serves as the bootstrap point that we can ¯nd in most P2P system designs. Additionally the website provides a link to a questionnaire where users can provide us with feedback. We will discuss the user feedback data later. Figure 5.2: AudioPeer O±cial Website 5.1.1.2 Client Software The core componentof theAudioPeersystem is the client program. Currently theimple- mentation is based on Microsoft ActiveX technology, which enables us to integrate our client software seamlessly with almost any website and run it on any web browser that 101 Figure 5.3: AudioPeer Client Interface support ActiveX technology, such as Internet Explorer and the newly popular FireFox browser. WewrapourActiveXcontrolwithaHTMLbasedinterface(asshownonFig.5.3)and use Javascript to connect the web interface with our ActiveX control. The majority area of the interface is taken up by a text chatting window which allows users to communicate with text in addition to voice. On the right side is a panel which shows the list of users currently in this chat room. The icons to the right of their names indicate the current status of the users (speaking or mute). Users can start talking with a click of a button at the bottom of the window. We also implemented our own silence detection algorithm, which proved to be very e®ective. If a user remains silent for more than three minutes, he/she will be automatically muted by the system so that other users who are speaking can get a chance to move to a better position with lower latency with respect to other speakers. 102 Another important function provided by each client is audio stream merging. As mentioned before, AudioPeer is designed to support large size audio chat group without linear growth in the network bandwidth. This great feature is achieved through audio merging at every node on the peer-to-peer network. The details of audio processing in AudioPeer is discussed in Section 5.1.2. 5.1.1.3 Rendezvous Point Server Most P2P systems provide peers with a bootstrap server to start the service. This server is usually called Rendezvous Point Server (RP server). In the AudioPeer system design, the functions of the RP server is minimized to just store session information and record user activities for administrative and research purposes. The session information is stored in a MySQL database and made available through website links. As we have discussed before, users can simply click a link on the website to start the chatting experience. After clicking a link for one chat room, the information aboutthesessionisretrievedfromthedatabaseandinsertedintotheAudioPeerActiveX control, which will use these information to start the join process, mostly without further help from the RP server. The RP database also persistently stores a variety of system-wide information includ- ing: debuggingandhistoryinformation,userandsessionmanagementdata,demographic user and course information, stored media descriptions for recorded sessions, and inter- active user information such as user ratings of the system. These collected data are very usefulnotonlyforresearch,butalsoforadministrativepurposes. Inthefollowingsection, 103 we will discuss the use of the data from the RP database to visualize the ongoing system architecture. 5.1.1.4 Administration Control Interface We have implemented an administration control interface (ACI) to e±ciently visualize the peer-to-peer overlay network in real-time for administrative purposes (Fig. 5.4). This tool is implemented with Visual C++ with OpenGL libraries running on the Windows operating system. Many useful system parameters are displayed on-the-°y along with the 3D represen- tation of the current streaming tree topology. ACI is an information visualizer that also allows active management. Besides functionalities such as displaying system-wide infor- mation about individual nodes, sessions and courses, ACI can also issue direct system commands to individual nodes and sessions to govern the system at large. For example, ACI can force a node to leave, mute, relocate to another chat room if necessary, or merge two chat rooms into one. ACI provides a 3D control interface for the AudioPeer system, in which there could bemanyconcurrentchatsessions. Weconcludedthatbyusinga3Dvisualization,system administrators can gain a much better perception of the system structure compared with a2Dversion. Alsoa3Drepresentationallowsustodisplaymoreinformationinthesame area, which becomes an important feature for a large scale system. 104 Figure 5.4: Administration Control Interface(ACI) 5.1.2 Audio Mixing The audio mixing algorithm focuses on minimizing the network utilization for our audio conferencing application. Unlike in video conferencing, it is possible to aggregate the un- compressed audio sources through simple arithmetic calculations, preserving the original audio bandwidth. Each peer node is equipped with an audio mixing module that relays the incoming audio from remote nodes to the outgoing connections. We use a software-based audio mixing algorithm called decode-mix-encode [46]. A linear mixing algorithm requires all the input audio bitstreams to be uncompressed for simple arithmetic additions and sub- tractions. Thus, all incoming encoded bitstreams are decoded into their uncompressed form,andtheresultinguncompressedbitstreamsaremergedintoamixedbitstream. This stream is later used when constructing the outgoing streams for each respective remote node. 105 Table 5.1 lists the currently supported audio media types and their characteristics: high-quality, low latency (PCM stereo); medium-quality, low latency (PCM mono); low- quality, low latency (GSM.610); and high-quality, high latency (MPEG-1 Layer 3). Supported audio media types Compression Type uncompressed compressed Audio Format PCM PCM GSM.610 MPEG Layer3 Bits per Sample 8 16 8 16 Channels mono stereo mono stereo Sampling Rate 8 KHz/16 KHz 48 KHz 8 KHz 48 KHz Delivery Rate 64 Kbps/128 Kbps 1.536 Mbps 13 Kbps 56 Kbps Usage Method LAN LAN dial-up modem cable modem, DSL Latency Requirement low low low high Audio Quality medium high low high Table 5.1: Audio types supported. 5.1.3 Implementation Of ACTIVE Protocol ACTIVE is an application-level multicast protocol designed to serve as a reliable audio streaming platform that provides minimal overall end-to-end delay among active nodes. Aimingathighscalabilityandlowlatency,theACTIVEprotocoldynamicallymaintainsa shared multicast tree among all peer nodes. For space reasons we restrict our discussion of the ACTIVE protocol implementation to those issues that have not been discussed before and also in relatively high-level description. We implemented the ACTIVE protocol using C/C++. At any given moment, a user can be in either of following states: Idle, Joining, Select, Setup, Joined, Leaving, Recover and Core (Fig. 5.5). We use TCP instead of UDP as streaming transportation protocol. The reason is that a lot of remote users now connect from behind a network address translation(NAT)devicesuchasahomeDSLmodemstotheInternet,whichcomplicates the building of a peer-to-peer network using UDP without a centralized server. We also 106 include network time protocol (NTP) functionality in our client software to synchronize our system in a distributed fashion. Idle Joining* Setup* Core Select* Leaving* Joined JOIN_CMD JOIN_RESP(OK) JOIN_REFER TIME_OUT LEAVE_CMD * Transient states with time out Recover* Figure 5.5: AudioPeer state diagram. 5.2 MStream 5.2.1 Application Overview Wireless networks are increasingly pervasive in many commercial and business environ- ments. Additionally, many portable computing devices now have built-in wireless net- working support. With these trends we are entering a new era of mobile computing and entertainment. In this section we introduce an application named MStream [32] for mobile handheld devices such as PDAs. MStream provides customized, location-aware streaming audio services through wireless IP networks to PDAs with GPS locators. 107 No. Question Options Feedback 1. Gender Male 73.33% Female 26.76% 2. Position Student 100% Others 0% 3. Age Range 18-22 40% 23-27 60% 4. How many hours per week do you use computers? 0-40 33.3% 41-80 53.3% 81+ 13.4% 5. How many hours per week do you use email? 0-10 40% 11-20 46.7% 21+ 13.4% 6. How many hours per week do you use online chat systems(e.g. Instant messenger) 0-5 46.7% 6-10 26.7% 11-20 13.3% 21+ 13.3% 7. What is the purpose of your use of Audiopeer? chat with friends 26.7% educational use 60% Others 13.3% 8. How many people did you chat with using au- diopeer? one other person 20% two or more 80% 9. Did you talk to two or more people at the same time? Yes 46.7% No 53.3% Table 5.2: Statistic of User Groups Withtheubiquitousavailabilityofwirelessnetworksinmanycommercialandbusiness settings, we are entering a new era of mobile computing and entertainment. In our demonstration we introduce a mobile application named MStream, which streams audio tracksthroughwirelessIPnetworkstoPDAs,customizedbasedontheirphysicallocation. The viability of such an approach has been enabled by the proliferation of a®ordable wireless networking, most notably the pervasive availability of 802.11-standards based networks. The bandwidth provided by all the di®erent forms of 802.11 is su±cient for audio streaming in a wide area, especially since modern audio compression algorithms 108 No. Question Feedback (Scale 1-7) 1. Once you had learned how to operate AudioPeer, did you ¯nd it easy to use? 5.06 2. Did you ¯nd AudioPeer a useful way to communicate? 4.4 3. Was AudioPeer helpful in your communications? 4.53 4. Do you ¯nd AudioPeer an e±cient way to communicate?? 4.2 5. Were you frustrated by any delays caused by communicat- ing with AudioPeer? 5.2 6. Did you like communicating using AudioPeer? 4.2 7. Did you enjoy communicating using AudioPeer? 4.13 8. Did you ¯nd AudioPeer fun to use? 4.13 9. Did AudioPeer make you feel connected to other people? 4.4 10. Did AudioPeer make you feel that people were at hand to answer your questions? 4.4 Table 5.3: User Feedback of AudioPeer provide excellent aural quality at a moderate data rate. We also leverage the research resultreportedin [39]topredictthelinkavailabilityofmobileunitsthatmovingaround. Finally, handheld devices such as PDAs now provide computing capabilities that were a few years ago only available on the desktop. Figure 5.6: MStream System Architecture The MStream system allows a personal sound environment to be created for each individual user. By obtaining GPS physical location information the audio ambience can 109 bespeci¯callytailoredtoaparticipant'senvironment. ThesignalstrengthofGPSissuch that it does not reliably work inside of most buildings. Therefore, MStream is currently applicable mostly in outdoors settings. We envision applications where the MStream system provides audio guidance in location sensitive applications such as guides for a university campus guide, a park guide, a zoo, a botanical garden, where users can get relevant audio information based on their physical location and time. (a) MStream Client with iPAQ H5555 (b)Pharos GPS (Model i360) Figure 5.7: MStream Devices 5.2.2 System Design MStream has been designed and implemented with o®-the-shelf hardware components suchthatitcanimmediatelybedeployedoncurrentlyavailablehandhelddevices. Onthis 110 platform, wehavedevelopedthecustomMStreamsoftware. Insteadofusingapurepeer- to-peer streaming design, the MStream system utilizes a hybrid approach. The system is composed of two main components: the server and the client. As shown in Figure. 5.6, the server streams music to the root of the each client tree in a server-client fashion; subsequently the clients within the same streaming area cluster themselves around the root client and construct a peer-to-peer topology. The MStream server provides several functions. First,itactsasarepositoryforalltheavailablemusic¯les. Second,itcontains the decision logic that determines which ¯le to stream to a client based on its location information and other parameters, such as the time of the day and the day of the week. The interaction between a client and the server unfolds in several phases as follows. Figure. 5.7(a) shows a MStream client on a HP iPaq h5555 with a Pharos GPS unit. Periodically the client application polls the GPS reading from an attached GPS unit, for example every one second. New GPS readings are re¯ned through an error-¯ltering process to remove occasional GPS reading errors. The GPS readings are then converted to xy-coordinates on a pre-de¯ned map. Subsequently, the most recent xy-coordinate is sent to the MStream server through a TCP connection in conjunction with a streaming request. After receiving the request from a client, the MStream server examines its table of streaming policy entries and determines which music ¯le to stream to the client. The steaming policy table is a data structure that contains tuples of xy-coordinates, location radii, time of day intervals and day of the week entries. If the client's xy-coordinate is within the radius of a tuple in the policy table, the server selects the music ¯le of the current time and date and streams it to the client. The MStream client runs on the Microsoft Windows Mobile 2003 platform and retrieves its location information from 111 an attached GPS unit, as shown in Figure. 5.7(b). After negotiating with the server, the client can either become the root of the current streaming area or connect to other clients in the same area to receive a music stream. Each MStream client bu®ers 500ms worth of data to accommodate wireless network jitters. If a wireless network temporarily unavailable, then default background music is played on the client. Instead of periodically updating the server with its current location, the mobile unit retrievestheportionofthestreamingpolicytableassociatedwiththecurrentmusictrack. The mobile unit detects if it has moved out of the current streaming area by checking the local copy of the policy table. If that is the case, a new request will automatically be sent to the server and the appropriate music ¯le will be selected and streamed to the client immediately. A fade-in and fade-out e®ect implements a smooth transition between the previous and the next music ¯le. Our initial tests of the MStream system show that it can support multiple mobile clients with ease and most of the network jitters can be smoothed out. In addition, even though the peer-to-peer streaming protocol among clients is somewhat more complicated comparedtoasimpleclient-serverdesign, theloadontheserverissigni¯cantlyreduced. 5.3 PartyPeer Partypeerisamulti-useronlinesocialgamethatuseACTIVEasthestreamingplatform. Party peer provides users from di®erent geographic locations a common place to interact virtually through audio and graphical signals ( As shown in Figure 5.8). 112 (a) bird view of the island (b) paint ball shooting Figure 5.8: Screen shot of PartyPeer The novelty of PartyPeer is demonstrated in its several components and the integra- tion as a whole. Its voice chat module includes several advanced features. Going beyond such examples as Yahoo Chat or Skype, it is designed for very large groups. Hence, hundreds or even thousands of people can join in the same space. To manage the conver- sations, advanced concepts such as spatial audio and proximity detection are used. The audio volume is adjusted based on the user's gaze direction and distance to other par- ticipants. This is achieved via the novel integration of a three-dimensional virtual world user interface that provides an experience which is much more advanced than existing chat systems. Also, unlike traditional voice-over-IP (VoIP) support for existing games, where a few friends get together to play as a team, the focus of PartyPeer is on enabling interactions among large numbers of users, so that participants who may not know each 113 other at the beginning will ¯nd new, like-minded individuals. With its peer-to-peer tech- nology, every end-user program acts as a client, a server and an audio mixing node. An overlay multicast tree is dynamically built and maintained in a fully distributed fashion. The latest NAT hole punching technique is incorporated to improve the system's compatibility with today's advanced ¯rewalls, broadband routers and home gateways while still bene¯ting from the low-latency advantages of user datagram protocol (UDP) media streams. Overall, PartyPeer provides a compelling and novel social community game that is designed to leverage the best features of today's broadband networks. PartyPeer fuses the concepts of a virtual world with our scalable media streaming engine to provide an entirely new and compelling experience for the users. 5.3.1 Game Overview PartyPeer is a social game that provides a virtual space where massive number of players can gather and interact with each other. The challenges of building a scalable, playable (low-latency) and yet practical massively multiplayer game using a P2P infrastructure have led us to devise an innovative P2P based low-latency streaming platform called ACTIVE. ACTIVE has evolved from our previous research [31, 33, 34] with signi¯cant changes to make DSM a practical and suitable platform for MMOG games. We chose to implement a social game to demonstrate the feasibility of using a P2P architecture for MMOG games because a social game is a very good platform to test a massive number of players in a session. Even though the delay requirement for a social game is less than for a ¯rst-person shooter game, we hope that our pioneering e®orts of implementing a playable game for many users using a P2P architecture will provide insights into many 114 Figure 5.9: Game Architecture Design (a) D 0 = 0m (b) D 0 = 30m (c) D 0 =60m Figure 5.10: Visibility prediction for maximum speed = 5m/s practical issues that so far have been largely ignored and inspire a new generation of P2P designs that could lead to successful P2P based MMOG games in the future. Due to the space limitations, we will focus on the design issues of DSM that have not been covered in our previous research [59, 31, 34, 39]. Interested readers can ¯nd techniques on how to build a low-latency audio streaming network in [59, 31]; approaches on how to arrange a P2P network based on player's virtual locations in the game world and the delivery of immersive sound in Section 4.3 and techniques on how to predict the future movement of players based on their current speed and direction in [39]. 115 5.3.2 Scalability As mentioned before, PartyPeer is designed to support thousands of concurrent players in one game session, so scalability is a very important issue. The fundamental idea is to reducethegamemessageexchangeasmuchaspossible. InPartyPeer,all3Dmodelsused inthegamearestoredonthelocalmachine{weonlyexchangegamestatusmessagesand audiodatathroughtheunderlyingP2Pnetwork(Figure5.9). Inthefollowingsectionswe will discuss our innovative design to greatly reduce the message tra±c among the players 5.3.2.1 Dynamic zoning Unlike the traditional area of interest (AOI) algorithm which usually divides the gaming spaceinto¯xedblockscalledzones,DSMdynamicallyupdatethemeshlinksbasedonthe changes of the location, which means the game zone is actually also dynamic, depending on the number of players online and how close they are together. By using probability- based message forwarding, a node receives more updates from neighboring nodes than updates from other nodes positioned at greater distances (hops). There is no cut-o® line for AOI in the DSM design, however the interest level drops and the mechanism places more interest on closer nodes and farther nodes receive less attention. The size of AOI for DSM is dynamic, but the number of objects in this AOI is bounded. 5.3.2.2 Visibility prediction We extended our previous research [39] to use the random walk model to predict the visibility of a player from another player in the game. If we de¯ne the range of visibility as R, the initial distance between two players as D 0 , and the rate of movement change as 116 (a) Direct Link (b) Relay Figure 5.11: P2P streaming using introducer server and relay server ¸, then we can predict the probability of visibility in the near future. Figure 5.10 shows some results to predict the visibility in the next 200 seconds given that R is 100 meters and the maximum speed for each player is 5 m/s. In an MMOG game, R and D 0 are directly available to each player, however ¸ is not immediately available and we use an average number based on the data collected in the past. Wewillupdatethisnumberoncewedeploythegametothelargerpublicandcollect more data. Visibility prediction is very useful to reduce the message exchange, because a player outside of the visibility range, even if he/she is still within our dynamic AOI, can avoid exchanging message with this player. We will collect more data to understand how much bandwidth we can save by using the visibility prediction in our future release of the game. 117 5.3.3 Latency Latency is a critical performance metrics for MMOG games. A large latency among players will make the interaction di±cult and eventually impossible. Game designs using aclient-serverarchitectureusuallyperformwellintermsoflatencybecausethecentralized serverprovidesaone-hopforwardingchannelamonganytwoplayers. Thisisincontrastto a P2P based design, where there are usually multiple hops between two players, resulting in longer delays. In our DSM mesh network, players are clustered based on their positions in the game world. Playersvirtuallyclosetoeachotherwillalsohavefeworevennointermediatehops between them. This will give us a comparative performance to a centralized approach for players within the same AOI. Also, research indicates that 150 to 200 ms delay among participants is acceptable for a social type activity such as voice conversations. This provides us with room to implement low latency applications using P2P based topology. 5.3.4 P2P Connectivity One important practical issue that has largely been ignored by the research community when designing the P2P based game architecture is the accessability problem introduced by currently widely deployed network address translation (NAT) devices such as home DSL modems or cable modems. NAT devices are very convenient for consumers because they allow sharing of one IP address when connecting multiple machines to the Internet. However, this technique becomes a problem when an outside machine tries to initiate a connection to a machine behind a NAT device because the local IP address acquired 118 by that machine is not globally addressable. A P2P based platform must solve the P2P connectivity issue to make an implementation feasible. There are several types of NAT devices based on how \friendly" they are to P2P type connections. Most NAT devices maintain the IP/port binding for a local socket and we caninitiateaP2PconnectionbetweentwomachinesbehindNATboxusinganintroducer server with a well-known IP address, as shown in Figure 5.11(a). Please note that the introducer server is only required during the connection setup stage and all future data streamingisperformedthroughdirectlinkbetweenthosetwomachines. However,asmall fraction of NAT devices create a new binding of IP/port for each di®erent destination address. Thesedevices,suchassymmetricNATdevices,arenotcompatiblewithexisting P2P protocols and a relay server/peer is required to build a forwarding channel between the two machines, as shown in Figure 5.11(b). Due to space limitations, we defer the details of the protocol to setup the P2P connection to a future publication. 5.3.5 Immersive Sound Design In this paper, we are concerned with the design of an immersive 3D-audio service over a Distributed Virtual Environment (DVE). 3D audio rendering is out of the scope of this paper since there are many rendering APIs that can perform this task. Our focus here is on the delivery of this service over a peer-to-peer architecture. The rendering of the spatial sound requires several parameters such as the relative positions of the avatars, orientation of the sound, avatar velocities, etc. If we send audio streams in discrete channels for each player, then it is easy to render the 3D e®ect since all original data are available on the terminal. However this approach is de¯nitely not 119 scalable. As we de¯ned in Table 3.1, if we send out discrete streams, then for a group of jNj players the bandwidth for both sending and receiving is jNj£F i (t)£D bps. Our DSM platform keep the audio stream from each source separate until before the insertion to the 3D audio rendering engine. This approach reduces the bandwidth required at the cost of losing the original data fromremoteplayers. Fortheneighborplayerswhodirectly connected, wecanstillrender the3De®ectssincealldataarereceiveddirectly. Forremoteplayers,weneedamechanism to somehow retain some of the spatial information while performing mixing with other streams. We propose to address this challenge by maintaining a stereo stream while mixing. A node will encode the audio stream with spatial information by processing them based on the information of the source of the audio stream. By providing the left andrightchannelswithadi®erentaudiostream,wecanretainsomeofthespatialfeatures fromtheremoteplayersevenafterafewhopsofmixing. Wewillpublishuserfeedbackon our immersive audio design after collecting more data from forthcoming public release. 5.4 LiveRH: A Tele-rehabilitation System Healthcare is one of the fastest growing sectors of the economy and providing cost- e®ective healthcare service to an aging population with a declining number of hospitals is a formidable challenge. Through recent technological advances it is now possible to migrate some of the services from centralized locations into the home. We describe a prototype architecture that we built to support novel, pervasive and easily deployable information technology applications in healthcare segment where outpatient treatment 120 can be a cost-e®ective alternative. Our proposed architecture is conceived as a °exible platformthatallowsapplicationbuilderstorapidlydesign,createanddeployapplications that require the transmission of delay-sensitive media streams such as audio, video, and haptic data. As an initial example we applied our framework to tele-rehabilitation where a therapist remotely monitors the exercise regimen and progress of a patient who, for example, previously su®ered from a stroke. Figure 5.12: Input Device: Phantom 6DOF Figure 5.13: Screen shot from one excercise 121 We have designed a virtual patient/clinician interactive platform called ACTIVE based on our innovative audio streaming protocol. The new ACTIVE platform was devised to distinguish among di®erent characteristics (e.g., bandwidth and processing re- quirements) of di®erent streaming data and handle them accordingly. The ACTIVE ar- chitecturedynamicallymaintainsandoptimizesapeer-to-peeroverlaystreamingnetwork so that time-sensitive data, e.g., a remote voice stream representing verbal instructions giventoapatientbyaclinicianduringrehabilitation,canbedeliveredinatimelyfashion. We are excited by ACTIVE's capability to provide a general and °exible platform that presents a universal interface to its applications such that multiple media channels can be allocated, each with potentially di®erent characteristics. Figure 5.14: LiveRH Clinic Trial We are using neuro-rehabilitation as our application case study to investigate the e®ectiveness of the ACTIVE approach. We have designed an exercise environment which can be host to a progressive set of training tasks from precise ¯ne motor movements to reaching movements that involve full arm and shoulder activity, the screen shot of once 122 of the exercises we developed is shown in Figure 5.13. We are leveraging our earlier work that makes use of the PHANToM haptic device (Figure: 5.12), which is a small, desk-grounded robot that can simulate the sense of touch on a virtual object through forcefeedback. ByusingtheACTIVEplatform, thetherapistcanremotelymonitorboth the actions and progress of the patient and, if necessary, provide the needed assistance through the voice channel. The metadata stream that contains the haptic information and user feedback are stored and analyzed later. We conducted a preliminary trial of our ACTIVE prototype starting in 2005 (see Figure 5.14 and a subset of the results were reported in [35]. As a work in progress, our prototype received a positive feedback with average user rating of 4.5 on a scale of 1 to 7. Our proposed virtual patient/clinician application is the ¯rst of its kind that is using a peer-to-peer based streaming network. Its main bene¯ts are that no expensive, centralizedinfrastructureisrequiredandthewholesystemiseasytodeployandscalable. Our innovative work shows that it is becoming feasible and cost-e®ective to utilize this type of heterogeneous network to conduct certain healthcare tasks remotely. 123 Chapter 6 Conclusion InthischapterIwillbrie°ysummarizethedissertationmaterialthatIjustdiscussedand conclude this dissertation. Afewthoughtsofpossiblefutureresearchisalsomentionedattheendofthischapter. 6.1 Summary Of The Dissertation During this dissertation, I have presented the challenges to build a peer-to-peer based streaming system for interactive application. Then two performance metrics, scalability and end-to-end latency, are identi¯ed as the problem to be solved. I presented the AC- TIVE protocol which is designed to cluster users with common topic of interest (TOI) and thus reduce the overlay latency among them. Performance is analyzed and ACTIVE is shown to gain signi¯cant improvement in terms of dynamically reduce the end-to-end delay among active users. We further extended our research to enable immersive audio streaming in large scale interactiveapplicationswithnetworkedvirtualenvironment. Thedesigniscalleddynamic streaming mesh since it is a mesh based design and the the mesh links are dynamically 124 update based on the virtual location of users. Our simulation shows that DSM out- performance other leading mesh-based solution which doesn't update their virtual links based on the virtual location. ACTIVE protocol is implemented and integrated to several applications and it is proven to be robust and serves great impact in the new breed of interactive applications that built on top of peer-to-peer topology. AudioPeer is a multi-user audio conference tools that allow many participants in one online session, speakers are identi¯ed and au- tomatically clustered to provided appropriate low latency among them. MStream is a location based music streaming system with a simpli¯ed version of the ACTIVE protocol built inside so that nearby mobile units can share a single stream from the server. Part- Peer is a massivelymultiplayeronline game (MMOG) that built on DSM protocol, where players with close virtual coordinate readings are clustered so that even a ¯rst-person shooting game can be implemented using P2P architecture without a centralize server to audit the tra±c. Lastly we discussed LiveRH, which is a remote rehabilitation plat- form for stroke patients with neurologic impairment to take exercise at home. ACTIVE platform is served as the delay critical communication channel between patients and the physical therapists. I believe that this dissertation, as reported here at the Integrated Media Systems Center of University of Southern California, along with many other research e®orts being carried on by research institutions and facilities around the world strongly shows that peer-to-peer topology is merging as a viable platform not only for scalable ¯le sharing and downloading, but also a feasible design for low latency applications such as large scale audio conferencing and MMOG games. 125 I am grateful being able to conduct my research on this exciting area and I will continue to work on future research topics in the hope that one day, everybody could interact with each other in an secure, fast and reliable immersive networked environment freely. 6.2 Future Research 6.2.1 Video Streaming As so far, ACTIVE protocol has been proven to be suitable for audio streaming, and as a general purpose streaming platform, I'd like to look into the research topics for streaming video streams over ACTIVE platform and evaluate the performance in terms of bandwidth usage, control overhead and also end-to-end delay in the system. 6.2.2 P2P Security One of the design concerns that we mentioned earlier but has not addressed so far is the security issue on P2P networks. As part of the future research, I will pursue to attack the security issue in following areas: ² design a distributed P2P authentication system utilizing the combination of PSK and MD5 methods that wildly used by current online commercial venders. The key research challenge is how to minimize the centralized part in this design but not at the cost of lowering the total security level. ² Evaluate the overhead cost when introducing the encrypted data packet to the live streaming system. Try to look into alternatives of current implementations that encrypt all data packets. ² Try to itemize the possible security attack that a hacker can launch against the system and precautions we can take into the design of the system to mitigate the aftermath of such attacks. 126 ² Design a well-balanced security scheme that not only convenient for regular users to use, but also e®ective enough to detect a DoS attack in very early stage so that management team can take actions quickly after a attack begins. 127 References [1] NS: the Network Simulator. Information about NS is availabale at http://www.isi.edu/nsnam/ns/. [2] Skype Internet Telephony. http://www.skype.com. [3] USC Distance Education Network. Information about DEN is availabale at http://den.usc.edu/. [4] Marios Assiotis and Velin Tzanov. A distributed architecture for mmorpg. In NetGames '06: Proceedings of 5th ACM SIGCOMM workshop on Network and sys- tem support for games, page 4, New York, NY, USA, 2006. ACM Press. [5] W. Xu B. Hopkins B. Knutsson, H. Lu. Peer-to-peer support for massively multi- player games. INFOCOM, 2004. [6] Farnoush Banaei-Kashani and Cyrus Shahabi. Brief announcement: e±cient °ood- ing in power-law networks. In PODC '03: Proceedings of the twenty-second annual symposium on Principles of distributed computing, pages 336{336, New York, NY, USA, 2003. ACM Press. [7] S. Banerjee, B. Bhattacharjee, and C. Kommareddy. Scalable Application Layer Multicast. Technical report, UMIACS TR-2002, 2002. [8] S. Banerjee, C. Kommareddy, K. Kar, B Battacharjee, and S. Khuller. Construction ofanE±cientOverlayMulticastInfrastructureforReal-timeApplications. In IEEE INFOCOM 2003, 2003. [9] Salman A. Baset and Henning Schulzrinne. An analyis of the skype peer-to-peer internet telephony protocol. In Technical Report CUCS-039-04, Computer Science Department, Columbia University, New York, NY, September. [10] Supratik Bhattacharyya, Christophe Diot, and Jorjeta Jetcheva. Pop-level and access-link-level tra±c dynamics in a tier-1 pop. In IMW '01: Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, pages 39{53, New York, NY, USA, 2001. ACM Press. [11] bio Reis Cecin, Rafael de Oliveira Jannone, Claudio Fernando Resin Geyer, Mer- cio Garcia Martins, and Jorge Luis Victoria Barbosa. Freemmg: a hybrid peer-to- peer and client-server model for massively multiplayer games. In NetGames '04: Proceedings of 3rd ACM SIGCOMM workshop on Network and system support for games, pages 172{172, New York, NY, USA, 2004. ACM Press. 128 [12] M. Blum, P. Chalasani, D. Coppersmith, B. Pulleyblank, P. Raghavan, and M. Su- dan. The minimum latency problem. In ACM Symposium on Theory of Computing, 1994. [13] K. Calvert, M.B. Doar, and E.W. Zegura. Gt-itm: Modeling internet topology. In IEEE Communications Magazine, June 1997. [14] M.Castro,P.Druschel,A.Kermarrec,andA.Rowstron. SCRIBE:ALarge-scaleand Decentralized Application-level Multicast Infrastructure. IEEE Journal on Selected Areas in communications (JSAC), 2002. [15] M. Castro, M.B. Jones, A.-M Kermarrec, A. Rowstron, M. Theimer, H. Wang, and A. Wolman. An evaluation of scalable application-level multicast built using peer- to-peer overlays. In INFOCOM 2003. Twenty-Second Annual Joint Conference of the IEEE Computer and Communications Societies., pages 1510{1520, 2003. [16] AlvinChenandRichardR.Muntz. Peerclustering: ahybridapproachtodistributed virtual environments. In NetGames '06: Proceedings of 5th ACM SIGCOMM work- shop on Network and system support for games, page11, NewYork, NY,USA,2006. ACM Press. [17] T.H. Cormen, C.E. Leiserson, and R.L. Rivest. Introduction to algorithms. MIT Press, 1997. [18] Yi Cui, Baochun Li, and Klara Nahrstedt. ostream: Asynchronous Streaming Mul- ticast in Application-layer Overlay Networks. IEEE Journal on Selected Rreas in Communications (JSAC), 22(1):191{196, 2004. [19] P. Francis. Yoid: Your Own Internet Distribution. Available online at http://www.aciri.org/yoid/, April 2000. [20] Xiaohui Gu, Zhen Wen, Philip S. Yu, and Zon-Yin Shae. Supporting multi-party voice-over-ip services with peer-to-peer stream processing. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 303{306, New York, NY, USA, 2005. ACM Press. [21] Thorsten Hampel, Thomas Bopp, and Robert Hinn. A peer-to-peer architecture for massive multiplayer online games. In NetGames '06: Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games,page48,NewYork, NY, USA, 2006. ACM Press. [22] Jan-Ming Ho, D. T. Lee, Chia-Hsiang Chang, and C. K. Wong. Minimum diameter spanning trees and related problems. SIAM J. Comput., 20(5):987{997, 1991. [23] Shun-Yun Hu. A case for 3d streaming on peer-to-peer networks. In Web3D '06: Proceedings of the eleventh international conference on 3D web technology, pages 57{63, New York, NY, USA, 2006. ACM Press. 129 [24] Shun-YunHuandGuan-MingLiao. Scalablepeer-to-peernetworkedvirtualenviron- ment. In NetGames '04: Proceedings of 3rd ACM SIGCOMM workshop on Network and system support for games, pages 129{133, New York, NY, USA, 2004. ACM Press. [25] Yang hua Chu, Sanjay G. Rao, Srinivasan Seshan, and Hui Zhang. Enabling Con- ferencing Applications on the Internet Using an Overlay Multicast Architecture. In ACM SIGCOMM 2001, San Diago, CA, Augest 2001. ACM. [26] Yang hua Chu, Sanjay G. Rao, and Hui Zhang. A case for end system multicast. In SIGMETRICS '00: Proceedings of the 2000 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, pages 1{12, New York, NY, USA, 2000. ACM Press. [27] A.K.Jain, M.N. Murty, and P.J.Flynn. Dataclustering: areview. ACM Comput. Surv., 31(3):264{323, 1999. [28] C. Jin, Q. Chen, and S. Jamin. Inet: Internet topology generator, 2000. [29] B. Knutsson, H. Lu, W. Xu, and B. Hopkins. Peer-to-peer support for massively multiplayer games. In IEEE INFOCOM, March 2004. [30] B. Knutsson, Honghui Lu, Wei Xu, and B. Hopkins. Peer-to-peer support for mas- sivelymultiplayergames. InINFOCOM2004.Twenty-thirdAnnualJointConference of the IEEE Computer and Communications Societies, page 107, 2004. [31] Leslie S. Liu and Roger Zimmermann. Active: Adaptive Low-latency Peer-to-Peer Streaming. InMultimedia Computing and Networking(MMCN),SanJose,CA,2005. [32] Leslie S. Liu and Roger Zimmermann. Mstream: Position-aware Mobile Music Sta- tion. In The Third International Conference on Mobile Systems, Applications, and Services(Mobisys), Seattle, WA, June 2005. [33] LeslieS.LiuandRogerZimmermann. AdaptiveLow-latencyPeer-to-PeerStreaming and Its Application. Special Issue of the Multimedia Systems Journal, 2006. [34] Leslie S. Liu and Roger Zimmermann. Immersive Peer-to-Peer Audio Platform for Massive Online Games. In IEEE International Workshop on Networking Issues in Multimedia Entertainment(NIME), Las Vegas, NV, 2006. [35] M. McLaughlin, R. Zimmermann, L. Liu, Y. Jung, W. Peng, S. Jin, J. Stew- art, S. Yeh, W. Zhu, and B Seo. Integrated Voice and Haptic Support for Tele- rehabilitation. First Workshop on Ubiquitous and Pervasive Healthcare (UbiCare), 2006. [36] Alberto Medina and Ibrahim Matta. Brite: A °exible generator of internet topolo- gies. In Technical Report BU-CS-TR-2000-005, Boston University, Boston, MA, 2000. 130 [37] Raymond T. Ng and Jiawei Han. E±cient and e®ective clustering methods for spatialdatamining. InVLDB '94: Proceedings of the 20th International Conference on Very Large Data Bases, pages 144{155, San Francisco, CA, USA, 1994. Morgan Kaufmann Publishers Inc. [38] Misbah Qidwai. A survey of real-time 3d audio rendering techniques for virtual environments. 2006. [39] Min Qin, Roger Zimmermann, and Leslie S. Liu. Supporting Multimedia Streaming Between Mobile Peers with Link Availability Prediction. In ACM International Multimedia Conference(ACM Multimedia), October 2005. [40] Sylvia Ratnasamy, Mark Handley, Richard Karp, and Scott Shenker. Application- level Multicast Using Content-Addressable Networks. In Third International Work- shop Networked Group Comm., November 2001. [41] R. Ravi, M. V. Marathe, S. S. Ravi, D. J. Rosenkrantz, and III H. B. Hunt. Many birds with one stone: multi-objective approximation algorithms. In STOC '93: Pro- ceedings of the twenty-¯fth annual ACM symposium on Theory of computing, pages 438{447, New York, NY, USA, 1993. ACM Press. [42] A.E. Rhalibi and M. Merabti. Interest management and scalability issues in p2p mmog. In NIME 06': 3rd IEEE Consumer Communications and Networking Con- ference. CCNC 2006, pages 1188 { 1192, 2006. [43] J. Rosenberg. Interactive connectivity establishment (ICE): a methodology for nettworkaddresstranslator(NAT)traversalforthesessioninitiationprotocol(SIP). Internet draft, Internet Engineering Task Force, July 2003.Work in progress. [44] J. Rosenberg, J. Weinberger, C. Huitema, and R. Mahy. STUN - Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs), RFC3489. Internet draft, Internet Engineering Task Force, March 2003. [45] Sherlia Y. Shi, Jonathan S. Turner, and Marcel Waldvogel. Dimensioning server access bandwidth and multicast routing in overlay networks. In NOSSDAV '01: Proceedings of the 11th international workshop on Network and operating systems support for digital audio and video, pages 83{91, New York, NY, USA, 2001. ACM Press. [46] K. Singh, G. Nair, and H. Schulzrinne. Centralized Conferencing using SIP. In Internet Telephony Workshop, April 2001. [47] Kundan Singh and Henning Schulzrinne. Peer-to-peer internet telephony using sip. InACM International Workshop on Network and Operating System Support for Dig- ital Audio and Video (NOSSDAV), June 13 - 14 2005. [48] The Internet Society. SDP: Session Description Protocol, RFC2327. Internet draft, Internet Engineering Task Force. 131 [49] The Internet Society. SIP: Session Initiation Protocol, RFC3261. Internet draft, Internet Engineering Task Force. [50] Kunwadee Sripandikulchai, Aditya Ganjam, Bruce Maggs, and Hui Zhang. The Feasibility of Supporting Large-scale Live Streaming Applications with Dynamic Application End-Points. SIGCOMM, 2004. [51] HiroakiHazeyamaTakujiIimuraandYoukiKadobayash. Distributedscalablemulti- player online game servers on peer-to-peer networks. IPSJ Jornal, February 2005. [52] Su-Wei Tan, Gill Waters, and John Crawford. Meshtree: reliable low delay degree- bounded multicast overlays. In ICPADS 05': IEEE 11th International Conference on Parallel and Distributed Systems, 2005. Proceedings., pages 565 { 569, 2005. [53] Swee Ann Tan, William Lau, and Allan Loh. Networked game mobility model for ¯rst-person-shooter games. In NetGames '05: Proceedings of 4th ACM SIGCOMM workshoponNetworkandsystemsupportforgames,pages1{9,NewYork,NY,USA, 2005. ACM Press. [54] Duc A. Tran, Kien A. Hua, and Tai T. Do. A Peer-to-Peer Architecture for Media Streaming. Journal on Selected Areas in Communications(JSAC), Special Issue on Advances in Service Overlay Networks, 2003. [55] Knut-Helge Vik, Carsten Griwodz, and Pål Halvorsen. Applicability of group communication for increased scalability in mmogs. In NetGames '06: Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games, page 2, New York, NY, USA, 2006. ACM Press. [56] Shinya Yamamoto, Yoshihiro Murata, Keiichi Yasumoto, and Minoru Ito. A dis- tributed event delivery method with load balancing for mmorpg. In NetGames '05: Proceedings of 4th ACM SIGCOMM workshop on Network and system support for games, pages 1{8, New York, NY, USA, 2005. ACM Press. [57] Anthony (Peiqun) Yu and Son T. Vuong. Mopar: a mobile peer-to-peer overlay architecture for interest management of massively multiplayer online games. In NOSSDAV '05: Proceedings of the international workshop on Network and oper- ating systems support for digital audio and video, pages 99{104, New York, NY, USA, 2005. ACM Press. [58] B. Zhang, S. Jamin, and L. Zhang. Host Multicast: A Framework for Delivering Multicast to End Users. In Proceedings of IEEE Infocom, New York, June 23-27, 2002. http://citeseer.ist.psu.edu/zhang02host.html. [59] RogerZimmermann, LeslieS.Liu, BeomjooSeo, RahulS.Hampole, andBrentNash. Audiopeer: A Collaborative Distributed Audio Chat System. In Distributed Multi- media Systems, San Jose, CA, January 2004. 132

Abstract (if available)

Abstract Peer-to-peer (P2P) streaming is emerging as a viable communications paradigm. Recent research has focused on building efficient and optimal overlay streaming platforms at the application level.Peer-to-Peer systems, either fully distributed or hybrid, are well known for features such as high scalability and low cost to deploy. Popular applications such as Skype and Bittorrent can serve tens of thousands of concurrent users with only a few centralized servers. However, it is still a challenge job to design a P2P protocol to support large interactive streaming group where both low latency and high scalability are vital. We identified two important issues that could lead to a successful design for interactive streaming applications. First, many existing designs that aim to build a low-latency streaming platform often make the unreasonable assumption that the processing time involved at each node is zero. However in a real implementation, these delays can add up to a significant amount of time after just a few overlay hops and make interactive applications difficult. Second, scant attention has been paid to the fact that even the number of participants in a group could be large, each user may only actively interact with a subset of them at any given time. We term this sub-group of users who are actively interacting with each other inside this sub-group, an active group. Depending on the streaming scenarios, there can be one or more than one active groups in an application. Some applications have only one active group at any given time, for example, audio conferencing or class session for remote education. Some applications usually have more than one active groups, for example, a BBQ party hosted in a networked virtual environment where participants can wandering freely in the virtual space and chat with anybody they met. In this dissertation, we will present two peer-to-peer based platforms, namely ACTIVE and DSM, for low latency streaming.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Scalable reputation systems for peer-to-peer networks

PDF

Supporting multimedia streaming among mobile ad-hoc peers with link availability prediction

PDF

Techniques for peer-to-peer content distribution over mobile ad hoc networks

PDF

Location-based spatial queries in mobile environments

PDF

Edge indexing in a grid for highly dynamic virtual environments

PDF

Satisfying QoS requirements through user-system interaction analysis

PDF

Scalable machine learning algorithms for item recommendation

PDF

Efficient updates for continuous queries over moving objects

PDF

Efficient pipelines for vision-based context sensing

PDF

Managing multi-party social dynamics for socially assistive robotics

PDF

City-scale aerial LiDAR point cloud visualization

PDF

Adaptive and resilient stream processing on cloud infrastructure

PDF

Advanced machine learning techniques for video, social and biomedical data analytics

PDF

Improving efficiency, privacy and robustness for crowd‐sensing applications

PDF

Statistical inference for dynamical, interacting multi-object systems with emphasis on human small group interactions

PDF

Distributed resource management for QoS-aware service provision

PDF

Dynamic graph analytics for cyber systems security applications

PDF

Learning controllable data generation for scalable model training

PDF

Ubiquitous computing for human activity analysis with applications in personalized healthcare

PDF

Incorporating aggregate feature statistics in structured dynamical models for human activity recognition

Asset Metadata

Creator Liu, Leslie Shihua (author)

Core Title Scalable peer-to-peer streaming for interactive applications

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 08/02/2007

Defense Date 05/09/2007

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag dynamic overlay streaming,MMOG,multimedia streaming,OAI-PMH Harvest,peer-to-peer streaming,VoIP system

Language English

Advisor Zimmermann, Roger (committee chair), Kuo, C.-C. Jay (committee member), Neumann, Ulrich (committee member)

Creator Email lleslie@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m751

Unique identifier UC1319326

Identifier etd-Liu-20070802 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-540014 (legacy record id),usctheses-m751 (legacy record id)

Legacy Identifier etd-Liu-20070802.pdf

Dmrecord 540014

Document Type Dissertation

Rights Liu, Leslie Shihua

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu