University of Southern California Dissertations and Theses

Understanding and optimizing internet video delivery

(USC Thesis Other)

Understanding and optimizing internet video delivery

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content UNDERSTANDING AND OPTIMIZING INTERNET VIDEO DELIVERY by Zahaib Akhtar A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (DEPARTMENT OF COMPUTER SCIENCE) December 2019 Copyright 2019 Zahaib Akhtar Dedication Dedicated to my parents Samina and Akhtar Ali for their love, encouragement and support. ii Acknowledgments Throughout the course of my PhD, I have been very fortunate to learn from a number of remarkable people. First and foremost, my advisor Professor Ramesh Govindan, to whom I owe a great deal of gratitude for his guidance at each step of this journey. This work would not have been possible without his continuous support. Second, I would like to express deep gratitude to a number of great collaborators including fellow PhD student Yun Nam at Purude as well as Professors Sanjay Rao and Ethan Katz-Bassett. I’m also indebted to collaborators at Conviva including Jibin Zhan and Hui Zhang for their support and last but not the least, collaborators at AT&T Research including Emir Halepovic, Shubho Sen and Shuai Hao for their continued guidance. Thank you all. I would like to thank members of my dissertation committee: Professors Antonio Ortega and Barath Raghavan for agreeing to review this manuscript and oering their valuable guidance. A special mention goes out to the folks at NSL for being valuable mates in sharing this journey and for providing the environment where this work was possible. Finally, I would like to thank my parents Samina and Akhtar Ali for their unwavering support, my wife Ayesha for putting up with the vagaries of a PhD student’s life and my three year old son Daneer for adding colour to days of hard slog. iii Contents Dedication ii Acknowledgments iii Contents iv List of Tables vi List of Figures vii Abstract xi 1 Introduction 1 1.1 Dissertation Overview ................................. 3 1.1.1 Understanding Video Management Planes.................. 3 1.1.2 Oboe: Specializing Adaptive Bitrate Algorithms .............. 4 1.1.3 AViC: Specializing CDN Caching....................... 5 1.2 Dissertation Organization ............................... 6 2 Understanding Video Management Planes 7 2.1 Introduction....................................... 8 2.2 The Video Management Plane ............................ 10 2.3 Goals, Methodology & Dataset ............................ 13 2.4 Characterizing Video Management Planes ...................... 17 2.4.1 Packaging.................................... 19 2.4.2 Device Playback ................................ 23 2.4.3 Content Distribution.............................. 27 2.4.4 Summary .................................... 30 2.5 Understanding Management Complexity....................... 32 2.6 Management of Syndication .............................. 35 3 Oboe: Specializing Adaptive Bitrate Algorithms 42 3.1 Introduction....................................... 43 3.2 Background and Motivation.............................. 46 3.2.1 Background on ABR Algorithms ....................... 47 3.2.2 Ensuring High QoE for All Users ....................... 48 3.3 Oboe Design ...................................... 49 3.3.1 Representing Network State.......................... 50 3.3.2 Oine Mapping of Network States ....................... 51 3.3.3 Online ABR Tuning .............................. 54 iv 3.4 Evaluation........................................ 57 3.4.1 Metrics ..................................... 57 3.4.2 Methodology .................................. 59 3.4.3 Oboe with RobustMPC ............................ 61 3.4.4 Oboe vs. Pensieve ............................... 63 3.4.5 Oboe with other ABR Algorithms...................... 68 3.4.6 Sensitivity experiments ............................ 69 3.4.7 Oboe Across Various Settings.......................... 71 3.4.8 Oboe Overhead................................. 73 3.5 Deployment Considerations .............................. 73 4 AViC: Specializing CDN Caching for Adaptive Bitrate Video 76 4.1 Introduction....................................... 77 4.2 Properties of Video Delivery.............................. 80 4.3 AViC Design ...................................... 85 4.3.1 Overview and Challenges ........................... 85 4.3.2 Eviction Policy ................................. 86 4.3.3 Admission Control ............................... 89 4.3.4 Performance Optimizations .......................... 92 4.4 Evaluation........................................ 95 4.4.1 Methodology .................................. 95 4.4.2 Performance Comparison ........................... 96 4.4.3 Ablation Study ................................. 103 4.4.4 Time and Memory Complexity ........................ 104 4.4.5 Sensitivity Analysis .............................. 105 5 Related Work 108 5.0.1 Understanding Video Management Planes.................. 109 5.0.2 Oboe: Specializing Adaptive Bitrate Algorithms .............. 110 5.0.3 AViC: Specializing CDN Caching for Adaptive Bitrate Video ........ 111 6 Conclusion and Future Work 113 Reference List 116 v List of Tables 2.1 Streaming protocol file extensions and sample URLs ................ 14 4.1 Features used by admission control classifier. ..................... 91 4.2 Byte hit ratio comparison for residential clients ................... 97 4.3 Object hit ratio comparison for residential clients.................. 97 4.4 Byte hit ratio comparison for cellular clients..................... 97 4.5 Object hit ratio comparison for cellular clients.................... 98 4.6 Byte hit ratio comparison for all clients. ....................... 98 4.7 Object hit ratio comparison for all clients. ...................... 98 4.8 Impact of AViC’s components on performance.................... 102 vi List of Figures 2.1 A video delivery pipeline. ................................ 11 2.2 Dataset characteristics with respect to view-hours and number of publishers. .. 14 2.3 Streaming protocols used in terms of percentages of publishers and view-hours for past 27 months ..................................... 18 2.4 CDF across publishers of percentage of view-hours served via DASH and HLS in the latest snapshot.................................... 19 2.5 Number of streaming protocols used by publishers (by % of publishers and by their view-hours)........................................ 20 2.6 Target platforms for video publishers......................... 22 2.7 Percentage of publishers supporting each platforms................. 22 2.8 Over time, percentage of view-hours, view-hours excluding 3 largest publishers, and views on each type of platform.......................... 23 2.9 CDF of individual view duration for each platform ................. 24 2.10 Analysisofnumberofplatformssupportedbypercentageofpublishersandview-hours 25 2.11 Percentage of view-hours served by specific devices belonging to the same platform. 26 2.12 Analysis of CDNs based on percentage of publishers and view-hours for past 27 months ......................................... 27 2.13 Analysis of number of CDNs used by publishers and the view-hours. ....... 28 vii 2.14 Correlation between dierent measures of complexity and publisher view-hours . 32 2.15 Content syndication is prevalent in our dataset, with some content owners syndi- cating to nearly half the full syndicators in our dataset. .............. 35 2.16 AveragebitrateperformanceofCaliforniabasediPadclientsofownerandsyndicator across dierent ISPs and CDNs. ............................ 36 2.17 Rebuering performance of California based iPad clients of owner and syndicator across dierent ISPs and CDNs. ............................ 37 2.18 Bitrate selection decisions for an episode of a popular series by the owner and ten syndicators. ....................................... 38 2.19 Storage savings under dierent syndication models for content served by an owner and two syndicators. .................................. 40 3.1 Performance of ABR algorithms using dierent configurations for two sessions with dierent throughput behaviors ............................ 45 3.2 Illustrating how policy for setting discount factors in MPC impacts performance for dierent traces ................................... 45 3.3 The logical diagram of the oine pipeline used by Oboe ............... 51 3.4 Logical diagram of Oboe’s online pipeline ...................... 55 3.5 A scatter plot of average bitrate and rebuering ratio between the VirtualPlayer and real Dash.js player................................. 58 3.6 The percentage improvement in QoE-lin of MPC+Oboe over RobustMPC for the Testbed experiment. The distribution of average bitrate, rebuering ratio and bitrate change magnitude for the schemes is also shown. .............. 59 3.7 QoE-lin of MPC+Oboe compared to RobustMPC ................ 59 3.8 An example session showing how MPC+Oboe is able to outperform RobustMPC by reconfiguring the discount parameter when a network state change is detected. 62 viii 3.9 The percentage improvement in QoE-lin of MPC+Oboe over Pensieve for the 0-6 Mbps throughput region. The distribution of average bitrate, rebuering ratio and bitrate change magnitude for the schemes is also shown. ........... 63 3.10 Validation of our training methodology for Pensieve. ................ 64 3.11 QoE-lin of MPC+Oboe compared to Pensieve................... 64 3.12 Benefits of specializing Pensieve models. Each curve shows the QoE improvement of MPC+Oboe relative to each Pensieve model.................... 64 3.13 QoE improvement of MPC+Oboe over two ways of dynamically selecting from specialized Pensieve models............................... 64 3.14 Percentage improvement in bitrate and rebuering of BOLA+Oboe over BOLA (a),(b) and HYB+Oboe over HYB (c), (d)...................... 66 3.15 Average QoE-lin of MPC+Oboe with various throughput predictors ...... 66 3.16 Comparing HYB with multiple fixed configurations and HYB+Oboe for various settings ......................................... 68 3.17 Avg. of avg. bitrate and fraction of sessions with rebuering for HYB+Oboe and dierent publisher preferences ............................. 72 3.18 Avg. of avg. bitrate and fraction of sessions with rebuering for RobustMPC and dierent publisher preferences ............................. 72 3.19 Comparing prototype Oboe with commercial client side ABR implementation in average bitrate and rebuering ratio. ......................... 74 3.20 Time between consecutive bitrate switches for two commercial ABRs....... 74 3.21 Variance in bitrate levels across videos from two content publishers. ....... 74 4.1 Serving video over the Internet ............................ 78 4.2 Chunk size distribution for cellular and residential clients .............. 81 4.3 Intra-session request interval of client players ..................... 81 ix 4.4 Caching potential of requests ............................. 82 4.5 Popularity of videos which contributed singletons.................. 82 4.6 Bitrate usage across dierent types of clients. .................... 82 4.7 Hierarchical arrangement of max-heaps........................ 92 4.8 Performance comparison of a modified GDSF algorithm .............. 99 4.9 Savings in bandwidth and requests to the CDN Origin................ 100 4.10 Savings in bandwidth and requests to the CDN Origin................ 103 4.11 Byte hit ratio sensitivity to N............................. 103 4.12 Importance of features to GBDT classification. ................... 106 4.13 Correlation of video popularity over time. ...................... 106 x Abstract In this work, we strive to understand and optimize the performance of systems which serve video over the Internet. Towards this end, we make the following three contributions. First, we perform a systematic study of the video management plane – the systems which form the video delivery pipeline. Second, we propose an approach called Oboe to improve the performance of client side video players by tuning Adaptive Bitrate Algorithms to the characteristics of a network. A range of dierent algorithm when augmented with Oboe improve their performance by upto 25%. Finally, we propose a caching algorithm called AViC which is specifically tailored for video workload. AViC outperforms state of the art caching algorithms by exploiting various properties of adaptive bitrate video. In particular, a LRU cache requires up to 3.5◊ the cache size used by AViC to match its performance. xi Chapter 1 Introduction 1 Video forms overwhelming majority of the Internet trac today and is further expected to grow in the future. This deluge in video trac is driven by well known services such as YouTube and Netflix as well as a number of rapidly growing services which provide TV-like experience over the Internet [37, 24]. For these video services, providing high quality of experience to users is of great importance. On the surface, a video stream over the Internet consists of a series of HTTP requests between a server and an end user device. However, underneath, it requires an intricate pipeline which may include a content publishing system, a Content Delivery Network (CDN), an ISP and finally the client’s video player. The performance of each of these components has an impact on the end user’s quality of experience. While much progress has been made, video systems still suer from sub optimal tail perfor- mance, causing loss of revenue for service providers. A common reason behind these failures is that these systems often do not specialize themselves. This lack of specialization can manifest in dierent ways. For example, client side adaptive bitrate algorithms which operate over highly variable throughput regimes do not specialize to dierent types of network conditions, hence performing well on average but not always. Another example is caching algorithms commonly used by content delivery networks. While CDNs serve a large fraction of video content, they use general purpose caching algorithms which perform sub optimally for video due to their inability to exploit properties of adaptive bitrate video. The central thesis of this dissertation is to explore the following hypothesis: if systems that serve Internet video are specifically tuned for it, then better performance can be achieved from them. The thesis proves this hypothesis. It does this by first performing a comprehensive analysis of the video delivery pipeline to identify its important components, which include (i) the client side adaptive bitrate algorithms, (ii) the content delivery network and (iii) the content publishing system. In the second phase, the thesis demonstrates performance improvement for two of these 2 components by specializing them. First, it picks adaptive bitrate (ABR) algorithms and shows that user quality of experience can be improved significantly by specializing a range of dierent ABR algorithms to network conditions. Second, it picks content delivery networks and shows that significantly higher cache performance as well as network eciency can be achieved by designing caching algorithms specialized for adaptive video workload. 1.1 Dissertation Overview In this dissertation we follow a two step approach to improve systems that serve video. In the first step, we make an attempt to systematically understand and characterize the Internet video delivery pipeline. In the second step we pick two concrete components in the pipeline and demonstrate how specializing them achieves performance improvement. 1.1.1 Understanding Video Management Planes ForavideopublishertoservevideoovertheInternet, itneedsto(i)preparemultiplebitratesof the video, break the bitrates into chunks, encapsulate them using a streaming protocol (ii) develop and maintain video player softwares for end user devices and (iii) distribute the video chunks to content delivery networks. These tasks can be collectively described as video management plane operations. Publishers today assemble their own video management planes which can be very diverse. They may use dierent technologies or third party services from a range of providers for each of management plane operations described above. To understand how publishers organize these management planes, we analyze 100 dierent commercial video publishers over a period of two years. Our analysis reveals great diversity in the management planes of today as well as their various shortcomings. For example, we find that most premium content publishers today use more than one instance of streaming protocol, content delivery networks and player softwares. Further, 3 we also find that troubleshooting failures in these management planes can be quite complex and this complexity grows with the publisher size. Our analysis of the video management plane not only gives us a comprehensive picture of the current day practices but also highlights important components which we can improve through specializations. For example, one important component we identify are the client side adaptive bitrate (ABR) algorithms. Publishers today encode videos using bitrate levels spread over a large range so that client side players can intelligently choose from a range of these available bitrates. Adaptive bitrate (ABR) algorithms are hence central towards determining user’s quality of experience. So we first focus on specializing these ABR algorithms. 1.1.2 Oboe: Specializing Adaptive Bitrate Algorithms ABR algorithms are employed by client side video players to select appropriate bitrates given the network’s throughput. These ABRs try to maximize the average bitrate while minimizing player stalls. In recent years a number of ABR designs have been published which can be broadly categorized as either (i) designed adaptation: algorithms which use heuristics or (ii) data-learnt adaptation: algorithm which use machine learning techniques. A key limitation of the ABRs belonging to designed adaptations category today is that while these ABR algorithms have tunable parameters, they use fixed values for these parameters. As a result, these ABRs tend to perform well only over a limited range of network conditions encountered in the wild. However, content publishers are interested in providing high quality of experience to all their users. To increase the range of these ABRs, we propose a system called Oboe. Oboe is a meta-system which can augment an ABR in the designed adaptations category. It does this by dynamically tuning its parameters to the current network conditions in an online manner. We demonstrate that this form of specialization imparts superior performance to three qualitatively dierent ABR 4 algorithms. We also demonstrate that that when augmented with Oboe, a designed adaptations based algorithm can outperform a state of the art data-learnt ABR. 1.1.3 AViC: Specializing CDN Caching During our analysis of the video delivery pipeline, another important component we identify are Content Delivery Networks (CDN). CDNs rely greatly on caching to not only reduce response times to end users but also to shield their origins from a high request rate. As such, caching algorithms attain great importance for achieving high CDN performance. Using the same theme of specializing video systems, we next move on to specializing the caching algorithms used by these Content Delivery Networks. While the majority of trac served by CDNs today comprises of video, these CDNs use general purpose caching algorithms which do not perform optimally for video workload. Most caching algorithms assume the well-known independent reference model [88]. However, this assumption is not necessary suitable for video resulting in low object and byte hitrates. To improve the caching performance for video, we propose an algorithm called AViC 1 . AViC is a specialized caching algorithm for a video workload. It is based on the insight that gradual pacing of requests in a video session makes it possible to estimate in advance, when a chunk will be requested again in the future. This estimation can then be exploited to evict those chunks first which are unlikely to be requested again soon. We show through trace driven evaluations that AViC is able to outperform a range of classical caching algorithms as well various state of the art research prototypes on both object and byte hitrates. 1 AViC stands forAdaptive bitrateVideoCache 5 1.2 Dissertation Organization This dissertation is organized as follows. In Chapter 2, we present a detailed analysis of the video management plane. We define what a management plane comprises of and then analyze how publishers set up dierent components of the management plane. We also quantify the complexity of dierent management plane operations and end the discussion with an exploration of potential improvements that can be made to the management plane. In Chapter 3, we discuss Oboe which is a system to dynamically tune various ABR algorithms to network conditions. We provide background on ABR algorithms as well as their shortcomings. We then outline Oboe’s technical design and finally evaluate its performance using trace based simulations and testbed experiments. We also present preliminary results from a pilot deployment of Oboe. In Chapter 4, we discuss AViC which is a caching algorithm for video workload. We motivate why designing algorithms specifically tailored for video is important for CDNs. We then describe key characteristics of video which can be exploited for caching purpose. Next, we present design of AViC in the light these insights. Finally, we evaluate AViC using trace based simulations and demonstrate the eectiveness of its approach compared to other caching algorithms. Chapter 5 presents related work and Chapter 6 concludes with final remarks on future work. 6 Chapter 2 Understanding Video Management Planes 7 2.1 Introduction Video forms the overwhelming majority of Internet trac [ 49, 52, 55, 42]. The deluge in video tracisduebothtothepopularityoflargeserviceslikeYouTube,Netflix,andFacebook[ 48,31,36] and to the significant increase in Internet video services provided by publishers who traditionally produced content for broadcast television [56]. An Internet video publisher must (i) split the video into chunks, encode each chunk at one or more bitrates, and encapsulate chunks using a streaming protocol; (ii) develop and maintain playback software for the wide range of user devices; and (iii) distribute video to Content Delivery Networks (CDNs). We refer to these tasks as video management plane operations (§2.2), as distinct from control plane operations that involve selecting which CDN to direct a user to and what bitrate to choose for each chunk, and data plane operations that involves transporting each chunk to the end user. Whereas the data and control planes have received much attention (e.g., [110, 156, 125, 142, 100, 114, 132, 62]), video management plane decisions have been relatively unexplored, even though they impact how many users and devices a publisher can reach, the computation and storage requirements of content publishers, the complexity of troubleshooting, application performance, and the eort needed to incorporate control plane innovations such as new bitrate selection algorithms [110, 156, 125, 142, 116]. This chapter characterizes aspects of video management planes for more than 100 content publishers (§2.3), including 7 of the top 10 subscription video publishers [12], as well as prominent sports and news broadcasters and on-demand video publishers. Our dataset comes from a CDN broker [131] and contains metadata for over 100 billion video views, including metadata about the client (device and application used), video (publisher, URL, playback duration), and delivery (CDN, performance metrics). The aggregate daily view-hours across all our publishers are comparable to reported values for Facebook and Netflix. 8 Two aspects make our data unique relative to published industry reports [12, 54, 17](§5). Our data spans 27 months, enabling analysis of management plane practices over time. It also lets us assess view-hours (the total number of hours content is viewed) and views (the total number of video sessions) for any slice of the data (e.g., how many view-hours or views can be attributed to mobile apps). Contributions. First, we characterize video management planes along three key dimensions (§2.4): streaming protocols, playback devices and platforms, and CDNs. For each dimension, we characterize (i) how each instance (e.g., a specific streaming protocol, or a specific platform category such as the set-top box) has evolved across publishers, and over time; and (ii) the number of instances of each dimension used by a given publisher and its evolution, and how this correlates with the publisher’s view-hours. Several common themes run across our analysis of these three dimensions. We find that, despite significant changes over the 27 month period, no single dominant alternative has emerged along any dimension. Among streaming protocols, HLS and DASH have significant usage, while view-hours are almost evenly distributed across 3 CDNs and across 3 platforms (browser, mobile, and set-top). Moreover, more than 90% of view-hours can be attributed to publishers who support more than 1 protocol. The same is true of publishers who use more than 1 CDN, and publishers who support more than 1 platform. Publishers with more view-hours tend to support more choices of protocols, platforms, and CDNs. Beyond these, our analysis uncovers some new findings: set-top boxes dominate by view-hours; almost 80% of view-hours are from publishers that support 4-5 CDNs; and a significant fraction of publishers who use multiple CDNs segregate live and on-demand trac by CDN. Our analysis also adds color to known findings. For example, DASH usage has increased, but this growth is being driven entirely by large publishers. Moreover, while mobile app views have indeed increased, view-hours have not proportionally increased because view durations on mobile tend to be short. 9 Second, we take an initial step towards quantifying the impact of three dimensions of diversity on the complexity of management plane operations such as software maintenance, failure triaging, and packaging overheads (§2.5). We find that metrics that approximate the complexity of these operations for a publisher are sub-linearly correlated with the publisher’s view-hours. For example, a publisher with 10◊ as many view-hours as another will tend to maintain 1.8◊ as many versions of its video playback software. Third, we demonstrate that today’s management plane practices may not be well suited for content syndication (§2.6), in which syndicators license and redistribute content from a content owner. Syndication is prevalent in Internet video, yet syndicators run video management planes that are independent from those of content owners. As a result, we find cases where, for the same content, owners’ clients observe significantly dierent delivery performance than syndicators’ clients. We also find that more integrated management planes between owners and syndicators can reduce CDN origin storage requirements for a popular video series by 2◊ . Our results further our understanding of video management planes and open the door for research into new syndication models, complexity metrics, and approaches to cope with diversity and reduce management complexity. 2.2 The Video Management Plane Avideo publisher makes available online live and/or stored video content. Video content is encoded in dierent formats, delivered by one or more CDNs, and delivered to playback software on user devices. The video control and data planes together achieve chunked adaptive streaming: the data plane streams video chunks over HTTP, and the control plane adaptively determines, based on network conditions, which bitrate a chunk is downloaded at, and from which CDN. Each publisher operates a video management plane, a term we use for a pipeline of automated systems (some with humans in the loop), Fig. 2.1 shows one such pipeline, that perform two 10 Figure 2.1: A video delivery pipeline. primary functions. The first function prepares video content for delivery to users. Preparation involves packaging the video content and distributing content to CDNs for delivery to users. The second function is to develop and maintain playback software for the wide range of devices on which video is consumed by users. 1 Packaging. Packaging achieves two goals: 1) preparing content for adaptive streaming and 2) generating the necessary information for an end user device to perform playback. Encoding. The first packaging step transcodes the master video file into multiple bitrates of encodings such as H.264 [25], H.265 [26] or VP9 [46]. A video bitrate encodes the video at a certain resolution and a certain quality. A given resolution can be encoded at dierent qualities, which dier in the degree of lossy compression applied to trade-o perceptual quality for reduced bandwidth. Publishers optionally use DRM (Digital Rights Management) software to encrypt the video so that only authenticated users can access it. 2 Each encoded bitrate of the video is then broken into chunks (a chunk is a fixed playback- duration portion of the video) for adaptive streaming and encapsulated using a Streaming Protocol 1 Other management plane functions, including accounting, billing, and fault isolation, are beyond the scope of this chapter. 2 This is orthogonal to TLS encryption of the video during transmission over HTTPS. 11 (discussed below). Some publishers support byte-range addressing, where clients can request an arbitrary byte range for a given bitrate instead of chunks. Streaming protocols. Streaming protocols define the encapsulation format for video chunks to enable delivery over the network. A number of streaming protocols are in use today including Apple’s HLS [5], Microsoft’s SmoothStreaming (MSS [29]), Adobe’s HDS [1] as well as an open standard MPEG-DASH protocol (DASH) [14]. Of these, Apple’s devices only support HLS, though recent Apple devices allow limited support for DASH [53]. Some protocols like DASH [14] can support any video encoding format, while others like HLS only support a fixed set of codecs [7]. Streaming protocols also specify metadata about the video necessary for adaptation by the control plane. This metadata is stored in a manifest file. The manifest contains information about a number of attributes including the values of available bitrates for adaptation, the audio bitrates, the time duration of an individual chunk and the URLs to fetch video chunks etc. Device Playback. The next function of the management plane is to support the range of devices on which a user can view the publisher’s content. To enable playback on them, publishers either provide Video Players embedded in web pages to permit browser-based viewing or Apps on devices that permit app-based content delivery. Browser-based video players today are either implemented using JavaScript inserted into webpages using native HTML5 support, or using external plugins such as Flash or Microsoft’s Silverlight. However, several types of devices such as set-top boxes (e.g. Roku, AppleTV), game consoles (e.g Xbox), smart TVs (e.g. Samsung TV), and mobile devices use app-based playback. To build these apps, publishers use device-specific SDKs (Software Development Kits, sometimes called Application Frameworks) which provide support for frame rendering, user controls etc. as well as bitrate adaptation logic [110, 156, 142, 125, 64, 116]. Because publishers may have to support dierent devices, and, for a given device, dierent SDK versions (since users may take 12 time to upgrade their device SDKs), at any given time publishers may have to maintain several versions of their app (one for each device-SDK version combination). Content Distribution. Publishers employ Content Distribution Networks (CDNs). Some content publishers such as YouTube and Netflix deploy their own CDNs. The publishers in our dataset serve their videos via third-party CDNs (though some also use private CDNs). To improve performance and availability, some publishers serve content through multiple CDNs [100, 114, 117]. Some publishers use a CDN broker to select the best CDN for a given client view [130]. Even some publishers who only use a single CDN use a CDN broker for management services such as monitoring and fault isolation. Most publishers proactively push content to CDNs. A publisher may either push packaged chunks to each of its CDNs, or may use a packaging service provided by a CDN. In the latter case, the publisher pushes the master video file (or live video stream), and the CDN performs the packaging on behalf of the publisher. The client playback software retrieves chunks using URLs in the manifest file. 2.3 Goals, Methodology & Dataset Goals. We want to characterize, at scale, publisher video management plane practices (with respect to packaging, CDN use, and device support) and how they have evolved over time. We also present preliminary analyses to understand the implications of these findings on the complexity of video management, and the performance of video delivery. Prior industry reports have explored related aspects of video management planes [54, 50, 17, 33, 28](see§5 for details). These studies have four shortcomings that we address in this chapter. First,theylackapublisher-centric focus,eventhoughpublishersmakevideomanagementdecisions. As a simple example, these studies do not reveal how many streaming protocols or how many CDNs a publisher uses, but these factors aect management complexity (§ 2.5). Second, these 13 Protocol Extension Sample URL HLS .m3u8, .m3u http://[...].akamaihd.net/master.m3u8 DASH .mpd http://[...].llwnd.net//Z53TiGRzq.mpd SmoothStreaming .ism, .isml http://[...].level3.net/56.ism/manifest HDS .f4m http://[...].aws.com/cache/hds.f4m Table 2.1: Streaming protocol file extensions and sample URLs (a) CDF of percentage of view-hours a day across publishers in March 2018 (b) Normalized number of publishers per month for past 27 months (c) Aggregated average view-hours a day for past 27 months Figure 2.2: Dataset characteristics with respect to view-hours and number of publishers. studies do not contextualize their results. For example, consider a finding that few publishers use DASH. If these publishers are large, they are more likely to drive adoption of DASH than if they are all small publishers. Third, most studies were one-o, but the video landscape is continuously evolving. Longitudinal trends can help understand how the video delivery ecosystem is likely to evolve in the short to medium term. Fourth, in part because they lack a publisher-centric focus, these studies do not shed light on a common practice in video delivery, content syndication (§2.6). ExtractingmanagementplanepracticesfromaCDNbrokerdataset. We use data from one of the largest brokers in the world. The broker is used by publishers to monitor playback 14 quality and to select CDNs. The broker provides a monitoring library which publishers integrate with their video players. The monitoring library reports per-view information to the broker’s backend. The broker collects data for devices including desktops, mobiles, and smart TVs. Our dataset spans 27 months (January 2016 to March 2018) and contains over 100 billion views. For each view, it identifies the video publisher; the manifest’s URL; device model (e.g., iPhone, Roku); the operating system (e.g., iOS, Android); HTTP user-agent (for browser views) or SDK and SDK version number (for app views); the CDN(s) that were used to deliver the content; 3 viewing time; and delivery performance (average bitrate and rebuering time). From this data, we can extract, for any given time window (e.g., a month), and for each publisher: which CDNs the publisher uses, and which devices the publisher’s content was viewed on. We also infer which streaming protocols a publisher uses by parsing the URL for the manifest file. Dierent streaming protocols use pre-defined file extension types for their manifest files (Tab. 2.1): for example, HLS manifest files typically use the .m3u8 file extension 4 . For each of these dimensions, we can associate three measures that can help contextualize the dimension. Our primary measure, and one used most often in the video industry [48, 36, 31], is the number of view-hours (i.e., the total viewing time in terms of hours). Using our data, we can examine, for example, the number of view-hours of a publisher’s content delivered from a given CDN, over HLS, to iPhones. In some of our analysis, we use the number of views associated with a particular dimension (e.g., the number of video plays delivered to Roku players). This is helpful to understand if view-hours were accumulated from a few long video views or many short video views. In some analyses, we also measure importance by the number of distinct videos seen. 3 During a single view, chunks may be downloaded from multiple CDNs. 4 There are two exceptions to this. RTMP can be detected from the protocol specification in the URL (RTMP instead of HTTP). Progressive downloading uses file extensions corresponding to video encodings, like .mp4 or .flv. 15 We do not have this data for all our publishers (some do not report video titles), so when we use this measure, it is an under-estimate of the total number of distinct videos. Dataset limitations. We only have data when publishers have integrated the broker’s monitoring library. So, we do not have data from publishers that do not use the library, including the 3 largest video publishers, YouTube, Netflix and Facebook. For publishers that do use the library, we cannot definitively dierentiate whether the appearance (or increase in view-hours) of a publisher in our data is due to actual growth or due to progressive on-boarding of the publisher. Publishers that use the broker may be predisposed towards using multiple CDNs, and the broker’s role in CDN choice means that trends in CDN usage in our data may or may not be indicative of trends in the larger Internet video ecosystem. The dataset does not include data necessary to investigate three aspects of video management: Digital Rights Management (DRM) usage, monetization, and encoding format. To study the video encoding codec, we could have downloaded all unique manifest URLs in our dataset. This is not only logistically dicult (13 million unique URLs in March 2018 alone) but also often requires user authentication, and we do not have subscriptions for all the publishers in our study. A macroscopic view of the dataset. Video consumption today is dominated by YouTube, Facebook, and Netflix, which contributed (according to 2016/2017 studies [42, 31]) 1 billion, 0.1 billion and 0.14 billion view-hours per day respectively. Beyond these three, there are a large number of video publishers that deliver online content. In March 2018, our dataset had more than hundred of publishers each with at least 3000 view-hours in a month. Across all our publishers, the aggregate daily view-hours is 0.06 billion per day, comparable to Facebook and Netflix in 2016 and 2017. Finally, the publishers in our study together serve 180 countries. Prominence. Our dataset includes many prominent publishers, including 7 of the top 10 US subscription-based video publishers [12]. Further, the publishers in our dataset are diverse, and include international sports broadcasters, major news outlets, on-demand movies/subscription 16 TV providers and entertainment content providers. Within these broad categories, the publishers in our dataset are prominent, including 3 of the top 5 Sports broadcasters and 4 of the top 5 Breaking News publishers according to Alexa category rankings [3]. Distribution of view-hours. In March 2018, the average daily view-hours of publishers in our dataset ranged from hundreds to tens of millions (actual numbers omitted due to confi- dentiality). Fig. 2.2(a) shows a CDF of percentage of view-hours a day across publishers. 84% of publishers contribute less than 1% of total view-hours. However, the top 5% of publishers contribute 65% of the total daily view-hours, while the biggest publisher contributes 33% of the daily view-hours. This trend is similar to studies of the web that indicate that the most popular web pages account for a large fraction of accesses [65, 76]. Longitudinal view. Fig. 2.2(b) shows the number of publishers in our dataset over time (for these, and subsequent analyses, we only count publishers that exceed 3000 view-hours per month). Over the course of 27 months, the number of publishers increased by 32%. Fig. 2.2(c) shows the average daily view-hours aggregated across publishers over the entire period. The total number of view-hours (solid line) grew by 5◊ over 27 months. Some of this growth was driven by new publishers that appeared over time, some by publishers on-boarding their viewers to our broker over time, and the rest by growth in viewership of existing publishers. The dotted line shows the view-hours over time for the approximately 45% of publishers that are present for the entire time period. These publishers experienced a 3◊ increase in view-hours. 2.4 Characterizing Video Management Planes We characterize video management planes along three dimensions: packaging, measured by the streaming protocols used; content distribution, measured by the CDNs used; and device playback, measured by types of devices and number of application frameworks. For each dimension d,we ask: 17 (a) Percentage of publishers that supported each streaming protocol over time (b) Percentage of view-hours by each streaming protocol over time (c) Percentage of view-hours (excluding largest publisher) by each streaming protocol over time Figure 2.3: Streaming protocols used in terms of percentages of publishers and view-hours for past 27 months • How has d evolved across publishers? • How has d evolved in terms of view-hours? For example, does a dominant practice result from a few big publishers or many small publishers? • What is the distribution across publishers of number of instances of d? Is it correlated with publisher view-hours? Our two-year dataset is too large to process every view, so we use a sequence of two-day snapshots taken bi-weekly. We use the last snapshot, taken in March 2018, for the third question. 18 2.4.1 Packaging Understanding the prevalence of dierent streaming protocols is important for several reasons. First, the amount of work/resource needed to package content is proportional to the number of streaming protocols supported by a publisher. Also the time taken to package content can add delay to live content distribution. Second, in some cases, support for a streaming protocol can directly impact the set of devices that can be supported: e.g., until recently publishers needed to support HLS to work with Apple devices (§2.2). Prevalence by streaming protocol. Streaming protocols include HTTP-based protocols as well as RTMP, a protocol for low latency video streaming services [141, 149, 18]. In our dataset, RTMP only accounted for 1.6% of the view-hours in January 2016 and 0.1% in March 2018. RTMP has compatibility issues with network middleboxes, scalability limitations [141, 149], and limited device support. For these reasons, our publishers prefer HTTP-based streaming protocols even though these protocols may add a few seconds of encoding and packaging delay to live streams. The rest of our analysis focuses on HTTP-based protocols. Across publishers. Fig. 2.3(a) shows the percentage of publishers that supported a given streaming protocol over time (the sum of percentages at any given point in time exceeds 100% because publishers can support multiple protocols). The rightmost point of each curve indicates the latest snapshot. In this latest snapshot, 91% of publishers support HLS, likely because many devices and players support it [38, 43, 27]. DASH and SmoothStreaming are currently supported Figure 2.4: CDF across publishers of percentage of view-hours served via DASH and HLS in the latest snapshot. 19 (a) Number of protocols supported by publishers, as%ofpublishersandwhenweightedbytheir view-hours (b) Number of protocols supported by publishers, bucketed by publisher view-hours (c) Average number of protocols supported per publisher over time Figure 2.5: Number of streaming protocols used by publishers (by % of publishers and by their view-hours). by around 40% of publishers, but HDS is only supported by 19% of the publishers. Over time, support for DASH has increased from 10% of publishers to 43%, corroborating a recent survey of video developers [54]. HDS has steadily lost support. The growth of DASH has not been at the expense of HLS or SmoothStreaming. Over time, HLS and SmoothStreaming support across publishers has remained steady. By view-hours. We can quantify usage of streaming protocols in terms of view-hours,unlike existingindustrysurveys[54,17]. Fig.2.3(b)showsthepercentageofview-hoursservedbydierent protocols over time. In our latest snapshot, HLS and DASH are dominant, each accounting for about 38-45% of the view-hours, with the other two being relatively small. Longitudinally, the most noticeable trend is the growth in use for DASH from 3% to 38% view-hours. We found that 20 this is due to a single large publisher that starts to become visible in the dataset from March 2017 onwards. This increase in DASH support noticeably reduces the fraction of view-hours attributable to the other protocols. When we remove this publisher (Fig. 2.3(c)), we observe that DASH support from other publishers only accounts for less than 5% of view-hours overall. To explain this further, Fig. 2.4 shows the distribution across publishers of the percentage of view-hours that used a given protocol, only considering publishers that support that protocol. Even though 40% of the publishers support DASH (Fig. 2.3(a)), half of them employ it for at most 20% of their view-hours (Fig. 2.4). In contrast, among the 90% of publishers that support HLS, half use it for at least 85% of their view-hours. Finally, in Fig. 2.3(b), the growth in DASH due to a single publisher also illustrates the complexity of a management plane operation: onboarding. In this case, this publisher slowly moved their clients to our broker over a period of one year, by gradually integrating the broker’s monitoring libraries into each of its device players. Number of protocols per publisher. Fig. 2.5(a) explores how many streaming protocols each publisher supports in the latest snapshot. Each group of bars corresponds to a given number of protocolsn and shows the percentage of publishers that usedn protocols (left) and the percentage of view-hours from publishers that used n protocols (right). While 38% of publishers support 1 protocol, these publishers account for less than 10% of view-hours. The use of 2 protocols is dominant (38% of publishers, accounting for nearly 60% of view-hours), and the use of 3 protocols is also significant. Fig. 2.5(b) presents the number of protocols used by publishers when bucketed by their view-hours. The left most bar corresponds to publishers with X view-hours or less (we do not specify X for confidentiality reasons). The second bar corresponds to publishers with X to 10X daily view-hours, the next bar to publishers with 10X to 100X view-hours, and so on. Each bar 21 Figure 2.6: Target platforms for video publishers corresponds to the percentage of publishers in a given bucket, broken down by the number of protocols used by the publishers. The tallest bar indicates that (i) over 35% of publishers have 100X to 1000X daily view-hours; and (ii) publishers in this bucket use from 1 to 4 protocols. Outside of publishers in the bucket with the least view-hours, more than 50% of publishers in all buckets use at least 2 protocols, with all publishers in the 10 4 X≠ 10 5 X bucket (right-most bar) using 2 protocols. A significant number of publishers in the intermediate buckets use 3 or 4 protocols. Fig. 2.5(c) shows changes in the number of protocols used over time. The lower curve shows the number of protocols averaged across publishers. The upper curve is the average weighted by the publisher’s view-hours. The weighted average is always higher, indicating larger publishers tend to use more protocols. Despite fluctuations, the average number of protocols has remained a bit below two, and the weighted average higher than two. This consistency is likely because the growth in DASH has coincided with the decline of HDS. Figure 2.7: Percentage of publishers supporting each platforms 22 (a) Percentage of view-hours on each platform (b) Percentage of view-hours on each platform, excluding 3 largest publishers (c) Percentage of views on each platform Figure 2.8: Over time, percentage of view-hours, view-hours excluding 3 largest publishers, and views on each type of platform 2.4.2 Device Playback Understanding the set of user devices that a publisher supports is important because (i) implementing and maintaining video players for a range of platforms requires significant eort; and (ii) the popularity of a platform can impact the publisher’s decision of whether to support it. Prevalencebyplatform. Video is consumed (Fig. 2.6) on a variety of devices which can broadly be classified into two platform types: browsers and apps. Video is consumed on desktops, laptops, tablets and mobile devices using browsers on these devices. Video is also consumed using apps on mobile devices, smart TVs, set-top boxes and gaming consoles. We now explore the prevalence 23 of video consumption across these 4 app-based platform categories, and across browsers (which includes browser usage on mobile devices 5 ). Across Publishers. Fig. 2.7 shows that, over the 27 month period, support has grown most significantly for set-top boxes and smart TVs (from under 20% of publishers to above 50% and 60% of publishers respectively today). There has also been growth in mobile applications, and, as expected, almost all publishers support browsers and mobile apps today. By view-hours. Fig. 2.8(a) shows that, by percentage of view-hours served by dierent platforms, browser viewership has declined from nearly 60% to less than 25% today. Despite publishers aggressively increasing support for smart TVs during the last two years, their share of view-hours has stayed at less than 5%. Interestingly, the set-top category has grown the most, with the largest share of view-hours (nearly 40%) in the latest snapshot, while mobile app viewership has stayed steady at about 20-25%. To understand whether large publishers bias our observations, Fig. 2.8(b) shows view-hour trends when we remove the three largest publishers, which account for half the view-hours in our dataset. There are some dierences in trends, with mobile app viewing surpassing all other platforms over time, and set-top viewing growing at a slower rate. However, the results overall are 5 While smart TVs and set-top boxes support browsers, we see very little video consumption on these browsers. Figure 2.9: CDF of individual view duration for each platform 24 (a) Percentage of publishers and view-hours by number of platform they supported in the latest snapshot (b) Percentage of publishers by number of plat- form they supported classified by view-hours in the latest snapshot (c) Average number of platform supported across publishers over time Figure 2.10: Analysis of number of platforms supported by percentage of publishers and view-hours qualitatively similar, indicating that platform usage trends in our dataset are not being driven by the largest publishers alone, unlike the trend with DASH adoption. By views. The growth in view-hours with set-top boxes could be caused by longer view durations or by more views. To investigate this further, Fig. 2.8(c) depicts the fraction of views served across dierent platforms (including the three large publishers). While views with set-top boxes have grown to 20% of views in the latest snapshot, they lag behind the set-top view-hour growth (to nearly 40% in Fig. 2.8(a)). Taken together, Fig. 2.8(a) and Fig. 2.8(c) suggest that mobile app views are of shorter duration, while set-top views are of longer duration. Fig. 2.9 confirms this intuition, depicting the CDF of individual view duration (in hours, with the X-axis 25 (a) Web browsers (b) Mobiles/Tablets (c) set-top boxes Figure 2.11: Percentage of view-hours served by specific devices belonging to the same platform. truncated at 1 hour) for each platform in our latest snapshot. Only 24% of mobile and browser views last longer than 0.2 hours, while more than 60% of set-top views last longer than 0.2 hours. Trends within platforms. An examination of trends of device usage within each of the top 3 platforms also shows interesting trends, some well-known, others less so. Among browsers, the view-hours for HTML5 increased from about 25% to nearly 60% within the two year period (Fig. 2.11(a)). This increase came at the expense of a reduction of view-hours in other browser- based players, especially Flash. Among mobile devices, view-hours for Android devices have increased significantly (Fig. 2.11(b)), and both Android and IOS have comparable viewership in the latest snapshot. Finally, among set-top boxes (Fig. 2.11(c)) Roku devices are dominant in terms of view-hours, but AppleTV and FireTV account for a non-negligible percentage of view-hours. Overall, the results indicate that a publisher must not only cope with multiple 26 platforms but also multiple devices within each platform, which can contribute to significant management complexity (§2.5). Number of platforms per publisher. Fig. 2.10(a) characterizes the number of platforms supported by publishers. Over 85% of publishers support more than one platform, with over 95% of the view-hours attributable to platforms that support more than one platform 30% of publishers support all 5 platforms and these publishers account for over 60% of the view-hours. As with other dimensions, the number of platforms supported increases with view-hours (Fig. 2.10(b)). For instance in the bucket corresponding to 10 3 X to 10 4 X view-hours, the vast majority of publishers support at least 3 platforms, and nearly half support all 5 platforms. Finally, Fig. 2.10(c) shows that the average and the view-hour weighted average of the number of platforms supported by publishers have increased by 48% and 37% respectively over the two year period. Publishers support more than 3 platforms on average in the latest snapshot, with the weighted average being nearly 4.5. 2.4.3 Content Distribution Once the content is packaged, it is distributed to end users using content distribution networks (CDNs). CDNs work by situating the content closer to the end user. Understanding this dimension is important because CDN usage can have significant performance impact [100, 114, 117, 132]. (a) Percentage of publishers by a CDN they used over time (b) Percentage of view-hours by each CDN over time Figure 2.12: Analysis of CDNs based on percentage of publishers and view-hours for past 27 months 27 (a) Percentage of publishers and view-hours by number of CDNs they used in the latest snap- shot (b) Percentage of publishers by number of CDNs theyusedclassifiedbyview-hoursinthelatest snapshot (c) Average number of CDNs used across publish- ers over time Figure 2.13: Analysis of number of CDNs used by publishers and the view-hours. Further, publishers can employ multiple CDNs which can lead to complexity in video management (§2.5). Prevalence by CDN. Across all publishers we observed 36 dierent CDNs in our dataset. This list included both regional and international CDNs. Further, some publishers had their own internal CDNs (sometimes used in conjunction with external CDNs). Of these, over 93% of the view-hours were served by 5 CDNs, indicating that video viewership is concentrated among a handful of CDNs. We analyze the opportunities arising from this consolidation in §2.6. Across publishers. Fig. 2.12(a) shows the percentage of publishers across time that use each of the top 5 CDNs (anonymized). One CDN, A dominates, with nearly 80% of the publishers 28 using it, while only 30% use the second most dominant CDN C. Longitudinally, these numbers have remained more or less steady. By view-hours. Fig. 2.12(b) shows the percentage of the view-hours served by each of these CDNs. In the current snapshot, 3 CDNs (A, B and C) each account for 20-35% of the view-hours, while the other 2 account for about 5% or less each. Some CDNs use anycast to direct a client to a particular server [81], but anycast is susceptible to BGP route changes that sever ongoing TCP connections, raising concerns that anycast may not be suitable for large transfers [150]. We discovered that 1 of the top 3 CDNs in our dataset uses anycast, suggesting that anycast route instability has not been a blocking factor in the reliable delivery of video chunks. Longitudinally, CDN A’s share of view-hours has halved, while CDN B’s share has increased from a few percent to about 20%. We examined the data a little further, and found that this change is driven by one large publisher who recently became our broker’s customer. This publisher uses CDNs B and C,drivingdown A’s share of view-hours. Again, these results highlight the importance of considering view-hours in the analysis, an aspect not considered by prior work (§2.3). Number of CDNs per publisher. We next characterize publishers by the number of CDNs they use. Fig. 2.13(a) shows the percentage of publishers that use a given number of CDNs, and the percentage of view-hours that may be attributed to these publishers in our latest snapshot. While more than 40% of publishers use only a single CDN, they account for less than 5% of the view-hours. In contrast, less than 10% of publishers use 5 CDNs, but these publishers account for more than 50% of view-hours. Fig. 2.13(b) shows the number of CDNs used by publishers classified by their view-hours. The results indicate that the percentage of publishers that use multiple CDNs increases with the number of view-hours attributable to the publisher. For example, at the extremes, all publishers 29 with more than 10 5 X view-hours use at least 4 CDNs, while all publishers with less than X daily view-hours use a single CDN. In the 10 3 X≠ 10 4 X bucket, the number of CDNs used ranges from 1 to 3, while in the 10 4 X≠ 10 5 X bucket, the number of CDNs ranges from 1 to 5. Finally, Fig. 2.13(c) shows the longitudinal trend for the average number of CDNs used by publishers, and the weighted average (weighted by the publisher’s view-hours). While there is some growth in the average CDNs per publisher (with the average exceeding 2 in the latest snapshot), the weighted average grows much faster and is nearly 4.5 in the latest snapshot. Live video has some dierent demands than video-on-demand (VoD), especially low end-to-end latency from video capture to viewing, and so we were interested in whether publishers favored particular CDNs for one type of content versus the other, perhaps due to dierent CDN features or latency. Of publishers which use multiple CDNs and serve both live and VoD trac, 30% use at least one CDN only for VoD trac, and 19% use at least one CDN only for live trac. In one extreme case, a publisher used one CDN for all its VoD trac and a dierent CDN for all its live video. However, most CDNs that were used exclusively for live content by one publisher were used exclusively for VoD content by another publisher. Thus, no CDN dominated others for live video, and our results seem to reflect opaque management plane decisions of publishers. We have left it to future work to explain the rationale for CDN choices amongst publishers. 2.4.4 Summary Several common themes run through our analysis of these three dimensions of management complexity. • In no dimension does a single alternative dominate in terms of view-hours. View-hours are roughly equal between HLS and DASH; across browser, mobile, and set-up; and across three large CDNs. 30 • More than 90% of view-hours can be attributed to publishers who support more than 1 protocol. The same is true of publishers who use more than 1 CDN, and publishers who support more than 1 platform. • Publishers with more view-hours support more choices in each dimension. The average number of choices, weighted by view-hours, is 2.2 for protocols, 4.5 for CDNs and 4.5 for platforms. • At least two of our trends (increase in DASH usage, and the emergence of 3 CDNs with comparable view-hours) are driven by a large publisher. However, large providers alone do not drive trends in platform usage. By assessing the distribution of view-hours, we observe new trends and provide additional insights on known trends: • By view-hours, set-top box usage is significant, even exceeding browsers and mobile apps. This sharp rise of set-top box usage is not well documented and can drive the adoption of higher resolution video such as 4K video. • Prior work has not quantified the distribution of multi-CDN usage. We find that almost 80% of view-hours are from publishers using 4 and 5 CDNs. While 2 or 3 CDNs are sucient for resilience or load balancing, additional CDNs appear to be necessary for improved coverage. • Given industry excitement with DASH [54], we expected to find significant DASH support among our publishers. While over 40% of our publishers support DASH, one large publisher accounts for almost all DASH view-hours. In working with publishers, we have experienced quality issues with DASH implementations, so it might take some more time for the DASH ecosystem to mature to the point where small publishers also use more DASH. • Our dataset shows negligible use of RTMP even though several of our publishers serve live content. RTMP provides low latency live streaming, but it has some scalability issues and lacks widespread device support. 31 (a) Management plane combinations (b) Protocol-titles (c) Unique SDKs Figure 2.14: Correlation between dierent measures of complexity and publisher view-hours • Other studies report high mobile view shares [33], and we find that these have indeed risen over time, but mobile app view-hours have not increased by a corresponding amount, because view durations on mobile devices tend to be small. • Prior work has quantified the demise of Flash, reporting a 96% drop in Flash views for one browser [41]. We find a much more modest drop, with about 40% of browser view-hours attributable to Flash, down from 60% at the beginning of our study. 2.5 Understanding Management Complexity Our results in §2.4 have shown that publishers must deal with significant diversity across all components of the management plane. This diversity can impact the complexity of management tasks. In this section, we propose measures to capture this complexity, and explore how these 32 measures correlate with publisher view-hours, an approximate indication of publisher size. A correlation indicates that management complexity is higher for large publishers and low for small publishers. If, however, even small publishers incur high management complexity, this may indicate a high barrier to entry since a publisher who targets modest viewership early in its business growth must still pay for high management costs. Some examples of video management tasks include: Software development and maintenance. Video publishers must build and maintain players for dierent devices and browsers. Typically, content publishers use device specific SDKs (released by device vendors, or third parties) [6, 32, 47]. A publisher may not only need to maintain multiple code bases corresponding to the dierent supported devices (§ 2.4), but may also need to support multiple versions of the SDKs to support legacy devices. Besides the one-time development cost, there is an ongoing maintenance cost associated with rolling out new features, and fixing software bugs. Packaging. For each video title, a publisher needs to package the content for dierent streaming protocols. Packaging may be performed by the CDN or other third parties [45, 17, 40, 4], but the associated overheads remain irrespective of who does the packaging. Failure Triaging. Troubleshooting video performance problems is challenging, and poor perfor- mance may be due to a CDN, the network or the user’s device [102, 115], or a combination of these factors [115]. In addition, performance problems may be associated with a particular streaming protocol (e.g., manifest files may have errors for specific protocols). Measures of management complexity. Each of these tasks has an associated complexity that depends on the three components we study (protocols, CDNs, and devices). We list below com- plexity measures for video management, motivated in part by prior work on quantifying complexity 33 in other domains such as web page complexity [77], and router configuration complexity [73]. Other measures are possible, and we have left an extended exploration to future work. Management plane combinations. One measure of complexity is the number of unique combinationsofCDN,streamingprotocol,andtheenduser’sdevicethatapublishersupports. This relatestothecomplexityoftriagingfailures. Afailurecanbecausedbyoneofthecomponents(e.g., CDN or protocol), an interaction between two components (e.g., a specific CDN’s implementation of HLS), or an interaction across all three components (e.g., we have observed a failure caused by the interaction between a Chromecast implementation using SmoothStreaming on a specific CDN). In the worst case, it might be necessary to examine all combinations to triage a failure: indeed, the broker whose data we use triages failures automatically by aggregating failure reports across all management plane combinations [11]. More broadly, failure triaging may also depend on other factors such as the choice of ISP which we do not consider in this chapter. Protocol-titles. The product of protocols used by a publisher and the number of unique video titles for a publisher captures the packaging costs for the publisher’s content 6 .Intuitively, each publisher has to package each title separately for each protocol. This measure determines the compute and storage resources needed to package the publisher’s contents and can impact the lag experienced by users for live content. Unique SDKs. Defined as the number of unique versions of SDKs and browsers supported by a publisher across all devices, this measure captures the software development and maintenance complexity. The metric may also relate to the complexity of triaging a failure related to the device, if the failure is specific to an SDK version or browser. Correlation between management complexity and publisher view-hours. Fig. 2.14(a) presents a scatter plot that shows how the management plane combinations metric (on a log scale) correlates with the view-hours (on a log-scale) served by the publisher. We also add a 6 To a first approximation. Packaging cost may also depend on the length of each title, which we do not have. 34 Figure 2.15: Content syndication is prevalent in our dataset, with some content owners syndicating to nearly half the full syndicators in our dataset. line of best fit using linear regression. The slope of the line shows that when the view-hours increase by a factor of 10, the number of management plane combinations increases by a factor of 1.72◊ , indicating a sub-linear growth in complexity with publisher size. Fig. 2.14(b) and Fig. 2.14(c) respectively show similar scatter plots and lines of best fit for both the Protocol-titles metric, and the Unique SDKs metric. Again, both graphs indicate that the complexity measures increase sub-linearly with publisher view-hours: when view-hours increase by an order of magnitude the Protocol-titles grows by 3.8◊ ,whilethe Unique SDKs metric grows by 1.8◊ with the biggest publishers having to maintain up to 85 dierent code bases. In each case, the linear fit is statistically significant, with p-values at the 0.05 level of significance smaller than 10 ≠ 9 . 2.6 Management of Syndication We next explore how today’s structure of management planes, where each publisher makes independent decisions on the choice of protocols, CDNs and playback devices, can result in sub- optimal performance when content is syndicated. Syndicators license and serve content obtained 35 from multiple content owners. Conversely, content owners may distribute their content through multiple syndicators. The prevalence of syndication. By manually inspecting the websites of each of our publishers, we classified about 44% of them as content owners and 43% as full syndicators who syndicate the entire content from an owner. The rest are partial syndicators which syndicate some, but not all content of an owner (e.g., a single series instead of an entire video library). By inspecting full syndicators’ websites, we can determine, for a given content owner, what fraction of full syndicators have syndicated that owner’s content. Fig. 2.15 shows the CDF of the percentage of syndicators used by each content owner. The figure shows that syndication is prevalent–morethan80%ofcontentownersuseatleastonesyndicator,and20%ofcontentowners syndicate their content to almost 1/3rd of all full syndicators. These numbers are conservative, since a content owner may use syndicators not in our dataset, and since we have not considered partial syndication. Overall, these numbers suggest that content syndication is significant in online video delivery. Incorporating syndication in management planes. Today, because each publisher runs an independent management plane instance, the easiest way to syndicate content is that the content owner provides a master or "mezzanine" copy of the content to each of its syndicators which then packages and distributes the content through its video management plane. (a) On ISP X, CDN A (b) On ISP Y, CDN B Figure 2.16: Average bitrate performance of California based iPad clients of owner and syndicator across dierent ISPs and CDNs. 36 (a) On ISP X, CDN A (b) On ISP Y, CDN B Figure 2.17: Rebuering performance of California based iPad clients of owner and syndicator across dierent ISPs and CDNs. In this independent syndication model, sub-optimal outcomes might result because syndicators can make independent decisions on video packaging choices. In this chapter, we illustrate two such outcomes to motivate why it is important to study video management planes: (a) dierent performance for the same syndicated content resulting from dierent bitrate choices; (b) redundancy in CDN storage usage because multiple copies of the same content can be stored on a CDN using dierent encodings or protocols. To quantify these, we focus on a popular series. In our dataset, we observe that this series is served independently by the content owner and by 10 other syndicators. For our bitrate analysis, we focus on a particular episode in the series across all publishers since the bitrates used to encode the video may vary based on the content [51]. Bitrate choices for syndicated content. As described before, bitrate choices decide, for each platform, the resolutions and qualities at which the video is available. With independent syndication, an owner and a syndicator can make dierent bitrate choices for the same episode of the same series. To obtain these bitrate choices, we downloaded the manifest files for the episode of interest. The manifest files provide information about the audio and video bitrates used in the encoding, as well as other information such as the duration of each chunk. In general, each publisher may have several manifest files for the same video based on factors such as the streaming protocol, type of device, and network connectivity (WiFi, 4G, Wired). For fair comparison, we 37 compare the manifest files served to the same type of device over the same type of Internet connection (WiFi, 4G, Wired). Bitrate choices can vary widely. Fig.2.18showsthevideobitratesusedbythesyndicators (S1 to S10), as well as the bitrates oered by the original content owner (O) for iPad devices over a WiFi network. The figure shows a significant dierence both in the number of bitrates, the range of bitrates, and individual bitrate choices. At one extreme, S2 encodes video into only 3 dierent bitrates, while at the other, S9 employs 14 bitrates. The owner uses 9 dierent bitrates and oers a bitrate that exceeds 8192 Kbps, while the highest bitrate oered by S1 is 7x lower and only a little above1024Kbps. Wehavealsoperformedasimilarcomparisonforotherdevicetypes,andobserved similar heterogeneity in bitrate decisions made by the content owner and various syndicators. Bitrate choices can impact performance. From our dataset, we can also observe the performance achieved by clients of some of these syndicators. Two widely used [92, 68] measures of video delivery performance are the average bitrate of each view, and the rebuering ratio (the fraction of the view that experiences rebuering). Figure 2.18: Bitrate selection decisions for an episode of a popular series by the owner and ten syndicators. 38 Fig. 2.16 shows the distribution of average bitrates observed by iPad clients of a syndicator (S7 in Fig. 2.18) and the owner, for our selected episode, across two dierent ISP/CDN combinations in March 2018. Further, we restricted the geo-location of clients to California, USA. Consistently, the content owner’s clients get much better average bitrates: at the median, the average bitrate of the owner’s clients is 2.5◊ that of the syndicator. Interestingly, clients of the owner also perceive lower rebuering ratios (Fig. 2.17), with almost 40% lower rebuering at the 90-th percentile. We observed similar results for other device, ISP and CDN combinations. While we cannot comprehensively answer why syndicators select widely varying bitrates, in this case, discussions with the owner and syndicator revealed that the owner selected its bitrates to provide better experience for its users, while the syndicator’s choices were dictated in part by the increased storage and encoding costs required for higher bitrates. Redundancy in CDN storage. Independent syndication can also result in redundant storage in CDNs. In this section, we explore this for a popular series syndicated by two syndicators from the owner. In this case, the owner stores the series content on two CDNs A and B, and uses 9 bitrates. One of its syndicators stores the same series on 3 CDNs A, B and C,butencodesthe content using 7 bitrates. Another syndicator stores the content on A, B and another CDN D, but encodes the content using 14 dierent bitrates. These dierent management choices for the same content arise largely because of the independent syndication model. We focus on a setting where publishers proactively push video content to a CDN origin server which serves cache misses from CDN edge servers [107]. This setting is commonly used in practice, especially for popular video content. We quantify the redundancy in storage in CDN origin servers. While there is likely some redundancy in edge servers as well, this is harder to quantify as that depends on content access patterns. Quantifying storage redundancy with independent syndication. To quantify redundant storage, we first compute the total storage required for the entire video series by (i) 39 Figure 2.19: Storage savings under dierent syndication models for content served by an owner and two syndicators. downloading the manifest files of each episode of the series for each of the three publishers; (ii) multiplying each bitrate in the manifest file by the duration of the episode, and summing over all bitrates to obtain storage per episode; and (iii) summing across episodes. This results in a total storage requirement of 1916 TB across the 3 publishers (content owner and two syndicators) for each of the common CDNs (A and B). We next explore the storage savings achievable for this series if a CDN removes redundant copies of chunks with the same, or similar bitrates (those within a small tolerance factor). This is motivated by the observation that the syndicators and content owners often have similar or identical bitrates (Fig. 2.18). This occurs in practice because, even though publishers make independent bitrate choices, they tend to follow guidelines recommended by streaming protocol specifications. For example, the HLS specifications recommend that publishers make available at least one bitrate under 192 Kbps and that each successive bitrate be within a multiplicative factor of 1.5-2◊ of the previous [7]. 40 Fig. 2.19 (left three bars) shows the absolute and percentage storage savings with the above model. Even with a 5% tolerance, CDNs A and B can save 316.1 TBs (16.5%) each, and at 10%, they can each save 865 TBs (45.2%). Integrated syndication. Our broker reports that at least one or two of the publishers in our dataset have attempted integrated syndication, in which the owner’s content delivery mechanism is integrated into the syndicator’s playback software. There are two variants of integrated syndication: (i) API integration, where the syndicator uses the owner’s manifest file and CDN; and (ii) app integration, where the syndicator embeds the owner’s app into its own. The rightmost bar of Fig. 2.19 shows that with integrated syndication, A and B each save 1257 TB (65.6%). In addition, especially with app integration, syndicators cannot choose dierent bitrates than content owners, so performance dierences similar to Fig. 2.16 are unlikely to arise. While integrated syndication has potential, many logistical challenges must be addressed to make it a reality. For instance, with app integration, syndicators have to integrate apps for every owner they syndicate from. While API integration is potentially logistically easier, accounting mechanisms must be developed to distinguish CDN usage by clients of the syndicator and the owner. Future work should explore better ways to improve the management of syndication. 41 Chapter 3 Oboe: Specializing Adaptive Bitrate Algorithms 42 3.1 Introduction Internet video forms a major fraction of Internet trac today [ 52], and delivering high quality of experience (QoE) is critical since it correlates with user engagement and revenue [49, 92, 109]. To deliver high quality video across diverse network conditions, most Internet video delivery uses adaptive bitrate (ABR) algorithms [111, 142, 157], combined with HTTP chunk-based streaming protocols (e.g., Apple’s HTTP Live Streaming, Adobe’s HTTP Dynamic Streaming). ABR algorithms (a) chop a video into chunks, each of which is encoded at a range of bitrates (or qualities); and (b) choose which bitrate level to fetch a chunk at based on conditions such as the amount of video the client has buered and the recent throughput achieved by the client. Within this general framework, ABR algorithms dier in how bitrate level selection decisions are made, and these decisions impact metrics such as the average bitrate or the rebuering ratio. We call these QoE metrics, because they have been shown to correlate well with QoE [92], but other perceptual video quality metrics [44] may also influence QoE. ABR algorithm design remains an active research area because content providers continue to be interested in improving the performance of video delivery. Current ABR algorithms perform well on average, but some users can experience poor delivery performance as measured by the QoE metrics. These users suer because ABR algorithms have limited dynamic range:theydo not perform uniformly well across the range of network conditions seen in practice because their parameters are sensitive to throughput variability (§3.2). Contributions In this chapter, we present the design of Oboe 1 (§3.3), a system that takes the first step towards overcoming these hurdles. Oboe improves the dynamic range of ABR algorithms by automatically tuning ABR behavior to the current network state of a client connection, specifically to throughput and throughput variability. 1 In orchestras all instruments tune to the Oboe. 43 Oboe’s design is based on the observation made by prior work [69, 159, 124, 118, 147] that TCP connections are well-modeled as traversing a piecewise-stationary sequence of network states (§3.3.1): the connection consists of multiple non-overlapping segments where each segment is in a distinct stationary network state. For each possible network state, Oboe pre-computes, oine, the best parameter configuration for a given ABR algorithm (§3.3.2). It does this by subjecting the algorithm, for each state, to dierent parameter values, and picking the one that results in the best performance. Then, during video playback, Oboe continuously uses a change-point detection algorithm to detect changes in network state and selects the parameter identified by the oine analysis as best for the current state. Thus, if a video session encounters varying network state during its lifetime, Oboe automatically specializes the ABR parameter to each state (§3.3.3). We have implemented Oboe and demonstrated several aspects of its performance through testbed experiments and trace driven simulations. First, Oboe significantly improves performance of QoE metrics for three qualitatively dierent ABR algorithms, one that makes bitrate switching decisions on buer occupancy alone (BOLA) [ 142], another that uses both throughput and buer occupancy (HYB, a widely deployed algorithm), and a third that also optimizes decisions across a finite lookahead horizon (RobustMPC) [157]. In each of these cases, Oboe results in significant improvement. For instance, Oboe reduces sessions with rebuering from 33.3% to 5.3% relative to RobustMPC while also significantly improving a composite QoE metric. Oboe, when applied to RobustMPC, also performs significantly better than a newly proposed approach called Pensieve that learns, from real traces (using reinforcement learning), how to adapt to a variety of network conditions. For nearly 80% of the sessions in our dataset, Oboe improves the same composite metric, with benefits exceeding 20% for 25% of the traces. Compared to Oboe, which can specialize parameters to individual network states, Pensieve is unable to specialize across the entire range of network throughputs. We have tried training specialized Pensieve models for dierent ranges of network throughputs and dynamically switching models based on 44 Figure 3.1: Performance of ABR algorithms using dierent configurations for two sessions with dierent throughput behaviors Figure 3.2: Illustrating how policy for setting discount factors in MPC impacts performance for dierent traces estimated session throughput. This helps, but a significant gap between the two approaches still remains (§3.4.4). While a variety of viable pathways exist to deploying Oboe, we focus on an architecture where Oboe and the entire ABR logic are deployed on the cloud which enables rapid evolution and fine-grain customizability. We show the viability of this architecture with results from a pilot deployment. 45 3.2 Background and Motivation The Internet video delivery ecosystem consists of hundreds of content publishers and hundreds of client side applications that stream video content to diverse user devices. Publishers, content delivery networks, and users all seek to improve user quality of experience (QoE). There are many factors that aect QoE including start up latency, the average bitrate for a video session, as well as the rebuering ratio (the percentage of time playback is stalled because of drained buer) [ 92]. Video players improve QoE using adaptive bitrate (ABR) algorithms which select bitrates for each chunk while (1) ensuring the bitrate seen by the user is as high as possible and (2) avoiding rebuering events at the client. Some ABR algorithms may also try to minimize the number of bitrate switches to make the playback smooth. Content publishers serve dierent types of content including VoD (Video on Demand) or Live broadcasts. They may also serve streams of dierent qualities ranging from HD (high definition) to SD (standard definition). These dierences impact how they serve videos. For example, publishers who serve VoD content can use player buers as large as 4 minutes [ 111], whereas publishers serving live content may have a time-to-live 2 requirement between 15-45 seconds. Similarly, based on the quality of streams they serve, publishers may use dierent bitrate levels or chunk sizes. Further, publishers may have dierent QoE objectives. For example, some may strictly prefer to minimize rebuering and others may relax their tolerance for rebuering to prioritize higher bitrates. We use the term publisher specifications to denote their choice of bitrate levels, chunk sizes, content type, and rebuering tolerance. 2 For live content, the time between the event and its broadcast to users. This bounds the maximum buer that aplayerstreamingaliveeventcanbuild. 46 3.2.1 Background on ABR Algorithms ABR algorithms fall in two broad categories: (i) those that use both prediction of network throughput and buer occupancy [ 157, 116, 145]; and (ii) those that are primarily based on buer occupancy [111, 142]. Within the above two categories, ABR algorithms can be designed using approaches ranging from heuristics to stochastic optimization. In §3.4,wediscussarecently proposed ABR algorithm based on a qualitatively dierent approach, reinforcement learning [ 125]. MPC:Throughputpredictionandbueroccupancywithlook-ahead. Selectsbitrate by solving an optimization problem MPC [157] predicts throughput of future chunk down- loads based on throughput samples of recently downloaded chunks, then uses this predicted throughput to select bitrates to optimize a given QoE function (§3.4) over a look-ahead window of 5 future chunks. The aggressive version of the algorithm (FastMPC) directly uses a throughput estimate obtained using a harmonic mean predictor. To compensate for throughput prediction errors, a more conservative version, RobustMPC, reduces predicted throughput by a discount factor1+d,where d is the maximum error in throughput predictions experienced in the last five chunk downloads. BOLA: Buer occupancy, selects bitrate by solving an optimization problem BOLA is a buer-based algorithm used in Dash.js [ 16], so it does not employ throughput prediction in making bitrate decisions [142]. It also models bitrate selection as an optimization problem which it solves for a given value of the buer. It uses a parameter “ which is a ratio of (i) a minimum buer threshold, below which it downloads the lowest bitrate and (ii) a target buer threshold which it tries to maintain. Conceptually “ controls how strongly the ABR should avoid rebuering [ 142]. Higher values of “ make the algorithm conservative. 47 HYB: Throughput prediction without lookahead. Selects bitrate using a simple heuristic An algorithm widely used in production (§3.5), HYB considers both the predicted throughput and current buer occupancy (HYB is short for hybrid). For each chunk, HYB picks the highest bitrate that can avoid rebuering. Specifically, if S j (i) denotes the size of chunk j encoded at bitrate i, B is the predicted throughput based on past samples, and L the length of the buer. HYB picks the largest bitrate i such that Sj(i) B <L◊ — . Here, — can have values between 0 and 1 (higher values represent aggressive ABR behavior). — can be tuned to oset prediction errors in throughput and to compensate for the greedy nature of the approach which may make it susceptible to future buering events owing to unexpected throughput dips. 3.2.2 Ensuring High QoE for All Users Despite widespread deployment, ABR algorithms continue to be an active area of research [111, 142, 157, 116, 145, 125]. This is because, while deployed ABR algorithms work well on average, they do not work uniformly well across all network conditions. A key reason for this is that ABR algorithms have parameters (which we henceforth refer to as configurations) that must be set in a manner sensitive to network conditions. ABR algorithms need to run on many dierent networks, ranging from cellular and WiFi networks at one end, to high-speed broadband connections at the other. Given this diversity, network conditions can vary significantly. Packet loss conditions can vary by an order of magnitude or more across the globe [97]. Network throughputs can also vary widely: for 90% of traces in a large dataset, the trace’s maximum throughput is more than twice its average throughput. Yet, unfortunately, most ABR algorithms today either employ fixed configurations or simple heuristics to adapt these configurations (§3.2.1). Figures 3.1(a) and 3.1(b) show how the choice of ABR configuration depends on network conditions. Figure 3.1(a) shows the bitrate and rebuering ratio for two client sessions with the HYBalgorithmforthreedierentvaluesofits — parameter, Cons(Conservative), Mod(Moderate), 48 and Aggr (Aggressive). The throughput behavior of the two sessions is shown in Figure 3.1(c). If a publisher prefers to eliminate rebuering, Mod is suitable for session A, but Cons is better for session B. Figure 3.1(b) shows that BOLA behaves similarly, with Mod being the preferred setting for session A and Cons for session B, to avoid rebuering. Figures 3.2(a) and 3.2(b) show the diculty in setting the discount factor with MPC, by comparing the performance of FastMPC (no discount factor), and RobustMPC (discount factor set by local heuristic) for two throughput traces with dierent characteristics. In each figure, the top subgraphs show the available throughput (green curve) and the throughput estimate of FastMPC (red) and RobustMPC (blue). For the left graph, although the throughput is generally good, the sudden variations force RobustMPC to make overly conservative bitrate decisions, as well as incur more bitrate switches. (bottom subgraph). In contrast, in Figure 3.2(b),the quicker and more frequent throughput changes (top subgraph) result in FastMPC experiencing rebuering (middle subgraph), while RobustMPC does not. This is just one example illustrating the diculty in picking parameters – in our evaluations (§ 3.4), we found that RobustMPC was itself too aggressive when selecting discount factors for some traces. While this section uses synthetic traces for illustrative purposes, our evaluations with real traces (§3.4) more extensively demonstrate the limitations of current approaches with respect to selecting parameters and the benefits of automatically tuning ABR parameters to network conditions. 3.3 Oboe Design Oboe aims to ensure good QoE for all users by enabling ABR algorithms to perform better across a wide range of network conditions. The configurations of many ABR algorithms are sensitive to network state, specifically to the value and variability of the available throughput between the client and the video server. For example, — in HYB should be smaller when available 49 throughputishighlyvariable, while“ inBOLAshouldbehigher. Thisexplainswhythealgorithms perform dierently for dierent values of parameters on a given client trace (§ 3.2.2). However, a line of prior work [69, 159, 124, 118, 147] has observed that network connections are piecewise stationary: that is, connections can be in one of several distinct states (§3.3.1), where each state is distinguished by stationarity in the statistical sense (informally, a process is stationary if its statistical properties including mean and variance do not change over time - see [135] for a more formal definition). Oboe leverages the piecewise stationarity of network connections to address the key challenge of sensitivity of configurations to network conditions. It does so using a two stage design: (a) an oine stage where it pre-computes the best configuration choice for each (stationary) network state (§3.3.2), and (b) and an online stage, where during a session, it detects changes in network state and applies the pre-computed best configuration for the current (stationary) state (§3.3.3). Oboecanalsoaccommodatepublisherspecificationssuchassessiontype(livevs.video-on-demand, time-to-live requirements), bitrate levels or any explicit QoE tradeos ( e.g.,preferencebetween rebuering and average bitrate) (§ 3.3.2), by using these to influence the selection of the best configuration for each (stationary) network state in the oine stage. 3.3.1 Representing Network State Most ABR algorithms today adapt bitrates based on the throughput (more precisely, goodput) achieved by recently downloaded chunks. This perceived throughput already accounts for network delays and loss-rates, as well as the dynamics of the underlying transport protocol. The network throughput along a path is not necessarily a stationary process [69, 159, 124, 118, 147]: flows at the bottleneck along a path may change over time resulting in changes to available throughput, or the bottleneck itself may shift [118]. An analysis of the throughput traces used in our evaluations (§3.4) confirms the lack of stationarity when applied to the entire trace. We 50 Figure 3.3: The logical diagram of the oine pipeline used by Oboe analyze throughput traces using the Augmented Dickey-Fuller (ADF [99]) test, a hypothesis test to check for stationarity in a time series. Our evaluations on a dataset of 15,000 video streaming throughput traces show that 59.5% were non-stationary (see §3.4.2 for details of the dataset), implying the presence of distinct mean and/or variance in dierent segments of the traces. However, prior work![69, 159, 124, 118, 147] shows that TCP connection throughput can be modeled as a piecewise stationary process; the connection consists of multiple non-overlapping segments where each segment is stationary and often lasts for tens of seconds or minutes (e.g., Figure 3.8). Moreover, Zhang et al. [159] show that the throughput in each segment may be modeled as an i.i.d. process. Motivated by these observations, Oboe defines network state s by a tuple<µ s ,‡ s >,where µ s is the mean and ‡ s the standard deviation of the client-perceived throughput in a (stationary) segment of the underlying TCP connection. 3.3.2 Oine Mapping of Network States To map network states to their optimal ABR configurations, Oboe uses a pipeline (Figure 3.3) consisting of three components – the ConfigEvaluator,the VirtualPlayer and the ConfigSelector. The ConfigEvaluator takes a stationary throughput trace as input, which represents a particular network state, and drives the exploration of dierent ABR configurations over this trace. It does so by using the VirtualPlayer which models the dynamics of an actual video player. The 51 VirtualPlayer interfaces with the ABR algorithm implementation and outputs the performance of dierent configurations of the ABR. Finally, the ConfigSelector compares the performance of dierent configurations and builds a ConfigMap, which maps a given network state to the best configuration. Generating throughput traces for ConfigEvaluator To explore configuration space of an ABR algorithm on each network state s, ConfigEvaluator needs a stationary throughput trace to represent s. To generate such a trace, we explored two dierent approaches. In one approach, we extracted stationary segments from real traces using oine change point detection ([ 8], described in §3.3.3). Change points capture points where the distribution changes. However, because we are not guaranteed coverage (i.e., not all states might be observable in real traces), we also explored a second approach which involved generating a synthetic trace for each s with s’s mean and standard deviation, assuming a Gaussian distribution for the throughput samples. This was motivated by Dinda et al. [124] who showed that the throughput of TCP flows of the same size in a given stationary segment may be modeled as a Gaussian distribution (also see §3.3.1). More recent work also shows that TCP throughput is well modeled as a Markov process, each of whose states may be modeled as a Gaussian distribution [143]. We found that Oboe with synthetic traces performed comparably to stationary segments from real traces. So, ConfigEvaluator uses synthetic traces. Specifically, ConfigEvaluator quantizes both mean and standard deviation of throughput using a quantum (in our experiments, of 50 Kbps), resulting in states (in our experiments, 10,000), spread over a two dimensional space (in our experiments, 0.05-10 Mbps) of throughput and standard deviation. For each state, we generate a synthetic stationary trace. We found that the benefits of finer quantization are marginal. 52 EstimatingABRperformancewithVirtualPlayerandpublisherspecifications Oboe uses VirtualPlayer, a trace-based simulator that mimics the behavior of an actual video player without downloading or rendering actual videos. It takes as input a throughput trace and outputs the QoE performance metrics of a video session for a specified ABR algorithm. We have validated VirtualPlayer in §3.4.7. In designing VirtualPlayer, we have decoupled ABR logic (Figure 3.3), so the same implementation of the ABR logic can be used in Oboe’s oine and online stage. Further, this design provides an interface to the ABR designer through which they can easily integrate their ABR algorithm with Oboe without having to know about Oboe’s internals. The VirtualPlayer also takes into account publisher specifications for bitrate levels, player buer sizes (determined by time-to-live requirements) and chunk size. These specifications are used by VirtualPlayer when it executes ABR algorithms on the input traces, ensuring that the resulting ConfigMap meets the publisher specifications. Finally, Oboe also allows the publisher to optionally express an explicit QoE tradeo such as maintaining the rebuering under a desired threshold x%. Oboe derives a ConfigMap that meets the rebuering threshold in a best eort manner. We evaluate the ecacy of this flexibility in § 3.4.7. Building the ConfigMap using ConfigSelector To build the ConfigMap, the ConfigEval- uator drives the exploration of dierent configurations for an ABR algorithm. For a given network state s, ConfigEvaluator sweeps through possible configurations of the ABR algorithm using the VirtualPlayer. For example, the — parameter in HYB can take values from 0 to 1, so ConfigEvaluator plays the trace for state s for multiple values of — (quantized for eciency, see below) in this range. Foreachparametervaluec i , VirtualPlayeroutputsaperformance vector V i =<v 1 ,v 2 ,...v m > where each v k corresponds to the values achieved by c i for a QoE metric (e.g., bitrate, rebuering ratio, and more generally join time and frequency of switching bitrates [92]). This set of 53 performance vectors with the corresponding parameter values are then sent to ConfigSelector for picking the best configuration. ConfigSelector takes the set of performance vectors and determines the best configuration from them using vector dominance. A configuration c i is said to dominate c j if V i element-wise dominates V j (i.e., each element of V i is better than or equal to the corresponding element of V j ). This step also takes into account any rebuering tolerance, and ConfigSelector applies this tolerance to select the maximal performance vector. Deferring the selection of the maximal vector for a given rebuering tolerance to this stage (instead of filtering vectors in the previous step) is beneficial: it minimizes recomputation by allowing Oboe to quickly compute a new maximal vector if the publisher changes the rebuering tolerance. At the end of this stage, Oboe obtains the ConfigMap, a complete mapping of each network state to its corresponding optimal ABR configuration. Two optimizations can be used to quicken the rate of exploration of the ConfigEvaluator. The first is to quantize the parameter sweep, so that configurations are evaluated at a coarser granularity. This trades o some performance for lower computational complexity. The second optimization is based on the observation that there is generally a monotonic relationship between parameter values and the performance. For instance, for HYB (§3.2.1), the rebuering ratio and average bitrate are monotonically non-decreasing with the parameter — . Based on this observation, we can instead use an O(logn) binary search of the configuration space instead of doing a full O(n) sweep of all configurations. 3.3.3 Online ABR Tuning Oboe uses the ConfigMap generated oine, and live throughput measurements from the video player to dynamically change ABR configurations during a video playback. It does this by using an online change point detection algorithm [58]. This algorithm identifies, in an online fashion, if 54 Figure 3.4: Logical diagram of Oboe’s online pipeline the distribution of the throughput samples has changed significantly, signaling a state transition. When a change point is detected, the algorithm also provides the new states’s mean and standard deviation. Oboe’s ChangeDetector (Figure 3.4) implements the change point detection algorithm, and the ReconfEngine is responsible for updating the ABR configuration based on a new network state and the ConfigMap. Change point detection algorithms Such algorithms analyze a time series and check if there are regions in the time series where the underlying distribution of the data changes to a dierent set of parameters. Oine change-point methods require the full time series to be available, whereas online methods work with a continuous stream of samples as they become available. We focus on online methods, since Oboe identifies change points for an in-progress session and dynamically changes configurations. While several techniques exist for change point detection [151, 90, 113, 155, 120, 136], we focus on probabilistic methods [70, 58, 95, 154]. Further, we use a Bayesian online probabilistic change-point detector [58] for two reasons. First, in [58], a sequence of observations can be partitioned into non-overlapping states such that the observations are i.i.d. conditioned on a given network state s. This view aligns well with the way we have defined a network state (§3.3.1). Further, the algorithm is fast and requires no prior knowledge about the data stream, matching our scenario. We use the implementation provided in [8] and integrate it with the ChangeDetector. 55 Detecting changes in network state During a video session, ChangeDetector is continually fed with a series of observations of the network throughput, which it uses to detect state changes. ChangeDetector calculates throughput and standard deviation by only considering those samples which belong to the current state. To generate inputs to ChangeDetector, one approach is to use each downloaded chunk to obtain a single throughput sample. However, this may be too coarse-grained, and prevent detection of changes in network state that occur during the chunk download. Instead, we use fine grained samples recorded at periodic intervals (tens of milliseconds) during the download of each chunk. Players such as Dash.js already periodically log intermediate throughput samples during a chunk download, so obtaining these samples does not incur any additional overhead. We only need to modify players to report these samples to Oboe. The set of samples are provided to ChangeDetector after the chunk download, and any change in state is only detected at the end of the chunk download. This is acceptable since any action that can be taken by the ABR algorithm (such as a bit rate switch) only impacts subsequent chunks. In the rarer case that an ABR algorithm abandons the download of a chunk that takes too long, the report is sent when the chunk download is abandoned. §3.4.8 evaluates the overheads of ChangeDetector. An alternative approach to changing configurations is to use an exponentially weighted moving average (EWMA) of the mean and standard deviation of throughput samples and to lookup the corresponding configuration. We experimented with such an approach and found its performance unsatisfactory. The approach can result in continual and unnecessary reconfigurations, since throughput may vary across samples even when the network is (statistically) stationary. Damping these changes can result in slow reaction times when a reconfiguration is actually beneficial. In contrast, Oboe (i) models the underlying TCP connection as a sequence of states; (ii) does not make changes to the configuration within a given network state; and (iii) only reconfigures when a state change is observed. 56 Reconfiguring ABR Algorithm When a change in the network state is detected, the ChangeDetector signals the change and the new network state s to the ReconfEngine. The ReconfEngine then searches a neighborhood of radius r in the ConfigMap to select the config- uration to use for state s. Specifically, if state s is a point in a 2-dimensional space of average throughput and standard deviation of throughput, then it picks the most conservative ABR configuration within a search radius r around s. It does this for two reasons. First, because Oboe quantizes the network states, it might not have precomputed the best configuration for s. Second, the estimated new network state s may have some error, for example, due to ineciencies in the client download stack [102]. Given these sources of uncertainty, Oboe chooses to be safe in its selection of the best configuration for s. Finally, ReconfEngine configures the ABR algorithm, and the reconfigured ABR algorithm is ready to compute the bitrate decision to be used for the next chunk at this point. 3.4 Evaluation In this section, we demonstrate Oboe’s ability to auto-tune three existing algorithms: RobustMPC, BOLA and HYB. We also compare an Oboe-tuned RobustMPC to Pensieve [125]. 3.4.1 Metrics Theperformanceofavideosessiondependsonmultiplefactors. Average bitrate andrebuering ratio were found to have the most impact on user quality of experience [92], though other factors such as changes in bitrates during a session can play a role [92]. There is no consensus on how to best capture a user’s QoE. Consequently, ABR algorithms today are designed to optimize dierent metrics. For instance, HYB and BOLA primarily maximize average bitrate subject to low rebuering. In contrast, other algorithms [ 157, 125] have been designed to optimize a QoE metric which is a linear combination of bitrate, rebuering and bitrate changes (smoothness). 57 Figure 3.5: A scatter plot of average bitrate and rebuering ratio between the VirtualPlayer and real Dash.js player With Oboe, our primary evaluation goal is to demonstrate the extent to which it can improve the underlying metrics that an ABR algorithm is designed for. Thus, our evaluations with BOLA and HYB focus on average bitrate and rebuering, while those with MPC+Oboe focus on the linear combination of QoE (which we refer to as QoE-lin,[157]), defined as follows. For a video with N chunks, let R i be the bitrate chosen for chunk i. Then, the magnitude of bitrate changes M may be defined as M = q N≠ 1 i |R i+1 ≠ R i |. If the session experiences a total of T seconds of rebuering, then, QoE-lin(p,c)= 1 N ú q i (R i ≠ pT ≠ cú M),where p and c represent scaling penalties applied to rebuering and changes in the session. This function may be viewed as the session QoE averaged over the number of chunks. For our videos that had a maximum bitrate of 4.3 Mbps, we use p=4.3 and c=1 as our default parameters (following previous work that set default rebuering penalty equal to the maximum bitrate value [ 157, 125]). Even when an algorithm optimizes a metric such as QoE-lin, it is important to understand the distributions of underlying factors. The underlying factors represent concrete application performance that publishers understand how to reason about. Moreover, a unified metric like QoE-lin can obscure important dierences. For example, two sessions may have the same QoE-lin but dierent performance in underlying metrics, leading to varied user experience. So, we also present graphs of these metrics. 58 Figure 3.6: The percentage improvement in QoE-lin of MPC+Oboe over RobustMPC for the Testbed experiment. The distribution of average bitrate, rebuering ratio and bitrate change magnitude for the schemes is also shown. Figure 3.7: QoE-lin of MPC+Oboe compared to RobustMPC 3.4.2 Methodology Implementation For RobustMPC, we used the implementation available at [35]. Our imple- mentation of BOLA [142] is from the Dash.js player. The implementation of HYB is a variant of the algorithm used in a large-scale deployment. These ABR algorithms and Oboe’s online stage (change point detection and ABR reconfiguration) run on the server in our experiments. Our client runs the Dash.js video player (version 1.2), a reference player implemented in JavaScript by 59 the MPEG-DASH forum [16]. We modified Dash.js to send client player state information (e.g. buer length, video play state and throughput measurements) to Oboe (§ 3.5). This player runs on the Google Chrome browser (version 61) in our experiments. In §3.5, we show that Oboe can also be run as a cloud service. Testbed setup Our evaluations measure ABR performance by delivering a video stream (the “EnvivioDash3” video from the MPEG-DASH reference videos [15]) from a video hosting server to a client, while varying network conditions using throughput traces from real user sessions. We use bitrates {300,750,1200,1850,2850,4300}kbps with a 4 second chunk duration and total length of 192 seconds. We focus on this video as it has been used in prior work [125], and we do not consider videos of longer duration because we only have throughput traces available for a video publisher that serves short music videos (as we discuss below). The video is hosted on an Apache server. Both the server and client software run on the same 8-core, 4 Ghz, Intel i7 commodity desktop with 12 GB RAM running Ubuntu 16.04. Between server and client, we emulate dierent network conditions using the Chrome DevTools API [23]. This allows us to control the upload/download throughput as well as latency using the Chrome-Remote-Interface based on throughput traces [9]. We use 571 throughput traces 3 from our dataset (discussed below) for this emulation. All our testbed experiments use a client buer of 2 minutes. Datasets We use throughput traces from real user sessions collected over a three month period. Each trace contains the individual chunk sizes and their download times for on-demand video sessions from a publisher that serves short (4-6 minute) music videos. We derive throughput by dividing the chunk sizes by their download durations. The traces contain sessions that used desktops with wired connections and also sessions on mobile devices using WiFi or cellular connections. Like previous work [157, 125], we primarily focus on traces that have less than 3 Available at https://github.com/USC-NSL/Oboe 60 6 Mbps average throughput, since this is the regime where bitrate switching decisions are likely to have QoE impact. We filtered out traces which were too short for playing our entire 192 second video, after which we obtained 5K traces from wired desktops and 4K sessions from WiFi or 3G/4G mobile devices. Our testbed experiments use a subset of 571 traces with roughly the same number of traces sampled from each of desktop and mobile clients. VirtualPlayer setup Recall that Oboe uses the VirtualPlayer to obtain a ConfigMap for any ABR algorithm. Since the majority of our results use an actual testbed with the Dash.js player, the benefits of Oboe in our evaluation results already arise despite any inaccuracies in building the ConfigMap on account of using the VirtualPlayer. That said, we have also verified that the VirtualPlayer does a good job of tracking the performance of the actual ABR algorithms. For instance, Figure 3.5(a) and 3.5(b) demonstrates this for the HYB algorithm. The figures shows the correlation for the average bitrate and rebuering ratio for 100 throughput traces randomly sampled from our dataset using HYB on the VirtualPlayer compared to using an actual Dash.js player. For both metrics, the graph closely tracks the y = x line indicating close correlation. Given these close correlations, we use the VirtualPlayer in §3.4.7 to explore Oboe’s performance over a larger range of diverse settings and our entire set of traces. 3.4.3 Oboe with RobustMPC We now demonstrate that Oboe can be used to auto-tune RobustMPC, the best performing variant of the MPC algorithms. The resulting MPC+Oboe uses the best value of the discount parameterdcorrespondingtothecurrentnetworkstate, replacingRobustMPC’sonlineadaptation based on throughput estimates obtained over the past 5 chunks (§3.2). 61 Figure 3.8: An example session showing how MPC+Oboe is able to outperform RobustMPC by reconfiguring the discount parameter when a network state change is detected. Figure 3.6(a) shows the CDF of the percentage improvement in QoE-lin of MPC+Oboe over RobustMPC. 4 MPC+Oboe improves QoE-lin for 71% of sessions, with an overall average QoE-lin improvement of 17.62% across all sessions. In particular, for 19% of the sessions, QoE-lin improves by more than 20%. For the sessions MPC+Oboe is unable to improve RobustMPC, its performance degradation is mostly under 8%. Figures 3.6(b), 3.6(c) and 3.6(d) show the constituent QoE metrics. While MPC+Oboe achieves distributionally similar bitrates as RobustMPC as shown in 3.6(b),it significantly reduces rebuering across sessions: the number of sessions with rebuering reduces from 33.2% to 5.3%. Further, it also achieves better playback smoothness by improving the median per chunk change magnitude by 38% (Figure 3.6(d)). Finally, Figure 3.7 shows the CDF of QoE-lin for MPC+Oboe and RobustMPC, and indicates MPC+Oboe distributionally performs better. Figure3.8illustrates,usingasinglesession,whyMPC+OboeperformsbetterthanRobustMPC. The top graph shows throughput as a function of time, which includes an initial stable state followed by a drop in throughput. The middle graph shows how the discount factor d of both RobustMPC, and MPC+Oboe vary (the predicted throughput for each system is reduced by a 4 The increase in QoE-lin over RobustMPC relative to the absolute QoE-lin value of RobustMPC expressed as a percentage. 62 Figure 3.9: The percentage improvement in QoE-lin of MPC+Oboe over Pensieve for the 0-6 Mbps throughput region. The distribution of average bitrate, rebuering ratio and bitrate change magnitude for the schemes is also shown. factor of 1 1+d ,where d is shown on the y-axis). During the initial stable state, when prediction errors are low, RobustMPC steadily lowers its discount factor leading to more aggressive bitrate selections (not shown). This results in a rebuering event 44 seconds into the session (lowest graph shows buer occupancy with 0 indicating a rebuering event). In contrast, MPC+Oboe does not incur a rebuering event and maintains a fixed d during the initial stable state. At 29 sec, it detects a change in the network state and adapts its discount factor, leading to more conservative bitrate selections. 3.4.4 Oboe vs. Pensieve Pensieve [125] uses deep reinforcement learning [129, 128], a combination of deep learning with reinforcement learning [144], and has been shown to outperform existing ABRs, including RobustMPC [125] in some settings. Since MPC+Oboe outperforms RobustMPC as well, we 63 Figure 3.10: Validation of our training method- ology for Pensieve. Figure 3.11: QoE-lin of MPC+Oboe com- pared to Pensieve Figure 3.12: Benefits of specializing Pensieve models. Each curve shows the QoE improvement of MPC+Oboe relative to each Pensieve model. Figure 3.13: QoE improvement of MPC+Oboe over two ways of dynamically selecting from spe- cialized Pensieve models. explore how MPC+Oboe performs relative to Pensieve. Our experiments use the Pensieve implementation provided by the authors [35]. Pensieve Re-Training and Validation Before evaluating Pensieve on our dataset, we retrain Pensieve using the source code on the trace dataset provided by the Pensieve authors [35]. This helps us validate our retraining given that deep reinforcement learning results are not easy to reproduce [105]. Weexperimentedwithfivedierentinitialentropyweightsintheauthorsuggestedrangeof1to 5, and linearly reduced their values in a gradual fashion using plateaus, with five dierent decrease rates until the entropy weight eventually reached 0.1. This rate scheduler follows best-practice [152]. From the trained set of models, we then selected the best performing model (an initial entropy weight of 1 reduced every 800 iterations until it reaches 0.1 over 100K iterations) and 64 compared its performance to the pre-trained Pensieve model provided by the authors. Figure 3.10 shows CDFs of QoE-lin for the pretrained (Original) model and the model trained by us (Retrained). The performance distribution of the two models are almost identical over the test traces provided by the Pensieve authors, thereby validating our retraining methodology. Having validated our retraining methodology, we trained Pensieve on our dataset with the same complete strategy described above. For this, we pick 1600 traces randomly from our dataset with average throughput in the 0-6 Mbps range. The number of training traces, the number of iterations per trace, and the range of throughput are similar to [125]. We then compare Pensieve and MPC+Oboe over a separate test set of traces also in the range of 0-6 Mbps (§3.4.2). Comparison with Pensieve Figure 3.9(a) shows the CDF of the percentage improvement in QoE-lin for MPC+Oboe over Pensieve. MPC+Oboe outperforms Pensieve for 81% of the sessions, with aQoE-lin improvement of 23.9% in average across all sessions. 25% of the sessions achieve more than 20%QoE-lin improvement. For the sessions MPC+Oboe is unable to improve over Pensieve, the performance dierence is mostly less than 5%. Figures 3.9(b), 3.9(c) and 3.9(d) show that MPC+Oboe distributionally outperforms Pensieve with respect to all underlying metrics. It reduces the number of sessions with rebuering from 10.7% to 5.3%, reduces the median per chunk change magnitude by 43.9%, and improves median and 95th percentile average bitrate by 2.6% and 4.7% respectively. Finally, Figure 3.11 shows the CDF of QoE-lin for MPC+Oboe and Pensieve, and indicates MPC+Oboe performs distributionally better. Analyzing Pensieve performance To understand where these performance improvements were coming from, we examined the relative performance of these two schemes in the 0-3 Mbps range (i.e., traces having an average throughput between 0-3 Mbps). In this more constrained range of network conditions, we found that MPC+Oboe achieves bigger gains over Pensieve (average QoE-lin improvement in 0-3 Mbps is 46.23%). We hypothesize that this performance 65 Figure 3.14: Percentage improvement in bitrate and rebuering of BOLA+Oboe over BOLA (a),(b) and HYB+Oboe over HYB (c), (d) Figure 3.15: Average QoE-lin of MPC+Oboe with various throughput predictors dierence stems from the fact that Pensieve builds a single model which does not specialize to dierent throughput ranges. To test this, we trained a separate Pensieve model only with traces that have an average throughput between 0-3 Mbps range and compared it with MPC+Oboe. Figure 3.12 shows the per session QoE-lin improvement of MPC+Oboe compared to Pensieve models trained for 0-3 Mbps (which we refer to as Pens-Specialized) and for 0-6 Mbps. The median QoE-lin improvement 66 with MPC+Oboe over Pens-Specialized is 10.49%, while the median improvement over Pensieve is 19.9%. This indicates specializing the model does improve Pensieve’s performance. Thus, Pensieve’s model is as yet unable to create specialized versions of itself based on the session characteristics. By contrast, Oboe specializes parameters for every network state and therefore performs better. We have also validated Pensieve’s inability to specialize in several other ways: building a model for the 3-6 Mbps and showing that it performs better with test data in that range compared to a 0-6 Mbps model; checking that a 0-6 Mbps model performs better for data in that range compared to a 0-100 Mbps model; and ensuring that these results hold even when the training set is doubled. It is hard to pinpoint exactly why Pensieve is unable to learn to be more conservative in the 0-3 Mbps range; deep neural network models remain a black box despite eorts by the machine learning community to make these models more transparent [137], and obtaining such understanding may need further advances in interpretable deep learning models. A model selector with Pensieve One way to improve Pensieve’s specialization might be to train dierent models for dierent throughput ranges and use the model more suited to the network conditions. To test the ecacy of this approach, we used two models (specialized for 0-3 Mbps and 3-6 Mbps), and tried two dierent model selectors.Pens-SelMultipleswitches models throughout the session, using the average throughput of the past 5 chunks. Pens-SelOnce starts with the 0-6 Mbps model, selects either the 0-3 Mbps or 3-6 Mbps model based on the average throughput of the first 5 initial chunks, and does not switch thereafter. Figure 3.13 shows CDFs of per-session QoE-lin improvement of MPC+Oboe over these selectors. MPC+Oboe is able to outperform both Pens-SelMultile and Pens-SelOnce, with average QoE-linimprovementsof14.2%and24.32%respectively. Eventhoughoneofthemodelselection schemes oers some improvements over the 0-6 Mbps Pensieve model, the benefits are modest. We hypothesize that this behavior is due to the dynamic selection of distinct Pensieve models, 67 Figure 3.16: Comparing HYB with multiple fixed configurations and HYB+Oboe for various settings which can interfere with reinforcement learning’s decision choices, since, during training, the reinforcement learning algorithm assumes there is no such third party intervention. 3.4.5 Oboe with other ABR Algorithms Oboe can also improve other existing ABR algorithms such as BOLA and HYB, which are designed to maximize average bitrate while minimizing rebuering. BOLA BOLA+Oboe tunes “ (§3.2), which determines how much the algorithm strives to avoid rebuering. BOLA, as implemented in Dash.js, uses a fixed default value of “ =≠ 10.28. Figure 3.14(a) and 3.14(b) show CDFs of per session performance improvement over BOLA with respect to average bitrate and rebuering ratio. BOLA+Oboe maintains the rebuering ratio of BOLA while improving average bitrates for more than 83% of sessions with an overall increase of 7.2% 68 in average across all sessions. For sessions where BOLA+Oboe does not outperform BOLA, its degradation is less than 3.1%. HYB The performance of HYB is sensitive to the choice of — parameter, which HYB+Oboe tunes. In production, HYB uses — =0.25, determined using A/B tests in a large-scale deployment. Figure 3.14(c) and 3.14(d) show CDFs of per session performance improvement of average bitrate and rebuering ratio over HYB. As with BOLA, HYB+Oboe maintains similar rebuering ratios as shown in 3.14(d), but improves bitrates for 98% of sessions with an overall average bitrate improvement of 8.32% in average across all sessions. 3.4.6 Sensitivity experiments Alternative throughput traces To understand how Oboe works on throughput datasets beyond those discussed in §3.4.2, we evaluated Oboe on two other datasets, FCC [19] and HSDPA [138] that have been used in recent work [157, 125]. FCC is a broadband dataset, while HSDPA contains throughput traces collected from video streaming sessions over 3G networks in Norway using mobile devices. Our comparisons use the traces and a Pensieve model pre-trained for those traces available at [35]. We focus our evaluations on MPC+Oboe and Pensieve, given that Pensieve has been shown to out-perform existing ABR schemes including RobustMPC on these traces. Our results show that MPC+Oboe continues to perform better than RobustMPC on these traces. Further, relative to Pensieve, MPC+Oboe improves QoE-lin by an average of 6.94% across the FCC dataset and 10.92% across the HSDPA dataset. These improvements are more modest than those in Figure 3.9(a). The vast majority of traces in the FCC and HSDPA set have an average throughput under 3 Mbps (over 95% for FCC and 98% for HSDPA). The results corroborate Figure 3.12 which indicates that MPC+Oboe provides more modest gains over Pensieve when the latter is trained and evaluated on datasets with a narrow throughput 69 range. MPC+Oboe provides larger gains in settings like the traces discussed in §3.4.2, where only 41% traces are under 3 Mbps and 59% are in the 3-6 Mbps range. Alternative throughput prediction methods Our experiments with RobustMPC rely on throughput prediction based on the harmonic mean of prior throughput samples (following earlier work [157, 125]), with Oboe tuning the configuration to compensate for prediction errors. We next consider if Oboe’s benefits hold if RobustMPC were to have more accurate throughput predictions, potentially by using alternate prediction methods [143]. Rather than using a specific prediction technique, we consider an ideal (and unachievable) approach that we denote as Ideal(T), which can exactly predict the average throughput over the next T seconds. Our experiments were conducted in simulation, using the VirtualPlayer, and the testbed experiment traces (§3.4.2). Figure 3.15 shows the average QoE-lin across the traces for RobustMPC and MPC+Oboe using both the default harmonic mean approach and Ideal(T) for dierent values of T. Although RobustMPC performs better with an ideal predictor, Oboe still provides benefits, achieving an average improvement in QoE-lin of 6.34% for Ideal(5) and of 1.8% for Ideal(10), compared to a 16.1% improvement with the harmonic mean estimator. While the magnitude of benefits is smaller with the ideal prediction approach, in practice Oboe will likely result in larger benefits, since even more sophisticated schemes [143] cannot achieve the ideal predictions, and the errors are likely to grow with larger T. Oboe can improve performance over RobustMPC even when an Ideal(T) prediction method is used for two reasons. First, T may not match the duration of chunk downloads with RobustMPC, which depends on the exact sequence of bitrates chosen during the look-ahead window. The durationisnotknownapriori, sinceRobustMPCitselfdeterminesthebitratesbasedonaprovided prediction. Second, the decisions made by RobustMPC are over a small look-ahead window, which may not guarantee optimality over the entire session duration. 70 3.4.7 Oboe Across Various Settings In §3.4.5 we have shown that Oboe outperforms other ABR algorithms when compared to their default configurations. We now explore, for HYB, whether Oboe outperforms all parameter settings of HYB and whether it can tune ABRs based on content type and publisher specifications. For these experiments, we use the VirtualPlayer described in §3.3.2. Comparison against all fixed configurations To explore dierent fixed configurations, we runHYBwith10dierentfixed — sandcomparewithHYB+Oboe. Wesummarizetheperformance for each configuration by considering the (i) median of the average bitrate and the (ii) 90th percentile of the rebuering ratio across test traces. In this experiment, we also consider an Oracle which is the best fixed configuration for each throughput trace with respect to two metrics that HYB tries to optimize. Figure 3.16(a) and 3.16(b) compare HYB, HYB+Oboe and Oracle over desktop, and mobile traces respectively. While Oracle and HYB+Oboe are depicted as single dots since their per- formance is uniquely determined, we present a frontier for HYB that shows its performance for dierent fixed configuration. Figure 3.16(a) shows that HYB+Oboe outperforms HYB in the sense that there is no fixed configuration for HYB that does better than HYB+Oboe performance. HYB+Oboe improves the average bitrates of the median session by 3.2%, while achieving similar rebuering ratio. Alternately, it reduces the 90th percentile rebuering ratio from 1.9% to 0%, while maintaining similar bitrates. A similar result holds for mobile traces (Figure 3.16(b)). Thus, even if publishers were to find the best fixed parameter choice for HYB, Oboe would outperform that choice because it dynamically adapts the parameters. Comparison under dierent publisher specifications Our results so far are for a VoD (video on demand) setting with a maximum buer size of 2 minutes. Figure 3.16(c) depicts performance for live video (which uses a maximum buer size of 20 seconds to mimic live settings). 71 Figure 3.17: Avg. of avg. bitrate and fraction of sessions with rebuering for HYB+Oboe and dierent publisher preferences Figure 3.18: Avg. of avg. bitrate and fraction of sessions with rebuering for RobustMPC and dierent publisher preferences HYB+Oboe outperforms HYB for this setting, though we note that the bitrate of both approaches degrades relative to the VoD setting since the baseline HYB switches to higher bitrates more conservatively owing to the smaller buer sizes. Finally, Figure 3.16(d) depicts performance for higher bitrate levels ({1002,1434,2738,3585,4661,5886}kbps) and a chunk size of 5 seconds. Even for these choices, HYB+Oboe outperforms HYB, demonstrating its ability to adapt to dierent publisher specification. Accommodatingpublisher’srebueringtolerance Oboeallowsthepublishertooptionally specify explicit rebuering preferences (§ 3.3). ABR algorithms such as RobustMPC which use the QoE-lin function may permit this indirectly by adjusting QoE-lin weights (§3.2.1). Figure 3.17 shows the eectiveness of these approaches, showing the average of average bitrates, and the fraction of sessions with rebuering for HYB+Oboe. As the publisher makes its rebuering preference stricter (from 2%-0%), HYB+Oboe achieves lower rebuering ratios close to the target rebuering tolerance. In contrast, Figure 3.18 shows that RobustMPC is less eective at controlling rebuering by adjusting its rebuering penalty when the weight on the rebuering term is varied between 100 (strictly avoid rebuering) to 4.3. 5 We find that even with a very high 5 We used a change penalty of 0 for fair comparison. 72 rebuering penalty of 100, RobustMPC causes rebuering in 11% of the sessions. This shows the benefit of Oboe’s approach which gives direct control over the underlying metrics. 3.4.8 Oboe Overhead Computing the ConfigMap incurs a one-time cost, since the map can be reused across all clients once the it is built. Computing the best parameter configuration for one network state takes about 12 seconds on a single core. This task is perfectly parallelizable, so computing 10K network states (§3.3) will take approximately 3.5 hours to explore with two machines of 48 cores each. We have also analyzed the processing overhead incurred by the ChangeDetector module of Oboe. We measure the time taken by ChangeDetector for every decision cycle across our experiments, and the measurement indicates that the median processing time of ChangeDetector is around 14 ms. Since each decision is made at a chunk boundary and chunks are 4 seconds, ChangeDetector accounts for less than 0.35% overhead. 3.5 Deployment Considerations The oine stage of Oboe can be run on the cloud, but several choices exist for the online stage, ranging from embedding the online stage entirely in the client player, or moving some or all of the online stage to the cloud. In our implementation, Oboe’s components run on the server side. This mimics a cloud implementation, which has the benefits of other cloud software: fast update deployment, device independence, etc. [34]. We leave a detailed comparison of these choices to future work, but explore, in this section, the feasibility of running the online stage on the cloud. To this end, we have implemented a restricted version of HYB+Oboe on AWS. This limited version of Oboe implements HYB and incorporates tuning based on publisher specifications but not network state. In our implementation, a client player periodically reports player state (such as buer length and current bitrate) and throughput samples to a Oboe cloud server and receives 73 Figure 3.19: Comparing prototype Oboe with commercial client side ABR implementation in average bitrate and rebuering ratio. Figure 3.20: Time between consecutive bitrate switches for two commercial ABRs Figure 3.21: Variance in bitrate levels across videos from two content publishers. bitrate decisions in return. For 10 player features and two chunk downloads per second, the communication overhead is 6.4 Kbps, negligibly small compared to the size of video chunks. Figures 3.19(a) and 3.19(b) compare the performance of this implementation against a client player running HYB over 20K sessions collected during a two-week pilot deployment. Oboe is comparable in performance to the client side player and even improves bitrate slightly (because it was tuned to this publisher’s specification). We expected a cloud implementation would perform worse because of the latency induced by client-server communication. However, we found that most of the bitrate switching decisions occur on timescales much longer than the client-server latencies. Figure 3.20 shows the CDF of the time interval between consecutive bitrate switches for ABR algorithms in two widely used video players(Adobe’s Flash [2] and Microsoft Smooth Streaming [30]). The figure shows that 74 over 95% of switching decisions occur at intervals higher than 1 second for both players. This suggests that a cloud-based deployment is viable. 75 Chapter 4 AViC: Specializing CDN Caching for Adaptive Bitrate Video 76 4.1 Introduction Video forms the majority of the Internet trac today and is likely to continue its rapid growth in the near future [55, 52]; by one estimate [55], 82% of all IP trac in 2022 will be video. Video delivery background. Regardless of the type of network or device used, most video delivery in the Internet employs adaptive streaming.Thistechniquedividesavideointo chunks each of which has a fixed playback duration. Each chunk is then encoded at dierent qualities (or bitrates) that trade-o reduced perceptual quality for lower network bandwidth. Thus, the term chunk denotes a segment of video of fixed playback duration encoded at a specific bitrate. Video content providers (also called publishers [63]) select parameters for the chunk duration and the bitrates to maximize user experience over a variety of devices and network conditions. The video player software on a client device requests successive chunks from a server, stores retrieved chunks in a playback buer, and displays chunks on the device. The player runs an adaptive bitrate (ABR) algorithm [64, 156, 125, 111, 142] to determine the bitrate at which to download a chunk: this decision, usually based on measurements of network state (e.g., available bandwidth), attempts to deliver the highest quality video to users while minimizing stalls. The role of CDNs. Video content publishers use CDNs to deliver video-on-demand 1 to end- users [63] (Fig. 4.1). CDNs consist of a small number of origin servers that store video content and a large number of front-end servers positioned topologically closer to users. When a client device requests a chunk of video, the CDN redirects it to the nearest front-end server, which, in turn, contacts the origin server for the requested bitrate of the chunk. The front-end server then caches the chunk to serve subsequent requests. This architecture enables the CDN to situate content closer to the end user and reduces response latency. Equally important, front-end servers can serve a significant fraction of requests from the cache, thereby shielding origin servers and reducing the CDN’s wide-area bandwidth usage. 1 In this chapter, we focus on video-on-demand; we leave caching policies for live video to future work. 77 Figure 4.1: Serving video over the Internet Caching algorithms. Given a cache of a fixed size, caching algorithms strive to retain those objects requested often. An object is the unit of cached information; examples include memory pages, web pages or video chunks. Caching algorithms achieve this using a caching policy based on recency (when the object was last accessed), frequency (the rate of accesses to the object), object size, or a combination of these. Earlier caching policies focused on eviction — which object to remove from the cache when a new object arrives. Recent work [75] has shown the importance of admission control — deciding whether to admit an object into the cache — especially for workloads where object sizes vary. Cachingalgorithms dierin complexity (the processing time neededto make a caching decision) and performance (measured by the cache’s hit ratio, the fraction of requests served from the cache). In general, higher complexity results in higher performance. The choice of algorithm design is usually constrained by the request arrival rate. For instance, operating systems need to make page replacement decisions at sub-millisecond timescales, so they often use simple algorithms like Least Recently Used (LRU) eviction. Web proxies see lower request arrival rates, so they can use slightly more complex algorithms [158, 85, 82] that take recency, frequency and size into account in making caching decisions. Why video caching is dierent . In this chapter, we argue that caching algorithms that leverage the properties of video workloads are likely to be more eective than existing caching 78 algorithms. Earlier algorithms such as LRU (Least Recently Used) and GDSF(Greedy-Dual Size Frequency) [85], or more recent ones like LHD [71] and AdaptSize [75], are designed under the Independent Reference Model (IRM) [88], which assumes that every request is independent of others. This assumption does not hold for video. When a user starts a video session, they can begin watching from any location in the video, but each session generally makes forward progress; once a session requests chunk k,(k<N), of an N-chunk video it is highly likely to request chunk k+1 soon in the future. Caching algorithms can leverage this property to improve cache hit ratios, thereby helping shield the origin server better, lower CDN wide-area bandwidth requirements, and provide low average response time for users. Contributions. In this chapter, we explore the design of a video caching algorithm for CDNs that leverages properties of today’s video delivery infrastructure to achieve better performance than state-of-the-art caching techniques. Our first contribution is to understand which properties of the delivery infrastructure can help improve caching. Using insights from a large video delivery service, we find that chunk requests are roughly periodic, with average inter-arrival time corresponding to the chunk playback duration. We also find the existence of a significant number of singletons: chunks cached upon the first request but never referenced thereafter. Singletons reduce the ecacy of caching by occupying cache space that more popular objects could use. Our second contribution is the design of AViC 2 which leverages these two observations and combines an eviction strategy with admission control. Eviction uses a simple heuristic to estimate a chunk’s next request time and evicts the chunk with the farthest estimate. Admission control uses a data driven approach to train a binary classifier that predicts whether a chunk is a singleton. However, naive implementations of eviction and admission control can significantly increase the computational and memory complexity of the caching algorithm. When a chunk arrives, 2 AViC stands forAdaptive bitrateVideoCache 79 AViC needs to update its reference estimates for every other chunk in the video, which can be computationally expensive. Moreover, for eective admission control, AViC may need to maintain metadata about videos and chunks long after their eviction. Our third contribution is a suite of optimizations that reduce computational complexity to well below the chunk inter-arrival time, and the memory complexity to that of competing state-of-the-art algorithms. Using request traces from a large video delivery service, we show that AViC outperforms existing caching algorithms. It improves upon state of the art algorithms such as LHD and AdaptSize by up to 15% on byte and object hit ratios. These savings allow AViC to reduce wide-area trac by more than 39% over competing algorithms, and to reduce requests to origin servers by more than 18%. Many CDNs use LRU, and AViC improves upon LRU significantly: an LRU cache requires up to 3.5◊ the size of an AViC cache to match its hit ratio. This finding is important because CDNs have thousands of front-ends and they partition their caches at each front-end across video content providers, so requiring a smaller cache for achieving the same hit ratio enables the CDN to serve more customers with the same front-end infrastructure. 4.2 Properties of Video Delivery We analyzed a large trace of video requests to understand properties of video delivery that we could exploit to design a caching algorithm for video. Dataset. We used a dataset containing CDN HTTP GET request logs from a popular video service in the US. This service provides access to a large library of on-demand videos. Clients access content from the service either using set-top boxes connected to a residential broadband networkorthroughmobile applications forAndroidandiOSplatforms whichmayconnectthrough a 3G/4G cellular network or WiFi. 80 Figure 4.2: Chunk size distribution for cellular and residential clients Figure 4.3: Intra-session request interval of client players The dataset is from a single CDN over a period of 5 weeks and has 585 M requests for 620 TB of aggregate video data transferred, where 120 M requests are from clients in 3G/4G cellular network and 465 M from residential broadband. Each entry in the dataset describes a single HTTP GET request for a video chunk. It records the request arrival time, a unique ID for the video, the chunk number within the video, the bitrate, a unique session ID, and the size of the response returned for the request. It does not contain any form of user-identifiable or personal information. In analyzing this dataset, we uncovered three facets of video delivery pertinent to the design of caching algorithms: (a) Large variability in object (chunk) sizes; (b) Predictability of chunk inter-arrivals; and (c) The prevalence of singletons (chunks requested exactly once). The following paragraphs quantify these observations. Chunk size variability. Recall from §4.1 that publishers encode chunks at multiple bitrates, so clients can adapt download quality using ABR algorithms [64, 156, 125, 111, 142]. The video service we study uses 7 encoding bitrates. Fig. 4.2 shows the distribution of sizes of requested chunks separately for the cellular and residential clients and combined for the full set of clients. Cellular downloads see a range of chunk sizes, from 150 KB to about 800 KB, whereas residential customers see chunk sizes up to 1800 KB, 12 times the smallest chunk size. 81 Figure 4.4: Caching potential of requests Figure 4.5: Popularity of videos which contributed singletons Figure 4.6: Bitrate usage across dierent types of clients. Size variability is an important factor in cache ecacy. The earliest caching algorithms (for page replacement or file buer management in OSs) cached fixed size objects. Optimal caching algorithms (like Belady’s MIN [72]) exist for this setting. However, when objects vary in size, especially by an order of magnitude as is the case in our dataset, caching decisions must take object size into account when making admission control and eviction decisions. This is because admitting a large object in the cache sacrifices the opportunity cost of instead caching smaller objects with potentially higher aggregate hit ratio. Finally, an algorithm that caches objects of varying sizes must perform well both by object hit ratio, the fraction of requested objects served from the cache, and byte hit ratio, the fraction of requested bytes served from the cache. This is especially important for caching video: an algorithm that sacrifices byte hit ratio for high object hit ratio results in lower bandwidth savings over the wide area network, while one that sacrifices object hit ratio for higher byte hit ratio can result in a higher fraction of requests reaching the origin server. Request arrival predictability. As discussed in §4.1, client players gradually download successive chunks in a video. These players use playback buers which limit advanced buering to a few tens of seconds of video. Once the playback buer is full, a player makes another request only when space becomes available in the buer. This suggests that request arrivals should be roughly periodic: if most clients watch videos at the normal playback rate (instead of, for example, fast forwarding or rewinding), the average request inter-arrival time should be roughly the duration of a chunk. More precisely, it should be slightly smaller than the chunk duration, 82 because the player anticipates buer availability and issues the next request before the playback completes. Fig. 4.3 shows the CDF of request intervals within a single session, separately for cellular and residential clients as well as the combined dataset (Cell + Res). Almost 89% of cellular requests and 79% of residential requests arrive within 4 seconds, the chunk duration used by this video service. Furthermore, the tail of all distributions stretches out beyond 10 seconds. We conjecture (but cannot conclusively verify, because we lack player-side instrumentation) that these larger intervals may be due to stalls, paused playbacks or players releasing more than one chunk from the buer before re-filling. The mean inter-arrival times for the cellular devices is 3.01 s, while for the full workload is 3.17 s, and the standard deviations are respectively 1.89 s and 2.85 s. While there is some variability in these inter-arrival times, the fact that the mean inter-arrival time matches our intuition about player behavior (see above) is encouraging, and suggests that future chunk request times might be predictable. Such predictability is important for caching algorithms: intuitively, caching algorithms strive to evict objects likely to be referenced farthest in the future [72]. For most workloads, estimates of future reference times are dicult to obtain, but that may not be the case for video, as our data suggests. The prevalence of singletons. An unusual aspect, at least in our data set, is the prevalence of singletons: chunks requested exactly once over a long period of time. Fig. 4.4 shows the CDF of the number of references to a chunk, within an interval of 12 hours for dierent types for clients. For cellular clients over this 12 hour period, 51% of requests belonged to chunks not referenced again within that time interval. For residential clients 36% of requests over a 12 hour period belonged to chunks which were not referenced again. We also experimented with a longer interval 83 of 24 hours (graph omitted for brevity). Over a 24 hour interval, 43% of cellular and 29% of residential requests belonged to chunk which were only referenced once. A singleton can pollute a cache, by excluding more popular objects from the cache, so a well-designed caching algorithm must strive to exclude singletons. Size variability exacerbates the problem: large singletons can exclude several smaller objects from the cache, thereby adversely aecting both object and byte hit ratios. Singletons can occur because the corresponding chunk belongs to a highly unpopular video (as with web pages, video popularity is heavy-tailed [66, 89]). Consider Fig. 4.5,whichshowsthe CDF of the number of user sessions for videos which contributed singletons for dierent types of clients. Among cellular clients 46% of singletons belong to videos with a single session whereas for residential clients only 38% of singletons belong to videos with a single session. Among residential clients, up to 18% of singletons are from videos which saw at least 10 sessions over a 12 hr period. To understand how videos with more than one session can result in singletons, consider the following scenario. Suppose user A streams a video v and is shortly followed by another user B who also streams the same video. User A’s network is stable and well provisioned to allow viewing at a high bitrate whereas user B’s network is unstable and can only support a lower bitrate. Due to user A’s session, the high quality bitrate gets cached at the server, however, requests from user B will result in cache misses because it is requesting a lower bitrate not used by user A,souser A’s chunk is a singleton. To quantify this intuition, Fig. 4.6 shows the distribution of requests which used a particular bitrate, for cellular and residential clients and for the combined dataset. Almost 65% of the cellular requests accessed bitrate 3, but few requests accessed bitrates 1, 5 and 6, likely resulting in singletons. Across the residential trace, bitrate 6 is the most preferred bitrate, accounting for 60% of all requests, but other bitrates account for smaller fractions of requests. This disparity suggests 84 that caching algorithms must make eviction and admission control decisions at the granularity of chunk bitrates. 4.3 AViC Design In this section, we describe how we leverage the video delivery properties uncovered in §4.2 to design an eective caching algorithm, called AViC, for video caching in CDNs. AViC has two components: eviction and admission control. 4.3.1 Overview and Challenges Eviction. AViC exploits the predictability of request arrivals within a session to estimate the arrival time of the next request for a given chunk (its request estimate). This enables us to design an eviction strategy that evicts the chunk whose request estimate is farthest in the future. The intuition for why such an algorithm would perform well comes from the design of MIN [72], a cache replacement algorithm that replaces objects farthest in the future. MIN is optimal for fixed size objects; we know of no optimality result for caching algorithms with variable sized objects (as in our setting). However, AViC must surmount several challenges in designing eviction policy: it must estimate request estimates for chunks for inactive sessions (those for which no chunk exists in the cache), must determine when to update request estimates and how to keep these estimates up-to-date, and also obtain request estimates at the granularity of chunk bitrates given chunk size variability. §4.3.2 describes how AViC addresses these challenges. Admission control. Based on the prevalence of singletons, AViC’s admission control component aims to prevent singletons from polluting the cache. This goal translates into a simple objective: determining whether a chunk is a singleton or not. This is a binary classification problem, and AViC uses trace data to train a machine learning classifier. The key challenge in the design of 85 the classifier is the choice of classification approach, and feature selection. In particular, feature selection must account for chunk size variability.§4.3.3 describes AViC’s admission control component. Eciency . AViC must make caching decisions faster than request inter-arrival times, and its memory footprint must be comparable to competing caching algorithms. Naive implementations of eviction can result in high processing times especially for updating request estimates. Similarly, both eviction and admission control require maintaining meta-data for videos whose chunks are no longer in the cache, which, without careful design, can inflate AViC’s memory footprint. §4.3.4 describes how AViC achieves its eciency goals. 4.3.2 Eviction Policy AViC evicts the chunk with the farthest next estimated request time. To achieve this, AViC’s eviction policy consists of three key design elements. First, it maintains chunk meta-data (e.g., request estimate, chunk ID) in sorted order by using a max-heap data structure keyed on the request estimate. It inserts this meta-data into the max-heap when the chunk is first requested and then maintains it as long as the chunk exists in the cache. Second, it keeps these estimates fresh by detecting, and correcting stale estimates. Finally, since multiple bitrates are available for each chunk, AViC takes this into account when updating request estimates in the max-heap. We now describe each of these design elements in detail. Maintaining sorted next request estimates. To calculate next estimates for chunks, the eviction policy relies on three key operations: Create, Estimate and Update. AViC invokes Create on each cache miss, and Estimate and Update on both cache hits and misses. Create. This operation instantiates meta-data associated with a chunk or a video. If the chunk belongs to a video v that has no other chunk currently cached, we create a new video record R v for v. R v stores information about a single video including the session IDs of all ongoing 86 sessions and the last chunk requested by each one of them as well as v’s mean session inter-arrival time. Estimate. This operation calculates the next request estimate for a chunk upon receipt of a request for the chunk. Suppose that a request arrives for the n-th chunk of video v on some session i. Let S v denote the set of all ongoing sessions for v. Estimate finds that session from S v which will next request the n-th chunk. Denote this session by j and further let the last chunk requested by j be the m-th chunk. Then Estimate calculates the next request estimate r n for the n-th chunk as follows. Whenm<n, t is the current time (at which the n-th chunk request arrived), and d is the chunk duration (§4.2): r n =t+ ! m≠ n " d (4.1) This leverages the predictability of request arrivals, and estimates the next arrival for n to be m≠ n chunk duration in the future. However, Estimate must also consider the case when the search through S v yields no existing session likely to request chunk n in the future. This happens whenm>n for all ongoing sessions. In this case, the request for n must come from a new session (in all our analyses, we assume rewinds and fast-forwards are rare). If the average session inter-arrival time for v is I,then: r n =t+I +nd (4.2) Thus, the estimate r n in this case is oset by the time it would take for a new session for v to arrive. Update. A single chunk request for any chunk in a video can cause the next estimates of all other chunks in the video to change. Hence all these chunks need to have their next request estimates re-computed. The Update operation achieves this. When a request arrives for a chunk 87 of video v, and regardless of whether the request results in a cache hit or miss, Update scans the max-heap and extracts the metadata of all chunks of video v, then calls Estimate for each chunk, obtains the updated request estimate, and updates the metadata associated with the corresponding chunk. Keeping request estimates fresh. AViC invokes Estimate for a cached chunk of a video only when a request arrives for it. Videos whose sessions were active in the recent past, but have since become idle, can have stale request estimates for their chunks. Chunks of these videos will have small request estimates and move to the bottom of the max-heap. These estimates are not updated unless additional requests arrive, so these chunks might continue to sit deep in the max-heap by virtue of the estimates they had acquired while popular, preventing their removal. This cache poisoning can adversely impact cache ecacy. To address this, AViC maintains video records in an LRU list. The LRU list allows it to quickly find out which videos have not had a recent session. On each request of a chunk, AViC picks the least recently used video and performs Update on its chunks as if the requested chunk belonged to this video (in addition to updating chunks belonging to the video of the requested chunk). This ensures that stale videos will eventually have their estimates recomputed. Request estimates for dierent bitrates . So far we have described the design of AViC implicitly assuming a single bitrate per video. However, publishers use multiple bitrates for their videos [63]. This implies that, when a chunk request for the m-th chunk arrives, AViC must estimate, for each chunk index i in the video, a request estimate for every bitrate b corresponding to that estimate. Specifically, consider a service that uses 3 bitrates, b 1 , b 2 and b 3 . Let’s say that a session requests chunk 1, and chunk 3, encoded at b 2 and chunk 5, encoded at b 3 are in the cache. Should AViC update request arrivals for both of these using Equation 4.1? One possible approach for future request estimation with multiple bitrates is to preempt the actions of a client player and prioritize the bitrate it is likely to request [140]. However, this 88 approach requires intricate knowledge of client player’s adaptive bitrate (ABR) algorithm. Given the range of ABR algorithms [156, 142, 111, 116, 125, 64] and a large user base comprising of range of heterogeneous devices [63] this can be challenging. AViC adopts a data driven approach instead. Fig. 4.6 shows that some bitrates in our dataset are more popular than others. AViC uses this relative popularity of the bitrates to weigh the estimates, based on the idea that less popular bitrates will have a request estimate farther in the future than more popular ones. It uses a simple heuristic, inflating the request estimate by the relative popularity. For example, suppose a video uses just a standard definition (SD) and a high definition (HD) bitrate. Further, assume that clients are 4 times as likely to prefer HD bitrates than SD. Then to calculate the request estimate for an SD chunk, AViC would compute it by inflating the estimate for the HD bitrate by 4 times. More precisely, say a video uses k bitrates b 1 ...b k . Let w j ,j œ (1..k) be the ratio of the number of accesses to chunks with bitrate b j to chunks with bitrate b i where b i is the most frequently accessed bitrate. Then, AViC adapts Equation 4.1 as follows to compute the next request time for chunk n encoded using bitrate b j : r n b j =t+ ! m≠ n " d w j (4.3) It adapts Equation 4.2 similarly, and we omit this for brevity. To compute w j , AViC maintains a counter of accesses for each bitrate b j for each video. 4.3.3 Admission Control To address the prevalence of singletons, admission control simply seeks to detect whether a chunk is likely to be a singleton. If so, it prevents that chunk from entering the cache. Admitting singletons not only takes up cache space for more useful chunks but can also evict chunks that contribute to higher hitrates. 89 WemodelsingletondetectionasasupervisedbinaryclassificationproblemandtrainaGradient Boosted Decision Tree (GBDT) [119] classifier. This classifier outputs a probability estimate for a future reference of a chunk; we apply a threshold · on this probability [119] to admit the chunk. We chose GBDT for several reasons. First, it is fast: training a single model takes less than 10 minutes on a dataset comprising of 100 M requests. Second, the size of the resultant model is small, with an average size of 175.86 KB. Third, GBDTs (and most decision tree based classifiers) are generally more interpretable than various deep learning techniques. Finally, they are quick to query and scale well with requests [74]. A rule-based alternative for admission control would analyze the video workload and determine admission control rules. For instance, Fig. 4.6 showed that cellular clients seldom used higher bitrates for videos. Using this information it may be possible to design an eective admission control which denies admission to high bitrate chunks. However, it is not clear whether such a rule based system can be easily designed manually. More importantly, it would require laborious manual analysis when the workload changes or when a publisher changes bitrate choices. The design of the classifier is non-trivial for two reasons. First, the definition of a singleton presupposes a time horizon, the time over which a requested chunk is never requested again. Choosing the time horizon is our first challenge. The second challenge is the choice of features. Choosing the time horizon. In a cache, an object o is a singleton if (a) o enters into the cache upon first access, and (b) the caching algorithm evicts o before its next request. The time between first access and eviction is the time horizon. Time horizons may depend upon cache size. An object o may be a singleton with respect to a 32 GB cache but may not be a singleton with respect to a 64 GB cache (the latter holds more objects, so the next request for o may arrive before the caching algorithm evicts o). This observation motivates AViC’s approach of training dierent classifiers for dierent cache sizes . This additional training does not impose a burden on 90 Feature Description Day The day of the week Time Time of day in 24 hours format Chunk size Size of chunk in bytes Chunk index Index of chunk in video Chunk bitrate Bitrate of requested chunk Total video sessions Global No. of video sessions Video sessions 24 hrs 24hr average no. of video sessions Video interarrival Mean interarrival of video sessions Video last interval Time since start of last session Table 4.1: Features used by admission control classifier. large CDNs with significant compute resources and GBDTs require minimal resources to begin with. But to train the classifier, we need to estimate the time horizon for a given cache size. We chose a simple estimator: the time to completely replace the contents of a cache using a FIFO policy. For each cache size, we can estimate the time horizon cheaply by simulating a short trace through a FIFO cache of the corresponding size. After estimating the time horizon T, we generate training samples from our traces by marking as a singleton any chunk request whose next request is more than time T in the future. Feature selection. The second challenge in classifier design is feature selection. Intuitively, several features of a chunk are likely to predict whether it is a singleton or not. For example, if chunks arrive infrequently, or video sessions arrive infrequently, then a chunk may likely be a singleton. If the chunk, or its corresponding video, are less popular, singletons might arise (§4.2). Finally, the size of a chunk can potentially be a signal indicating singleton status: as Fig. 4.6 suggests, some bitrates are more likely to be singletons. For this reason, our GBDT classifier uses these features (Tab. 4.1). In §4.4 we analyze the importance of each feature for classification. Other details. In our evaluations, we train a model for each day (24-hour period) using request logs from the previous 2 weeks (3 week windows only provided marginal benefits). This re-training accounts for changes in the video workload. We can do this quickly because GBDT trains fast. 91 Figure 4.7: Hierarchical arrangement of max-heaps 4.3.4 Performance Optimizations AViC must make caching decisions faster than requests arrive, and must have a memory footprint comparable to existing caching algorithms. This section describes optimizations that speed up Estimate and Update operations, and reduce metadata requirements for admission control. Optimizing Estimate. To calculate the request estimate for a chunk, Estimate must scan all ongoing sessions of a video to find the session likely to request the chunk earliest. However, searching though all sessions is unnecessary since some sessions may already have gone past the particular chunk. For example consider the scenario where a video v contains 30 chunks. The video has three ongoing sessions A, B and C. Assume that A last requested chunk 3, B requested 8 and C last requested chunk 27. When a request for chunk 9 arrives from B the only session that is likely to request chunk 9 again in the future is A (assuming A is unlikely to fast forward, and C unlikely to rewind the video). To narrow the search space of candidate sessions, AViC uses a nested hashmap (hashmap of hashmap) data structure to store sessionIDs. The outer hashmap uses a bucket of chunk indices as keys. Each bucket then maps to another hashmap which maps a chunk index in the bucket 92 to a list of sessionIDs which last requested the chunk. If we consider a bucket size of 5 for the outer hashmap, then in the example above, video v has a total of 6 buckets (30 chunks divided by 5) and each of these buckets maps to another hashmap with a maximum size of 5 (5 chunks in each bucket). So, A gets hashed to bucket 0 in the outer hashmap and then in the nested hashmap the chunk last requested by A (chunk 3) stores A’s sessionID. Similarly, B maps to bucket 1 in the outer hashmap and C to bucket 5. When a request for chunk 9 arrives from B, AViC starts searching from bucket 1 and moves backwards, searching till any other session that is going to request chunk 9 in the future is found. This results in a constant (O(1)) time search complexity, because after indexing in to the outer hashmap, the search space is restricted to the size of bucket. AViC uses a bucket size of 25 chunks, which requires a total of 432 buckets for a 3-hour video with 4-second chunks. Optimizing Update. This operation recalculates the request estimates for all chunks of a given video. This can be expensive since a video can have thousands of chunks. Observe that Update does not need to update every chunk. Instead, for a given video, it only needs to update that chunk with the farthest next estimated request. As long as this chunk’s request estimate is accurate, AViC can make eviction decisions. However, searching this local farthest chunk is expensive using a single max-heap. To optimize this lookup, AViC organizes the cache as a hierarchy of local max-heaps and a global heap (Fig. 4.7). There is one local heap for each video, which contains its chunks sorted according to the request estimates. The global max-heap has pointers to the root nodes of all the local video specific heaps. Then, finding the farthest chunk in a video is a single heap lookup. With this design, to evict a chunk AViC first consults the global heap to find the video which holds the chunk with the farthest next reference. Let c f denote this chunk and assume that video v holds c f . AViC then removes the meta data associated with c f from v’s local heap. After 93 removing the chunk, if v’s local heap is non-empty, AViC updates v’s node in the global heap with the new root of the local heap. It can then evict c f . These steps require three heap operations. Optimizing Update also reduces the amount of work AViC needs to do to avoid stale estimates (§4.3.2). On each chunk request, AViC only updates the request estimate for the farthest chunk of the least recently used video. Reducing metadata storage. AViC’s eviction and admission control policies use historical information associated with videos. AViC’s eviction algorithm orders chunks by their next request estimates (§4.3.2). To compute these estimates, Equation 4.2 needs the inter-arrival time of sessions for a video. Likewise, for admission control (§4.3.3), AViC assembles a set of features on each chunk request, then uses these to query the GBDT model. These features are video-specific (Tab. 4.1). As such, AViC might need to maintain state for every video, which can result in a high memory footprint. To reduce memory footprint, observe that AViC may not need to maintain accurate state for every video. For instance, consider a video v which has inter-arrival of I between successive sessions and further suppose that I is significantly large, e.g., > 24 hours. In this case, if AViC caches chunks of v, it would likely evict them much earlier than I — the interval after which another session for v arrives. In other words, chunks of v will be singletons. For such unpopular videos, maintaining accurate estimates of inter-arrivals does not contribute towards improving the performance of AViC. Based on this insight, instead of maintaining state for all videos, AViC maintains historical state for the N most recently requested videos. It does this by maintaining video records in two separate LRU lists. An Active LRU list which keeps track of the videos currently in cache and an Inactive LRU list which keeps track of videos requested recently but are no longer active. When AViC evicts a video v from the Active list it moves it to the Inactive list but preserves its state. If a session for v arrives while it is in the Inactive list, AViC moves v back to the Active list. If no 94 further sessions arrive while a video is in the Inactive list, AViC eventually evicts the video and removes its state. AViC uses a value of N = 5000 which allows it to achieve high performance while keeping AViC’s memory footprint comparable to or within the memory footprint of other algorithms. We evaluate AViC’s sensitivity to N in §4.4. 4.4 Evaluation This section compares AViC to several other caching algorithms and quantifies the benefits of our design choices. 4.4.1 Methodology Implementation. We have implemented AViC in Go [20]. This implementation takes as input a request trace and a cache size, processes the requests in the trace, and outputs the byte and object hit ratios. Dataset. We have a dataset containing 5 weeks worth of requests to a large video provider (§4.2). We use weeks 4 and 5 from our dataset to compare AViC with other alternatives, and the weeks 2, 3 to train the admission control classifier 3 . In weeks 4 and 5, the data contains 117 M requests with a total of 124 TB of aggregate requests. Of the 117 M requests, 24 M correspond to cellular clients and the remaining 93 M correspond to residential clients. We evaluate AViC and other alternatives separately for cellular and residential client requests, and also collectively over the 117 M requests. Metrics. We compare performance using both byte hit ratio and object hit ratio (§4.2). Both metrics are important for video: a high byte hit ratio reduces backbone trac from CDN front ends to origin servers and a high object hit ratio reduces the number of requests to CDN origin. 3 As discussed in §4.3 we also experimented with using the first three weeks to train the classifier, but it yielded marginal benefits. 95 Further, by showing both metrics together, our goal is also to show that AViC doesn’t compromise one metric while achieving high performance on the other (as some competing algorithms do). Baseline algorithms. We compare AViC against four algorithms: LRU, GDSF [85], Adapt- Size [75] and LHD [71]. We pick LRU since it is one of the most widely deployed cache algorithms. GDSF combines frequency and size with recency to improve over LRU. AdaptSize combines admission control with LRU. Its admission control uses an adaptive size threshold to prevent admission of large objects into the cache. LHD uses hit density to maximize those objects in the cache which contribute most to the hitrate given their size. Together, these four algorithms cover a range of designs. Finally, to present results with reference to an upper bound for the achievable hit ratios, we also show hit ratios of an Oracle which applies MIN [72] to our variable-sized workload (as discussed in §4.2, MIN is not optimal for variable-sized objects). To evaluate these algorithms, we have used author-provided implementations for AdaptSize and LHD [13, 10], and have implemented LRU and GDSF. 4.4.2 Performance Comparison Residential clients. Tab. 4.2 and Tab. 4.3 show the byte and object hit ratios over a range of caches sizes for residential clients. AViC consistently outperforms other algorithms both in byte and object hit ratios at dierent cache sizes. For instance, for a 128 GB cache, AViC improves over LRU by 24.0% and over GDSF by 17.6% in byte hit ratios (improvement as a percentage of Oracle’s hit ratio). To put it another way, LRU needs 3.5◊ the cache size to match the byte hit ratio performance of AViC. Finally, AViC is about 26.5% better than AdaptSize and about 10.5% better than LHD in byte hit ratio (with slightly lower gains on object hit ratio), both recently proposed algorithms, across the range of cache sizes we evaluate. Cellular clients. Tab. 4.4 and Tab. 4.5 show the byte and object hit ratios for the cellular clients. Cellular clients tend to prefer slightly lower bitrates than residential clients and therefore 96 Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 10.56 13.11 9.88 12.46 18.05 28.58 64 14.32 16.52 13.23 17.51 23.14 35.35 128 19.55 22.33 18.46 25.41 29.98 43.39 256 27.28 32.51 27.51 34.98 38.75 52.30 512 38.00 44.41 36.60 44.47 48.17 61.31 1024 50.22 54.58 41.28 52.23 57.71 69.58 Table 4.2: Byte hit ratio comparison for residential clients Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 9.60 12.77 9.64 11.74 15.75 25.01 64 12.93 15.93 12.64 16.12 20.01 30.73 128 17.41 20.87 17.27 22.70 25.75 37.63 256 23.93 29.35 24.97 30.97 33.41 45.55 512 33.02 39.73 32.88 39.92 42.38 54.04 1024 43.71 49.64 37.82 48.41 52.07 62.48 Table 4.3: Object hit ratio comparison for residential clients exercise the cache dierently. As with the previous results, AViC outperforms the competing algorithms, but by smaller margins. For a 128 GB cache AViC outperforms LRU by 18.9%, GDSF by 6.8%, AdaptSize and LHD by 21.2% and 7.2% for byte hit ratios. For cellular clients, an LRU cache requires approximately 2◊ the cache size used by AViC to match its performance. All clients combined. Tab. 4.6 and Tab. 4.7 show the byte and object hit ratios over a range of dierent cache sizes for our full trace dataset (cellular and residential client combined). AViC is able to consistently outperform all competing algorithms over the entire trace as well: up to 28.1% Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 16.27 18.87 14.44 18.46 23.76 38.08 64 22.30 26.93 21.32 27.39 32.60 46.18 128 31.02 37.60 29.75 37.40 41.30 54.50 256 42.03 46.53 39.85 45.64 50.54 62.02 512 53.34 55.62 46.34 55.26 58.19 67.28 1024 62.24 63.20 40.98 63.15 64.24 68.65 Table 4.4: Byte hit ratio comparison for cellular clients 97 Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 15.85 18.45 14.33 18.21 23.01 36.96 64 21.68 26.47 21.01 27.03 31.61 44.74 128 30.14 36.99 29.41 36.85 39.47 52.67 256 40.78 45.71 39.25 44.85 47.81 59.88 512 51.58 54.30 45.41 53.95 55.58 65.15 1024 60.09 61.47 40.77 61.47 61.93 66.53 Table 4.5: Object hit ratio comparison for cellular clients Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 9.97 12.37 8.92 10.86 16.94 27.73 64 13.53 15.60 11.60 14.70 21.72 34.27 128 18.35 20.35 16.34 20.39 28.16 42.05 256 25.58 29.10 24.64 29.43 36.78 50.85 512 35.64 41.45 35.47 39.40 46.47 59.97 1024 47.77 52.23 44.84 48.44 56.26 68.50 Table 4.6: Byte hit ratio comparison for all clients. better than LRU and AdaptSize, and up to 18.6% better than LHD and GDSF. At larger cache sizes, the performance dierence narrows: with a 1 TB cache, all algorithms perform comparably, and approach the Oracle. Explainingperformancedierences . WenowtrytounderstandwhatcausesAViCtoperform better than some of the competing algorithms. GDSF and LHD. Unlike AViC, GDSF and LHD prefer to evict larger chunks over smaller chunks. This strategy can help achieve higher object hit ratios at the expense of lower byte hit Cache Size (GB) LRU GDSF AdaptSize LHD AViC Oracle 32 8.82 12.28 9.32 11.01 14.37 24.12 64 12.01 15.38 12.21 14.79 18.27 29.74 128 16.09 19.84 16.80 20.41 23.68 36.62 256 22.03 27.54 23.71 29.04 31.33 44.75 512 30.46 38.52 33.19 39.07 40.90 53.89 1024 41.13 48.99 42.63 48.51 51.06 62.89 Table 4.7: Object hit ratio comparison for all clients. 98 Figure 4.8: Performance comparison of a modified GDSF algorithm ratios. When object sizes follow a normal distribution, this strategy may work well. However, the size distributions for video delivery may be significantly skewed, because video chunk sizes depend on factors such as the type of bitrates provided by the content provider, the network the clients are on, and the ABR algorithms used by client player. As Fig. 4.6 shows, 60% of residential client requests were for the highest bitrate chunks. As such, penalizing the size of the chunks can put GDSF and LHD at a disadvantage because they are likely to prefer higher bitrate chunks for eviction — the same chunks which can potentially deliver higher byte and object hit ratios if cached. To demonstrate this more concretely, we consider a variant of GDSF algorithm. This variant modifies the cost function used by GDSF by halving the size penalty to highest bitrate chunks. Fig. 4.8 shows the byte and object hit ratios comparison between the original GDSF algorithms and this modified version (GDSF-MOD). Notice that GDSF-MOD achieves higher byte hit ratios while achieving higher or comparable object hit ratios. Even with this improvement, however, GDSF-MOD does not outperform AViC. The penalty imposed on size also explains why the performance gap between AViC and baselines is smaller for cellular clients compared to residential clients. As Fig. 4.6 shows, cellular clients tend to prefer bitrate 3 whose chunk size is 4.1◊ smaller than bitrate 6. The adverse 99 (a) Per hour bandwidth savings comparison (b) Per hour request savings comparison Figure 4.9: Savings in bandwidth and requests to the CDN Origin. impact of chunk sizes impacts the cellular workload less, so competing algorithms fare slightly better for these clients. AdaptSize. Admission control in AdaptSize uses a dynamic size threshold to decide whether to admit an object to the cache or not. It assigns a low probability of admission to objects which are large. This results in high object hit ratios for a Web workload whose size varies over a large range (1 B to 1 GB [75]). However, our results for both residential and cellular clients show that AdaptSize is unable to outperform LRU, for two reasons. First, the size distribution of video chunks is in smaller range (150 KB to 1800 KB) than that of Web workloads. Second, unlike Web workloads which may have few large objects, in both the residential and the cellular traces, higher bitrates are more popular (Fig. 4.6). 60% of residential clients prefer the highest bitrate whereas 78% of cellular clients prefer the 3rd and 4th bitrate. Because of these two factors, AdaptSize’s admission control often prevents higher bitrate chunks from entering the cache, resulting in poor performance. Over the combined trace, AdaptSize narrowly outperforms LRU in object hit ratios (Tab. 4.7). A less skewed aggregate size distribution in the combined trace allows AdaptSize to find reasonable size thresholds to filter large chunks while still delivering high object hit ratios. Unlike AdaptSize, AViC does not solely rely on the size of the chunk to decide admission. Instead it uses other factors such as the bitrate of the chunk, its position in the video (chunk index), and its popularity, which allows it to make better admission decisions. 100 Implications of results. While AViC’s absolute gains over competing algorithms seem small, they are still significant. Many CDNs use LRU, and AViC’s performance gains over LRU are significant. In particular, the fact that LRU requires 2-3.5◊ bigger cache, especially at small cache sizes, to achieve the same hit ratios as AViC is important. CDNs can have thousands of front end servers [79], and may serve hundreds of video publishers [63]. For each publisher, at each front end, a CDN might wish to virtualize the cache — partition a large cache into smaller caches dedicated to each publisher, to avoid cache interference between clients of dierent publishers. Our results show that AViC oers significant cache consolidation opportunities: to achieve today’s hit ratios, a CDN need only provision half to a third size of caches in its front ends. Figure 4.9(a) shows the hourly average CDN bandwidth savings and Figure 4.9(b) shows the average hourly requests prevented from reaching the origin. At smaller cache sizes (32 GB, 64 GB, 128 GB), AViC on average saves up to 39.2% more bytes per hour than the closest competitor. Further, it saves up to 18.8% of requests per hour served by the CDN’s origin servers. These savings in bytes and number of requests are significant. Video is the dominant component of Internet trac, and wide-area bandwidth costs are significant [ 112], so even small reductions in wide-area trac can be important. These savings can also benefit origin server provisioning. Finally, recent trends suggest a move towards higher-quality video streaming formats, such as 4K and Ultra-HD [63, 49]. This means that in the future content providers are likely to oer even higher bitrates. As such, the gap between AViC and algorithms which penalize object size, such as GDSF, LHD and AdaptSize is likely to be higher in the future. Performance compared to Oracle. While AViC outperforms the competing algorithms, it exhibits a noticeable performance gap relative to Oracle. For example, at smaller cache sizes, AViC’s hit ratios are between 60≠ 65% of Oracle’s hit ratios. Three factors explain these performance dierences. 101 Cache Size (GB) AViC w/o adm ctrl w/o BR weights w/o stale avoidance Oracle 32 23.76 22.66 20.19 13.52 38.08 64 32.60 30.48 27.37 19.00 46.18 128 41.30 39.40 36.49 26.55 54.50 256 50.54 48.93 46.20 36.83 62.02 512 58.19 57.74 56.19 48.96 67.28 1024 64.24 64.05 63.29 60.73 68.65 Table 4.8: Impact of AViC’s components on performance AViC uses the mean inter-arrival time of sessions for videos for request estimates. However, inter-arrival times can take a few sessions to converge to a stable value, until which time estimates can be inaccurate. We found, using a 128 GB cache on the residential trace, that a significant fraction (3.42%) of non-singleton chunks, when they were evicted, had inaccurate estimates because not enough sessions had arrived to obtain a reliable mean inter-arrival time estimate. Even with good estimates for the mean inter-arrival time, AViC can have erroneous request time estimates for two reasons. AViC assumes that seeks or pauses in video, or users reneging, are relatively infrequent. However, this assumption can be easily violated in the real world, invalidating future request estimates. AViC also does not account for client activity resulting from adaptation to network conditions. For instance, it does not know when a client is about to experience re-buering, which can also induce error in request estimates. We computed the distribution of the estimation error for those chunks which had good estimates for the mean inter-arrival times. For a 128 GB cache on the residential trace, while the median error was only 4 s (about one chunk duration), the 99-th percentile error was 52.3 s. This long tail of the error distribution likely explains the dierence between AViC and the Oracle. 102 (a) Request inter arrivals and service latency (b) Memory complexity compared to LRU and GDSF. Figure 4.10: Savings in bandwidth and requests to the CDN Origin. Figure 4.11: Byte hit ratio sensitivity toN 4.4.3 Ablation Study Each component of AViC contributes significantly to performance. Tab. 4.8 shows byte hit ratios when we remove one of the three main components of AViC: admission control, correct request estimation for bitrates, and stale estimate avoidance. These results are for cellular clients; the object hit ratio and results for residential and the full set of clients are qualitatively similar, and we omit these for space. Especially at lower cache sizes, the mechanism to prevent stale estimates by updating the least recently used video contributes the most (over 10% gains at cache sizes up to 512 GB), since stale videos can potentially cause poisoning of the cache limiting its performance over the long run. Weighting requests estimates by bitrate popularity, and admission control, each improve performance by a few percentage points. 103 4.4.4 Time and Memory Complexity Time complexity. To be practical, AViC must service requests faster than the rate at which they arrive (§4.3.4). Fig. 4.10(a) plots two curves: the distribution of request inter arrival times and the distribution of AViC’s request processing latency. To compute the latter, we use a single 2.4 Ghz core of an Intel Xeon server. Because of its careful data structure and request estimate design, AViC is able to process requests 1-2 orders of magnitude faster than request arrivals. Memorycomplexity. To analyze the memory usage, we profile the amount of memory allocated to AViC by using Go’s run-time profiler [22]. This is the preferred approach for profiling Go programs and provides accurate statistics on how much memory Go’s run-time allocates to a process [21]. We periodically place calls using the run-time profiler to sample the memory size during a run of AViC Go. Figure 4.10(b) shows the average size of the allocated memory for AViC in comparison to LRU and GDSF. The error bars show the standard deviation. As expected, LRU is the most ecient in terms of memory usage, whereas AViC uses less than 2◊ the memory used by LRU. Notice that among the three algorithms, GDSF uses the most memory. This dierence arises from the way Go’s runtime allocates memory to dynamically sized data structures. Both AViC and GDSF use heaps to maintain the metadata associated with cached objects. Heaps in Go use dynamic arrays. The size of these dynamic arrays are not fixed at compile time, instead the Go runtime initializes them to a default size. To grow the heap, the runtime resizes the array typically by a factor of 1.25≠ 2◊ . The dierence in memory usage arises in the way GDSF uses heaps. By design, GDSF uses a single heap to track all chunks in the cache. The size of this single heap is much larger than the smaller heaps which result due to the hierarchical heap design used by AViC (§4.3). As a result, 104 GDSF’s single heap grows by larger amounts (in absolute terms) than AViC’s heaps: the runtime individually resizes the latter’s heaps. Summary. Together, these results demonstrate that AViC (i) can handle a high load of requests without compromising performance and (ii) keeps memory overhead low compared to other algorithms. Since we use standalone implementations of LHD and AdaptSize, we could not use the same profiling approach to estimate their memory usage. We have left it to future work to instrument their implementations to obtain accurate memory usage for these algorithms, but expect that their memory footprint will be no lower than LRU. 4.4.5 Sensitivity Analysis Global history. AViC uses an Inactive video list to maintain metadata associated with videos which do not have any chunks cached (§4.3). This metadata allows it to compute features needed for admission control as well as the inter-arrival times for videos needed for request estimates. We now analyze how sensitive are AViC’s results toN, the number of video records maintained in the Inactive video list. Fig. 4.11 show the byte hit ratios (object hit ratio omitted for brevity) for dierent cache sizes as a function of N. Large cache sizes (512 GB, 1024 GB) are not sensitive to N. This is because large caches are big enough to already hold unpopular videos in the Active list of videos and hence do not benefit from the Inactive list. Smaller caches are sensitive up to around N = 50. AViC chooses to be conservative and uses a value of N = 5000 since the memory cost of maintaining 5000 video records is small; at this value, AViC is not aected by discarding history. Admission control. Fig. 4.12 shows the relative importance of dierent features to the perfor- mance of GBDT’s classifier. The two most important features are the average number of sessions for a video over a 24 hour period and the inter arrival time of new sessions for a video. The bitrate of the requested chunk also plays an important role followed by other chunk level attributes such 105 Figure 4.12: Importance of features to GBDT classification. Figure 4.13: Correlation of video popularity over time. as the size of the chunk and the index of the chunk in the video. These relative importance results conform to intuition and suggest that the GBDT classifier learns request patterns well. To understand why the workload is amenable to learning, we analyze the correlation of popularity of videos over adjacent periods of 24 hours. Fig. 4.13 shows this correlation over two sets of adjacent days. It shows that the videos which are popular tend to remain popular the 106 next day and unpopular videos remain unpopular. We have also analyzed this correlation over 48 hour intervals and found similar results; we omit these for brevity. 107 Chapter 5 Related Work 108 This work touches a large body of related work on adaptive bitrate video. We now summarize it with reference to the chapters in this dissertation 5.0.1 Understanding Video Management Planes CharacterizingVideoServices. YouTube and Netflix alone have been the subject of numerous studies over the years [60, 80, 103, 161, 84, 96, 146, 91, 160, 59, 78]. This body of work has studied several aspects, including 1) architecture, serving strategy and its evolution, 2) characterization of videos in terms of encoded bitrates, total number of videos, popularity, caching, and 3) the user access patterns and quality of experience etc. Ghasemi et al. [102] conducted an in-depth study of Yahoo’s video serving infrastructure to reveal problems in dierent points in the video delivery pipeline. Other work has also examined dierent types of video services including a Pay-TV [57], cellular video [94], an on-demand service [123] and user-generated live streaming services [149, 141]. These papers focus on one or a handful of online publishers, but our work focuses on characterizing management plane practices across a large number of online video publishers. Industry surveys. Because of the growing interest in Internet video, several industry sur- veys [50, 54, 17, 33] have examined the video ecosystem. A 2017 industry study by Bitmovin surveyed380videodevelopers(individualsorcompaniesassociatedwiththeInternetvideobusiness in various ways). This study characterized streaming protocols, encoding formats, devices, DRM etc.. An earlier 2016 study [17] characterized aggregate distributions across many of the same dimensions as [54]. Another prior industry survey [28] and an anecdotal report [39]discussthe percentage of publishers that use multiple CDNs (but do not discuss the number of CDNs used, or thefactdierentCDNsmaybeusedforliveandVoD).Whilevaluable,noneofthesereportsweight findings by view-hours, or present trends across publishers categorized by view-hours, or present longitudinal analyses, as we do. Both of these methodological dierences result in new findings 109 and add insights to known trends. Finally, while [33] does presents some trends with respect to device usage, other dimensions are not considered. In addition, we go beyond all these reports by considering the implications of these trends for management complexity, and syndicated content. Quantifying diversity and complexity. Prior work has captured diversity of mobile users [98] and apps [134] and the complexity of web pages [77] and routers [73]. While we draw inspiration from these works, our focus is on video management planes, a dierent domain. 5.0.2 Oboe: Specializing Adaptive Bitrate Algorithms TuningABRAlgorithmConfigurations. The BBA2 algorithm [111] tunes its lower reservoir based on buer occupancy dynamics, while MPC [ 157] adapts its throughput discount factor based on past prediction errors (§3.2). In contrast to such ad-hoc heuristics, Oboe selects configuration parameters based on network state, and publisher specifications. The approach is generically applicable to many ABR algorithms. Newer congestion control protocols like BBR [83] estimate network throughput, which if exposed, could benefit Oboe. Learning ABR Algorithms. Among ABR algorithms that use Reinforcement Learning and other machine learning techniques [148, 126, 87, 86, 125], Pensieve [125] has been shown to perform the best. While Pensieve does not specialize to dierent throughput regimes, Oboe performs better by specializing parameter values for each network state independently. Otherworkinself-tuning. BeyondABRalgorithms, self-tuningapproacheshavebeenexplored inothercontexts. Winsteinetal. [153]usedsimulationstodetermineTCPparametersfordierent settings, while Semke et al. [139] proposed tuning TCP socket buers to ensure high throughput. More generally, Google Vizier [104] performs such black-box tuning as a service. While Vizier can potentially be used to implement the oine phase of Oboe, our work identifies underlying 110 principles (such as the piecewise stationarity of available throughput) that forms the basis for the tuning. Video QoE. Several researchers have pointed out that sub-optimal ABR performance can significantly impact user-engagement and hence revenue [67, 121]. Others have looked at quality issues that occur when multiple players start to compete for bandwidth [106, 61, 116, 109]. In contrast, Oboe improves the QoE performance of several ABR algorithms across a range of dierent network conditions by automatically tuning their parameters. 5.0.3 AViC: Specializing CDN Caching for Adaptive Bitrate Video Caching can improve the performance of databases, key-value stores, operating systems and web proxies, to name a few. So, caching algorithms have been extensively studied in the literature [93, 133, 127, 122, 101, 108, 158, 85, 75, 71]. Many of these designs combine recency and frequency using dierent techniques. For example, Facebook’s photo caching algorithm [ 108] uses four LRU lists and moves objects from lower LRU lists to higher lists based on hits. It evicts objects from the lowest of the 4 lists, which combines frequency and recency as objects that receive more frequent requests move up to the higher lists becoming more secure from eviction. Similarly, LRFU [122] augments a single LRU list with another list of objects in the least frequently used (LFU) order. This allows the evictions policy to leverage recent history from the LRU list as well as older history from the LFU list. Other algorithms also factor in the size of an object when making eviction decisions. For example, the Greedy Dual Size family of algorithms [158, 85] penalize objects based on their sizes. This design evicts large sized objects first. AdaptSize [75] uses admission control to prevent large sized objects from getting into the cache, which is useful when object sizes vary such that the admission of a single object can evict a large number of smaller objects. LHD [71] computes a hit density which is a function of the objects size. Between two objects which have similar hitrate 111 potential but dierent sizes, the object of larger size has a lower hitrate density. The algorithm maximizes the hitrate density of the cache. All algorithms assume the Independent Reference Model (IRM [88]) in which the request for a given object is independent of requests for other objects. This does not hold for video delivery. A recent work [140] uses this dependency to design a pre-fetching strategy. Pre-fetching allows the cache to maximize hit ratios, but suers from two limitations. It requires knowledge of the client side bitrate adaptation algorithm. It can transfer data between a CDN front-end and the origin that a client may never request (e.g., if the session terminates, or if network conditions change). AViC does not assume knowledge of the bitrate adaptation algorithm, and never pre-fetches chunks. 112 Chapter 6 Conclusion and Future Work 113 In this dissertation we have shown that by specifically tuning and designing two important systems that serve video, we can achieve significantly higher adaptive bitrate video performance. Towards this end, we first analyze the Internet video management plane to understand how video is served today and what are the components of this management plane which critically impact performance. The Internet video management plane, which is responsible for packaging video content and for ensuring playback across dierent devices, has received relatively little research attention. Using data collected by a large broker over a period of two years from over one hundred video publishers, we find that there exists significant diversity across the three aspects of video management we study (packaging, CDN usage, and playback device usage): large publishers support 3-4 protocols, 5 CDNs and 5 dierent device types. This diversity adds complexity to several management tasks such as failure triaging, software management, and encoding. We find that complexity metrics for these tasks are sub-linearly related to the number of view-hours. Finally, the structure of today’s management planes can lead to variable delivery performance for syndicated content. Integrating management planes for syndicated content can avoid this as well as reduce CDN origin server storage requirements. We thenpropose Oboewhichis a systemforautomaticallytuning ABR algorithmsbyadapting ABR configurations in realtime to match the current network state. Picking configurations in a manner informed by network state and publisher preferences distinguishes Oboe’s approach from heuristics used today that do not consider these factors. Oboe significantly improves the performance of BOLA, HYB and RobustMPC; further, for nearly 80% of the sessions in our evaluation dataset, Oboe integrated with RobustMPC improves QoE-lin relative to Pensieve and the improvements exceed 20% for 25% of the sessions. Finally, we propose AViC, a CDN front-end caching algorithm specifically designed for adaptive bitrate video. AViC leverages three properties of video delivery: chunk size variability, 114 predictability of request arrivals, and the prevalence of singletons. Its eviction policy uses predictability of request arrivals to estimate future chunk request times, and its admission control predicts singleton chunks. Both components take size variability into account. AViC incorporates performance optimizations to reduce time and memory complexity. Using request traces from a real-world video service, we show that AViC outperforms range of algorithms including LRU, GDSF, AdaptSize and LHD by up to 50% in byte and object hit ratios. Future work can attempt to close the gap between AViC and Oracle, by improving request estimates during pauses, stalls, or early in the session when players may request chunks back-to-back. Future Work. While we have covered some ground, there is room for future work in all three aspects of this dissertation. For example, we can explore mechanisms which allow better syndication of content between video publishers. We can also build new complexity metrics to characterize the complexity of video management planes. This could allow researchers to build strategies to better cope with diversity and reducing management complexity. Similarly, through Oboe we have shown the benefit of specializing adaptive bitrate algorithms to the network state. However, this idea can be broadened to other aspects of video delivery. For example, having specialized algorithms for live and on demand content. Additionally, while we performed a pilot deployment of Oboe, future work can focus on providing Oboe as a cloud service which potentially hosts an ensemble of ABR algorithms. Content publishers can select the ABR algorithm which best suits their need. Finally, through AViC we have demonstrated a cache for adaptive bitrate video. However, its not clear whether the ideas we based on our design can be generalized to upcoming forms of interactive media content such as 360 video. Caching for 360 video is likely to present additional challenges because of the greater amount of available streams. As such, future work may need to perform fresh analysis for a 360 video workload to answer the type of questions which go beyond the ones we do in this dissertation. 115 Reference List [1] Adobe: Adobe HTTP Dynamic Streaming. www.adobe.com/products/hds- dynamic-streaming.html. [2] Adobe: Adobe OSMF player. http://www.osmf.org. [3] Alexa: Top 500 sites on the web. https://www.alexa.com/topsites/category. [4] Amazon AWS Media Package. https://aws.amazon.com/mediapackage/. [5] Apple: Apple’s HTTP Live Streaming. https://developer.apple.com/ streaming/. [6] Apple: AVFoundation framework . https://developer.apple.com/av- foundation/. [7] Apple: Technical Note 2224 for HLS Streaming. https://developer.apple.com/ library/content/technotes/tn2224/_index.html. [8] Bayesian Changepoint Detection. https://github.com/hildensia/bayesian_ changepoint_detection. [9] Chrome Remote Interface. https://github.com/cyrus-and/chrome-remote- interface. [10] CMU-CORGI: LHD. https://github.com/CMU-CORGI/LHD. [11] Conviva: Precision Delivery Intelligence. https://www.conviva.com/whitepapers/. [12] Conviva: The 2017 OTT Streaming Market Year in Review. https://www.conviva. com/blog/2017-ott-streaming-market-year-review/. [13] dasebe: AdaptSize. https://github.com/dasebe/webcachesim. [14] DASH-IF: MPEG-DASH. https://mpeg.chiariglione.org/standards/mpeg- dash. [15] DASH Industry Forum. https://dash.akamaized.net/envivio/EnvivioDash3. [16] Dash Industry Forum: Dash.js. https://github.com/Dash-Industry-Forum/ dash.js. [17] encoding.com. http://1yy04i3k9fyt3vqjsf2mv610yvm-wpengine.netdna- ssl.com/files/2017-Global-Media-Formats-Report.pdf. [18] Facebook Live. https://live.fb.com/. 116 [19] Federal Communications Commission. Raw Data - Measuring Broadband America. www.fcc.gov/reports-research/reports/measuring-broadband-america/ raw-data-measuring-broadband-america-2016. [20] Golang. https://golang.org/. [21] Golang: DebuggingperformanceissuesinGoprograms.https://github.com/golang/ go/wiki/Performance. [22] Golang: runtime package. https://golang.org/pkg/runtime/. [23] Google-Chrome: Chrome DevTools Protocol. https://chromedevtools.github.io/ devtools-protocol/tot/Network/. [24] HBO NOW. https://www.hbo.com. [25] ITU: H.264. https://www.itu.int/rec/T-REC-H.264. [26] ITU: H.265. https://www.itu.int/rec/T-REC-H.265. [27] JW Player. https://www.jwplayer.com/. [28] Level2: Over the top video delivery. http://www.level3.com/~/media/files/ white-paper/en_cdn_wp_ovrtopvddlvry.ashx. [29] Microsoft: Microsoft Smooth Streaming. http://www.iis.net/downloads/ microsoft/smooth-streaming. [30] Microsoft Smooth Streaming. http://www.iis.net/downloads/microsoft/ smooth-streaming. [31] Netflix: 2017 on Netflix - A Year in Bingeing. https://media.netflix.com/en/ press-releases/2017-on-netflix-a-year-in-bingeingl. [32] Nexplayer: Nexplayer Software Development Kit. https://nexplayersdk.com/. [33] Ooyala: Global Video Index. http://go.ooyala.com/rs/447-EQK-225/images/ Ooyala-Global-Video-Index-Q4-2017.pdf. [34] Oracle: 5 Reasons to Consider SaaS for Your Business Applications. http: //www.oracle.com/us/solutions/cloud/saas-business-applications- 1945540.pdf. [35] Pensieve. https://github.com/hongzimao/pensieve. [36] Recode: Facebook Says Video Is Huge – 100-Million-Hours-Per-Day Huge. https://www.recode.net/2016/1/27/11589140/facebook-says-video- is-huge-100-million-hours-per-day-huge. [37] Sling TV: Stream Live TV Services Online. https://www.sling.com. [38] Streaming Learning Center: DASH or HLS? Which is the best format today? https://streaminglearningcenter.com/blogs/dash-or-hls-which-is- the-best-format-today.html. [39] Streamingmedia: Video: The Pros and Cons of a Multi-CDN Strategy. http://www.streamingmedia.com/Articles/Editorial/Short-Cuts/ Video-The-Pros-and-Cons-of-a-Multi-CDN-Strategy-112351.aspx. 117 [40] Telestream. http://www.telestream.net/. [41] The Chromium Projects: Flash Usage Trends. https://www.chromium.org/flash- roadmap/flash-usage-trends. [42] The Wall Street Journal: YouTube Tops 1 Billion Hours of Video a Day, on Pace to Eclipse TV. https://www.wsj.com/articles/youtube-tops-1-billion-hours-of- video-a-day-on-pace-to-eclipse-tv-1488220851. [43] Theoplayer. https://www.theoplayer.com/. [44] Toward A Practical Perceptual Video Quality Metric. https://medium.com/ netflix-techblog/toward-a-practical-perceptual-video-quality- metric-653f208b9652. [45] Unified Streaming. http://www.unified-streaming.com. [46] WebM: VP9. https://www.webmproject.org/vp9/. [47] Xbox: XDK Software Development Kit. https://www.xbox.com/en-US/ developers. [48] YouTube: You know what’s cool? A billion hours. https://youtube.googleblog. com/2017/02/you-know-whats-cool-billion-hours.html. [49] Cisco: It Came to Me in a Stream... . https://www.cisco.com/web/about/ac79/ docs/sp/Online-Video-Consumption_Consumers.pdf, 2012. [50] DASH-IF: Survey of European Broadcaster on MPEG-DASH. http://dashif.org/ wp-content/uploads/2015/04/Survey-of-the-European-Broadcasters- on-MPEG-DASH-Whitepaper-V2.1.pdf, 2013. [51] Netflix: Per-title Encode Optimization, Dec 2015. https://medium.com/netflix- techblog/per-title-encode-optimization-7e99442b62a2. [52] Sandvine: Global Internet phenomena report . https://www.sandvine.com/trends/ global-internet-phenomena/, 2015. [53] Apple: fMP4 support on Apple devices. https://developer.apple.com/ streaming/examples/, 2016. [54] Bitmovin: Video Developer Survey. https://bitmovin.com/whitepapers/ Bitmovin-Developer-Survey.pdf, Sept. 2017. [55] Cisco: Visual Networking Index: Global Mobile Data Trac Forecast Update 2016- 2021 . http://www.cisco.com/c/en/us/solutions/collateral/service- provider/visual-networking-index-vni/mobile-white-paper-c11- 520862.html, 2017. [56] comScore: OTT breaks out of its Netflix shell . https://www.comscore.com/ Insights/Blog/OTT-Breaks-Out-of-Its-Netflix-Shell, 2017. [57] H. Abrahamsson and M. Nordmark. Program Popularity and Viewer Behaviour in a Large TV-on-demand System. In Proceedings of the 2012 Internet Measurement Conference,IMC ’12, 2012. 118 [58] R. P. Adams and D. J. MacKay. Bayesian Online Changepoint Detection. In arXiv:0710.3742v1, 2007. [59] V. K. Adhikari, Y. Guo, F. Hao, V. Hilt, Z.-L. Zhang, M. Varvello, and M. Steiner. Measurement Study of Netflix, Hulu, and a Tale of Three CDNs. IEEE/ACM Trans. Netw., 23(6), Dec. 2015. [60] V. K. Adhikari, S. Jain, and Z.-L. Zhang. YouTube Trac Dynamics and Its Interplay with a Tier-1 ISP: An ISP Perspective. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC ’10, 2010. [61] S. Akhshabi, L. Anantakrishnan, A. C. Begen, and C. Dovrolis. What Happens when HTTP Adaptive Streaming Players Compete for Bandwidth? In Proceedings of the 22Nd International Workshop on Network and Operating System Support for Digital Audio and Video, NOSSDAV ’12, 2012. [62] S. Akhshabi, A. C. Begen, and C. Dovrolis. An Experimental Evaluation of Rate-adaptation Algorithms in Adaptive Streaming over HTTP. In Proceedings of the Second Annual ACM Conference on Multimedia Systems, MMSys ’11, 2011. [63] Z. Akhtar, Y. S. Nam, J. Chen, R. Govindan, E. Katz-Bassett, S. Rao, J. Zhan, and H. Zhang. Understanding video management planes. In Proceedings of the Internet Measurement Conference 2018, IMC ’18, 2018. [64] Z. Akhtar, Y. S. Nam, R. Govindan, S. Rao, J. Chen, E. Katz-Bassett, B. M. Ribeiro, J. Zhan, and H. Zhang. Oboe:Auto-tuning video ABR algorithms to network conditions. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, SIGCOMM ’18, 2018. [65] V. Almeida, A. Bestavros, M. Crovella, and A. De Oliveira. Characterizing reference locality in the WWW. In Parallel and Distributed Information Systems, 1996., Fourth International Conference on, pages 92–103. IEEE, 1996. [66] M. F. Arlitt and C. L. Williamson. Internet web servers: workload characterization and performance implications. 1997. [67] A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, and H. Zhang. Developing a Predictive Model of Quality of Experience for Internet Video. In Proceedings of the ACM Conference on Special Interest Group on Data Communication, SIGCOMM, 2013. [68] Balachandran, Athula and Sekar, Vyas and Akella, Aditya and Seshan, Srinivasan and Stoica, Ion and Zhang, Hui. Developing a predictive model of quality of experience for internet video. Aug. 2013. [69] H. Balakrishnan, M. Stemm, S. Seshan, and R. H. Katz. Analyzing Stability in Wide-area Network Performance. ACM SIGMETRICS Performance Evaluation Review, 25:2–12, 1997. [70] D. Barry and J. A. Hartigan. A Bayesian Analysis for Change Point Problems. Journal of the American Statistical Society, 88(421):309–319, 1993. [71] N. Beckmann, H. Chen, and A. Cidon. LHD: Improving Cache Hit Rate by Maximizing Hit Density. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), 2018. [72] L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Syst. J., 1966. 119 [73] T. Benson, A. Akella, and D. Maltz. Unraveling the Complexity of Network Manage- ment. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation, NSDI’09, 2009. [74] D. S. Berger. Towards lightweight and robust machine learning for cdn caching. In Proceedings of the 17th ACM Workshop on Hot Topics in Networks, HotNets ’18, 2018. [75] D. S. Berger, R. K. Sitaraman, and M. Harchol-Balter. Adaptsize: Orchestrating the Hot Object Memory Cache in a Content Delivery Network. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation, NSDI’17, 2017. [76] L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and Zipf-like distributions: Evidence and implications. In INFOCOM’99. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE, volume 1, pages 126–134. IEEE, 1999. [77] M. Butkiewicz, H. V. Madhyastha, and V. Sekar. Understanding Website Complexity: Measurements, Metrics, and Implications. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, 2011. [78] T. Böttger, F. Cuadrado, G. Tyson, I. Castro, and S. Uhlig. A Hypergiant’s View of Internet. In SIGCOMM Computer Communication Review, CCR ’18, 2018. [79] M. Calder, X. Fan, Z. Hu, E. Katz-Bassett, J. Heidemann, and R. Govindan. Mapping the Expansion of Google’s Serving Infrastructure. In Proceedings of the 2013 Conference on Internet Measurement Conference, IMC ’13, 2013. [80] M. Calder, X. Fan, Z. Hu, E. Katz-Bassett, J. Heidemann, and R. Govindan. Mapping the Expansion of Google’s Serving Infrastructure. In Proceedings of the 2013 Conference on Internet Measurement Conference, IMC ’13, 2013. [81] M. Calder, A. Flavel, E. Katz-Bassett, R. Mahajan, and J. Padhye. Analyzing the Perfor- mance of an Anycast CDN. In Proceedings of the 2015 Internet Measurement Conference, IMC ’15, 2015. [82] P. Cao and S. Irani. Cost-aware WWW Proxy Caching Algorithms. In Proceedings of the USENIX Symposium on Internet Technologies and Systems on USENIX Symposium on Internet Technologies and Systems, USITS’97, 1997. [83] N.Cardwell,Y.Cheng,C.S.Gunn,S.H.Yeganeh,andV.Jacobson. BBR:Congestion-Based Congestion Control. ACM Queue, 14:20–53, 2016. [84] M. Cha, H. Kwak, P. Rodriguez, Y.-Y. Ahn, and S. Moon. I Tube, You Tube, Everybody Tubes: AnalyzingtheWorld’sLargestUserGeneratedContentVideoSystem. InProceedings of the 7th ACM SIGCOMM Conference on Internet Measurement, IMC ’07, 2007. [85] L. Cherkasova. Improving WWW Proxies Performance with Greedy-Dual-Size-Frequency Caching Policy. 1998. [86] F. Chiariotti, S. D’Aronco, L. Toni, and P. Frossard. Online Learning Adaptation Strategy for DASH Clients. In Proceedings of the International Conference on Multimedia Systems, MMSys, 2016. [87] M. Claeys, S. Latré, J. Famaey, T. Wu, W. V. Leekwijck, and F. D. Turck. Design and Optimisation of a (FA)Q-learning-based HTTP Adaptive Streaming Client. Connection Science, 26(1):25–43, 2014. 120 [88] E. G. Coman and P. J. Denning. Operating Systems Theory (Prentice-Hall series in automatic computation). Prentice Hall, 1973. [89] C. R. Cunha, A. Bestavros, and M. E. Crovella. Characteristics of www client-based traces. 1995. [90] F. Desobry, M. Davy, and C. Doncarli. An Online Kernel Change Detection Algorithm. IEEE Transactions on Signal Processing, 53(8):2961–2974, 2005. [91] Y. Ding, Y. Du, Y. Hu, Z. Liu, L. Wang, K. Ross, and A. Ghose. Broadcast Yourself: UnderstandingYouTubeUploaders. InProceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, 2011. [92] F. Dobrian, V. Sekar, A. Awan, I. Stoica, D. Joseph, A. Ganjam, J. Zhan, and H. Zhang. Understanding the Impact of Video Quality on User Engagement. In Proceedings of the ACM SIGCOMM 2011 Conference, SIGCOMM ’11, 2011. [93] G. Einziger, R. Friedman, and B. Manes. TinyLFU: A Highly Ecient Cache Admission Policy. volume 13, Nov. 2017. [94] J. Erman, A. Gerber, K. K. Ramadrishnan, S. Sen, and O. Spatscheck. Over the Top Video: The Gorilla in Cellular Networks. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, 2011. [95] P. Fernhead. Exact and Ecient Bayesian Inference for Multiple Changepoint Problems. Statistics and Computing, 16(2):203–213, 2006. [96] A. Finamore, M. Mellia, M. M. Munafò, R. Torres, and S. G. Rao. YouTube Everywhere: Impact of Device and Infrastructure Synergies on User Experience. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, 2011. [97] T. Flach, P. Papageorge, A. Terzis, L. Pedrosa, Y. Cheng, T. Karim, E. Katz-Bassett, and R. Govindan. An Internet-Wide Analysis of Trac Policing. In Proceedings of the 2016 Conference on ACM SIGCOMM 2016 Conference, SIGCOMM ’16, 2016. [98] K. Fukuda, H. Asai, and K. Nagami. Tracking the Evolution and Diversity in Network Usage of Smartphones. In Proceedings of the 2015 Internet Measurement Conference,IMC ’15, 2015. [99] W. A. Fuller. Introduction to Statistical Time Series. John Wiley and Sons, 1976. [100] A. Ganjam, F. Siddiqui, J. Zhan, X. Liu, I. Stoica, J. Jiang, V. Sekar, and H. Zhang. C3: Internet-Scale Control Plane for Video Quality Optimization. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI 15, 2015. [101] N. Gast and B. Van Houdt. Transient and steady-state regime of a family of list-based cache replacement algorithms. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’15, 2015. [102] M. Ghasemi, P. Kanuparthy, A. Mansy, T. Benson, and J. Rexford. Performance Charac- terization of a Commercial Video Streaming Service. In Proceedings of the 2016 ACM on Internet Measurement Conference, IMC ’16, 2016. [103] P. Gill, M. Arlitt, Z. Li, and A. Mahanti. YouTube Trac Characterization: A View from the Edge. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement, IMC ’07, 2007. 121 [104] D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. Karro, and D. Sculley. Google Vizier: A Service for Black-Box Optimization. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining, SIGKDD, 2017. [105] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep Rein- forcement Learning that Matters. In Proceedings of the Association for Advancement of Artificial Intelligence, AAAI, 2018. [106] R.HoudailleandS.Gouache. ShapingHTTPAdaptiveStreamsforaBetterUserExperience. In Proceedings of the 3rd Multimedia Systems Conference, MMSys ’12, 2012. [107] Q. Huang, K. Birman, R. van Renesse, W. Lloyd, S. Kumar, and H. C. Li. An Analysis of Facebook Photo Caching. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP ’13, 2013. [108] Q. Huang, K. Birman, R. van Renesse, W. Lloyd, S. Kumar, and H. C. Li. An analysis of facebook photo caching. InProceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP ’13, 2013. [109] T.-Y. Huang, N. Handigol, B. Heller, N. McKeown, and R. Johari. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. In Proceedings of the 2012 ACM Conference on Internet Measurement Conference, IMC ’12, 2012. [110] T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A Buer-based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. In Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM ’14, 2014. [111] T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A Buer-based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. In Proceedings of the ACM Conference on Special Interest Group on Data Communication, SIGCOMM, 2014. [112] S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Hölzle, S. Stuart, and A. Vahdat. B4: Experience with a globally-deployed software defined wan. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, SIGCOMM ’13, 2013. [113] D. R. Jeske, V. Montes De Oca, W. Bischo, and M. Marvasti. Cusum Techniques for Timeslot Sequences with Applications to Network Surveillance. Computational Statistics and Data Analysis, 53:4332–4344, 2009. [114] J. Jiang, V. Sekar, H. Milner, D. Shepherd, I. Stoica, and H. Zhang. CFA: A Practical Prediction System for Video QoE Optimization. In 13th USENIX Symposium on Networked Systems Design and Implementation, NSDI 16, 2016. [115] J. Jiang, V. Sekar, I. Stoica, and H. Zhang. Shedding Light on the Structure of Internet Video Quality Problems in the Wild. In Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, CoNEXT ’13, 2013. [116] J. Jiang, V. Sekar, and H. Zhang. Improving Fairness, Eciency, and Stability in HTTP- based Adaptive Video Streaming with FESTIVE. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, CoNEXT ’12, 2012. 122 [117] J. Jiang, S. Sun, V. Sekar, and H. Zhang. Pytheas: Enabling Data-Driven Quality of Experience Optimization Using Group-Based Exploration-Exploitation. In 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017, pages 393–406, 2017. [118] J. Jobin, M. Faloutsos, S. K. Tripathi, and S. V. Krishnamurthy. Understanding the Eects of Hotspots in Wireless Cellular Networks. In Proceedings of the Conference of the IEEE Computer and Communications Societies, INFOCOM, 2004. [119] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly ecient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30, 2017. [120] E. J. Keogh, S. Chu, D. Hart, and M. J. Pazzani. An Online Algorithm for Segmenting Time Series. In Proceedings of the IEEE International Conference on Data Mining,ICDM, 2001. [121] S. S. Krishnan and R. K. Sitaraman. Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-experimental Designs. In Proceedings of the 2012 ACM Conference on Internet Measurement Conference, IMC ’12, 2012. [122] D. Lee, J. Choi, J. H. Kim, S. H. Noh, S. L. Min, Y. Cho, and C. S. Kim. Lrfu: A spectrum of policies that subsumes the least recently used and least frequently used policies. [123] Z. Li, J. Lin, M.-I. Akodjenou, G. Xie, M. A. Kaafar, Y. Jin, and G. Peng. Watching Videos from Everywhere: A Study of the PPTV Mobile VoD System. In Proceedings of the 2012 Internet Measurement Conference, IMC ’12, 2012. [124] D. Lu, Y. Qiao, P. A. Dinda, and F. E. Bustamante. Characterizing and Predicting TCP Throughput on the Wide Area Network. In IEEE International Conference on Distributed Computing Systems, ICDCS, 2005. [125] H. Mao, R. Netravali, and M. Alizadeh. Neural Adaptive Video Streaming with Pensieve. In Proceedings of the ACM Conference on Special Interest Group on Data Communication, SIGCOMM, 2017. [126] V. Martín, J. Cabrera, and N. García. Design, Optimization and Evaluation of a Q- learning HTTP Adaptive Streaming Client. IEEE Transactions on Consumer Electronics, 62(4):380–388, 2016. [127] N. Megiddo and D. S. Modha. ARC: A Self-Tuning, Low Overhead Replacement Cache. In Proceedings of the 2Nd USENIX Conference on File and Storage Technologies, FAST ’03. [128] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning, ICML, 2016. [129] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level Control Through Deep Reinforcement Learning. Nature, 518(7540):529–533, 2015. [130] M. K. Mukerjee, I. N. Bozkurt, B. Maggs, S. Seshan, and H. Zhang. The Impact of Brokers on the Future of Content Delivery. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks, HotNets ’16, 2016. 123 [131] M. K. Mukerjee, I. N. Bozkurt, D. Ray, B. M. Maggs, S. Seshan, and H. Zhang. Redesigning CDN-Broker Interactions for Improved Content Delivery. In Proceedings of the 13th International Conference on Emerging Networking EXperiments and Technologies, CoNEXT ’17, 2017. [132] M. K. Mukerjee, D. Naylor, J. Jiang, D. Han, S. Seshan, and H. Zhang. Practical, Real-time Centralized Control for CDN-based Live Video Delivery. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM ’15. ACM, 2015. [133] E. J. O’Neil, P. E. O’Neil, and G. Weikum. The LRU-K Page Replacement Algorithm for Database Disk Buering. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, SIGMOD ’93, 1993. [134] T. Petsas, A. Papadogiannakis, M. Polychronakis, E. P. Markatos, and T. Karagiannis. Rise of the Planet of the Apps: A Systematic Study of the Mobile App Ecosystem. In Proceedings of the 2013 Conference on Internet Measurement Conference, IMC ’13, 2013. [135] H. Pishro-Nik. Introduction to Probability, Statistics and Random Processes. Kappa Research, 2014. [136] T.Rakthanmanon,E.J.Keogh,S.Lonardi,andS.Evans.TimeSeriesEpenthesis: Clustering Time Series Streams Requires Ignoring Some Data. In Proceedings of the International Conference on Data Mining, ICML, 2011. [137] M. T. Ribeiro, S. Singh, and C. Guestrin. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining, SIGKDD, 2016. [138] H. Riiser, P. Vigmostad, C. Griwodz, and P. Halvorsen. Commute Path Bandwidth Traces from 3G Networks: Analysis and Applications. In Proceedings of the ACM Multimedia Systems Conference, MMSys, 2013. [139] J. Semke, J. Mahdavi, and M. Mathis. Automatic TCP Buer Tuning. In Proceedings of the ACM SIGCOMM ’98 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM ’98, 1998. [140] S.-H. Shen and A. Akella. An information-aware qoe-centric mobile video cache. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, MobiCom ’13, 2013. [141] M.Siekkinen,E.Masala,andT.Kämäräinen. AFirstLookatQualityofMobileLiveStream- ing Experience: The Case of Periscope. In Proceedings of the 2016 Internet Measurement Conference, IMC ’16, 2016. [142] K. Spiteri, R. Urgaonkar, and R. K. Sitaraman. BOLA: Near-optimal Bitrate Adaptation for Online Videos. In Proceedings of the IEEE International Conference on Computer Communications, INFOCOM, 2016. [143] Y. Sun, X. Yin, J. Jiang, V. Sekar, F. Lin, N. Wang, T. Liu, and B. Sinopoli. CS2P: ImprovingVideoBitrateSelectionandAdaptationwithData-DrivenThroughputPrediction. In Proceedings of the 2016 ACM Conference on SIGCOMM, SIGCOMM 2016, 2016. [144] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction.MITpress Cambridge, 1998. 124 [145] G. Tian and Y. Liu. Towards Agile and Smooth Video Adaptation in Dynamic HTTP Streaming. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, CoNEXT ’12, 2012. [146] R. Torres, A. Finamore, J. R. Kim, M. Mellia, M. M. Munafo, and S. Rao. Dissecting Video Server Selection Strategies in the YouTube CDN. In Proceedings of the 2011 31st International Conference on Distributed Computing Systems, ICDCS ’11, 2011. [147] G. Urvoy-Keller. On the Stationarity of TCP Bulk Data Transfers. In Proceedings of the Passive and Active Measurement Conference, PAM, 2005. [148] J. van der Hooft, S. Petrangeli, M. Claeys, J. Famaey, and F. D. Turck. A learning-based algorithm for improved bandwidth-awareness of adaptive streaming clients. In IM, pages 131–138. IEEE, 2015. [149] B. Wang, X. Zhang, G. Wang, H. Zheng, and B. Y. Zhao. Anatomy of a Personalized Livestreaming System. In Proceedings of the 2016 Internet Measurement Conference,IMC ’16, 2016. [150] L. Wei and J. Heidemann. Does Anycast Hang up on You? In IEEE International Workshop on Trac Monitoring and Analysis , Dublin, Ireland, 2017. [151] L. Wei and E. Keogh. Semi-supervised Time Series Classification. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining, SIGKDD, 2006. [152] R. J. Williams and J. Peng. Function Optimization using Connectionist Reinforcement Learning Algorithms. Connection Science, 3(3):241–268, 1991. [153] K. Winstein and H. Balakrishnan. TCP Ex Machina: Computer-generated Congestion Control. InProceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM,SIGCOMM ’13. ACM, 2013. [154] X. Xiang and K. Murphy. Modelling Changing Dependency Structure in Multivariate Time Series. In Proceedings of the International Conference on Data Mining, ICML, 2007. [155] K. Yamanishi and J.-i. Takeuchi. A Unifying Framework for Detecting Outliers and Change Points from Non-stationary Time Series Data. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining, SIGKDD, 2002. [156] X. Yin, A. Jindal, V. Sekar, and B. Sinopoli. A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM ’15, London, United Kingdom, 2015. [157] X. Yin, A. Jindal, V. Sekar, and B. Sinopoli. A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP. In Proceedings of the ACM Conference on Special Interest Group on Data Communication, SIGCOMM, 2015. [158] N. E. Young. The K-Server Dual and Loose Competitiveness for Paging. CoRR, cs.DS/0205044, 2002. [159] Y. Zhang and N. Dueld. On the Constancy of Internet Path Properties. In Proceedings of the ACM SIGCOMM Workshop on Internet Measurement, 2001. [160] J. Zhou, Y. Li, V. K. Adhikari, and Z.-L. Zhang. Counting YouTube Videos via Random Prefix Sampling. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, 2011. 125 [161] M. Zink, K. Suh, Y. Gu, and J. Kurose. Characteristics of YouTube Network Trac at a Campus Network - Measurements, Models, and Implications. Comput. Netw., Mar. 2009. 126

Abstract (if available)

Abstract In this work, we strive to understand and optimize the performance of systems which serve video over the Internet. Towards this end, we make the following three contributions. First, we perform a systematic study of the video management plane—the systems which form the video delivery pipeline. Second, we propose an approach called Oboe to improve the performance of client side video players by tuning Adaptive Bitrate Algorithms to the characteristics of a network. A range of different algorithm when augmented with Oboe improve their performance by up to 25%. Finally, we propose a caching algorithm called AViC which is specifically tailored for video workload. AViC outperforms state of the art caching algorithms by exploiting various properties of adaptive bitrate video. In particular, a LRU cache requires up to 3.5× the cache size used by AViC to match its performance.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Improving reliability, power and performance in hardware transactional memory

PDF

Elements of next-generation wireless video systems: millimeter-wave and device-to-device algorithms

PDF

Efficient pipelines for vision-based context sensing

PDF

Improve cellular performance with minimal infrastructure changes

PDF

Global analysis and modeling on decentralized Internet

PDF

Cache analysis and techniques for optimizing data movement across the cache hierarchy

PDF

Adaptive resource management in distributed systems

PDF

Estimation of graph Laplacian and covariance matrices

PDF

Towards highly-available cloud and content-provider networks

PDF

Asynchronous writes in cache augmented data stores

PDF

Making web transfers more efficient

PDF

Coded computing: a transformative framework for resilient, secure, private, and communication efficient large scale distributed computing

PDF

Architectural innovations for mitigating data movement cost on graphics processing units and storage systems

PDF

Compression of signal on graphs with the application to image and video coding

PDF

Performant, scalable, and efficient deployment of network function virtualization

PDF

Transfer learning for intelligent systems in the wild

PDF

Domical: a new cooperative caching framework for streaming media in wireless home networks

PDF

Context-aware models for understanding and supporting spoken interactions with children

PDF

QoS-aware algorithm design for distributed systems

PDF

Efficient techniques for sharing on-chip resources in CMPs

Asset Metadata

Creator Akhtar, Zahaib (author)

Core Title Understanding and optimizing internet video delivery

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 10/03/2019

Defense Date 05/29/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag ABR algorithms,ABR video,adaptive bitrate streaming,caching,internet video,OAI-PMH Harvest,video caching,video management planes

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Govindan, Ramesh (committee chair), Ortega, Antonio (committee member), Raghavan, Barath (committee member)

Creator Email zahaib.akhtar@gmail.com,zakhtar@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-222867

Unique identifier UC11674126

Identifier etd-AkhtarZaha-7845.pdf (filename),usctheses-c89-222867 (legacy record id)

Legacy Identifier etd-AkhtarZaha-7845.pdf

Dmrecord 222867

Document Type Dissertation

Rights Akhtar, Zahaib

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA