University of Southern California Dissertations and Theses

Leveraging programmability and machine learning for distributed network management to improve security and performance

(USC Thesis Other)

Leveraging programmability and machine learning for distributed network management to improve security and performance

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content LEVERAGING PROGRAMMABILITY AND MACHINE LEARNING FOR DISTRIBUTED NETWORK MANAGEMENT TO IMPROVE SECURITY AND PERFORMANCE by Sivaramakrishnan Ramanathan A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulllment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2021 Copyright 2021 Sivaramakrishnan Ramanathan Dedication To Krutika. ii Acknowledgements I would like to thank my advisors, Dr. Jelena Mirkovic, and Dr. Minlan Yu for giving me an opportunity to pursue my Ph.D. When I came to the United States, I had no plans of pursuing a Ph.D., and neither did I have any research experience. They took a chance and guided me with unwavering support and encouragement throughout my graduate studies. Specically, Dr. Minlan Yu’s Computer Communication course and a summer internship with Dr. Jelena Mirkovic’s lab inspired me to get into research. Throughout my graduate studies, they were extremely supportive by encouraging me to do multiple internships, funding me to attend dierent conferences, and also guiding me through the job search process. I will be always grateful to my advisors who always found time for me, shaping my thoughts and overall making me a better researcher. I was fortunate to collaborate with many researchers during my Ph.D. I would like to extend my grat- itude to Dr. Yaron Kanza and Dr. Balachander Krishnamurthy who were kind enough to host me two times at AT&T Labs and actively playing an important role in SPred (Chapter 2) and SDProber (Chapter 3). Specically, they always emphasized solving real-world problems that would benet the community. Outside of my internship, Dr. Yaron Kanza spent signicant time helping me shape my research in both SDProber and SPred. He has always been patient to listen to my ideas and was always eager to give me insightful feedback. I had an amazing time at ICSI with Dr. Sadia Afroz and Annushah Hossain where iii we developed the technique of identifying reused addresses (Chapter 6). I was also fortunate to collabo- rate with Dr. Ran Ben Basat, Dr. Yuliang Li, Dr. Gianni Antichi, Dr. Michael Mitzenmacher, Dr. Pravein Govindan Kannan, and Dr. Alexander Rush. I’ve been very lucky to have been a part of two labs actively one at USC and the other at Harvard. The folks at both these labs were always supportive, ready to listen to my ideas, my mock talks, review my papers, and have oered dierent types of support throughout my Ph.D. Special thanks to Zhiying Xu, Junzhi Gong, Yang Zhou, Dr. Jiaqi Gao, Omid Alipourfard, Rui Miao, Xiyue Deng, Wei-Cheng Wu, Sima Arasteh, Rajat Tandon, Nicolaas Weideman, Jaydeep Ramani, Haoda Wang, Brandon Paulson, Dr. Simon Woo, Abdul Qadeer, Abhinav Palia, Ameya Hanamsagar, Dr. Hao Shi, Dr. Christophe Hauser, Fawad Ahmad, Jane Yen, and Sulagna Mukherjee. Special thanks to Joe Kemp, Alba Regalado, Jeanine Yamazaki, and Lizsl De Leon who ensured that I had no roadblocks during my graduate studies. I am very privileged to be in a position where I can fulll my dreams without any limitations because of the many sacrices my mother Jeyameenakshi and my father S.S. Ramanathan, made in their lives. I am fortunate to have a growing family, and for every single one to be constant support throughout my graduate studies. I would also like to thank my friends from Bangalore, Boston, and Los Angeles for doing a great job in checking my sanity for the past ve years. I am also grateful for my cats, Ginger and Ale, who have helped me in ways that they will never understand. Finally, all this would not have been possible without my then girlfriend, my current ancée, and my future wife Krutika. iv TableofContents Dedication ii Acknowledgements iii ListofTables ix ListofFigures x Abstract xiii Chapter1: Introduction 1 1.1 Challenges in Distributed Network Management . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Monitoring all network events is critical but expensive . . . . . . . . . . . . . . . . 2 1.1.2 Lack of coordination among networks to handle events . . . . . . . . . . . . . . . . 4 1.1.3 Unreliable events generated from dierent networks . . . . . . . . . . . . . . . . . 5 1.2 New Opportunities in Network Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 Programmable Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.2 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Leveraging Programmability and Machine Learning for Distributed Network Management 8 1.3.1 Measurement frameworks that eciently monitor and react to dierent types of network events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.2 Framework that allows coordination between victim and ISP to detect and mitigate attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.3 Framework to improve accuracy and coverage of blocklists . . . . . . . . . . . . . 10 1.4 Industrial Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Chapter2: SPred: MLPredictiononProgrammableSwitches 13 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3 SPred Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3.1 Complex Predictors in Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3.2 Address Network and Trac Changes . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 LSTM predictions at switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.4.1 Background on LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.4.2 Problems of running LSTM at switches . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4.3 Transforming LSTM into DFA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.4 From DFA to P4 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 v 2.4.5 Retraining and Rening the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5.1 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5.1.1 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5.1.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.5.1.3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.5.1.4 Training and Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.5.1.5 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.5.1.6 Test on Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.5.2 Microburst Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.5.3 Congestion Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.5.4 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.5.4.1 Constraints on Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.5.4.2 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.5.4.3 Choice ofp andw for congestion control . . . . . . . . . . . . . . . . . . 44 2.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Chapter3: SDProber: ASoftwareDenedProberforSDN 46 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2 Network and Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.3 Overview of SDProber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.3.1 Delay Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.3.2 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.2.1 Probe Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.2.2 SDN Controller and Open vSwitches . . . . . . . . . . . . . . . . . . . . 52 3.3.2.3 Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.4 Monitoring by Random Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.5 Weight Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.6 Baseline Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.7.1 Control over Inspection Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.7.2 Cost Eectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.7.3 Adjusting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.7.4 Detection Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.8 Weight Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.9 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Chapter4: SENSSAgainstVolumetricDDoSAttacks 65 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.2 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.2.1 Volumetric Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.2.3 SENSS vs First-ISP vs Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.3 SENSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.1 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.2 SENSS Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 vi 4.3.3 ISP Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.3.4 Client Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3.5 Security and Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.4.1 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.4.2 2016 attack on Dyn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.4.3 Eectiveness in Sparse Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.4.4 Comparison of SENSS and Cloud Defenses . . . . . . . . . . . . . . . . . . . . . . . 92 4.4.5 Delay, Trac and Message Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.4.6 Scalability within an ISP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Chapter5: BLAG:ImprovingtheAccuracyofBlacklists 97 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.2 Problems With Current Blacklists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.2.1 Fragmented Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.2.2 Re-oense Is Frequent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.2.3 Malicious Sources Are Co-located . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.4 Careful Aggregation and Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.3 BLAG Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 5.3.1 Relevance Scores: Evaluating Quality . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.3.2 Recommendation System: Estimating Future Misclassications . . . . . . . . . . . 107 5.3.3 Selective Expansion to Prexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.4 Why BLAG Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.4 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.4.1 Blacklist Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.4.2 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.4.2.1 Malicious Email Campaign or Spam . . . . . . . . . . . . . . . . . . . . . 114 5.4.2.2 DDoS on a University network . . . . . . . . . . . . . . . . . . . . . . . . 115 5.4.2.3 DDoS on DNS root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.4.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.5.1 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.5.2 BLAG is More Accurate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.6 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.6.1 Expansion approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.6.2 Contribution of recommendation system . . . . . . . . . . . . . . . . . . . . . . . . 125 5.6.3 Contribution of Individual Blacklists . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.6.4 Contribution of known-legitimate sources (L tr ): . . . . . . . . . . . . . . . . . . . . 126 5.7 Parameter tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.8 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 vii Chapter6: QuantifyingtheImpactofBlocklistingintheAgeofAddressReuse 132 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.3 Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.3.1 Identifying NATed Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 6.3.2 Identifying Dynamic Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.4 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.6 Understanding blocklists usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 Chapter7: Conclusions 151 Bibliography 153 viii ListofTables 2.1 Measure-control systems that leverage network events in the data plane. . . . . . . . . . . . . . . 16 2.2 Training and testing dataset for predicting microbursts at dierent trac congurations. Each cell consists of the number of samples for predicting microbursts at 50 th percentile and 90 th percentile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.1 SENSS APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.2 Key innovations of client programs and the SENSS features we use . . . . . . . . . . . . . 79 4.3 Lying ISP scenarios and their eect (TR: trac query reply, RR: route query reply) . . . . . . . . 87 4.4 Providers and peers of select clouds, which we use in our evaluation. . . . . . . . . . . . . 93 5.1 Scenario datasets used in this study. Each scenario dataset is split into three – training, validation and testing. The training dataset is collected chronologically before the validation and testing, and contains only legitimate sources (L tr ). The validation and testing datasets are collected during malicious events and contain malicious (M v and M tr ) and legitimate (L v and L tr ) sources. The validation dataset is used to tune the appropriate parameters used in BLAG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.2 Four types of blacklists, roughly categorized by the type of malicious activities they capture. Each row gives the number of blacklists and blacklist maintainers for that type. . 113 6.1 Each row shows the number of blocklists provided by the blocklist maintainer. We monitor 151 blocklists to determine the number of blocklisted addresses using NAT or are dynamically allocated. Blocklists used by network operators who took our survey are marked with (*). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.2 Summary of survey responses on usage of blocklists. Questions in (*) were only taken by 34 out of 65 respondants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 ix ListofFigures 2.1 Prediction by LSTM and EWMA+. The queuing delay exceeds the threshold (10s) at times 25–100s. Red circles (from 25–175s) denote an event (ground truth), and check marks and crosses above the graph denote accurate and missed predictions. . . . . . . . . 18 2.2 An example showing how setting the right threshold for EWMA+ is hard. Shades of red indicate incorrect prediction and shades of green indicate correct prediction. . . . . . . . . 19 2.3 SPred uses network traces to generate approximate LSTM models that are written as rules in the data plane. Periodically, the model is retrained from misclassications and periodic network traces from the data plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 An LSTM cell and its gates: input (i t ), forget (f t ) and output (o t ). The cell is presented for timest 1 andt to illustrate the self loop. At each timet, input featuresx t are provided to the LSTM and an hidden stateh t is returned. . . . . . . . . . . . . . . . . . . . . . . . . 23 2.5 The equations of the LSTM cell are applied to a feature vectorx t and return a hidden state h t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.6 From network abstraction to a DFA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.7 An automaton and the match-action tables produced in the transformation. . . . . . . . . . 27 2.8 Capturing prediction errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.9 F1 score when predicting 50 th percentile microbursts,p =25s, 50s and 100s. . . . . . . 34 2.10 F1 score when predicting 90 th percentile microbursts,p =25s, 50s and 100s. . . . . . . 35 2.11 The prediction accuracy for a microburst event and a non-microburst event. . . . . . . . . 38 2.12 Re-training based on misclassications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.13 Slowdown of DCTCP algorithm over dierent trac loads. . . . . . . . . . . . . . . . . . . 40 2.14 Slowdown of TCN algorithm over Websearch trac with varying workloads. . . . . . . . . 41 x 2.15 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.1 Schematic view of the mirroring process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.2 System architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.3 OpenFlow rules and group tables in SDProber. . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4 Actual probing rate versus expected rate, for links with dierent traversal probability. . . . 58 3.5 The average number probe packets per link when satisfying the min-rate constraints. . . . 58 3.6 The number of surplus probe packets per link when satisfying the min-rate constraints. . . 58 3.7 Detection time of delayed links per. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 3.8 Detection time of delayed links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.1 DDoS attacks and ways to handle them. (a) Attack trac mostly converges at one bottleneck, forming a tree-like pattern. (b) lter in a cloud, requires redirection and dedicated infrastructure (c) lter on the path of the attack, with ISP’s help, using SENSS . 67 4.2 SENSS architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.3 Illustration of trac query. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.4 Illustration of DDoS attack handling with SENSS. Yellow nodes are victims, blue are legitimate clients and red are attackers. White nodes are ISPs that do not deploy SENSS, and grey are ISPs that deploy SENSS. Red and blue lines represent attack and legitimate trac, respectively. Grey lines represent replies to spoofed requests. Numbers in the same color above the lines represent volume in Gbps. Black tags on links are ISP-specic, anonymized tags that will be used in some replies by SENSS. We show only relevant elds in SENSS messages, shown as text above the diagram. . . . . . . . . . . . . . . . . . . . . 81 4.5 DDoS attack ltered, versus the percentage of transit ASes deploying SENSS, given the top and the random deployment strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.6 Comparison of SENSS versus several cloud-based DDoS defenses, with regard to bandwidth consumed by attack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.7 SENSS client using our client program to mitigate oods w/o signature attack. . . . . . . . . . . 94 5.1 Blacklists have low coverage and tend to overlap with other categories of blacklists. Therefore, aggregating blacklists of dierent types can improve coverage. Also blacklisted addresses tend to be collocated, thus expanding IP addresses to prexes may further improve malicious source identication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 xi 5.2 Blacklisted addresses re-oend quickly. A possible solution is to expand addresses to prexes, but this causes misclassication of legitimate sources. . . . . . . . . . . . . . . . . 100 5.3 BLAG implementation consists of assigning relevance scores to IP addresses from dierent blacklists and creating a new misclassication blacklist (MB). The MB contains all possible IP addresses, but a score is only assigned to those that are misclassications from the known-legitimate sources dataset (L tr ). Then, a recommendation system generates missing scores for IP addresses in the MB. IP addresses that have a score higher than threshold (red blocks in MB) are pruned out and the remaining ones are used for aggregation. These IP addresses will be put on a new blacklist known as “Master blacklist candidates”. Finally, we selectively expand some candidates into /24 prexes, if we estimate that this expansion will not severely increase misclassication. . . . . . . . . . . . 103 5.4 Latent factorization of the score matrixR, aMN matrix, where M is the number of IP addresses and N is the number of blacklists. The cells indicate relevance scores. IP addresses not listed in a blacklist are assigned a zero score and IP addresses listed in misclassication blacklist (MB) are given a score of 1. Score matrix is factorized into two matrices ofMK andKN, and the cross product results in a new matrixR 0 , which updates the missing scores in MB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.5 Selective expansion of IP addresses into prexes. . . . . . . . . . . . . . . . . . . . . . . . . 111 5.6 Specicity and recall of BLAG with two competing approaches on trac datasets. . . . . . 118 5.7 Delay in reporting malicious IP addresses reported by BLAG. . . . . . . . . . . . . . . . . . 118 5.8 Specicity and recall of BLAG and four competing approaches with expansion. . . . . . . . 121 5.9 Evaluating BGP and AS expansion techniques. . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.10 Evaluating impact on specicty/recall due to various components in BLAG and contribu- tion of blacklist size to BLAG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.11 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.1 BitTorrent crawler to detect NATed reused addresses. In (a), the crawler identies BitTorrent users with same IP address and multiple port numbers. In (b) and (c), the crawler sendsbt_ping toIP 1 andIP 2 and receives replies. In (d), the crawler determines IP 1 is a NATed reused address. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 6.2 IP addresses allocated to RIPE Atlas probes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.3 CDF of blocklisted and reused addresses from each AS. . . . . . . . . . . . . . . . . . . . . 143 6.4 Detecting NATed and dynamic addresses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.6 Number of users behind NATed addresses in blocklists. . . . . . . . . . . . . . . . . . . . . 147 xii Abstract The rise in dierent types of applications has attracted many users to the Internet. Companies generate revenue from users and a key component for user retention is reliable network performance. Network operators are constantly scaling their infrastructure to provide tight network and security guarantees to their users. However, issues in networks such as packet drops, low utilization of links, and targeted attacks can violate these guarantees. Network operators use dierent network management tools to understand and diagnose problems in the network. As the network scales to support more users, tools that are traditionally used to understand and diagnose problems, also need to change. For instance, there exist transient events occurring at mi- crosecond granularity in datacenter networks that could aect the network’s performance. Traditional tools may miss such events as they work at coarser time granularities. As networks grow, securing the network has also become hard. In the past year, there have been a 776% increase in large volumetric denial of service (DDoS) attacks and networks have spent up to $50,000 to protect themselves. Moreover, most deployed defenses are reactive in nature, where a mitigation strategy is only developed when symptoms of attacks are seen. Proactively detecting attackers would not only block all attack trac but also reduce costs for victim networks. In this dissertation, we leverage recent advancements in programmable switches and machine learning to develop frameworks for better network management. We present SPred, which uses machine learning models in switches to detect transient events faster. We designed SDProber [170] to balance the cost of xiii monitoring with event detection time. We also built frameworks that help network operators to meet secu- rity guarantees. We present SENSS [173], which allows networks to coordinate with upstream networks to develop better detection and mitigation strategies against DDoS attacks. Finally, we present BLAG [171] that makes blocklists more suitable for emergency response by combining blocklists of dierent attack types and reducing the collateral damage by using a recommendation system and reused address detec- tion [168]. Our work has several industrial impacts. SENSS has been deployed in three academic networks to provide better detection of DDoS attacks. Our technique to identify reused addresses is being adopted by IPInfo [51], which maintains a large repository of IP address related information. We have open sourced BLAG and actively collect blocklists from dierent sources daily. Finally, AT&T has partially deployed SDProber in their network to detect persistent congestion and we hold two patents for SDProber and SPred. xiv Chapter1 Introduction To accommodate the increasing number of users and applications on the Internet, companies are con- stantly scaling their infrastructure. A key component in customer retention is the underlying network. Network events such as packet drops, low link utilization, and targeted attacks can break the service level agreements (SLA) provided by the network to the customer. Network management involves constantly monitoring distributed network events and taking neces- sary actions to maintain tight SLAs. For instance, there are many short-lived events in the datacenter such as microbursts, that cause a momentary surge in the queue occupancy, which degrades performance in the network. Quick detection or even prediction of these events can increase performance in the form of decreased ow completion time, or it can reduce the tail latency of ows. As another example, volumet- ric denial of service (DDoS) attacks originate from distributed sources on the Internet to cause expensive damages to a victim network. Stopping DDoS attacks near the source can prevent the attack trac from reaching the victim. Alternatively, proactively identifying these distributed malicious sources and blocking their trac can neutralize attacks before they inict serious damage at the victim. Recently, there are several advancements in network hardware and techniques that can be used to improve network security and performance. Switches, for instance, are becoming more programmable, 1 allowing network operators to customize their own dataplanes to support more exible routing and net- work trac manipulation. Machine learning algorithms and their supporting hardware have advanced considerably to extract key insights from network events at scale. 1.1 ChallengesinDistributedNetworkManagement Although networks want to provide strict guarantees on performance, availability, and security, network management tools have not kept up. This mainly arises due to various monitoring requirements, lack of coordination between networks to handle dierent types of events, or unreliable event observations generated from dierent networks. 1.1.1 Monitoringallnetworkeventsiscriticalbutexpensive Need to detect events at smaller timescales: To keep up with the service guarantees, networks are required to be more sensitive to small, transient events such as microbursts or high link utilization, which occur at small time granularities (usually at microseconds). Although the duration of these events is small, their impact often cascades to cause visible problems, including load imbalance, increase in ow completion times, reduced reachability, and security compromise. Traditionally, network events from switches are reported to an external controller, which decides an eective mitigation strategy. This works well for longer events, such as RTT uctuation, but it can be ineective for shorter, transient events for the following reasons. Firstly, programmable switches allow eective monitoring of transient events by frequently exporting the events to an external controller. But this could be expensive in bandwidth consumption and storage. For instance, monitoring queue occupancy experienced by every packet in a queue adds 6.8% bandwidth overhead [22]. Secondly, longer events are usually persistent and the time taken to export events from the switch may not be signicant in comparison to event’s duration. On the other hand, transient events can be detected 2 late in switch-to-controller paradigm, as the time taken to export events to an external entity could be longer than the event itself. Thirdly, some switches allow regular polling from an external controller to enable timely detection of transient events. Since transient events could occur at any time, this may require polling the switch at a very high frequency, thereby unnecessarily increasing the load on the switch’s CPU. Finally, measure and control systems, which are required to react to transient network events, would have sub-optimal performance when the transient events are detected late or not detected at all. Consider ECN based congestion control algorithms, such as DCTCP [8], that mark packets on congestion events to signal end-hosts to adjust their sending rates. In networks that predominantly contain short ows, such signal will not produce signicant congestion reduction, as short ows can be completed within 1 RTT, i.e., before congestion notication reaches the end host. This is noticeable in today’s data-center networks, where 55% of all ows are short ows [139]. Itisnotalwaysnecessarytomonitorallevents: Network operators use several monitoring tools to understand the network’s performance. Datacenters at Facebook, Microsoft, and Google have dedicated measurement infrastructure to constantly monitor networks. For instance, Pingmesh [83] a framework built on ping measures round trip times (RTT) between end hosts to identify hosts that experience high delays. Although, pingmesh probes the network infrequently (orders of minutes), it still generates up to 24 TB of data every day! Beyond transient events, traditionally monitored events, such as persistent delays, provide useful and actionable information for network management. However, monitoring such events at all times may be unnecessary. For instance, at a large backbone ISP, persistent congestion events are frequent and are distributed unevenly in the network. Some regions of the network frequently experience persistent con- gestion events, while at the same time, other regions may never experience any congestion event. This may be attributed to the time of day (night vs. day) or where the customers are located (rural vs. urban) [147]. 3 To monitor such persistent congestion events, network operators deploy periodic active probes such as ping or traceroute to capture delays between end hosts. However, there is a trade-o between detection time and the cost of monitoring. Increasing the inspection rates on links can reduce the detection time, but inspecting too often can hinder regular trac by putting additional load on switches. Moreover, existing tools are inexible, as they allow only end-to-end measurement of delay, and not measurement of delay per link. For instance, consider two groups of links, where the rst group needs to be inspected atx times per minute and the other group needs to be inspected at 2x times per minute. Such constraints cannot often be satised by traceroute via predened paths. Objective: Designmeasurementframeworksthatcapturetransienteventsforquickmitiga- tion and frameworks that balance the trade-o between the cost of monitoring and detection timeofevents. 1.1.2 Lackofcoordinationamongnetworkstohandleevents As the number of users on the internet increases, there exists a subset of malicious users that cause harm or compromise the security of other users. Volumetric DDoS attacks send a large volume trac, to overwhelm a victim’s network. Typically, such attacks originate from many geographically distributed, compromised sources. In 2020, Amazon Web Services (AWS) was hit by one of the largest recorded DDoS attacks of 2.3 Tbps [188]. Volumetric attacks are problematic, not only to the victim, but also to the trac ows that share the path with the attack. Ideally, networks upstream from a victim are in a favorable position to help the victim under attack. For instance, they could help the victim identify malicious trac towards them and also lter attack trac before it reaches the victim. However, today’s internet has no automated mechanism for victims to request help from upstream networks in isolating or mitigating attacks. Current mechanisms are crude, where communications be- tween victims and upstream networks occur via human channels, which are slow and error-prone. Even 4 when the upstream networks are able to help, they often use strategies that have high collateral damage like blackholing, where all of the trac towards the victim (including the legitimate trac) is dropped. Objective: Design frameworks that allow secure collaboration between the attack’s victim andupstreamnetworkstoimproveattackcharacterizationandmitigation. 1.1.3 Unreliableeventsgeneratedfromdierentnetworks Many networks maintain public blocklists that contain a list of attackers (identied by IP addresses). These blocklists are usually curated by observing attack events in the maintainer’s network. IP blocklists are simple and ecient to implement, where IP addresses of incoming trac are checked at line rate and matching trac is dropped or diverted. Blocklists can be potentially used as an emergency response, when a network is under a novel attack. Blocking or diverting trac from known oenders can act as the rst layer of defense. For instance, when an email server is hit by a large phishing campaign and the signature does not yet exist in the spam lters, blocklists can be used to drop emails sent from known malicious servers. Under such circumstances, it is crucial for blocklists to be accurate, as they are required to identify the majority of attacks and keep the misclassications low. However, individual blocklists are not accurate enough to serve as an emergency response for several reasons. First, individual blocklists miss many attackers, as they have limited vantage points and often monitor only a single type of attack. But compromised devices are used to send various types of attacks over time, that is, they can send DDoS attacks one day and can be involved in a spam campaign months later. Second, individual blocklists record attackers that are currently active, however, malicious entities can stay dormant to evade blocklisting and attack at a later time. Third, blocklists are reactive, that is, malicious entities are listed in them only after an attack event is observed by the blocklist maintainer. However, attackers often reside in mismanaged networks [239], and proactively identifying these networks could improve attack detection. 5 Blocklists can also lead to misclassications, by listing legitimate addresses. The presence of legitimate addresses in blocklists can be due to several reasons. First, blocklist maintainers use their own proprietary algorithm to include or exclude addresses, and such algorithms could be error-prone. Second, historical blocklist data could provide insights into potential re-oenders, but they also contain addresses that are no longer malicious. Dierent forms of misclassications could be amplied when historical blocklist data from dierent sources are used. Blocklists are widely used by network operators such as Cloudare or Akamai. However, using IP addresses as an identier for malicious activity can be problematic when an IP address is shared or reused. For instance, a user [48] who tried to access websites hosted on Cloudare was unnecessarily subjected to CAPTCHAs. On further inspection, they found that their public IP address was shared by many other users via Network Address Translation (NAT). One of the NAT users was running a spam campaign leading to the NAT’s IP address being listed in many blocklists. It is known that Cloudare uses its own IP reputation with the help of blocklists to protect its customers. Thus some users can be unjustly blocked, whenever users access a CloudFlare hosted website. Objective:Combinemultiplerecordedattackevents(blocklists)maintainedbydierentnet- works and dierent attack types, while minimizing misclassications, to make blocklists suit- able for emergency response. Also, identify reused addresses in blocklists to further reduce misclassications. 1.2 NewOpportunitiesinNetworkMonitoring Our identied objectives can improve measurement techniques, provide better attack identication and mitigation to victim networks, and also provide accurate and comprehesive blocklists, which can be used for emergency response. To support these objectives we would like to leverage new advances in pro- grammable switches and machine learning. 6 1.2.1 ProgrammableSwitches Switches play a crucial role in achieving exible monitoring of network events or provide better detec- tion/mitigation capabilities to victim networks. Traditional switches are purposefully designed to only forward packets at high speed based on pre-dened rules. Requiring switches to perform any other func- tions would aect their packet forwarding rates. Recent advances in programmable switches allow operators to build exible applications beyond packet forwarding. We can leverage the supported functionalities in programmable switches to build frameworks for better network management. Programmable switches can aid in monitoring network events, such as queue occupancy, counters, or delays at ne timescales. For instance, such ne-grained statistics could be used by upstream networks to make trac-related metrics available to victim networks, to aid in attack characterization. Switches also oer programmable match-action tables, which allow network operators to match on packet headers and take actions such as forwarding, diverting, modifying or dropping packets. For example, trac-related metrics can be used by victim networks to develop accurate attack signatures, which can be inserted into the match action tables of the upstream networks for attack mitigation. The challenge is that programmable switches are still restricted in functionalities, and support only ba- sic arithmetic, comparison, and bitwise operations. Consider using programmable switches for predicting transient events. This would involve implementing a machine learning inference algorithm in switches, which often requires complex operations such assigmoid ortanh, which are currently not supported in programmable switches. Moreover, the switch’s memory is often small to bound the memory access times. Therefore, additional supporting applications should work with these memory restrictions. 1.2.2 MachineLearning There is a need to detect and mitigate transient events with sub-RTT delays, and this can only be achieved with prediction. Machine learning has been eective in predicting events in time-series data and using 7 programmable switches to predict transient events in the network can aid measure-and-control systems to react faster detection to transient events. Although there are fundamental limitations in programmable switches, we could still utilize supported functionalities to build alternative representations of prediction models in switches. Network operators using these blocklists can unnecessarily drop legitimate trac. Machine learning can also be used to identify misclassications in blocklists and identify addresses that are likely to be attackers. This could help boost the performance of blocklists, making them more suitable for emergency response. 1.3 LeveragingProgrammabilityandMachineLearningforDistributed NetworkManagement Based on these new opportunities, we can build frameworks for better network management. 1.3.1 Measurementframeworksthatecientlymonitorandreacttodierenttypesof networkevents To capture events at dierent timescales in the network, my thesis develops frameworks to support ma- chine learning in programmable switches, allowing the network operator to eectively capture and predict network events and also leverage programmable match-action table to eectively monitor regions in the network that are prone to failures. SPred: Frameworkthatenablesmachinelearninginswitchesforquickerreactionstotran- sient events. As the bandwidth of networks grows and their latency gets ultra-low, their performance becomes increasingly sensitive to ne-timescale network events such as microbursts, congestion, and load imbalances. By relying on detection, networks react to events when they already occur, which leads to some trac loss, due to detection and signaling delay. In this work, we have developed SPred, which uses 8 an ML prediction model, executed in the data plane, to predict transient network events and reduce miti- gation delay. We use Long-Short-Term-Memory (LSTM)—the state-of-the-art prediction tool. Because it is challenging to deploy an LSTM model on programmable switches, due to their resource constraints, SPred uses several optimizations. We train the LSTM model oine, then transform it into a DFA-based approxi- mation, and translate it into a match-action rule system like P4 to predict network events at line rate. We also detect missed predictions and use them to retrain and improve SPred in real-time. Our evaluation with large-scale simulations and a P4 prototype shows that SPred can predict various network events, such as microbursts and congestion, with high accuracy, and improve network performance. SDProber [170]: Framework that automatically adjusts probing rates on links to focus on network regions prone to failures. Proactive measurement of the delay in communication networks aims to detect congestion and nd links on which the trac ow is obstructed. The goal is to detect links with delays as early as possible, without interfering with the network trac. There is, however, a tradeo between the detection time and the cost (e.g., bandwidth utilization). An adaptive measurement adjusts the inspection rate per each link, for eective monitoring with reduced costs, but it requires control over forwarding rules. In this work, we show how adaptive measurement can be implemented eectively in SDN. We develop SDProber—a tool for proactive measurement of delays in SDN. SDProber uses probe packets that are routed by adding tailored rules to the vSwitches. It adjusts the forwarding rules to route probe packets more frequently to areas where congestion tends to occur. To increase eciency, instead of computing complex routes for probe packets, SDProber uses a novel approach of probing by a random walk. Adaptation is achieved by changing the probabilities that govern the random walk. We show how SDProber uses OpenFlow to conduct the random walk, and we present experimental results to illustrate the cost-eectiveness of SDProber. The experiments show that SDProber provides control over the probe rates per each link and that it reduces measurement costs in comparison to baseline methods that send probe packets via shortest paths. 9 1.3.2 FrameworkthatallowscoordinationbetweenvictimandISPtodetectandmitigate attacks My thesis develops collaborative frameworks that leverage programmable switches and software-dened networking (SDN) to enable communication between a victim and ISP for better detection and mitigation strategies. SENSS[173]: Asecurityframeworkforbetterdetectionandmitigation. Volumetric distributed denial-of-service (DDoS) attacks can bring any network to a halt. Because of their distributed nature and high volume, the victim often cannot handle these attacks alone and needs help from upstream ISPs. To- day’s Internet has no automated mechanism for victims to ask ISPs for help in attack handling, and ISPs themselves do not oer such services. We have developed SENSS, a security service for collaborative miti- gation of volumetric DDoS attacks. SENSS enables the victim of an attack to request attack monitoring and ltering on demand, and to pay for the services rendered. Requests can be sent both to the immediate and to remote ISPs, in an automated and secure manner, and can be authenticated by these ISPs, without having prior trust with the victim. Simple and generic SENSS APIs enable victims to build custom detection and mitigation approaches against a variety of DDoS attacks. SENSS is deployable with today’s infrastructure, and it has strong economic incentives both for ISPs and for the attack victims. It is also very eective in sparse deployment, oering full protection to direct customers of early adopters, and considerable protec- tion to remote victims when deployed strategically. Deployment on the largest 1% of ISPs protects not just direct customers of these ISPs, but everyone on the Internet, from 90% of volumetric DDoS attacks. 1.3.3 Frameworktoimproveaccuracyandcoverageofblocklists My thesis has developed an approach that combines blocklists of dierent attack types to make them more suitable for emergency response. I have also developed techniques to identify reused addresses in blocklists to reduce misclassications. 10 BLAG[171]:Blocklistaggregatorthatusesmachinelearningtopruneoutmisclassications. IP address blocklists are a useful source of information about repeat attackers. Such information can be used to prioritize which trac to divert for deeper inspection (e.g., repeat oender trac), or which trac to serve rst (e.g., trac from sources that are not blocklisted). But blocklists also suer from overspecial- ization – each list is geared towards a specic purpose – and they may be inaccurate due to misclassication or stale information. We have developed BLAG, a system that evaluates and aggregates multiple blocklists feeds, producing a more useful, accurate and timelymasterblocklist, tailored to the specic customer net- work. BLAG uses a sample of the legitimate sources of the customer network’s inbound trac to evaluate the accuracy of each blocklist over regions of address space. It then leverages recommendation systems to select the most accurate information to aggregate into its master blocklist. Finally, BLAG identies por- tions of the master blocklist that can be expanded into larger address regions (e.g. /24 prexes) to uncover more malicious addresses with minimum collateral damage. Our evaluation of 157 blocklists of various at- tack types and three ground-truth datasets shows that BLAG achieves high specicity up to 99%, improves recall by up to 114 times compared to competing approaches, and detects attacks up to 13.7 days faster, which makes it a promising approach for blocklist generation. Techniques[168]toidentifyreusedaddressesinblocklists. Blocklists, consisting of known ma- licious IP addresses, can be used as a simple method to block malicious trac. However, blocklists can potentially lead to unjust blocking of legitimate users due to IP address reuse, where more users could be blocked than intended. IP addresses can be reused either at the same time (Network Address Translation) or over time (dynamic addressing). We have developed two new techniques to identify reused addresses. We built a crawler using the BitTorrent Distributed Hash Table to detect NATed addresses and used the RIPE Atlas measurement logs to detect dynamically allocated address spaces. We then analyzed 151 publicly available IPv4 blocklists to show the implications of reused addresses and found that 53–60% of blocklists 11 contained reused addresses, having about 30.6K–45.1K listings of reused addresses. We also found that reused addresses can potentially aect as many as 78 legitimate users for as many as 44 days. 1.4 IndustrialImpact In this dissertation, we present frameworks that leverage programmable switches and machine learning to monitor network events at various timescales, improve attack detection/mitigation strategies by en- abling collaboration between the victim and upstream networks, and improve performance of blocklists for emergency response. Our work has several real-world impacts. SENSS has been experimentally deployed in three research and education networks for better attack detection. Our technique of identifying reused addresses is being deployed at IPInfo [51], which is the largest repository for IP address-related data. BLAG has been open sourced, and blocklists are actively collected daily. Finally, AT&T has two patents on SDProber and SPred. SDProber was also partially deployed at AT&T to detect high-delay events in their network, and is further open-sourced to make monitoring tools used by large networks available to the community. 12 Chapter2 SPred: MLPredictiononProgrammableSwitches 2.1 Introduction As networks enter into tight service-level agreements (SLAs) on performance and availability, their oper- ations are increasingly sensitive to transient network events such as high link utilization, high queuing time, high queue occupancy, microbursts, packet losses, and attacks. These events happen frequently, last only a few microseconds or milliseconds, and often cause signicant problems such as load imbalances, high average and tail ow completion times, and even impact reachability and security. It is imperative to quickly mitigate these transient events, to minimize their impact on network performance. For example, the sooner we capture microbursts or high queue occupancy events, the faster we can inform congestion control system to adjust the sending rate [8, 16, 242, 125, 78]. For malicious trac events (e.g., TCP low rate attacks [189]), any delay in detection can result in signicant impact on legitimate trac and thus revenue loss. Traditionally, switches reported network events of interest to a remote controller [87, 42, 124], which could decide on mitigation. This incurs several overheads: (1) propagation delay from the switch to con- troller, (2) CPU overhead to poll counters in the switch ASIC at high frequency [244, 143, 245] and (3) added bandwidth for messages to controller [28, 22]. 13 With the sophistication of programmable switches and NICs, we can now piggyback network event reports in packets (e.g., ECN marks, in-band network telemetry (INT [28])), and automatically analyze and react to network events in the data plane. Such a dataplane measure-control system eliminates the overheads that the traditional approach has. Measure-control has been used for a wide range of tasks such as load balancing [7, 75]), congestion control decisions [8, 16, 242, 125], troubleshooting [143, 37, 22], failure recovery [167], and attack detection/mitigation [243, 173, 189]. Signals (piggybacked reports) sent by a measure-control system inform downstream hosts and switches, which then take mitigation action. It can thus take up to one round-trip time (RTT) from detection to mit- igation. As network speeds continue to grow [199, 96] even one RTT of mitigation delay may be too long. For example, for networks with 100Gbps links and 10s round trip time (RTT), ows with less than 125KB can be sent within one RTT. In production workloads 50-100% of ows are shorter than 100 KB [139], thus majority of production ows would be impacted by an unwanted network event, even if it gets mitigated with one RTT. This situation is exacerbated by link speed increasing much faster than switch buer ca- pacity [244, 78] – this makes it harder to handle bursty trac, which is even more sensitive to transient delays and drops. We need to mitigate transient events with sub-RTT delays, and we can only get there with prediction. If we could predict network events, and predict them ahead of the time when they occur, this would reduce or even eliminate mitigation delay. There are a few existing approaches that attempt to anticipate network events based on trend estima- tion [22, 143, 7] and static thresholds. These approaches use short and simple observations of the past to predict the future, while network trac and services change quickly and frequently in datacenters. This mismatch can lead to inaccurate prediction. Even when they predict correctly, such simple approaches can predict only a little ahead, which does not fully eliminate the mitigation delay. In this chapter, we propose SPred, an event prediction system which leverages complex time series predictors such as Long Short-Term Memory (LSTM [90]) to predict network events in the data plane. 14 LSTM has dedicated structures (such as input, forget and output gates) that retain long-term dependencies over streaming data [73, 241]. This longer view of the past allows LSTM to predict events much earlier than trends estimators, and the structure of LSTM allows for more complex prediction models, and thus more accurate prediction. A key challenge in using LSTM directly in switches is that LSTM requires complex functions and states which are not supported on switches with limited functions, pipeline stages, and memory [160]. We address this challenge by decoupling model training and use. We train an LSTM model oine, using labeled data (step 1). Then we transform the trained model into a deterministic nite automaton (DFA) (step 2), which can be expressed as a set of P4 rules (step 3). We deploy the P4 rules on the switch for transient event prediction (step 4). As trac uctuates, our prediction accuracy may decrease. We collect information of mis-predictions and use them to automatically retrain and re-deploy the model. We demonstrate the eectiveness of our prediction framework in ns-3 simulations, by detecting mi- crobursts and improving congestion control for two existing ECN-based algorithms (DCTCP [8] and TCN [16]. Our framework can detect microbursts with high accuracy (F1-scores up to 0.98). Our P4-based model em- beddings are within 2–17% of the trained LSTM model, and they outperform existing simple models, such as EWMA (by 43%) and random forests (by 28%). By applying predictions for proactive ECN marking, we improve FCT slowdown for congestion control algorithms up to 19% on average for all ows, and up to 33% for short ows (< 100K bytes) and 31% for long ows (> 100K bytes). To demonstrate the feasibility of our approach we implemented a prototype of our framework in a P4 switch with no additional trac overhead to the switch. 2.2 Motivation Table 2.1 shows how knowledge of network events such as high link utilization, persistent queuing time per packet or bursty trac observed in the data plane can be used for making automated control decisions 15 Measure-controlsystems Event Congestioncontrol HPCC [125] High/low link/queue utilization TCN [16] High queuing time per packet ECNSharp [242] Persistent high queuing time per packet DCTCP [8], DRILL [75] High queue occupancy HULA [108], HULL [107], Conga [7], Hermes [236] High/low link utilization Networkanalytics BurstRadar [102], Marple [143], ConQuest [37] Bursty trac SQR [167] Link utilization/link state Attackdetection Poseidon [243], Avant-Guard [189], SENSS [173] High trac from specic sources Table 2.1: Measure-control systems that leverage network events in the data plane. to increase network utilization and reduce ow completion times. For example, when link utilization in- creases on a path, ingress switches can quickly shift trac to alternative paths [7, 75]). By leveraging ECN marks or in-band network telemetry, we can send signals of high queuing delay or high queue occupancy to the sender to inform their congestion control decisions [8, 16, 242, 125]. By capturing network events in the data plane, we can also enable timely responses for troubleshooting, failure recovery, and attack detection/mitigation [167, 22, 143, 173, 189, 243]. In this section, we dene the event prediction problem and discuss opportunities for using LSTM for such prediction. Theeventpredictionproblem. At timet, our goal is to predict if an event will occur in a future window of sizep ([t + 1;t +p]) based on measurements during a monitoring window of sizew ([tw + 1;t]). We usex i to dene the measured value of a featurex (e.g., queue occupancy) at timei. We usex i as the predicted value ofx at timei. Prediction also needs ground truth to establish accuracy, and for training and validation. Operators can dene an event detection functionEvent(x t+1 ;x t+p ), to automatically measure when an event occurs and establish ground truth. The event detection function supports dierent simple operations over the selected featurex t+1 :::x t+p such as max, min, average or exists, to determine if an event of interest has occurred. We say an event occurs if the function meets some operator-set criteria. For instance, congestion control algorithms use queuing delays to determine when to send explicit notications 16 to end hosts. We can dene a high queuing delay event asx j 2 max(x t+1 :::x t+p )j x j threshold, wherethreshold can be 10s [16, 242]. Another example of event detection would be using queuing delays to detect microbursts, max(x t+1 :::x t+p ) threshold, where threshold can be 50s [244, 102] ∗ . To predict an event, we rst run a prediction function Pred that predicts the values forx t+1 :::x t+p based on monitored values of x tw+1 :::x t . We then calculate the event function Event based on the predicted valuesx t+1 :::x t+p . A labeling functionlabel() maps sequence of predicted valuesx t+1 :::x t+p to true if an event function returns true. BenetsofLSTM-basedprediction. Today, switches use simple models such as EWMA (Exponentially Weighted Moving Average) of recent history of the selected feature (e.g., delay, queue occupancy) as an estimation of the immediate future [143, 22, 7]. This is barely better than detection after an event occurs – to harness prediction power we must predict further into the future. We must also use more complex models than simple trend estimation. To illustrate why model complexity matters, we compare EWMA to LSTM. Because EWMA predicts a single value, while LSTM predicts a sequence of values, we extend EWMA into EWMA+ [127] by linearly extrapolating from the values in the measurement windowx tw+1 :::x t to predict values in the prediction window x t+1 :::x t+p † [6]. We then compare the last estimated value against a xed threshold, like in [7, 22], to predict an event. We will compare this approach with LSTM, which predicts over the same time interval. We aim to show that EWMA’s simple model (single feature trend) is limited in capturing complex time-series features in network events, compared to classic time-series prediction solutions such as LSTM [73, 241]. We take an example of predicting high queuing delay. For simplicity, we set both the measurement windoww and the prediction windowp as 25s, and obtain the denition of an event from ECNSharp [242] ∗ We use a similar metric for improving congestion control and detecting microbursts in Section 2.5. † We have also tried other extrapolation functions; they yield similar results. 17 where high queueing delay is detected if there exists a queuing delay during the prediction window that exceeds a threshold of 10s. LSTM EWMA+ Figure 2.1: Prediction by LSTM and EWMA+. The queuing delay exceeds the threshold (10s) at times 25–100 s. Red circles (from 25–175 s) denote an event (ground truth), and check marks and crosses above the graph denote accurate and missed predictions. We simulate a Clos topology with 9 leaf switches, 4 spine nodes, and 16 hosts connected to each leaf switch in ns-3. We run a popular workload FB_Hadoop [179] at 25% load with DCTCP as a congestion control algorithm and with buer size of 340KB. We run the simulation twice with random seeds. We use the rst run to collect training data and generate the LSTM model. We then use the second run to evaluate EWMA+ and LSTM predictions. Fig. 2.1 shows that EWMA+ predicts only 3/6 events whereas LSTM predicts all 6 events correctly (2x improvement). We now elaborate on reasons for LSTM’s superior performance. Using time series information instead of trend. Since EWMA+ uses the simple trend of recent past values to predict future, it does not predict congestion events that were not preceded by sequences of high queuing delays, trending towards congestion (time periods 75–100s in Fig. 2.1). There are several latent features that can inuence queuing delays that are not captured by the trend estimation. For instance, queuing delay can be controlled by DCTCP through explicit feedback, but there could be a persistent 18 queue build up (time periods 100–150s) that is lower than the DCTCP ECN marking threshold. Once the senders begin to ramp up the sending rate, a combination of persistent queue build up and increase in trac could create further congestion. This interplay of dierent inuences on queuing delay may not be visible in trend estimation over instantaneous history (time periods from 125–150s). However, a time-series prediction model such as LSTM can learn which sequences of values lead to congestion/no-congestion events, which improves prediction accuracy. Threshold-less operation. EWMA+ can be tuned to an extent by choosing a dierent threshold for prediction than is used for detection. Consider the prediction task depicted in Fig. 2.2 where we show a simplied example (extracted from a much longer series of queue length in our experiments). Here the measured values are ingress queue occupancy (second row) and set the value of monitoring and prediction windowsw = p = 2. An event occurs when the egress queue occupancy exceeds 70 and the third row of the table presents the ground truth labels (label(x t )). We evaluate EWMA+ with thresholds 70, 71 and 72, respectively. For EWMA+(70) there are 1 false negative case and 6 false positive cases, for EWMA+(71) there are 1 false negative and 4 false positive cases, and for EWMA+(72) there are 2 false negative and 4 false positive cases. This illustrates the sensitivity of EWMA+ to the threshold setting. LSTM models, in comparison, can learn complex patterns of measured value’s sequences, and do not rely on a threshold. Time (t) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ingress (x t ) 70 40 60 80 20 80 90 40 20 80 60 90 60 70 70 egress 10 5 0 0 5 10 70 10 5 5 10 80 10 10 70 label() F F F F F F T F F F F T F F T EWMA + 70 70 40 61 80 19 80 90 40 20 81 60 90 59 71 FP FN Pred 1 (x t ) T T F F T F T T F F T F T F T 6 1 Pred 2 (x t ) F F F F T F T T F F T F T F T 4 1 Pred 3 (x t ) F F F F T F T T F F T F T F F 4 2 EWMA 70 EWMA 71 EWMA 72 Figure 2.2: An example showing how setting the right threshold for EWMA+ is hard. Shades of red indicate incorrect prediction and shades of green indicate correct prediction. 19 2.3 SPredOverview In this section we provide overview of SPred and its operation. SPred implements LSTM or similar complex time series prediction function to predict network events tens to hundreds of microseconds in the future in the data plane. This allows us to trigger mitigation early and improve ow completion times. Our system design addresses two challenges: (1) how to implement complex predictors in switches, and (2) how to address network and trac changes, by retraining the model. We give a brief overview of these challenges and our solution in this section and provide details in the next section. While the rest of this chapter focuses on LSTM, other complex models could be similarly implemented in switches [30, 228]. 2.3.1 ComplexPredictorsinSwitches While complex prediction has signicant benets, it can be challenging to implement it in the data plane on programmable switches, because switches currently support only simple, integer arithmetic. Switches also lack resources such as CPU and memory to support large model training. To address this challenge (see Figure 2.3) perform model training oine, and we then simplify the model into discrete nite automaton (DFA) and deploy it on the switch. ModelTrainingatexternalmachinesbasedonlabeledtrac. We train the LSTM model (or other complex predictor) based on the labeled network trac in the control plane. We periodically collect the network trac with a sequence of measured values per each monitored entity—a value per time unit, as illustrated in Section 2.2, and network event labels based on detection (e.g., if a given value is above a certain threshold). Operators could capture events from their network using existing tools [28, 149, 22] and create labeled data for their own network. Deploytrainedmodelatprogrammableswitches. We transform the trained model into a determin- istic nite automaton (DFA) (see Section 2.4.3), which can be implemented as a set of match-action rules 20 Rules Verifier LSTM model generator Periodic network traces with events Retraining and refining models Control plane Data plane Network traces Transform Approximate LSTM model Write model Incoming Traffic Outgoing traffic Figure 2.3: SPred uses network traces to generate approximate LSTM models that are written as rules in the data plane. Periodically, the model is retrained from misclassications and periodic network traces from the data plane. installed on the programmable switch (see Section 2.4.4). The network operator sets the action that is performed when an event is predicted. 2.3.2 AddressNetworkandTracChanges To address network and trac changes, we must monitor the prediction accuracy and retrain the model when accuracy declines. While the prediction module operates, we use the verier module that compares predictions with the actual events, when they occur, and capture prediction errors (false positive and false negative predictions). The verier noties the control plane about prediction errors with the feature values and labels used for the prediction and triggers model retraining in the control plane. The control plane then export the model back to the switch for deployment (see Section 2.4.5). 21 2.4 LSTMpredictionsatswitches We now provide details about the training of the LSTM model and translation into match-action rules. 2.4.1 BackgroundonLSTM Long Short-Term Memory (LSTM) [90] is a Recurrent Neural Network (RNN) and the standard tool for prediction over streaming data. Recurrent Neural Networks provide a mechanism for prediction over a stream (x 1 ;x 2 ;:::;x t ;:::) of features. As features arrive, the network computes states, while taking into account a sequence of preceding observations. RNN consists of cells that receive streaming features as input and produce a stream of computed values. For example, RNN can receive a stream of measurements from a switch ingress and compute a stream of predictions regarding the future occupancy of the egress. Using self-loops, the cells retain some of the information from previously seen features, so that a prediction will be based on a history of the feature’s values – this add complexity and improves performance of the model. LSTM is an RNN that has been designed to retain long-term dependencies. To that end, an LSTM needs to learn which information to carry on and which information to forget when passing information in the self-loops. To implement this, LSTM cells have multiple gates and activation components to transfer information between cells. Aforgetgate is used to forget irrelevant information, for example, in the detec- tion of egress congestion, trac via the ingress is relevant for a short time and then should be forgotten if congestion does not persist. An input gate maintains relevant input. For example, in the detection of microbursts, this gate may not give the same weight to input regarding ingress congestion and input re- garding the number of network ows; or it may dismiss some input altogether. Anoutputgate determines what components to forward in the self-loop, for the next iteration. For instance, it may transform the output in such a way that it will be clearer whether a microburst is predicted, by mapping values closer to 0 or 1. This design helps to retain long-term dependencies. The gates are implemented using a sigmoid 22 function that typically maps inputs to be close to 0 or close to 1, so multiplying by it either retains the current value or sets it to zero. The three gates on an LSTM cell are arranged in a pipeline as shown in Figure 2.4. Each gate has associated weights (W i , W f , W o , W C ) and biases (b i , b f , b o , b C ). During the training phase, the LSTM classier calibrates the weights and biases. Then, during the application stage (prediction), the LSTM cell receives a feature vector,x t , the learned weights, and biases. It produces the outputh t (hidden state at time t) and transfers the cell stateC t to the next iteration, using the equations in Fig. 2.5. In these equations, is the logistic sigmoid function and is the element-wise product of vectors. The output can either be passed as an input to the following LSTM cell or can be the nal output of the network. LSTM can have more than one cell. The cells are connected and forward the cell state and hidden state to one another per each iteration. They also forward the values in a self-loop at the end of each iteration, as illustrated above for a single cell. The result of each step is a vector of hidden states, which are combined to provide the prediction. The training of the model determines the weights and biases for all the cells. x t   tanh     tanh h t  X t-1   tanh     tanh h t-1  f t h t-2 h t h t-1 i t o t C t-1 C t C t-2 Self loop: the output of the cell is sent back to the cell Input features Output hidden states The LSTM Cell at time t-1 The LSTM Cell at time t Figure 2.4: An LSTM cell and its gates: input (i t ), forget (f t ) and output (o t ). The cell is presented for times t 1 andt to illustrate the self loop. At each timet, input featuresx t are provided to the LSTM and an hidden stateh t is returned. 23 i t =(W i [x t ;h t1 ] +b i ) input gate f t =(W f [x t ;h t1 ] +b f ) forget gate o t =(W o [x t ;h t1 ] +b o ) output gate C t =f t C t1 +i t tanh(W C [x t ;h t1 ] +b C ) h t =o t tanh(C t ) Figure 2.5: The equations of the LSTM cell are applied to a feature vectorx t and return a hidden stateh t . 2.4.2 ProblemsofrunningLSTMatswitches It is impractical to implement LSTM in P4 at switches due to (1) limited functions, (2) limited pipeline stages, and (3) memory constraints. Limited functions. LSTM functions (hyperbolic tangent (tanh) and sigmoid) require oating point op- erations, which are not supported by programmable switches. Switches only support simple arithmetic operations including addition, subtraction, bit shifts, and read/write operations on registers [160]. Limited pipeline stages. The tanh and sigmoid functions could be approximated using match-action tables [187]. If tanh and sigmoid were supported (or somehow approximated), there would still be a need for a pipeline of at least seven operations (3 operations for computing the gatesi t ,f t ando t (the operations ; +;), 2 additional operations for computingC t ( and +) and 2 operations for computingh t (tanh and ). Since tanh and sigmoid are not supported, approximating them would further lengthen the pipeline, and lead to high packet processing delays. Limitedmemory. We could try to implement the LSTM directly using a match-action table, by match- ing each possible input to the learned value of the LSTM output, but this is impractical. Consider a sliding window over the streaming features where thel most recent features are kept for each one of thek pro- cessed features. The prediction function Pred() dened in Section 2.2, receivesk sequences ofl values. A comprehensive match-action table would determine for each combination whether an event should be predicted. However, even if there are only two possible values for each parameter, the match-action table 24 would have 2 kl entries. Maintaining and using such a table is not possible even for relatively small values ofk andl, and there is not enough memory in a P4 switch to store such a table. Due to the above limitations, the LSTM model is trained in an external machine with large memory and CPU resources. This yields a prediction model. 2.4.3 TransformingLSTMintoDFA c 0 c 1 h 0 h 1 h 2 1 3 4 5 2 6 7 8 9 10 11 12 13 14 q 00 q 11 q 01 q 12 q 11 1 2 3 4 8 6 7 11 Network abstraction State machine q 12 q 01 q 00 Figure 2.6: From network abstraction to a DFA. To execute prediction on a switch, the LSTM is transformed into a Deterministic Finite Automaton (DFA) that represents a state machine. A DFA is a 5-tupleA = (Q; ;;q 0 ;F ) that consists of a setQ of states, an alphabet , a transition function : Q ! Q, an initial state q 0 2 Q and a subset F Q of accepting states. Alphabet is a discretization of the measured values (features), e.g., the set f0; 5; 10;:::; 100g when relating to the percentage of ingress queue occupancy. The accepting statesF are states at which an event is predicted. 25 BasicTransformation. To illustrate the approach, we start by presenting a direct transformation of an LSTM into a DFA and then explain how to improve it and achieve better predictions. This transformation is a simplied version of the network-abstraction model of Omlin and Giles [157]. In a basic transformation, we rst create states based on the cell stateC t and hidden stateh t of the LSTM. The values ofC t andh t are in the range (1; 1) due to the use of of tanh. Consider a partition of the range (1; 1] intom partsf(1 +i 2 m ;1 + (i + 1) 2 m ]ji = 0; 1;:::;m 1g. The discrete state ofC t is a function that mapsC t to the appropriate range in the partition, state(C t ) =i i 1 +i 2 m <C t 1 + (i + 1) 2 m Similarly, state(h t ) =j i 1 +j 2 n <h t 1 + (j + 1) 2 n for a partition of (1; 1] inton ranges. The states of the automaton areQ =fq ij j 0 i m and 0 jng, i.e., all themn possible indexes of the partitions of the ranges ofC t andh t . The accepting states F are statesq ij for which there is an event prediction forh t in thej-th range. The transition function is computed from the LSTM as follows. For eachx2 , and each stateq ij , we consider the values ofC t andh t to be the midpoint of the rangesi andj. The valuesC t+1 andh t+1 are computed by applying the LSTM equations of Fig. 2.5. The new state is according to the indexes of the new values. That is,(q ij ;a) =q i 0 j 0 wherei 0 = state(C t+1 ) andj 0 = state(h t+1 ) for [C t+1 ;h t+1 ] = LE(C t =1 +i 1 m ;h t =1 +i 1 n ;x t =a); such that LE is the application of the LSTM equations to computeC t+1 andh t+1 . 26 The resulting automaton has at mostmn states. Hence, the size of the automaton can be controlled by setting the number of ranges in the partitions. The selection ofm andn is according to the space allocated for P4 rules in the switch (see Section 2.5.4). The following example illustrates the process. q 00 q 11 q 01 x ≥ 70 x ≥ 70 30 < x < 70 x ≤ 30 30 < x < 70 x ≤ 30 30 < x < 70 x ≥ 70 x ≤ 30 All cases Match Action state = q 00 && x ≥ 70 state = q 11 state = q 11 && x ≥ 70 state = q 12 state = q 11 && x ≤ 30 state = q 00 state = q 11 && 30 < x < 70 state = q 01 state = q 01 && 30 < x < 70 state = q 00 state = q 01 && x ≤ 30 state = q 00 state = q 01 && x ≥ 70 state = q 12 q 12 Match Action state = q 12 User defined action, state = q 00  = {x ≤ 30, 30 < x < 70 , x ≥ 70} Q = {q 00 , q 01 , q 11 , q 12 } q 0 = {q 00 } F = {q 12 }  = {(q 00 , x ≥ 70, q 11 ), (q 11 , x ≥ 70, q 12 ), (q 11 , 30 < x < 70 , q 01 ), …} Figure 2.7: An automaton and the match-action tables produced in the transformation. Example 2.4.1. Suppose the ingress queue occupancy is measured and yields the following percentage values. 70; 40; 60; 80; 20; 80;90; 40; 20; 80; 60;90; 60; 70;70 as illustrated in Section 2.2. Suppose that there are 2 cell statesc 0 andc 1 , and 3 hidden statesh 0 ,h 1 ,h 2 , resulting from the range partitions. Consider the following sequenceS of 3-tuples (x t ;c t ;h t ), wherex t is the measured feature at timet,c t andh t are the state and hidden state at timet, and the values in bold font are the times when an event occurred: (70;c 1 ;h 1 ), (40;c 0 ;h 1 ), (60;c 0 ;h 0 ), (80;c 1 ;h 1 ), (20;c 0 ;h 0 ), (80;c 1 ;h 1 ), (90;c 1 ;h 2 ), (40;c 0 ;h 0 ), (20;c 0 ; h 0 ), (80;c 1 ;h 1 ), (60;c 0 ;h 1 ), (90;c 1 ;h 2 ), (60;c 0 ;h 0 ), (70;c 1 ;h 1 ), (70;c 1 ;h 2 ) 27 For creating the automaton let us assume that the alphabet contains three elements: x 30, 30< x < 70 andx 70 (this is just a simplication for the example.) There are two states forC t and three states forh t , so the automaton has 6 potential statesQ =fq 00 ;q 01 ;q 02 ;q 10 ;q 11 ;q 12 g. Note that when an event occurred the hidden state wash 2 . In this example, the learned process is such that an event is predicted if there is a sequence of at least two values greater than or equal to 70 (x 70) separated by at most one value greater than 30 (30<x< 70). Based on the sequenceS, we create an array where one dimension consists of thec-values and the other of theh-values. Every two consecutive pairs form a transition, e.g., the rst two pairs correspond to the transition from (1; 1) to (0; 1). The transitions are depicted in Fig. 2.6. Each visited cell is considered as a state, and each step from one cell to another is becoming a transition in the DFA. For example, the transition from (1; 1) to (0; 1) is labeled as Step 1 in the network abstraction and it yields the transition fromq 11 toq 01 in the DFA. The transition from (0; 1) to (0; 0) labeled as Step 2 yields the transition from q 01 toq 00 , and so on. The resulting automaton in shown in Fig. 2.7. The states associated withh 2 are accepting states. The transition function is as depicted by the edges of the automaton in Fig. 2.7. OptimizedTransformation. If the constructed DFA is inconsistent with the LSTM model, we need to repeat the process shown in Fig. 2.6 with a ner partition into cells. To discover the regions, the input could be varied, to explore the search space of partitions. However, such a process does not scale. To cope with that, we adopted a renement process that was developed by Weiss et al. [221]. We only briey describe the method of Weiss et al., and refer to their paper [221]. Given an LSTM, or any RNNR, rst, a DFAA L is constructed using the L*-Algorithm of Angluin [9]. A second DFAA P is constructed for partitionP using network abstraction [157], like the process depicted in Fig. 2.6. Iteratively, the process looks for a sequence (word over the alphabet ) that is accepted by one of 28 the two automatons but not by the other. If such a sequencew is discovered, it is tested over the LSTM, to know which of the two DFAs misclassies it. IfA L needs a renement, new states are added to correctly classifyw. IfA P classiesw incorrectly, the partitionP is rened by partitioning the cell to whichw leads. When the DFAs agree on all tested sequences,A L is returned. For details, see [221]. Note that the renement process can be stopped when the size of the automaton reaches a predened limit or after a given time limit. 2.4.4 FromDFAtoP4Rules To execute the prediction module in a switch, the DFAA = (Q; ;;q 0 ;F ) is transformed into match- action rules, e.g., in P4. The transformation creates two match-action tables—one for implementing the transition function and one for the nite statesF . Algorithm 1 presents the transformation. TableT implements the transitions between the states. The rule (state =q ij &&x;state =q i 0 j 0) in step 3 is applied when the state isq ij and the input isx. The action is a change of the state to beq i 0 j 0. Hence, it implements a transition between the states for the inputx2 . TableT F implements the action that is executed when an event is predicted. The action is set by the network operator. The rule (state =q ij ;state =q 0 && action) in step 7 is applied when the stateq ij is in F . In this case, the dened action is executed and the state machine is transitioned into the initial state, to restart the prediction process. Example2.4.2. Consider the automaton created in Example 2.4.1. Fig. 2.7 illustrates the result automaton and the transformation into match-action tables (P4 rules). 2.4.5 RetrainingandReningtheModel Misclassications of network events may occur due to an imperfect LSTM model or due to imperfect trans- formation of the LSTM model into DFA. Detected misclassications can be used for improving the model. 29 Algorithm1 Transforming DFA into P4 Rules Input: A = (Q; ;;q 0 ;F ) Output: A tableT and a tableT F of P4 rules 1: letT andT F be match-action tables 2: for (q ij ;q i 0 j 0;x)2 do 3: r (state =q ij &&x;state =q i 0 j 0) 4: insertr intoT 5: endfor 6: forq ij 2F do 7: r (state =q ij ;state =q 0 && action) 8: insertr intoT F 9: endfor 10: returnT and aT F This is done by recording misclassications and the preceding feature values (e.g., queue occupancy), to be used as counterexamples for rening the DFA or retraining the model. This is done by constantly record- ing data in designated registers. When a prediction error occurs, the content of the registers is stored. Otherwise, the space is reused, in a round-robin fashion. Measurementregisters record thep measured values per prediction, wherep is the lookahead value. Predictionregisters keep the model predictions. The earliest measurement register stores the location of the oldest measurement in the measurement registers. Theearliest-predictionregister holds the locations of the oldest prediction in the prediction registers. M1 T T T T 0 M1 M2 T T 0 M1 M2 M3 T T F 0 M1 M2 M3 M4 T T F T 0 M1 M2 M3 M4 M5 M2 M3 M4 F T F T 1 M5 0 0 0 0 1 Figure 2.8: Capturing prediction errors. At timet, the prediction label in registert modp is the prediction madep time units ago. This label is compared with the current event status, and if there is no match, this is an incorrect prediction. The entire 30 content of the registers is recorded as a mismatch, where the earliest-measurement and earliest-prediction registers store a valuei such that the arrays are read as a sequencei;:::; (i +p) modp. After the comparison, the measured value is added to the measurement registert modp and the new prediction label is added to the prediction registert modp. Fig. 2.8 illustrates the process forp = 4, where the last addition ofM 5 indicates an incorrect prediction that can be used for renement. 2.5 Evaluation In this section, we present our experimental evaluation, which consists of the following. (1) We show that SPred has higher accuracy in predicting network events than simpler methods like EWMA+ and Random Forest. (2) We examine the eect of the transformation from LSTM to P4 rules on the prediction accuracy. (3) We evaluate the eect of prediction on improving congestion control. (4) We explore the eect of retraining. (5) We demonstrate that our approach is practical—the actual prediction using P4 rules works at line rate. 2.5.1 EvaluationSetup 2.5.1.1 Settings We used the following test settings. Networktopology. We used thens-3 simulator to generate simulated network events, including queuing delays, based on given network topology and trac ows. The simulator was used for capturing events at microsecond granularity, to test our models. We used a Clos topology consisting of 9 leaf switches and 4 spine switches, where 16 hosts were connected to every leaf switch. The link connecting the host to the leaf switch had a capacity of 10 Gbps, and the link connecting the leaf to spine switches had a capacity 31 of 40 Gbps. The maximum capacity of every queue was 350 KB. All the links had a propagation delay of 10s. We executed DCTCP [8] as the default congestion control algorithm. Tracloads. In the simulations, we usedWebsearch [246] andFB_Hadoop [179]—two popular workloads for data center networks. We adjusted the average trac load to be at 25% and 60%, respectively. Like in [125], for trac diversity we generated incast trac, in some tests, by randomly selecting 60 senders and one receiver and producing ows of 500 KB. 2.5.1.2 Applications We predicted microburst events and congestion events, and examined the eect of prediction on mitigating congestion. PredictingMicrobursts. We tested microburst prediction and compared our algorithm to three alterna- tives. We refer to the transformation described in Section 2.4.4 asP4-RepresentationofLSTM and denote it asSPred in our graphs. We compareSPred to three alternative approaches. (1) O-switch LSTM. Using the LSTM model outside the switch provides insights regarding the overall ability of LSTM to cope with prediction tasks. In Section 2.5.4, we describe how to congure the network for best predictions. (2) EWMA+. When using EWMA+ (see Section 2.2), an event is predicted when the EWMA+ of the mea- sured feature exceeds a predened threshold. We use the same threshold as is used for event detection. (3) Random Forest. Random Forest is a common classier [88], and it is simpler than LSTM and can be easily deployed in a switch. It consists of a collection of decision trees, where each tree performs a bi- nary classication (e.g., event/no-event), and the classications are combined, to prevent overtting. The implementation of Random Forests in P4 was studied in pForests [30], where the two main parameters were the number of decision trees and the maximum height of each tree. We used the same parameters as 32 p(s) Type Websearch (50 th /90 th percentile) FB_Hadoop (50 th /90 th percentile) Load:25% Load:60% Load:25% Load:60% 25 Train 239K/270K 742K/807K 260K/250K 798K/749K Test 40K/41K 111K/126K 43K/38K 132K/106K 50 Train 155K/173K 436K/493K 172K/157K 496K/434K Test 22K/24K 65K/76K 25K/21K 80K/60K 100 Train 91K/105K 256K/307K 105K/91K 309K/254K Test 12K/15K 38K/47K 15K/12K 50K/36K Table 2.2: Training and testing dataset for predicting microbursts at dierent trac congurations. Each cell consists of the number of samples for predicting microbursts at 50 th percentile and 90 th percentile. pForests [30], that is, 32 decision trees and a maximum tree height of 4. Note that the height of the tree is restricted by the switch pipeline. CongestionControl. We show how to use prediction for improving two ECN-based congestion control algorithms, namely, DCTCP [8] and TCN [16]. These algorithms use dierent techniques for detecting congestion, however, they all mark the explicit congestion notication (ECN) bits of the packet (typically bits 6 and 7) to notify the end-host that it should adjust its packet sending rate. SPred is used for predicting congestion, to mark the ECN bits early. SPred marks the ECN bits as follows. For every outgoing packet, it determines the amount of time the packet spends in the queue, namely queuing time. It records queuing times for a duration ofw (see Section 2.2) as (t 1 ;QT 1 );:::; (t w ;QT w ). The model predicts whether the queuing time of future packets will exceed a threshold in the prediction periodp. If an event is predicted, the ECN bits are marked. To ensure that no unnecessary feedback are sent to the end-host, our prediction layer does not mark the ECN bit if it has already been marked inp. We dene an event when queue occupancy threshold exceeds 10s in the prediction periodp. Also note that this predictor marks only ECN bits that the original ECN marking algorithm has not marked. We tested the eect of prediction on congestion control (Section 2.5.4). 33 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (a) FB_Hadoop Trac at 25% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (b) FB_Hadoop Trac at 60% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (c) Websearch Trac at 25% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (d) Websearch Trac at 60% load. Figure 2.9: F1 score when predicting 50 th percentile microbursts,p =25s, 50s and 100s. 34 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (a) FB_Hadoop Trac at 25% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (b) FB_Hadoop Trac at 60% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (c) Websearch Trac at 25% load. 25 50 100 Predictions into future (μs) 0.5 0.6 0.7 0.8 0.9 1.0 F1-score EWMA+ RF LSTM SPred (d) Websearch Trac at 60% load. Figure 2.10: F1 score when predicting 90 th percentile microbursts,p =25s, 50s and 100s. 35 2.5.1.3 Metrics We used F1-score to evaluate microburst prediction accuracy, and measured ow completion time (FCT) slowdown—the actual FCT normalized by the ideal FCT—to assess improvement for congestion control (see [125]). 2.5.1.4 TrainingandTesting We used ns-3 simulations to generate the training and testing datasets. We varied the network trac by generating trac patterns with dierent loads. For predicting microbursts, we used the rst 90% of the time for training and the remaining 10% for testing. After training the LSTM model and transforming it into P4 rules, we replayed the remaining 10% of trac, to estimate the prediction accuracy. Table 2.2 shows the number of samples used for training/testing for predicting microbursts. For congestion-control tests, we executed the same trac conguration three times. The rst run used the original congestion control algorithm to generate training data. The second run was similar to the rst, but used a dierent seed to generate a dierent trac pattern. It was used for determining the FCT slowdown, for the original algorithm. The third run was with the same seed as the second run, but it had a prediction layer combined with the original ECN-marking algorithm. In the third run, we measured the FCT slowdown by using our prediction layer. During testing, a high-F1 score denotes higher accuracy of the models. If our prediction model makes incorrect predictions, it would adversely aect the FCT slowdown. Therefore, by comparing the FCT slowdown before and after using the prediction layer, we could see the benets of prediction. For model training we used Keras with Tensorow [2]. Our DFA extraction module is written in Python (2.5 K lines of code). For the trac simulation, we used the ns-3 simulator, with modications done to 1.5 K lines of C++ code. 36 2.5.1.5 Parameters We used the DCTCP marking threshold ‡ ofK min =K max = 84KB as in [8]. We used the same param- eters for TCN as described in [16] and set the instantaneous ECN marking threshold as 70s, that is, the ECN bits are marked whenever the queuing delay exceeds 70s. For the prediction layer introduced, we setp andw to be 40s. This value is equal to half the RTT—lower intervals of RTT tend to react faster to bursty trac [242]. The eect ofp andw on the results are discussed in Section 2.5.4. 2.5.1.6 TestonSwitch. We tested a prototype of SPred on a Barefoot/Intel Tono programmable switch using registers in P4. We deployed a model that detects microburst for the FB_Hadoop trac with 25% load, consisting of eight match-action rules for implementing three DFA states. The model was executed at line rate, and the test showed that there was no delay to the trac—the deployment of the model aected neither the throughput nor the latency of the switch. SPred could further be implemented in hardware on architectures like DRMT [38] and Broadcom Trident 4 [207], which support [154]. 2.5.2 MicroburstPrediction To test the eectiveness of SPred in microburst prediction, we compared our prediction model with EWMA+ and Random Forest. We consider as microburst the cases where the queuing time for a packet exceeds a threshold. The threshold was selected as the 50 th and 90 th percentile of the delay in the Face- book datacenter [244], which resulted in a threshold of 50s and 80s. Although these values refer to link utilization, we can assume that these are upper bounds on the delay in the queue, per each percentile. Similar denitions were used by BurstRadar [102]. We predict if a microburst would occur 25, 50, and ‡ DCTCP’s suggested threshold is 60 packets per 10 Gbps. 37 25 50 100 Predictions into future (μs) 0.0 0.2 0.4 0.6 0.8 1.0 F1-score EWMA+ RF LSTM SPred (a) Predicting microburst 25 50 100 Predictions into future (μs) 0.0 0.2 0.4 0.6 0.8 1.0 F1-score EWMA+ RF LSTM SPred (b) Predicting no-microburst Figure 2.11: The prediction accuracy for a microburst event and a non-microburst event. 100s in the future (parameterp). We also set the measurement windoww to be equal top. Figures 2.9 and 2.10 present the F1-score of the prediction results. LSTMhasthehighestF1score. For microburst prediction, LSTM on average has 14.2–43.4% higher F1 score than EWMA+, for dierent values ofp. LSTM also outperforms Random Forest, with 9.8–28.3% higher F1 score. The P4 models have comparable performance to that of LSTM and higher accuracy than EWMA+ and Random Forest. The accuracy of SPred is slightly lower than that of the LSTM model due to the transformation into DFA. On average across dierentp values, SPred has 2.6–17.3% lower F1 score than LSTM models. (However, LSTM cannot be executed on the switch.) SPred accurately predicts microbursts. The accuracy of SPred for 50 th percentile microbursts is higher than that of EWMA+ or Random Forest—SPred has 8.5–36.4% and 3.5–23.8% higher F1-score than EWMA+ and Random Forest, respectively. For the 90 th percentile microbursts, prediction is easier because the queuing delay lasts longer, so all the methods are accurate. Note, P4 has 7.4–25.5% and 3–16% higher F1-score than EWMA+ and Random Forest. 38 30 60 90 120 150 180 0.86 0.87 0.88 0.89 0.9 0.91 original updated Time (ms) F1-score Figure 2.12: Re-training based on misclassications. LSTM and SPred have high accuracy. Fig. 2.11 analyzes the 90 th percentile prediction of mi- crobursts, for FB_Hadoop trac at 25% load. It presents the accuracy for two cases—predicting a mi- croburst and predicting there will not be a microburst. All the methods are mostly accurate in predicting no-microburst cases. However, EWMA+ fails in predicting cases of a microburst (F1-score of 0.34–0.41), because potential signals are hidden by averaging the input. For microburst prediction, the F1-score of Random Forest is only 0.45–0.51, whileSPred has an F1-score of 0.76–0.84 and LSTM achieves an F1-score of 0.91–0.93. Model retraining. Fig. 2.12 shows how retraining the model increases the accuracy when trac patterns change. In this test, we applied to the FB_Hadoop trac at 25% load a model that is trained for this trac pattern. Then, we increased the load to 60%, and every 30ms, we retrained the model based on the detected misclassications. Without retraining, the F1-score drops as the trac pattern changes, while continuous retraining keeps the F1-score high. The F1-score of the original model is in the range 0.85–0.89, while for the retrained model, the range is 0.88-0.91. 39 520 553 612 679 745 811 877 1K 1K 2K 34K 48K 81K 248K 11M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (a) FB_Hadoop trac, no incast. 5K 9K 21K 28K 38K 52K 68K 93K 226K 777K 1M 2M 4M 8M 31M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (b) Websearch trac, no incast. 520 553 612 679 745 811 877 1K 1K 2K 34K 48K 81K 249K 12M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (c) FB_Hadoop trac with incast. 5K 9K 21K 28K 38K 52K 68K 93K 230K 773K 1M 2M 4M 8M 31M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (d) Websearch trac with incast. Figure 2.13: Slowdown of DCTCP algorithm over dierent trac loads. 40 5K 9K 21K 28K 38K 52K 68K 92K 225K 777K 1M 2M 4M 8M 31M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (a) Websearch trac, no incast. 5K 9K 21K 28K 38K 52K 68K 93K 230K 773K 1M 2M 4M 8M 31M 3 5 10 20 50 100 Actual Load=60% Pred Load=60% Actual Load=25% Pred Load=25% Flow size (bytes) FCT Slowdown (log) (b) Websearch trac with incast. Figure 2.14: Slowdown of TCN algorithm over Websearch trac with varying workloads. 2.5.3 CongestionControl We tested the eect of prediction and early marking of the ECN bits on congestion control. Fig. 2.13 and Fig. 2.14 show the 95 th percentile FCT slowdown for the two dierent types of congestion control algorithms. ImprovementofDCTCP. Since prediction helps mark ECN earlier, it can reduce the queue build up. This is validated in our experiment–Fig. 2.13 shows thatSPred reduces short ow FCT, which is dominated by queuing delay. Specically, at 25% load,SPred reduces the 95 th percentile FCT by at least 32.9% for ows under 100KB. At 60% load, the improvement is smaller, because at a higher load, the original system already places many ECN marks. We also see thatSPred does not degrade long ows (>10MB). We believe this is becauseSPred-marked ECN helps reduce queuing and optimize throughput. Specically, the dierences brought bySPred in ECN marking arise when the original scheme does not mark butSPred predicts a longer queue. WithoutSPred, DCTCP continues growing the sending window size, and receives multiple ECN marks in the next RTT, which triggers large sending window reduction. With SPred, one ECN is marked in the current round, 41 0 500 1000 1500 2000 2500 0.82 0.84 0.86 0.88 0.9 25 µs 50 µs 100 µs Rules F1-score (a) The F1 score as a function of the size of the P4 program. 0 20 40 60 80 0 500 1000 1500 2000 2500 25 µs 50 µs 100 µs States Rules (b) The size of the P4 model as a function of the size of the DFA. 10 20 30 40 50 Number of nodes 0.95 0.96 0.97 0.98 0.99 1.00 F1-score events non-events all (c) The accuracy as a function of the LSTM size. 20 40 60 80 5 5.25 5.5 5.75 6 w/p Improvement of Avg FCT (%) (d) The eect of the monitoring windoww and look aheadp on the congestion control. Figure 2.15: Sensitivity analysis 42 which triggers slight window reduction. Therefore, although SPred causes sending window reduction earlier, at a lower queue, this ultimately optimizes the bandwidth use. Improvement of TCN. Fig. 2.14 presents the FCT slowdown for Websearch trac. Tests with FB_- Hadoop trac showed similar results and are omitted. TCN uses the sojourn time (or queuing delay) to mark the ECN bits instead of using instantaneous queue occupancy. Since SPred is also congured to predict based on sojourn time, it provides similar improvements like observed in DCTCP. Specically, for ows under 100KB,SPred reduces the 95 th FCT by at least 30.1%, without degrading long ows. 2.5.4 SensitivityAnalysis The results of machine learning models often depend on their conguration. In this section, we examine the eect of the conguration on the prediction accuracy and the selection of the best model. We show this only for a few congurations for microburst detection and congestion control. Other congurations yield similar results. 2.5.4.1 ConstraintsonRules Fig. 2.15a shows the accuracy (F1 score) of 90 th percentile microburst prediction as a function of the number of rules for FB_Hadoop trac at 25% load. The best DFA model required between 30 to 46 rules when transformed into P4 rules. The accuracy of the model increases when the number of rules grow until the SPred reaches the accuracy of the LSTM. More rules cannot improve the model further. Fig. 2.15b presents the size of the produced DFA and the number of P4 rules. There is a linear ratio between the number of DFA states and the number of P4 rules. So, controlling the size of the generated DFA can be used for controlling the size of the number of rules. For instance, it takes about 1193–1498 rules to represent a DFA with 50 states. Thus, the network operator can tune the accuracy of the model depending on the hardware limitations in the switch. 43 2.5.4.2 ModelSelection Increasing the number of LSTM cells can increase the accuracy of the prediction. We show this for conges- tion control experiment with DCTCP, at 25% load without incast over Websearch trac, where the number of LSTM cells was increased, starting with 10 cells and up to 50 cells. Fig. 2.15c shows the eect of the increase on the F1 score. In our tests, the accuracy of these models does not change much when increasing the number of cells from 10 to 50. Yet, LSTM with 50 cells has the highest accuracy (with F1-score of 0.97) for both microburst and non-microburst cases, so we used LSTM with 50 cells in our experiments. 2.5.4.3 Choiceofpandw forcongestioncontrol As described in Section 2.5, we predict if the queuing time exceeds a threshold in the nextps based on events observed in measurement periodw. To choose the appropriate value forp;w, we run our simulation with the DCTCP algorithm at 25% load without incast, for Websearch trac and varying p;w values. Settingp = w = 40 (which is half the RTT) yields the highest reduction in FCT (up to 6%). But also for dierentp values there is an improvements higher than 5.5%. 2.6 RelatedWork Due to their eect on network performance, network events were studied extensively in the literature, in papers on monitoring, event detection and classication of network trac. Event detection and monitoring. Detecting transient events is challenging, and it was studied in dierent contexts. Joshi et al. [102], Chen et al [37] and Narayana et al. [143] investigated detection of microbursts. PINT [22] introduced a framework for eectively utilizing INT [28] in network monitoring and measurement tasks. Active probing was studied in [169]. These, and many other papers in the area, study monitoring and detection methods, not prediction. 44 ML models outside the data plane. Dierent systems were developed for congestion control and events detection in end hosts. REMY [225], Aurora [100], and PCC Vivace [65] were devised for improving congestion control using ML, but their models cannot be deployed in the data plane. ML classication in the data plane. Dierent methods for deploying ML classiers in the data plane were developed. pForest [30] utilizes Random Forest for classication of the network trac. Jepsen et al. [101] introduced a binary decision tree implemented in P4. IIsy [228] showed how SVN, decision tree and Naive Bayes classiers can be approximated and deployed on a switch. Groléat et al. [81] implemented an SVM classier in FPGA. All these classiers, however, are not as eective as LSTM in prediction over streaming data because they are designed for classifying static data. ML-designated hardware. Taurus [200] proposed a new hardware architecture that can support prediction in the data plane. SPred in comparison can be deployed on existing programmable switches, with no need for new hardware. 2.7 Conclusion In this chapter, we introduce a frameworkSPred for prediction of transient network events in the dataplane using LSTM. SPred transforms LSTM into a DFA and then into a set of P4 rules that approximate it. The rules can be deployed on programmable switches and operate without delaying the network trac. During prediction, data about predicted events are collected and labeled to verify the accuracy of the model and retrain it, if necessary. SPred can potentially improve control-measure systems that require fast reactions to network events. We show thatSPred detects microbursts with a high accuracy (F1-score upto 0.98) and improves congestion control algorithms such as DCTCP and TCN (upto 19% decrease in FCT slowdown). Thisworkdoesnotraiseanyethicalissues. 45 Chapter3 SDProber: ASoftwareDenedProberforSDN 3.1 Introduction Measurements are crucial for managing communication networks, e.g., for troubleshooting and to main- tain service-level agreements (SLAs). However, measurements incur costs in the form of bandwidth, CPU and memory utilization. In this chapter we show how the central control in SDN can be used for reducing the costs that are involved in proactive delay measurements, and how SDN can facilitate adaptability of the measurements to varying conditions. Persistent delays in wide area networks are often perilous and can adversely aect the eectiveness of online commerce in the stock exchange, streaming video and audio, online games, and other online applications, where obstructed data transfer can cause a signicant loss of money or disrupt the quality of service. Thus, it is essential to proactively detect long delays as early as possible, and cope with the hindrance, as soon as it starts. Delays are detected by periodically inspecting the links of the network. However, there is a tradeo between the detection time and the cost. Increasing the inspection rate of a link can reduce the detection time of a delay, while inspecting a link too often could hinder trac via that links or via the nodes it connects. It is, thus, desirable to limit the inspection rate per each link. A lower bound would specify how often the link should be inspected, to prevent a congestion that is unobserved or detected late. An upper 46 bound would restrict the number of inspections per link, so that the measurement would not obstruct network trac. Network congestion events are frequent and are distributed unevenly. The frequency of congestion and high delays, thus, could be learned, and the inspection rates would be modied accordingly. Links that were not inspected for a long time and frequently-delayed links would receive a high priority when considering which links to examine next. The goal is to inspect each link at a rate that is within the spec- ied limits. Traditional tools, like ping and traceroute, however, are unsuitable for adaptive measurement, where dierent links should be inspected at dierent rates. To illustrate this, consider a network where the links are arbitrarily partitioned into two groups. The links of one group should be probed at a rate of x inspections per minute, and the others, at a rate of 2x. Such constraints often cannot be satised when measuring round-trip times via predened paths, e.g., when links from the two groups complete the round trip of one another. SDN’s central control over forwarding rules, however, allows for ecient implementation of adaptable delay monitoring (unlike ping’s predened path limitation). We present SDProber—a software-dened prober for proactive delay measurements in SDN. SDProber uses probe agents to emit probes and measure the travel times of the probes. To avoid complex route computations, SDProber employes a novel method that combines pseudo random walk of the probes with binary exponential backo. (In Section 3.3.2.2 we explain why a “pseudo” random walk is employed.) The probes traverse the network in a random walk over a weighted graph. The weights are adapted to route more probes via links whose lower limit is unsatised, and less probes via links whose the upper limit has been exceeded. Moreover, SDProber routes probes via paths rather than inspecting each edge independently, to reduce the number of emitted probe packets and to reduce the number of nodes that can dispatch probes. The goals of SDProber are as follows: (1) inspect links at specied rates,(2) reduce the total number of probe packets when monitoring the network,(3) minimize the number of excess packets through links, and 47 (4) detect delays as early as possible. In our experiments, we show that in SDProber the expected number of probe packets going through a link is close to the observed number, which means that the underlying probabilistic model of the random walk is a correct representation of the actual probing. We compared SDProber to two baseline methods that send probe packets through shortest paths. SDProber uses 4–16 times fewer probe packets than the baseline methods and sends 10–62 times fewer surplus packets through links. Notwithstanding this signicant cost reduction, SDProber is as fast as one baseline method and much faster than the other method, for detecting all the delays. 3.2 NetworkandDelays A network is represented as a directed graphG = (V;E) where the nodes are routers and switches. The edges are communication links between nodes. Given two nodess 1 ands 2 , delay(s 1 ;s 2 ) is the time it takes for a packet sent froms 1 to arrive ats 2 . When the measured delay for a link is higher than expected, in comparison to historical delays, we refer to the link as delayed. In proactive monitoring, the links of the network are inspected periodically, e.g., using probe packets. The rate of inspection per each link, i.e., probe packets per link, should not exceed a given upper bound, to prevent a waste of resources, and it should not go below a certain lower bound, to prevent a case where delays are not detected for a very long time. The network operator species the minimum and maximum rates of probe-packet dispatching per link. This is specied as a rate constraint of the form (min-rate;max-rate). Problemdenition: The input is a networkG with rate constraints on edges, and a cost constraint C that species the total probe packets per minute. The goal is to probeG such that probe rates would satisfy the rate constrains and the cost constraintC. Sending probes via predened paths could be problematic when paths consist of links that should be probed at dierent rates. Computing a set of paths that satises the probe-rate constraints is complex, 48 expensive in terms of running times (essentially, NP-hard), and inexible, in the sense that any change may require computing a dierent set. SDProber solves this by routing probe packets stochastically, based on the probe rates, in a random-walk fashion. 3.3 OverviewofSDProber In this section, we present an overview of SDProber. 3.3.1 DelayMeasurement The delay between two given nodes s 1 and s 2 is measured by SDProber using probe packets (similar to [213]). A schematic representation of the process is depicted in Fig. 3.1. 1 A probe agent sends a probe on a path vias 1 ands 2 . 2 When the probe packet arrives ats 1 , it is mirrored, and the clone is sent to a collector. 3 The probe packet is forwarded tos 2 . 4 Upon arrival ats 2 , the probe packet is mirrored and the clone is sent to the collector. Lett i be the time of arrival at the collector of the mirrored packet from s i , fori2f1; 2g. The estimated delay, is the time dierencet 2 t 1 . The measured time dierence depends on the travel time of the mirrored packets froms 1 ands 2 to the collector. Measuring this one-way travel is hard, however the round trip from the collector to a node and back can easily be measured using ping. Lett $ 1 andt $ 2 be the round trip times between the nodess 1 ;s 2 and the collector. Lett ! 1 andt ! 2 be the one-way trip times from the nodess 1 ands 2 to the collector. Then, t 2 t 1 = delay(s 1 ;s 2 ) +t ! 2 t ! 1 . Clearly,t $ 1 t ! 1 andt $ 2 t ! 2 . So, t 2 t 1 delay(s 1 ;s 2 ) +t ! 2 delay(s 1 ;s 2 ) +t $ 2 t 2 t 1 delay(s 1 ;s 2 )t ! 1 delay(s 1 ;s 2 )t $ 1 49 Accordingly, delay(s 1 ;s 2 )t $ 1 t 2 t 1 delay(s 1 ;s 2 ) +t $ 2 That is, t 2 t 1 t $ 2 delay(s 1 ;s 2 )t 2 t 1 +t $ 1 This provides a bound on the error for a measureddelay(s 1 ;s 2 ). For example, if the measured valuest 2 t 1 , t $ 1 andt $ 2 are 10 milliseconds, 2 milliseconds and 1 millisecond, in correspondence, then delay(s 1 ;s 2 ) is at least 9 milliseconds and at most 12 milliseconds. See in [204] how to bound delay estimation errors in ISP networks. Alternatively, delay(s 1 ;s 2 ) can be measured using timestamps. When all the network nodes support the In-band Network Telemetry (INT) framework [109], and are synchronized, the Ingress timestamp can be added to the mirrored packets ats 1 ands 2 , and the collector will compute the dierence between the timestamps. 3.3.2 SystemArchitecture SDProber sends probe packets repeatedly to measure delays in dierent parts of the network. The mirrored packets are collected at the collector, to compute the expected delay per link or path. This is used for creating a basis of comparison to detect anomalous delays. The architecture of the system is presented in Fig. 3.2. Next, we describe the dierent components of SDProber. 3.3.2.1 ProbeAgent The probe agent is responsible for crafting and dispatching probe packets, by activating probe clients. The number of probe clients can vary. Currently, a probe client is attached to every node. Alternatively, a small 50 Figure 3.1: Schematic view of the mirroring process. number of probe clients can be used. Each probe packet will be emitted from a probe client to the destined starting nodes 1 and continue along the probing route. The TTL should be set appropriately. The probe packets are marked to distinguish them from genuine trac and to associate mirrored pack- ets to their probing task. Each probe has a unique ID, in its payload. The collector groups mirrored packets by this ID and distinguishes groups of mirrored packets of dierent probes from one another. This allows the collector to reconstruct the path for each probe packet. Dierently from probe packets, mirrored pack- ets should not be mirrored (on their way to the collector). In SDProber this is done by assigning a unique destination MAC addresses to probe packets and a unique destination MAC addresses to mirrored packets (the MAC-address eld was chosen arbitrarily, and any available eld can be used in lieu of it). To further restrict the traversal of the probe packets, the probe client sets the time to live (TTL) eld in the packet header to a predened limit. 51 Figure 3.2: System architecture. 3.3.2.2 SDNControllerandOpenvSwitches The underlying elements of the network are Open vSwitches (OVS) with an OpenFlow programming interface. Each OVS is identied by a datapath ID (DPID) known to the SDN controller. For adaptive measurements, SDProber routes probe packets in a random walk fashion. To do so, it uses a combination of group tables and match-action rules. OpenFlow’s group tables are designed to execute one or more buckets for a single match, where each bucket consists of one or more actions. Group tables can be operated in dierent modes: ALL, SELECT, INDIRECT, and FAST FAILURE. SDProber uses group tables in ALL and SELECT modes. Group tables in ALL mode execute all the buckets, and group tables in SELECT mode 52 Figure 3.3: OpenFlow rules and group tables in SDProber. execute a selected bucket. The selection is based on eld values of the packet and weights that are given to the groups. Note that the selection is uniform when the weights are all equal. Fig. 3.3 illustrates how OpenFlow forwarding rules and group tables handle probe packets. SDProber adds two match-action forwarding rules, FR1 and FR2, to each OVS. The rules forward probe packets to group tables or to the collector. When a probe packet arrives at a switch, rule FR1 forwards the probe packet to a group table in ALL mode, containing two buckets. First, a mirrored packet is created, with a destination MAC address of a mirrored packet and a UDP source port equal to the DPID of that switch. The mirrored packet is routed to the collector. Second, the TTL of the probe packet is decremented and the probe is forwarded to the group table that operates in SELECT mode, where each bucket has a weight. For each forwarded packet, the OVS chooses a bucket and executes the actions in the bucket. Each bucket contains a forwarding rule to a dierent neighbor node (a dierent port). The buckets are selected arbitrarily, per a hash of eld values, in proportion to the weights. OVS hashes several elds including MAC addresses, IPv4/IPv6 addresses, VLAN ID, Ethernet type, protocol (UDP in probe packets)—to choose a bucket in a group table. The hash values are cached by the OVS, to uniformly select buckets per each ow. Hence, to add randomness to the bucket selection, the probe agent assigns a unique source MAC address to each probe packet. Note, however, that in repeating visits of a probe packet at a node, the same actions are applied at each visit. Hence, the traversal is apseudo 53 random walk. A real random walk can be implemented by applying rules that take the TTL into account, but this would require adding many more rules to each node (it will multiply the number of rules by the TTL limit), and in practice, this is too expensive and unnecessary. When a mirrored probe packet arrives at a switch, match-action rule FR2 forwards the probe packet to the collector. This prevents the unnecessary mirroring of already mirrored packets. 3.3.2.3 Collector Mirrored probe packets reach the collector. The collector records the arrival time, extracts the UDP source from the header and gets the unique identier from the payload. The mirrored packets are grouped by the identier of the probe. If all the mirrored packets arrive at the collector, then the number of groups is equal to the total number of probe packets, and the number of packets in each group is equal to the initial TTL limit. After grouping, the collector computes the traversed path of each probe packet by ordering the mirrored packets of each group based on DPID values and the known network topology. The recorded times of arrival of the ordered mirrored packets are used for estimating the delay for each link on the path. When INT is used, the collector extracts the timestamps from the payload to estimate the delay. The collector stores the measured times and uses the stored data to compute the mean and the variance of the delay. By using reservoir sampling (see [216]), a random sample can be be used in lieu of the complete history, to reduce the storage space. 3.4 MonitoringbyRandomWalk SDProber needs to satisfy the min-rate and max-rate constraints when routing probe packets. Computing a set of paths that satises all the constraints is computationally expensive and not always possible. Instead, in SDProber the probe packets perform a random walk (see [197]) over a weighted graphs. 54 In the random walk, the initial node and each traversal step are selected randomly, per probe. The link-selection probabilities are proportional to the weights of forwarding rules. The path length is limited by setting the TTL eld to a particular value, say 10 steps. It determines the number of inspected links per probe packet, to reduce the rate at which probe packets are crafted and dispatched. Increasing the TTL also allows reducing the number of probe clients. The initial node is selected randomly, possibly non-uniformly, as follows. Letn be the number of nodes in the networkG, i.e.,jVj =n, where each nodev i 2V has a weightw i . LetW = P n i=1 w i be the sum of weights. The probability of selecting nodev i isw i =W . To implement this, in each selection of a node, a numberx in the range [0; 1) is picked uniformly. Thei such that P i1 j=1 w j W x< P i j=1 w j W is discovered, andv i is the selected node. When forwarding probe packets, the link (port) for the next step is chosen proportionally to the weights assigned to forwarding rules in the OpenFlow’s SELECT-mode group tables. (See Section 3.3.2.) To control the inspection rates, we need to estimate the number of probes passing through each link for a given number of emitted probes. This is done as follows. First, we compute visit probabilities for nodes. LetP 0 be a vector such thatP 0 [i] is the probability of selectingv i as the initial node, for 1in. The transitionmatrix ofG is annn matrixM = (p ij ) 1i;jn , wherep ij is the probability of forwarding the probe packet fromv i tov j . Note that for each nodev i , the array (p i1 ;:::;p in ) species the probabilities for the next step after reachingv i . Thus,p ij = 0 ifv i andv j are not neighbors, and P n j=1 p ij = 1, for all v i 2V . Given the initial probabilitiesP 0 and the transition matrixM,P 1 = (M T )P 0 is the vector of probabili- ties of reaching each node after one step of the random walk. ByP t = (M T ) t P 0 we denote the probability of reaching each node aftert steps of the random walk. The probability of traversing a link (v i ;v j ) in a random walk ofk steps is the probability of reaching nodev i in stept and proceeding to nodev j in stept + 1, for some 0t<k. We denote this probability 55 by p-traverse ij . That is, p-traverse ij = P k1 t=0 (P t ) i (p ij ), since (P t ) i is the probability of reaching nodev i at stept, andp ij is the probability of forwarding tov j a packet that arrived atv i . In the random walk approach, we do not need to conduct complex computations to craft probe packets or change them as they traverse the graph. If network changes require adjustments of probe rates, we merely have to alter the node weights of the initial node selection or the weights in the group tables. 3.5 WeightAdaptation In adaptive monitoring, weights that aect the random walk are adjusted to aid satisfying the rate con- straints. For example, consider a link that should be probed at a rate of 2 to 4 times per minute. If during the last minute it was probed less than 2 (more than 4) times, its visit probability should be increased (decreased). SDProber modies the weights iteratively using binary exponential backo. The iterations continue indenitely, as long as the monitoring continues. Link-weight adaptation. The weights are adapted to satisfy the probing constrains. Weights are doubled (halved) when the probing rate is below (above) the minimum (maximum) rate. Adding (reducing) weight increases (decreases) the probability of selecting the link in the next iteration. Rates within the limits specied by the rate constraints are adjusted after each iteration. The weight of a link with a normal delay is decreased by half. The weight of each uninspected link is multiplied—by if it was delayed during the lastk iterations, and by 2, otherwise. The valuek is congurable. By that, historically delayed links could receive a higher weight than links with no history of delays, to be visited more frequently. Node-weight adaptation. Node weights should be modied to reect the changes in links. The weight of a node with a link below the minimum rate is doubled, to increase the chances of visiting this node in the next iteration. Otherwise, the node weight is halved, to prevent traversal via links whose min-rate constraint is already satised. The weight is also halved for nodes with (1) a link whose probing rate is already greater than max-rate and (2) no links whose rate is less than min-rate. 56 3.6 BaselineMethods Commonly, probe packets are sent via the shortest path between two nodes [234, 83]. Hence, we compare SDProber to two baseline methods in which each probe traverses the shortest path between selected nodes. We present now the baseline methods. Random Pair Selection (RPS). In each iteration of RPS, pairs of source and destination nodes are selected randomly. The probe packets are sent via the shortest path from the source to the destination. The probe packets are mirrored to the collector at each hop, and the collector uses the arrival times of packets to estimate the delay on links, as in SDProber. In each iteration, the pair of source and destination nodes is selected uniformly from the set of pairs that have not been selected previously, till all the links are probed. This is repeated for the duration of the monitoring. Greedy Path Selection works in a greedy fashion, iteratively. Initially an empty set Visited is cre- ated. In each iteration, for each pair of nodes, the weight of the shortest pathP between these nodes is P e2P ande= 2Visited min-rate(e), that is, the sum of the min rate values of all the unvisited links on the path. The path with the maximal weight is selected and its links are added to Visited. The probe packet is sent from the source to the destination of the selected path. The process ends when Visited contains all the links of the network. This is repeated while the monitoring continues. 3.7 Evaluation To test SDProber, we conducted a series of experiments. The goals are to show that(1) SDProber provides control over the link probing rates, (2) SDProber satises the min-rate constrains with a lower cost (in terms of the total number of packets) than the baseline methods, without increasing the delay deection time, and (3) given historical delay data, can be tuned to improve performances. 57 0 20 40 60 80 100 120 140 160 180 200 0 0.002 0.004 0.006 0.008 0.01 0.012 Avg. number of probe packets Probabilities Small Medium Large Expected Figure 3.4: Actual probing rate versus expected rate, for links with dierent traversal probabil- ity. 0 200 400 600 800 1000 1200 1400 1600 Small Medium Large Avg. number of probe packets per edge SDProber Greedy RPS Figure 3.5: The average number probe packets per link when sat- isfying the min-rate constraints. 0 200 400 600 800 1000 1200 1400 1600 Small Medium Large Avg. number of excess probe packets sent per edge SDProber Greedy RPS Figure 3.6: The number of surplus probe packets per link when sat- isfying the min-rate constraints. Setting. The tests were executed on Mininet [60] with Open vSwitch 2.7.2 and a RYU controller [180]. We used a publicly-available real topology (the largest one in TopologyZoo [111]), with 196 nodes and 243 links. A probing iteration was launched every 30 seconds. After every iteration, the weights of nodes and links were adjusted (see Section 3.3.2.2) and the rules were updated (see Section 3.5). We tested three probing proles, in which min-rate, max-rate and the the dierence between them vary. The proles are (small, 16, 20), (medium, 8, 28) and (large, 20, 60), where each tuple contains the name of the prole, min-rate and max-rate, respectively. Rates are in packet per minute (ppm). In each prole, the specied rates were assigned to all the links. In actual networks, the network operator can set probing rates based on SLA, the frequency of delays, etc. In the presented results, the TTL limit was 10. In additional experiments, which are not presented here, we tested the eect of the TTL on the results and saw that as we increase the initial TTL, the detection time of delays decreases, but as the TTL approaches 10, the decrease gets smaller, and beyond 10 it is negligible. This is due to the hashing implementation by OVS, as discussed in Section 3.3.2.2. 3.7.1 ControloverInspectionRates SDProber controls the number of probe packets through each link by adjusting the link weights. The probability of a packet to go through a link (p-traverse) is as described in Section 3.4. An important question 58 is whether the probabilistic model is correlated with the actual trac ow. To test this, we measured the number of probe packets going via each link, and compared the expected count with the measured one. In this test, the total number of emitted probes per iteration was equal to the sum of the min-rates over all the links. Each measured number of packets is the average of ve executions and each execution is limited to ten iterations. Fig. 3.4 presents the results for the three probing proles. The gray line depicts the expected values, which is the probability multiplied by the number of probe packets and the TTL. The graphs show that pseudo random walk provides inspection rates that are close to the expected values. For all the proles, the error is within a range of around10% from the expected value. Thus, the probabilistic model provides a reliable description of the trac ow. We conducted additional experiments, with other probing rates, and the results were the same. The probabilities and their range depend on the topology and the rate constraints. Also, a small number of packets per iteration can lead to weight adjustments and an increase in the probabilities. 3.7.2 CostEectiveness A main advantage of SDProber over the baseline methods is its ability to monitor the network with a low cost, in terms of the overall number of emitted probe packets. The next experiments show this. For each method, we increased the number of emitted probe packets per iteration till satisfying all the min- rate constraints of the links. Emitting fewer packets means that the constraints are satised with a lower cost. The results are presented in Fig. 3.5. SDProber sends fewer probe packets than Greedy and RPS, to satisfy the min-rate constrains. A reduction by a factor of 4.48–11.77 and of 10.52–12.44 is achieved in comparison to Greedy and RPS, respectively. The eectiveness of SDProber is achieved by adapting the weights to avoid probing links whose min-rates were already satised. SDProber satises max-rate constraints more strictly than Gready and RPS. To show this, Fig. 3.6 de- picts the average number of excess probe packets per link, i.e., packets that cause the probing rate to exceed 59 max-rate. SDProber sends 10.46–36.31 and 18.69–62.29 times fewer surplus packets than Greedy and RPS, respectively. 3.7.3 Adjusting The parameter provides control over the weight adaptation, to inspect frequently-delayed links (FDL) at a higher rate than other links. We tested the eect of on a network with 10% delays, wheref% are FDL, i.e., delays repeat for these links in dierent iterations. We varied the values from 2 to 4 with steps of 0.4 for thesmall probing prole and show its inuence on the detection time forf2f0%; 50%; 100%g in Fig. 3.7. The number of packets per iteration was small (200), to emphasize the eect of frequent probing of FDL. Each reported time is the average of ve executions and each execution is limited to ten iterations. When there are FDL, i.e.,f2f50%; 100%g, increasing reduces the detection time, by 2.84% and 5.58% on an average. But the reduction in detection time decreases with the increase inalpha because other parameters (number of probe packets, f and rate constraints) inuence the detection time as well. However, when there are no FDL, i.e.,f2f0%g, decreasing increases the detection time by 3.91% on an average and = 2 yields the fastest detection time. 3.7.4 DetectionTime To test how fast delays are detected, we evaluated the time it takes to detect all the delayed links. The percentage of delayed links in dierent runs was varied from 10% to 100%, at increments of 10% (in all cases, the entire network should be inspected to detect all delays). In each experiment, the number of packets available per iteration was equal to the sum of min-rate values over all the links—this ensures that there are enough packets per iteration to probe all the links at min-rate. Fig. 3.8 presents the results for themedium probing prole. The results are similar for the other two probing proles. Each reported time is the average of ve executions and iterations are continued till all the delayed links have been detected. 60 0 10 20 30 40 50 60 70 80 90 0% FDL 50% FDL 100% FDL Time (s) α 2 α 2.4 α 2.8 α 3.2 α 3.6 α 4 Figure 3.7: Detection time of delayed links per. 0 20 40 60 80 100 120 140 160 10 20 30 40 50 60 70 80 90 100 Time (s) % of delayed links Greedy RPS SDProber Figure 3.8: Detection time of delayed links. RPS detects delays slightly slower than SDProber, but requires many more packets to satisfy the min-rate constraints. RPS and SDProber detect delays about twice faster than Greedy. SDProber conducted less than 3 complete iterations (weight updates) during this test. The reason for the slow detection by Greedy is that links with a low weight are visited last, and a delay on such a link is detected behindhand. In SDProber, the weight adjustment guarantees frequent probing of all the links, according to the min-rates. Note that RPS ignores the weights, so regardless of where the inspection rates are violated, the entire network is inspected, including links which have already been inspected. So, detecting all the delays with RPS is relatively fast, but it has a high cost. 61 3.8 WeightAllocation SDProber can be congured to use dierent weight-allocation strategies, e.g,static oradaptive—we inves- tigated the adaptive case. Previous studies [134, 76] have shown that link failures can be learned. Accord- ingly, a failure probability can be associate with each link, to determine static weights for the random walk (future work). However, in static weight allocation, links with a low failure probability would be inspected at a very low rate—this reduces costs but may lead to late detection of failures in links with a low weight. As an example use case of dynamic weight allocation, consider AT&T FlexWare—aNetworkFunctionon Demand architecture. A customer could deploy dozens or hundreds of virtual network functions (VNFs) on FlexWare devices, and the network between them would be monitored by SDProber. Based on full knowledge of the VNFs, the connections between VNFs will be monitored at rates that are specied by the customer, to comply with the SLA. When a new service is deployed, it can be accommodated by weight adaption, easily and promptly. A similar approach can be applied to a hybrid environment where only some of the nodes are OVS. 3.9 RelatedWork Network measurements, including delay measurement, received a lot of attention over the years [25, 118, 140]. However, these papers do not show how to use SDN capabilities or centralized control for eective proactive measurement of the delay. Recently, with the advent of SDN, there is a growing interest in measurements that are adapted to SDN [54, 126, 223, 232, 206, 13, 219]. Several systems utilize mirroring for measurements. NetSight [85] uses mirroring to gather informa- tion about the trajectories of all the packets in a network. However, their method is only suitable for small networks, and does not scale. Everow [247] provides a scalable sampling of packets in datacenter networks. They, however, require specic hardware, e.g., multiplexers, to support their sampling method. 62 Furthermore, their method is reactive, to be used in a response to an event, while SDProber is proactive, to monitor the network in an economic way and detect high delays early. Using probe packets to measure latencies in OpenFlow-based networks was studied in [213, 231]. SLAM [231] uses the time of arrival of OpenFlow packetin messages at the controller to estimate the delay between links. However, this approach requires knowledge of trac patterns, e.g., to distinguish between lack of packetin messages and delays that obstruct packetin messages, so it may not be applicable to networks which are not a datacenter. The OpenNetMon [213] system provides per-ow metrics, like throughput, delay and packet loss, for OpenFlow networks. It uses probe packets and a collector for delay measurement. Pingmesh [83] uses ping to measure delays in a datacenter. Unlike SDProber, OpenNetMon and Pingmesh do not provide an adaptive measurement framework. 3.10 Conclusion We presented SDProber—a prober for proactive measurement of delays in SDN. SDProber provides control over the inspection rates of links, by routing probe packets according to min-rate and max-rate constraints, while utilizing each probe packet to inspect several links. To satisfy the constraints without expensive computations, the dispatched probes conduct a random walk over a weighted graph. The weights are modied using binary exponential backo to achieve the desired probing rate for each link. Furthermore, weight adaptation can take into account the history of delays, to inspect frequently-delayed links at a higher rate than other links. We demonstrate how random walk is implemented using OpenFlow. In our evaluation, we show that SDProber is more eective than baseline methods by sending signicantly fewer probe packets without increasing the time it takes to detect delays and while reducing the number of excess packets through links. 63 The focus of this project is on measurement of the delay. However, SDProber can be used to measure other variables of the network as well, e.g., packet-drop rate, jitter, etc. Implementing and testing this is future work. 64 Chapter4 SENSSAgainstVolumetricDDoSAttacks 4.1 Introduction Volumetric distributed denial-of-service (DDoS) attacks can overwhelm even the largest networks. In 2016, Dyn, a large and geographically distributed DNS management network, was hit by 1.2 Tbps attack [230], from more than 100,000 sources. In 2018, Github was hit by a 1.35 Tbps attack, from thousands of au- tonomous systems and tens of thousands of hosts [174]. Distributed attacks create problems not just for their victims, but also for the ISPs and bystanders, which share the path with attack trac [46]. Volumetric DDoS attacks cannot be handled by the victim alone, and require help of other networks. When the attack’s volume is so high that it overwhelms the victim’s link to its ISP, or even a link within the ISP, this ISP or other upstream ISPs must help in the attack’s mitigation. Similarly when the attack is indistinguishable from legitimate trac, ltering at the victim has a high collateral damage. Upstream networks can help identify locations close to attack sources, and place lters at these locations, where they inict a lower collateral damage. Today’s Internet has no automated mechanism for victims to ask their peers or remote networks for help in attack handling, and has low incentives for ISPs to oer such services. While most ISPs will help when engaged, asking for help occurs through human channels today, and the ISP’s response is often limited to blackholing all trac to the victim [209]. 65 We propose SENSS, a framework for collaborative defense against volumetric DDoS attacks. Using SENSS, the victim of an attack can request help from direct and remote networks (called “ISPs” in the rest of the chapter) in an automated and secure manner, and pay on demand for the services rendered. The victim sends messages to SENSS-enabled ISPs (“SENSS ISPs” for short), asking for trac and/or route monitoring or control actions. Each message is signed by the victim and authenticated by SENSS ISPs, which prevents misbehavior and misuse. Replies to monitoring messages bring information to the victim, which it can combine with its own knowledge of its trac and business priorities, to devise custom attack mitigation. The victim then issues control messages to SENSS servers to mitigate the attack. SENSS APIs at ISPs can be readily implemented by today’s infrastructure, which makes SENSS cheap for ISPs to deploy and incentivizes deployment. Ourrstcontribution is the novel design of the SENSS framework, for secure, automated, on-demand collaboration between victims and ISPs (Section 4.3). SENSS has two main design principles: 1. Intelligenceatthevictim,simplefunctionalitiesatISPs. Many collaborative defenses place intelligence (e.g., deep packet inspection – DPI) in the cloud or at an ISP, which makes defense functionalities complex and costly. In SENSS, the victim drives all the decisions, such as what to monitor and which actions to take to mitigate attacks. This allows for simple functionalities at ISPs, which are easily implemented in the current ISP infrastructure. Intelligence at the victim makes SENSS cheap for ISPs, and eective in sparse deployment. It also limits the impact of misbehaving participants (Section 4.3.5). Victims that lack technical skills to make decisions about attacks can delegate SENSS control to their ISPs or to third parties via SENSS’s proxy mechanism (Section 4.3.5). 2. Versatile,evolvableandcustomizabledefense. Many defenses today work against one or a few attack variants. SENSS’s simple APIs can be used as building blocks by the victims to create customized defenses against many DDoS avors. As attacks evolve, these simple APIs can be used to build new defenses on top of the existing SENSS infrastructure. 66 leaves root = bottleneck (a) DDoS tree – trac converges (b) Cloud defense (c) SENSS Figure 4.1: DDoS attacks and ways to handle them. (a) Attack trac mostly converges at one bottleneck, forming a tree-like pattern. (b) lter in a cloud, requires redirection and dedicated infrastructure (c) lter on the path of the attack, with ISP’s help, using SENSS Oursecondcontribution lies in novel algorithms for handling of direct oods, reector attacks and cross-re attacks, all described in Section 4.3. These algorithms and SENSS enable the following novel functionalities that are not available in today’s defenses: (1) mitigation of direct oods close to attackers, (2) surgical mitigation of reector attacks, (3) mitigation of cross-re attacks. Our third contribution is a thorough evaluation of SENSS using simulation on the Internet’s AS- level topology and emulation on the Deterlab testbed [17] (Section 4.4). SENSS is very eective in sparse deployment, oering full protection to direct customers of early adopters, and considerable protection to remote victims when deployed strategically. In case of 2016 attack on Dyn, SENSS deployment at Cogent, Level 3, Zayo and Comcast would have ltered 100% of the attack. In general case, deployment on the largest 1% of ISPs would protect not just direct customers of these ISPs but everyone on the Internet from 90% of direct oods and reector attacks. SENSS further lters attacks closer to their sources, and saves twice as much bandwidth as cloud-based defenses. SENSS overhead within an ISP remains constant irrespective of attack’s complexity or its volume, and SENSS’s handling of client requests requires just a fraction of a second. 67 4.2 BackgroundandRelatedWork In this section we discuss types of DDoS attacks that we would like to address, and the need for a collab- orative defense between victims and ISPs. We also survey related work in the operational and research realms, and position SENSS in this landscape. 4.2.1 VolumetricAttacks DDoS attacks overwhelm the victim with excessive trac, sent from many sources. In this work we focus on three variants of volumetric attacks, which usually cannot be handled at the victim: direct oods, reector attacks and cross-re attacks. We use the termoods to refer to all three of these variants. Direct oods send trac directly to the victim. Reector attacks send request trac to public servers (e.g., DNS servers), spoong the victim’s address, and the servers reply to the victim, overwhelming it. Cross-re attacks send trac to many destinations. This trac congests some bottleneck link in an ISP, which is also shared by the victim’s inbound trac. In all three cases, there is a tree-like pattern of trac, with attack and legitimate sources being the leaves and the bottleneck link being the root. This is illustrated in the Figure 4.1a. Because volumetric attacks only need to generate high enough volume to DoS the victim, they do not have to have a specic signature. They can also be spoofed. Further, they can be launched from many distributed sources. The 2016 Dyn attack [230] and 2018 Github attack [174] were launched from 10,000 – 100,000 of sources, distributed over thousands of networks. 4.2.2 RelatedWork DDoS defense is a mature eld. In this Section we focus only on defenses, which address volumetric attacks, and on collaborative DDoS defenses. 68 Victim-enddefenses, such as Bro [163] or Arbor APS [148] are often used to detect and lter smaller DDoS attacks. However, large volumetric attacks cannot be handled by the victim, because the bottleneck is upstream from the victim’s network. First-ISP. The victim can engage its ISP through human channels, to help in mitigation, i.e., to lter the attack. Most ISPs oer only crude ltering of all trac to the victim, aka remotely-triggered black hole (RTBH) [209]. This protects ISP infrastructure, but cuts the victim o the Internet and exacerbates the impact of the attack. Bohatei [186] is a research defense within a single ISP. It uses software-dened networking (SDN) and network function virtualization (NFV) to oer exible, elastic DDoS defense. It instantiates a number of VMs on demand, and steers victim’s inbound and outbound trac through these VMs, which imple- ment custom mitigation programs. Bohatei’s custom programs oer ner-grain trac analysis at a higher resource cost (statistics storage and application header access) than SENSS. SENSS is complementary to Bohatei, as it handles spoofed and cross-re attacks, which cannot be handled by Bohatei. Cloud-based defenses, such as CloudFlare [46] and Zenedge [233], defend against attacks through geo-replication of the victim’s resources. To eectively defeat oods, this geo-replication needs to be extensive, requiring equipment and peering contracts at many locations and a lot of excess bandwidth to withstand high-volume attacks. When a cloud wants to grow, it has to enter into more peering agreements and buy and install more equipment and ber; both are expensive. Clouds attract all the victim’s inbound tractothemselves, and apply “packet scrubbing” to identify an attack signature and lter the attack trac. Clean trac is then tunneled to the victim. The scrubbing process is proprietary, but it often involves deep- packet inspection (DPI), and thus has a high processing cost. Compared to clouds, SENSS enables ISPs on the attack path to oer a distributed attack response, through victim-ISP collaboration. ISPs deploy SENSS on their existing infrastructure, leveraging existing 69 functionalities like SDN, Flowspec [135], Netow and access control lists (ACLs). This makes SENSS’s de- ployment costs smaller than that of clouds. On the other hand, SENSS lters are coarser-grain (network- and transport-level but not application- or payload-level) than those of clouds, which may lead to a higher collateral damage for application-level attacks. For this reason, SENSS focuses only on handling of volu- metric attacks, which usually randomize elds after the transport header. SENSS infrastructure grows when more networks adopt SENSS and start supporting it on its existing infrastructure. This makes SENSS growth more sustainable than cloud growth. We illustrate handling of network oods by clouds and by SENSS in Figures 4.1b and 4.1c respectively. While clouds divert the victim’s inbound trac to their infrastructure, SENSS empowers ISPs on the traf- c’s path to help in attack diagnosis and mitigation. This saves more bandwidth (see Section 4.4.4). Collaborativedefenses. The need for collaborative defense against evolving DDoS attacks was elab- orated by Kang et al. in [104]. There are several collaborative research approaches to DDoS mitigation, such as Pushback [97], traceback [77], StopIt [129], AITF [11], DefCOM [156], CoDef [122], SIBRA [20], SCION [1], TVA [229], Defense by Oense [218] and SPIFFY [105]. Some of these approaches are not deployable today, because they require Internet redesign – TVA and SCION. Other approaches are less deployable than SENSS, because they require router hardware changes. Traceback, StopIt, AITF, CoDef and DefCOM require packet marking at routers, and StopIt requires cryptographic operations on trac. Further, some approaches apply victim-to-source collaboration model – StopIt, Defense by oense and CoDef – which has the victim ask the source networks (legitimate or attack) for help in attack mitigation. Victim-to-source collaboration cannot be eective in sparse deployment, because many sources would reside in legacy networks. SENSS, on the other hand, is very eective in sparse deployment, thanks to its victim-to-ISP collaboration model. Even a few ISPs on the attack’s path, collaborating with the victim, enable that victim to lter much of the attack trac. 70 Pushback [97] appliesISP-to-ISP collaboration. Such collaboration is dicult in general, because an ISP handles a lot of trac for many customers. Moderate oods, which may not be noticeable at the ISP level may overwhelm a customer on a small downlink. To detect these oods the ISP would have to continuously monitor all its customers’ trac, which is costly. It would also have to make guesses as to which trac to lter, while a victim can make a better decision (and implement it with SENSS) based on its knowledge of its inbound trac patterns. CoDef [122] focuses on handling of cross-re attacks, through collaborative rerouting and rate lim- iting. Collaboration occurs between the aected network and the networks that host legitimate sources. CoDef’s mitigation (rerouting) resembles SENSS’s route control messages, but its diagnosis requires packet marking, while SENSS does not require this. SENSS is thus more deployable than CoDef. Further, CoDef’s collaboration model (victim-to-source) means that CoDef requires a wider deployment than SENSS to achieve a given eectiveness target. SIBRA [20] introduces a new mechanism for legitimate sources and destinations to reserve bandwidth on inter-AS links, and thus isolate themselves from DDoS attacks. SIBRA is complementary to SENSS and attempts to prevent DDoS, while SENSS attempts to mitigate it. SIBRA also requires more extensive changes to ISPs than SENSS and is thus less deployable. SPIFFY [105] introduces implicit collaboration between the bottleneck and sources of trac, by tem- porarily expanding the bottleneck link and identifying sources, which do not expand their sending rate, as attackers. SPIFFY only handles cross-re attacks and is thus complementary to SENSS. 4.2.3 SENSSvsFirst-ISPvsClouds We now briey discuss how SENSS could complement current operational solutions: rst-ISP and clouds. 71 SENSS’s messaging mechanism oers a way for remote networks to collaborate on demand, without prior trust. As such, SENSS mechanisms could be used to remotely trigger rst-ISP or cloud-based solu- tions. This could transform cloud defenses from pre-arranged into on-demand solutions. SENSS could further be used when rst-ISP or cloud defenses fail or are overwhelmed (e.g. Spamhaus attacks in 2013 [12] and Dyn attacks in 2016 [230]). A cloud or an ISP can use SENSS to achieve a collabo- rative response with other ISPs in the Internet, to lighten its load. SENSS can also benet from clouds and rst-ISP solutions, which could act as victim proxies, and enable adoption of SENSS among non-technical victims. 4.3 SENSS In this section, we discuss challenges of collaborative defense, provide a high-level overview of SENSS operation, and detail how SENSS would be implemented at ISPs and at attack victims. 4.3.1 Challenges When designing a collaborative defense, we must address the labor-division challenge (what will be done where), the deployment challenge (how to motivate deployment) and the security challenge. Labordivision: we must decide which functionalities are required for attack handling and where to implement them. SENSS uses victim-to-ISP collaboration, which places all the intelligence at the victim and requires only simple functionalities of ISPs. This enables a victim to request help from any ISP for diagnosis and mitigation, but remain fully in control over the attack handling. The victim can combine its internal knowledge of its business and customers, with observations received from SENSS, to create customized defenses. Because the victim can directly contact any SENSS ISP, neighbor or remote, SENSS is very eective in sparse deployment (see Section 4.4) and robust to misbehavior (see Section 4.3.5). 72 Deployment: collaborative defenses must oer signicant benets to networks that deploy them. SENSS brings clear benets to the victims of attacks, but the challenge lies in making it appealing to ISPs. We address this challenge in several ways. First, SENSShandlesseveralattackvariants, thus an ISP can oer it to current customers as added value, and many customers may nd it useful. Second, SENSSworkswith existinghardware and thus has low cost to deploy. Third, SENSSoerssignicantbenetstoearlyadopters, thus ISPs can oer immediate protection to their customers against DDoS, even in sparse deployment. Security: collaboration should not introduce new vulnerabilities even under direct attacks, or when some collaborators misbehave. SENSS has multiple security layers to protect against misuse: (1) It allows the victims to only observe/control trac and routes for their prexes, whose ownership is proved via digital certicates. SENSS ISPs validate these certicates before processing each request. (2) All communi- cation between the victim and the SENSS ISPs is secured using TLS, and is thus protected against message sning, forgery, modication or replay, (3) SENSS operation is driven by the victim, with each SENSS ISP reporting its own observations of the victim’s trac and routes. Such design severely limits the impact of a misbehaving ISP (see Section 4.3.5). 4.3.2 SENSSArchitecture Figure 4.2 illustrates SENSS operation. SENSS consists of client code, which runs at an end network or an ISP, and server code running at an ISP. Clients are illustrated as blue circles and servers as grey circles in the Figure. We refer to the client-deploying network as thevictim, but it may decide to engage SENSS even in absence of attacks, e.g., to learn about its normal trac patterns. An ISP can decide to provide SENSS services for free or for a fee, which could be a at-rate or per-message fee. Economically, a per-message fee makes the most sense as the ISP incurs cost for running SENSS only when it handles SENSS messages, and a victim is only interested in paying for SENSS when they are under attack. 73 SENSS server SENSS client victim attacker SENSS server 2. Query and reply 3. Control 2. Query and reply 3. Control SENSS directory 1. Lookup SENSS servers Figure 4.2: SENSS architecture. An ISP deploys SENSS by implementing SENSS APIs on a server, and exposing them to the public. These APIs can be implemented as a Web service, and thus deployed at the ISP’s Web server. This lever- ages existing approaches for service robustness (Web server replication), message security (HTTPS), and charging for service (e-commerce). The ISP also implements automated mechanisms within its network, which act on SENSS client’s requests by implementing trac/route handling in border routers. This is shown as grey lines in the Figure. These mechanisms can be implemented as a collection of scripts, which communicate with switches/routers using Netow, Openow, Flowspec or switch-specic management language (e.g., for ACL setup). We assume that all client-server communication occurs at the network level and not at the host or user level. This limits the cost of communication and the state at SENSS servers. A victim under attack runs the SENSS client. The client rst uses a public SENSS directory to identify multiple SENSS servers (step 1 in the Figure). One way to implement this directory is to assign a common DNS name to each SENSS server. The victim sends queries to SENSS servers about the victim’s inbound trac or the routes to the victim’s prexes (step 2 in the Figure). For security reasons, clients are only allowed to ask about and manipulate trac and routes for the prexes their network owns. While some 74 Type Message Matching Fields Action Reply/Response Trac query trac_query predicates, otime start/stop rule_id and a list of<tag, direction, #bytes/#pkts> matching the predicate during otime Route query route_query prex none rule_id and AS paths from the SENSS ISP to the prex Trac control lter/allow predicates, tag, dur start/stop rule_id and lter/allow all trac matching the<predicate, tag> for duration dur Route control demote prex, seg, dur start/stop rule_id and reduce pref. of routes to prex if they contain seg in AS path, for duration dur Abort abort prex none delete all rules for the prex, and blacklist the public key Table 4.1: SENSS APIs attack diagnostics may be easier if we allowed clients to ask about anyone’s trac, this would create grave privacy concerns, and jeopardize adoption of SENSS. Each query is accompanied by a digital certicate, which proves the that the client is authorized to issue query and control messages about the IP prexes contained in the query. We call this certicate proofofprexownership and provide more information about it in Section 4.3.5. The SENSS server validates the certicate and the message, performs the required service, charges the victim for it, and returns the response to the client (step 2 in the Figure). The server may also decide not to perform a given action, e.g., because it would be against some internal policy or because it would consume too many resources. In this case the server does not charge the victim and simply returns a reject message to the client. The client analyzes responses obtained from SENSS servers, identies the best action and the best locations for mitigation (e.g., ltering, rerouting) and issues control messages to chosen SENSS servers (step 3 in the Figure), which authenticate them, charge for and perform the actions. The query and control steps can be repeated multiple times, until the desired mitigation eect is achieved. 75 4.3.3 ISPImplementation APIs. Table 4.1 summarizes SENSS APIs, used by the client to request SENSS services. Each message has anaction eld, which species if the trac/route handling rules should be installed (start) or removed (stop) at the SENSS ISP. The server enacts these handling rules after each message, by installing them in the appropriate border routers at the ISP. The server then replies with a unique identier (rule_id) to the client. All rules expire after the duration (otime or dur parameter) specied in the message. When a rule expires at the server, the server removes the rule from the routers where it was installed. The client may choose to remove a handling rule prior to its expiry, by setting theaction eld tostop and specifying rule’s identier. Additionally, the client can use a specialabort message, which removes all the rules for the given prex. A SENSS server may also decide not to handle the client’s message, and return areject reply to the client. The basic building block of trac messages (trac_query and trac_control) are predicates, which match a ow based on header elds. For example, the predicate “(src_ip=10.0.0.1j src_ip=10.0.0.2) & src_- port=53” matches ows with source IP 10.0.0.1 or 10.0.0.2 and source port 53. Predicates support conjuction (&), disjunction (j), negation (!) and wildcard (*) operators. Predicates can also specify trac direction (SELF – generated by the ISP, IN and OUT), and they can specify a tag, denoting a peering link to which the query applies. A trac_query asks for monitoring of trac matching the predicate for time otime. On receiving a trac_query, a SENSS server installs the appropriate trac collection rules (Openow or Netlter) to some or all border routers. After time otime the server collects observations from the routers and removes the collection rules. It then aggregates these observations into a single reply, and sends it to the client. The reply contains rule_id, and a list of tuples. Each tuple contains a tag, identifying the neighbor, trac direction dir and the amount of trac observed in packets or bytes. The tag is the ISP- specic, unique identier per neighbor, and should be anonymized in a manner, which is reversible only by the ISP. For example, the ISP could encrypt the neighbor’s identity with a secret key. If the client later 76 A B C D E F client server traffic_query(dst=1.2.3.0/24, 10s, start) traffic_reply(id=1, (Tc,IN,1000)(Td,IN,10)(Tf,IN,10000)) prefix 1.2.3.0/24 Figure 4.3: Illustration of trac query. requests ltering of trac with a given tag, the ISP decrypts the tag to identify the appropriate peering link and the switch, where the lters should be installed. One trac_query and its reply are illustrated in Fig. 4.3. The client A asks for monitoring trac to its prex, 1.2.3.0/24, for 10 seconds. The server B replies with the list containing three records – one for each neighbor that forwards trac to B with destination IP in 1.2.3.0/24. Atrac_control message requests a SENSS ISP tolter orallow trac matching the specic predicates and tags, for the duration dur. If the client sends multiple messages for the same prex, they will be matched in the order in which they are received. The server translates each rule into a ltering rule that its routers understand (e.g., ACL, Flowspec, Openow) and installs it at the routers matching the tag eld from the message. Rules are removed after time dur. A route_query asks an ISP about the AS paths in its best routes to the prex. The server collects the best routes from all border routers, and replies with the set of unique AS paths. A route_control message asks an ISP to demote all its routes for the select prex that contain the AS- path segment seg, for the duration dur. The server collects all the routes from each border routers for the specied prex. It then decides for each router if the route should be demoted. This decision can take into account the ISP’s internal route selection policy and the cost of demotion (e.g., if a customer route is being demoted and the next best route is peer or provider route there will be an associated cost). If the server 77 decides to demote the route, it issues appropriate messages to the router’s BGP daemon. After time dur these messages are reversed. Identifying deployment routers. After receiving and validating (see Section 4.3.5) a SENSS message, the SENSS server identies the routers or switches, which should implement the required functionality. For trac queries with IN direction, and for route queries and route control, all ingress routers would be used. For trac lters, the server only needs to identify those ingress routers that were specied in the tag eld of the client’s message. If no tag is specied, the egress router to the specic prex can be used for lter installation. Implementingmonitoringandcontrolatrouters. If the ISP supports SDN, SENSS can implement all trac observation and control functionalities by issuing OpenFlow messages to the SDN controller. For ISPs that do not support SDN, a SENSS server can enact trac observation by installing Netow collection rules in routers. For cost reasons, the server can install rules that use packet sampling, and then extrapolate the actual packet/byte rates for the trac_reply. Finally, each router supports access control lists (ACLs). SENSS server can enact trac monitoring of a ow by installing an ACL rule with “permit” target, and later querying the number of packets that matched the rule. The server can implement trac control at non-SDN ISPs by using ACLs or Flowspec [135]. The SENSS server implements route observation and route control by using BGP software. For route observation, it queries the given router for its best route to the select prex. For route demotion, it rst queries the given router for all its routes to the select prex. Then it decides which, if any, routes to demote and issues commands that lower the values of the routes’ local preference attributes. Practicalissues. SENSS messages lead to rules being installed at a router’s TCAM, whose space is usually limited. We nd that many distributed attacks can be handled eciently with very few rules per router, using coarse ow specications on elds such as neighbor tag, destination IPs, transport ports, protocols, etc. On the other hand, the client may rarely want to observe and control ows at a ne-grained level, 78 Attacks KeyIdea SENSSfeaturesweuse Flood w sig Filter trac based on signature lter Flood w/o sig Identify geographical dierences between usual and attack trac paths; use for lter placement trac_query before and during attack, lter Reector DDoS Victim marks trac; ISPs lter non-marked trac lter, allow Cross-re Locate bottleneck links via trac queries and route around them. Combines route/trac info. route_query, trac_query, demote Table 4.2: Key innovations of client programs and the SENSS features we use using the source IP eld. Since the number of rules, which use source IP eld, could quickly skyrocket, ISPs could discourage use of such rules by pricing them much higher than other rules. In Section 4.4 we show that DDoS attacks can be handled well with coarse-grained rules. An ISP may have privacy concerns about others learning about their trac handling and routes. We have carefully designed SENSS to avoid leakage of information that ISPs consider private or sensitive, such as trac load balancing and peering. 1. Route replies reveal only public peering information, which is already visible in BGP advertisements. 2. Trac replies are anonymized or aggregated. An ISP can return encrypted tags in trac replies that ask only about IN direction, as we illustrate in our examples for ood w/o signature in Section 4.3.4. When the victim sends lter requests, the ISP decrypts the tags to identify where to place lters. In trac queries that ask about IN and OUT trac (e.g., cross-re example) the ISP can return aggregated trac (“all” tag in our example for cross-re) or omit trac volume. This protects condential load balancing information. An ISP may not be open to others changing its routing decisions in any way. Such an ISP can always return thereject response fordemote messages. This will make it unable to aid in cross-re attack mitigation, but it will still be able to help in handling direct oods and reector attacks. 79 4.3.4 ClientPrograms We now illustrate how a victim could design custom attack mitigation programs using SENSS APIs, by presenting four programs for mitigation of the following attacks: direct oods with and without trans- port/network signature (we use the terms ood w sig and ood w/o sig), reection and cross-re attacks. All are novel contributions, made possible by the SENSS APIs (Table 4.2). Detection of DDoS attacks and identication of the attack signature is out of scope of our research. The victim can use existing tools, such as AMON [103], Packetscore [110], Bro [163] or Arbor APS [148]. Floods w sig. If the SENSS client has a network/transport signature (e.g., the target destination IP and port number) to separate legitimate from attack trac, it uses SENSS to send trac_lter messages containing the signature and the target prex to SENSS servers. Figure 4.4a illustrates handling of oods with signature. To preserve budget, the client can install lters only on some servers that are likely to carry most of the attack, e.g., Tier 1 and Tier 2 ISPs, and install them one by one, until mitigation is achieved. Floods w/o sig. When DDoS attack trac is very similar to the legitimate trac or very diverse or spoofed, the victim cannot derive a suciently specic TCP/IP header signature, to separate the attack from the legitimate trac. Using SENSS the client can observe the dierences in the geographical distri- bution of the victim’s inbound trac prior and during the attack. Such dierences usually exist, because both the legitimate and the attack trac’s distribution over source networks are heavy-tailed. This can be used for location-based ltering, where the client identies ISP-ISP links (tags) where lter placement would minimize collateral damage. During normal operation, the client periodically issuestrac_queries, and extracts the tags and the volume of trac for the ISP-ISP links –pre i . Letpre T be the total inbound trac prior to the attack. During attack,trac_queries are repeated, to learn the tags and volume of links that now carry a mix of attack and legitimate trac –post i . Comparingpre i andpost i , our program iden- ties candidates for lter placement (i) as those tags wherepost i is large butpre i was small or zero. The 80 V A B C D E F G H I J K L S3 S4 S5 S6 1 40 2 8 t1 t2 t3 t6 t5 t7 V to A: filter(dst_IP=v1 & dst_port=25, *, 5min, start) V to G: filter(dst_IP=v1 & dst_port=25, *, 5min, start) (a) Flood w sig V to A, D, G, J, L: traffic_query({dst_IP=v1,dir=IN}, 10s, start) A to V: t5 (1) D to V: t6 (1) L to V: t2 (2) J to V: t1 (2) V A B C D E F G H I J K L S3 S4 S5 S6 1 40 2 8 t1 t2 t3 t6 t5 t7 Before attack: During attack: V to A, D, G, J, L: traffic_query({dst_IP=v1, dir=IN}, 10s, start) A to V: {t5, IN, 1} {t7, IN, 40} D to V: {t6, IN, 1} G to V: {t3, IN, 8} L to V: {t2, IN, 2} J to V: {t1, IN, 2} V to A: filter(dst_IP=v1, t7, 5min, start) V to G: filter(dst_IP=v1, t3, 5min, start) (b) Flood w/o sig V A B C D E F G H I J K L S1 S2 S4 S5 S6 100 10 2 100 8 5000 400 500 S4 S3 V NATs its traffic to dst_port 53 to use IP vn and range of ports [a,b] V to A-L: allow(dst_IP=vn & src_port=53 & dst_port=[a,b], *, 5min, start) V to A-L: filter(dst_IP=V & src_port=53, *, 5min, start) (c) Reector V A B C D E F G H I J K L S1 S2 S3 S4 S5 S6 100 50 1 10 50 10 D1 D2 D3 t1 t2 t3 t4 t5 t6 t7 t8 t10 t11 t12 V to A, D, G, J, L: route_query(dst_IP=v1) A to V: V D to V: CBAV G to V: FEV L to V: KJEV J to V: EV CPPS={DCBAV, GFEV, LKJEV} V to A, D, G, J, L: demote(dst_IP=v1, seg={AV} D starts using path HGFEV Control-path discovery Bottleneck detection V to A, D, G, J, L: traffic_query({dst_IP=v1, dir={IN,OUT}}, 10s, start) A to V: {all, IN, 100} {all, OUT, 50} {all, SELF, 0} D to V: {all, IN, 100} {all, OUT, 100} {all, SELF, 0} J to V: {all,IN, 1} {all,OUT, 1} {all, SELF, 0} L to V: {all, IN, 1} {all, OUT, 1} {all, SELF, 0} Mitigation (d) Cross-re Figure 4.4: Illustration of DDoS attack handling with SENSS. Yellow nodes are victims, blue are legitimate clients and red are attackers. White nodes are ISPs that do not deploy SENSS, and grey are ISPs that deploy SENSS. Red and blue lines represent attack and legitimate trac, respectively. Grey lines represent replies to spoofed requests. Numbers in the same color above the lines represent volume in Gbps. Black tags on links are ISP-specic, anonymized tags that will be used in some replies by SENSS. We show only relevant elds in SENSS messages, shown as text above the diagram. 81 program uses a parameter2 (0; 1) to crudely control anticipated collateral damage. It orders tags by theirpre i values, in an increasing order, and selects rstN so thatpre i1 +pre i2 +::: +pre iN pre T . Figure 4.4b illustrates handling of oods without signature, with tag names starting with letter “t” (anonymized) and reported trac volume shown in parentheses. We highlight the tags that are candidates for ltering, assuming = 0:2. This example also illustrates how the client can patch together a partial view of the DDoS tree even in sparse SENSS deployment. Reector attacks. Reector attacks are challenging to handle, because the victim wants to receive replies to its legitimate service requests, and must lter replies to spoofed trac, but these two kinds of replies are very similar at the network level (e.g., in Figure 4.4c the victim wants to receive the blue trac and lter the grey). We handle reector attacks by the victim network “marking” all its query trac, by NATing all its legitimate requests for targeted service S, using a small range of port numbers R=[a, b]. NATing can be done at the border router of the victim’s network. NATing creates an articial TCP/IP signature, which persist in replies. We can use this signature to lter out replies caused by attack trac, because these replies will not go to the NAT IP address nor to the port numbers in the range [a,b]. The SENSS client installs two lters at each selected SENSS server, which will be applied in this order: (1) to allow all trac to the NAT on ports in range [a,b], from service port for S, (2) to deny all other trac to victim’s prexes coming from service port for S. This surgically removes all of the attack and has no collateral damage. Range [a,b] must be chosen so that it is small enough to make guessing hard, and large enough to accommodate regular request trac. The client further must frequently change the range [a,b] to avoid guessing and DNS hijacking attacks. We analyzed trac from public traces (MAWI [82]) and found that 99.9% of networks could keep around 300 ports in the range [a,b], and change them each 10 seconds to satisfy these conditions. Figure 4.4c illustrates handling of reector attacks. 82 Cross-re. The cross-re attack [106] creates congestion at an ISP upstream from the victim. For example, in the Figure 4.4d, attack networks S1, S2, S4 and S6 send trac to D1, D2 and D3, and the congestion occurs at A, causing V’s inbound trac drops. SENSS client can: (1) identify AS path segments that host the bottleneck and (2) mitigate the attack by re-routing around the bottleneck. Toidentify the bottleneck AS path segment the client rst issuesroute_- queries to select SENSS servers, and extracts the AS paths from the replies, forming the control plane path set – CPPS. The client then traverses each path in the set issuing a trac_query for SENSS server in each AS (if such server exists) on the path. If a SENSS ISP has multiple ASes, the server replies separately for each AS specied in the query. The client compares the outgoing trac from the upstream reports with the incoming trac from the downstream reports. It identies the bottleneck segment as the segment where the upstream AS reports sending more trac than was received by the downstream AS. The client mitigates the attack by issuing demote messages containing the bottleneck link to select SENSS servers. To be robust against lying servers, the client includes the reporting AS in the bottleneck segment. Figure 4.4d illustrates handling of cross-re attacks. We have highlighted the identied bottleneck, which will be demoted. Query replies have aggregated trac in reports to protect ISP’s load balancing decisions. 4.3.5 SecurityandRobustness In this Section we detail how we have secured SENSS against direct attacks and misuse. Securingcommunication: SENSS only allows the victim to issue messages about its own prexes. The proof of prex ownership is publicly veriable, unforgeable information, which binds the SENSS client’s public key with the list of its prexes. The client includes its prex-ownership certicate with each request. Upon successful certicate verication, the ISPs cache the {public key, prexes owned} information for some short time. 83 We can create proof of prex ownership by using RPKI ROA (Route Origin Authorization) certicates. SENSS ISPs deploy RPKI certicate verication. Using RPKI certicates enables SENSS servers to verify requests from remote clients, with whom they have no prior trust. While a client would still have to establish a payment mechanism to pay for services, this can be automated using e-commerce solutions. RPKI is today deployed in a limited fashion, mostly because networks see no special benet of RPKI in sparse deployment. If SENSS used RPKI to very prex ownership, this may provide added incentive for deployment. Today’s increasing deployment of MANRS [133] also motivates larger deployment of RPKI. As an alternative to RPKI certicates, which contain much additional information not needed by SENSS, we could design new certicates, which bind a public key with prexes owned. These certicated would be issued by the same entities that today issue RPKI ROA certicates, i.e., authorities that have assigned a given address space to the victim. The communications between a SENSS client and a SENSS server are secured using TLS [63], and occur via HTTPS. The victim uses the public key from the proof of prex ownership in the TLS key exchange process. If the private key corresponding to the proof of prex ownership gets compromised, the attacker could control all of the victim’s inbound trac. To reduce the risk of this, the SENSS client would issue the abort message to all SENSS servers as soon as it becomes aware of the compromise. This message, when successfully authenticated, purges all trac/route rules for the given prex at a given SENSS server, and the server removes all corresponding rules from the routers. The SENSS server also blacklists the public key, which it used to authenticate the message. This stops the use of the stolen key. The SENSS client can access SENSS servers again when it acquires a new certicate. The reverse scenario, where the attacker issues the abort message using a stolen private key, gives no advantage to attacker. It cuts the attacker o SENSS, while the client can still access it using its new certicate. 84 Robustness and SENSS proxy: During attacks, the victim’s network may become unable to receive replies from SENSS servers. The victim can outsource its decision making power, along with its customized mitigation programs, to a SENSS proxy — a machine in a dierent ISP, e.g., a public cloud, that will act as SENSS client. The victim may set up one or several proxies, prior to any attack, as a backup service. The victim generates a prex-ownership certicate for the proxy using its private key, binding the proxy’s public key with one or more of the victim’s prexes. The proxy monitors the victim’s service availability, e.g. by periodically issuing requests for some public service oered by the victim. When the availability declines, the proxy starts attack mitigation. If the victim has information about the attack (e.g., type, TCP/IP header signature) it can communicate it to proxy using one-way messages, e.g., using DOTS [59] or custom UDP. These messages carry a unique ID to avoid replay, and are encrypted by a key shared between the victim and the proxy. The proxy waits for the signature for some limited time. If the signature is received, the proxy activates the custom program for handling ood with signature. Otherwise, it activates the the custom program for handling ood without signature. The proxy includes both its and the victim’s original certicate in the SENSS messages it sends. The SENSS server validates the certicates by validating the entire certicate chain. Some victims may not be suciently technically savvy to detect attacks or make mitigation decisions. These victims can ooad their attack mitigation to their rst-hop ISP or to a cloud-based DDoS defense, by creating a SENSS proxy there. An attacker could target a SENSS server to disable its operation. An ISP can replicate SENSS server functionality for robustness, just as it is done today with other services. HandlingMisbehavior: SENSS clients have low incentive to misbehave. Clients are unlikely to send excessive requests to the server because they need to pay for each request. SENSS security mechanisms further ensure that client actions only aect trac and routes to their own prexes. However, a SENSS ISP cannot check if a victim 85 is under attack, and it does not need to. Any network can use SENSS to require its trac and routes to be handled in certain way by upstream ISPs, even when it is not under attack. It is up to ISPs to set the pricing scheme in such a way to fully recoup their costs of running SENSS, and to discourage excessive messages. SENSS servers could lie about their observations or fail to implement control actions, for which they have charged the victim. In replies to trac_queries a SENSS server may claim that it sends or receives more or less trac than it does, and it may report a fake distribution of trac over real or fake tags. In replies to route_queries a SENSS ISP may make up AS path segments. Table 4.3 lists all the possible ways a server may lie in its reports (column 2), and the wrong decisions the victim may make (column 3). The overall eect that a lying server has on SENSS operation falls into only two categories: Legacy or Dropper. A Legacy liar has no eect on victim’s trac, but the victim pays for its reports. For example, if a SENSS server lies about its trac to make it smaller, its eect is that of a legacy ISP. Legacy liars prolong the attack mitigation and make it more costly for the victim, but they cannot make the victim drop legitimate trac, postpone attack mitigation indenitely or inuence victim’s actions at other ISPs. A Dropper liar can drop some victim’s trac due to its lying, e.g., through the victim installing unnecessary lters at the lyingISP. For example, if a SENSS server reports a higher trac on its links, it may create a dropper eect. Dropper liars are already on the data path and can drop trac even without SENSS, so SENSS does not make the situation any worse. A server may fail to render the requested services, but still charge the victim for them, thus increasing own prots. A server could also extort the victim by dropping its trac and then charging for diagnosis and mitigation. Both of these attacks are made possible by SENSS, because it allows the ISPs to charge victims when handling SENSS requests. While we cannot prevent these attacks, we sketch here how a victim can build a reputation score for each server, and use it to avoid underperforming or extortionist servers. The victim can monitor the eect of each control message by measuring the trac it receives after the message 86 Attack Message(Lie) Action,eectatliar Flood w and w/o sig TR (IN< Actual) No lter, Legacy TR (IN> Actual) Filter, Dropper Reector None Cross-re TR (OUT = IN) None, Legacy Cross-re TR (OUT> Actual) Demote larger seg., Legacy RR (fake seg.) Demote fake seg., Legacy TR (fake seg.) Demote fake seq., Legacy Table 4.3: Lying ISP scenarios and their eect (TR: trac query reply, RR: route query reply) was accepted and processed by a SENSS server. Control messages, which fail to reduce attack trac would indicate underperforming servers. The victim can use each instance of underperformance to internally assign some bad reputation points to the given SENSS server. After a while these would accrue, allowing the victim to identify and avoid such servers in the future. Similarly, the victim can detect Dropper ISPs by running our client program for cross-re handling. If a given AS is a part of the bottleneck segment, the victim can assign some bad reputation points to this AS. When the reputation declines signicantly, the victim can conclude that that AS is a Dropper ISP and the victim can use SENSS to demote routes containing this AS. 4.4 Evaluation In this Section we rst evaluate SENSS’s eectiveness in sparse deployment, using a simulation. We use an AS-level simulator of the Internet, developed in Perl. We rst illustrate how SENSS would help in an attack scenario similar to the 2016 attack on Dyn [230]. We then evaluate SENSS’s eectiveness in sparse deployment and show that: (1) direct customers of SENSS ISPs have immediate benets and are fully protected against direct oods and reector attacks – this is an excellent deployment incentive for ISPs, (2) strategic deployment at 0.1–0.8% of all ISPs (0.3–3% of transit ISPs) protects everyone against 90% of oods, and (3) SENSS outperforms cloud-based DDoS defenses, saving 2–4 times more bandwidth. 87 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS uni-dir-single real-dir-single uni-dir-multi real-dir-multi uni-remote real-remote (a) Flood w sig and reector – top 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS uni-dir-single real-dir-single uni-dir-multi real-dir-multi uni-remote real-remote (b) Flood w/o sig – top 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS uni-all real-all (c) Cross-re – top 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS dir-single dir-multi remote (d) Flood w sig and reector – ran- dom 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS dir-single dir-multi remote (e) Flood w/o sig – random 0 20 40 60 80 100 0.01 0.1 1 10 % attack fltered % transit ASes deploying SENSS all (f) Cross-re – random Figure 4.5: DDoS attack ltered, versus the percentage of transit ASes deploying SENSS, given the top and the random deployment strategy. We next evaluate SENSS’s response speed, scalability and overhead, using a SENSS prototype on the DeterLab testbed [17], in the 186-switch, Cogent topology from TopologyZoo. SENSS’s message processing delay scales linearly with the number of switches and concurrent requests and it is 0.05–0.25 seconds per request. In this section we use AS and ISP terms interchangeably, for simplicity. 4.4.1 EvaluationMethodology Simulation methodology. For our eectiveness tests, we infer AS-level topology and routing from CAIDA’s [33] AS relationships dataset, from May 1st, 2017. There are 57,552 ASes, 114,018 customer- provider links and 133,795 peer-to-peer links. Routing is inferred using the no-valley, customer-prefer approach [72]. In each attack instance we deploy SENSS on some ASes, following one chosen deployment strategy, and we simulate legitimate and attack trac ows as aggregates on inter-AS links. We simulate just the 88 aggregate rate of each ow and not its packets, nor details (e.g., source and destination addresses, ports or transport protocol). When we simulate ood w sig, we assume that attack trac is so dierent from the legitimate trac, that it is possible to devise a header-level signature for its ltering. Installing a lter on some SENSS ISP in the simulation of ood w sig only removes attack trac going to the victim. When we simulate ood w/o sig, we assume that attack and legitimate trac are so similar that no header level signature is possible. Installing a lter on some SENSS ISP in the simulation of ood w/o sig removes both legitimate and attack trac going to the victim. We measure the amount of legitimate and attack trac dropped and the bandwidth consumed by the attack on inter-AS links. We perform 1,000 random trials for each data point and show the median (lines) and 25th and 75th percentile (errorbars). Emulation methodology. For our evaluation of response speed, scalability and overhead we have developed a SENSS prototype, including client and server functionalities, and the client customization programs described in this chapter. We deployed our prototype on the DeterLab testbed [17] and measured the time it takes to serve a SENSS request under many concurrent requests. We replicated the Cogent topology (186 nodes) from the Topology Zoo[4]. On each node we ran Quagga as router software and Open vSwitch as the SDN software. We used RYU as the SDN controller. There was one SENSS server in the topology. For control plane requests, it connected to the Quagga software on switches via telnet, and for data plane requests it sent OpenFlow messages. We emulated large, 100 ms, end to end propagation delays between the SENSS server and each victim. On the Cogent topology we use gravity model [177] to emulate legitimate and attack trac. We gener- ate legitimate trac using iPerf (TCP mode) and attack trac using a custom tool to generate UDP ood. We generate sucient legitimate trac to ll a 1 Gbit link from our victim to the ISP and then launch the attack. 89 4.4.2 2016attackonDyn We reproduce the attack on Dyn in 2016 by reproducing locations of Mirai bots, using bot IP addresses from [145]. We divide 1.2 Tbps equally among bots, then allocate each bot to its AS, based on the bot’s IP address. We use 7015 and 13977 as victim ASes for Dyn. We then strategically deploy SENSS on only four ASes, which are close to Dyn and on path between most Mirai bots and Dyn. These are AS 174 (Cogent), 3356 (Level 3), 6461 (Zayo Bandwidth) and 7922 (Comcast). With that deployment, and if attack signature were possible, SENSS would lter 100% of the attack trac with only four ltering rules (one per ISP). Cogent lters 56% of the attack, and Zayo lters 40%, so even two SENSS ISPs would lter almost all the attack. We also consider the case when attack signature is not possible. We simulate legitimate trac to Dyn by assuming that it ows mostly from large US connectivity providers. We extract the top 10 providers from a 2017 survey [18], and simulate legitimate trac proportionally to each provider’s number of customers. Using=5%, SENSS lters 99% of trac to AS 13977, and 90% of trac to AS 7015. This requires around a 1,000 ltering rules, with 60% being at Cogent and 40% being at Level 3, one rule per POP. 4.4.3 EectivenessinSparseDeployment We now investigate how deployment strategy and the number of deployment points inuence eec- tiveness. We investigate two deployment strategies. In top strategy, we deploy SENSS at the top N = (1:::10; 000) ASes, ordered in decreasing order by their customer cone size. Inrandomstrategy, we deploy SENSS at the randomN = (1:::10; 000) ASes. In both cases, we only consider deployment at ASes that have at least one customer link, i.e. transit ASes. There are 13,123 such ASes in our topology – 23% of all ASes. Uniformtrac: Since we cannot anticipate where the attackers, legitimate clients and the victim may reside, we deploy them at random. We rst randomly select a victim, and then randomly select 1,000 ASes 90 to host attackers and the additional 1,000 ASes to host legitimate clients. We distribute attack/legitimate trac equally among attackers/legitimate clients. Realistic trac: We randomly select a victim but distribute attackers at Mirai bot locations, and legitimate clients at large US residential ISPs, as we have done in Section 4.4.2. We show results only for the top deployment, because the random deployment results do not change with trac distribution. We show the median percentage of attack trac ltered, for ood w sig and reector attacks (Fig 4.5a and 4.5d), ood w/o sig (Fig 4.5b and 4.5e), and for cross-re (Fig 4.5c and 4.5f). For ood w/o sig we use = 5%. Other values of lower SENSS’s eectiveness for small number of deployment points (up to 1,000 ASes) by up to 50%, and we omit them for space reasons. They have no eect for deployments higher than 1,000. For ood w and w/o sig and for reector attacks we show separately the benets to direct customers of the SENSS-deploying ISP, and to remote victims that may request SENSS services when under attack. Cross-re creates disturbance far from the attack’s victim, and we show benet to all ASes. Floodwsigandreectorattacks. In case of these attacks, our results are consistent for both uniform and realistic trac patterns. Direct, single-homed customers of SENSS ISPs receive almost total protection at all times, both in top and in random deployments, and under both uniform and realistic trac. Direct, multi-homed customers receive lower protection than single-homed, because some of their trac traverses non-SENSS ISPs. During an attack, a multi-homed victim could temporarily become single- homed, to increase its protection. It could do so by selectively announcing its prexes only to SENSS ISPs. Remote ASes that request SENSS services (aka “remote customers”) receive protection only after a certain number of deployment points is reached. This is wheretop deployment is vastly superior torandom. Deployment at only 0.7% top ASes achieves 90% protection for remote customers. On the other hand, 64% of randomly chosen ASes must deploy SENSS to achieve 90% protection for remote customers. Floodw/osignature. In this simulation we assume that SENSS deploys coarse-grained rules, ltering all trac owing to the attack’s victim at select lter locations. For uniform trac pattern, SENSS needs 91 �� (a) Flood w sig – top �� (b) Flood w sig – random Figure 4.6: Comparison of SENSS versus several cloud-based DDoS defenses, with regard to bandwidth consumed by attack. large deployment for eective attack mitigation. This is because SENSS must lter close to attack sources to meet the requirement for collateral damage (controlled by = 5%). At small deployment, SENSS either misses some attack, or its lters would cause too high collateral damage. Deployment at 1.5% top ASes is needed to achieve more than 90% protection for direct, single-homed customers and deployment at 3.8% top ASes is needed for 90% protection for direct, multi-homed and remote customers. Random deployment does much worse against ood w/o sig, where 70% deployment is needed for 90% protection for everyone. For realistic trac patterns, SENSS is very eective for direct, single-homed customers, even with 1–10 deployment points (0.01–0.1 on x-axis). This is similar to our result for the 2016 Dyn attack. Cross-reattacks. Results are similar for both trac patterns. Top deployment again vastly outper- formsrandom deployment. Deployment at the top 1% of ASes can mitigate 90% of cross-re attacks, while 66% of randomly selected ASes must deploy SENSS to achieve the same eectiveness. 4.4.4 ComparisonofSENSSandCloudDefenses In this section we compare SENSS’s performance to that of clouds, with regard to bandwidth saved during an attack. We assume the same kind of ltering deployed by SENSS and clouds, to isolate the eect of 92 Cloud ASes Providers Peers Avg. AS-pathlength CloudFlare 7 41 185 3.2 Google 20 12 187 2.8 Akamai 38 374 199 3.5 Incapsula 1 69 130 3 Zenedge 1 14 0 4 Table 4.4: Providers and peers of select clouds, which we use in our evaluation. deployment points. This evaluation shows that on-path deployment is superior to the case when trac is diverted to a cloud. We rst calculate the amount of bandwidth consumed by attack trac on inter-AS links, as follows. For each of our eectiveness scenarios (Section 4.4.3) whenever an attack ow crosses an inter-AS link we add its volume to the total consumption. We assume a perfect defense, which drops all attack trac when it reaches the defense. The dierence between bandwidth consumption without and with defense becomes the saved bandwidth. We report it as percentage of the attack bandwidth when there is no defense. An ideal defense would save close to 100% of the bandwidth because it would be deployed close to the sources. We select ve clouds to compare to, which currently oer DDoS defense – CloudFlare, Google, Akamai, Incapsula and Zenedge. While we do not know their peering agreements, we can easily nd out their AS ownership from public records. We then obtain peering for those ASes from the CAIDA’s AS-level topology. When a cloud owns multiple ASes, we assume that all such ASes deploy the defense. Table 4.4 shows the number of ASes each cloud owns in our topology, number of providers and peers in our AS topology, and the average AS path length from all other ASes to the cloud. Figure 4.6a shows the saved bandwidth under top SENSS deployment (median shown with line and 25% and 75% with errorbars), and Figure 4.6b shows it under random SENSS deployment, for ood w sig attack. In both Figures we also show with colored horizontal bars 25% and 75% of bandwidth consumption when the victim is defended by clouds, and the lines show the median of the same measure. Intuitively, 93 0 200 400 600 800 1000 1200 5 10 15 20 25 30 35 Speed in Mbps Time in Seconds Legitimate Traffic Measured Attack Traffic Measured Figure 4.7: SENSS client using our client program to mitigate oods w/o signature attack. bandwidth savings will be largest when attack is ltered close to its sources. There are signicant dier- ences among cloud defenses – ranging from 13% bandwidth saved by Zenedge to 38–46% by Google. These dierences occur because some clouds have long AS paths (e.g., Zenedge), i.e., they are far from sources, while others (e,g,. Google) have short paths. SENSS outperforms all clouds after 0.4% of top transit ASes (52 ASes, on par with Google’s and Akamai’s AS count) or after 15% of random transit ASes deploy SENSS. This is because SENSS’s on-path defense stops the attack closer to its sources, than when attack trac must be diverted to clouds. 4.4.5 Delay,TracandMessageCost All communication between a SENSS client and a SENSS server occurs in one session, over SSL. It takes two round-trip times for SSL establishment. After this, the client would send a query and wait for a reply. Finally it may send a control message to the server to mitigate attacks. Each of the four attack types we studied require 1–3 messages per SENSS ISP for mitigation. The client can strike the balance between achieving a fast response (ask all SENSS servers for help) and saving money (ask servers one by one). The client can rst communicate with Tier-1 and Tier-2 ASes simultaneously, and then switch to iterative communications. This yields on the average 10-second delay, and 300–400 messages for full mitigation. When using SENSS to mitigate oods w/o sig, the victim must periodically issue trac queries to learn legitimate trac distribution, and identify links that carry a lot of legitimate trac. During attacks the 94 SENSS client uses its most recent observation to estimate collateral damage for a given lter deployment. More frequent queries increase message cost but may reduce collateral damage if trac uctuates a lot. We investigated the impact of query period on collateral damage, by replacing legitimate trac in our eectiveness experiments by trac volumes and destinations obtained from 24 hours of trac logs from a large US CDN. We calculated trac volumes per each minute in the logs and used earlier observations to make SENSS decisions about lter locations, then used later observations to estimate collateral damage. We summarize results of this experiment: (1) observation periods of up to 12 h only slightly increase collateral damage over the ideal case, when we observe immediately prior to attack, (2) ltering at large ASes has higher uctuation of collateral damage, due to higher trac aggregation, than ltering at smaller ASes. 4.4.6 ScalabilitywithinanISP SENSS functionalities in switches are implemented on the fast path, and incur no per-packet overhead. Further, each SENSS request results in one rule per switch. In our emulation experiments it took only 0:15 sec to handle a single trac_query and 0:26 sec to handle a route demote. This includes the propagation delay between the victim and the SENSS server (0:05 sec), RPKI validation (0:02 sec), and SENSS processing of the query (0:03 sec). When handling 100 concurrent requests, it took 4:32 s to fully serve trac_- queries, and 24:95 s to serve route_queries followed by demote messages. The delay mostly comes from the concurrent telnet requests to switches, and can be further reduced if we parallelize this communication. In Figure 4.7 we illustrate one experiment when ood w/o sig attack is handled by SENSS. The SENSS client at the victim uses our client program from Section 4.3.4 for handling oods w/o sig. This includes two types of SENSS messages: a trac_query and a lter. The attack is fully handled within 7 seconds. 95 4.5 Conclusion Volumetric DDoS attacks cannot be handled by the victim, because they usually create congestion up- stream from the victim. We have proposed SENSS – a framework for collaborative diagnosis and mitiga- tion of volumetric attacks. SENSS’s simple but powerful interfaces enable victim-customized solutions to several DDoS variants. SENSS mitigates attacks instead of withstanding them. SENSS is implementable in today’s ISPs with SDN, it is very eective in sparse deployment, much faster than manual inter-AS collaboration, and has small message overhead. We hope that these good features will encourage its wide adoption. 96 Chapter5 BLAG:ImprovingtheAccuracyofBlacklists 5.1 Introduction IP blacklists (“blacklists” for short), which contain identities of prior known oenders, are usually used to aid more sophisticated defenses, such as spam lters or security information and event management (SIEM) systems, in identifying trac that warrants further analysis [52]. Blacklists do not have to be very accurate to be used in thisadvisory mode, since they merely aid another, more sophisticated system, which decides trac’s destiny. In fact, prior works [151, 116, 123, 165, 192] show that blacklists are often not very accurate or useful. They do not misclassify many legitimate sources, but they miss the majority of malicious sources. What if a network suers a novel attack, which bypasses its sophisticated defenses? Or what if the attack has such a large volume that the more sophisticated system cannot keep up? In these situations, networks usually resort to crude defenses, such as rate-limiting trac or ltering all incoming trac to a given port number or a destination [141, 185, 113, 64]. IP blacklists could be used as emergency response, in case of a novel or large-scale attacks, to lter attacks from prior known malicious sources, and act as the rst layer of defense. For example, an email server hit by a heavy phishing campaign, whose signature does not yet exist in the server’s spam lters, could use blacklists to drop emails sent by prior known malicious sources. Or a network under a distributed 97 denial-of-service attack could use blacklists to drop all trac sent by prior known malicious sources, to lessen its load. Blacklists are easy to implement since only the source IP address of incoming trac is checked, which can be cheap at line speed. In this emergency mode, blacklists would have to be very accurate, since they would actively drop trac. This means that blacklists should be able to identify the majority of attack sources while keeping misclassication of legitimate sources low. This chapter explores how to generate suciently accurate blacklists for emergency use. Individual blacklists today suer from several drawbacks that limit their accuracy in malicious source identication. Firstly, individual blacklists miss many malicious sources. This eect may come from their limited vantage points – e.g., blacklist maintainers may have honeypots in the United States but not in India – or from their limited scope – e.g., blacklists are created for specic attack classes, like spam. On the other hand, compromised devices are constantly being drafted into botnets and misused for dierent attacks, such as sending spam one day and participating in denial-of-service attacks on a dierent day. Aggregating blacklists from dierent maintainers and across various attack types can improve the accuracy of malicious source identication over any individual blacklist. Secondly, blacklists are snapshots of malicious sources at a given time. Attackers are known to evade blacklisting by staying dormant, only to resume malicious activities later [151]. Historical blacklist data can provide additional intelligence on active past oenders that are likely to re-oend in the future. Finally, malicious sources have historically been known to concentrate in a few mismanaged net- works [240]. Thus,expandingcertainblacklistedIPaddressesintoIPprexes could improve the accuracy of malicious source identication. But the aggregation of data from multiple lists and past periods, and expansion of addresses into pre- xes may greatly increase misclassications of legitimate trac if applied naïvely. We propose BLAG (BLocklist AGgregator), which performs smart aggregation of blacklist data, and tailors this data to the 98 0 50 100 150 0 10 20 30 40 50 60 routable /24 unique routable /24 (#) of blacklists (%) of blacklisted routable /24 (a) Blacklist coverage in routable /24 prexes that are blacklisted. Malware Reputation Spam Attack Malware Reputation Spam Attack Blacklist category overlapping percentage 100% 14.63% 13.2% 10.74% 1.67% 100% 71.6% 9.87% 0.15% 7.08% 100% 0.37% 9.56% 77.09% 29.56% 100% (b) Blacklist overlap across dier- ent blacklist categories. 50 100 150 200 250 0 20 40 60 80 100 Number of blacklisted address in the same /24 prefix (%) of /24 (c) IP addresses blacklisted in the same /24 prex. Figure 5.1: Blacklists have low coverage and tend to overlap with other categories of blacklists. There- fore, aggregating blacklists of dierent types can improve coverage. Also blacklisted addresses tend to be collocated, thus expanding IP addresses to prexes may further improve malicious source identication. customer network ∗ . BLAG’s blacklists have a much higher accuracy of malicious source identication and they keep collateral damage to legitimate sources low. BLAG overcomes problems of existing blacklists as follows: 1. Aggregation: BLAG aggregates IP addresses from 157 blacklists of dierent attack types to improve coverage. BLAG also includes historical listings from these blacklists and assigns relevance score that determines which historical IP addresses are more likely to re-oend. 2. Estimate misclassications: BLAG uses a recommendation system, together with a sample of sources that send inbound trac to a customer network to tailor its blacklist to this customer. BLAG identies portions of individual blacklists that may lead to the legitimate source misclassications and prunes them out. Other portions are aggregated into the master blacklist for this customer. 3. Selectiveexpansion: BLAG selects a subset of IP addresses on the master blacklist to expand into /24 prexes. Only those IP addresses are expanded, where the expansion is not likely to increase legitimate source misclassications for the given customer. ∗ A customer network is a network, which is deploying BLAG for its own emergency response. 99 0 50 100 150 0 20 40 60 80 100 (#) of blacklists (%) of reported addresses that reoffend in the same blacklist (a) IP addresses in blacklists that re-oend. 0 50 100 150 200 0 20 40 60 80 100 Days between reoffense (%) of reoffenses (b) Days between reoense. 0 10 20 0 20 40 60 0 50 100 150 0 3 6 9 Naïve aggregation Naïve aggregation expansion (#) of blacklist (%) of misclassifications Scenario 1 Scenario 2 Scenario 3 (c) isclassication observed in in- dividual blacklists across three dif- ferent legitimate addresses. Figure 5.2: Blacklisted addresses re-oend quickly. A possible solution is to expand addresses to prexes, but this causes misclassication of legitimate sources. We present three real-world deployment scenarios for BLAG † covering dierent attacks, where cus- tomer networks can reduce the burden on resource-intensive technologies. BLAG improves existing black- listing approaches by increasing recall (malicious source identication) from 0.1–18.4% to 6.4–69.7%, while maintaining high specicity (legitimate source identication) of 95–99.5%. BLAG also outperforms PRESTA [222], a proposed blacklist aggregation approach by achieving 11.5–84.4% higher specicity, with comparable re- call. BLAG also improves the detection delay, discovering malicious sources 8.8–13.4 days faster than competing approaches. 5.2 ProblemsWithCurrentBlacklists In this section, we illustrate the drawbacks that blacklists have, that limit their usefulness in emergency scenarios. We then discuss possible solutions to improve blacklisting and some challenges that we must address. We rst show that blacklists generally have low coverage and blacklists of dierent attack types tend to overlap with one another (Section 5.2.1). This motivates the need for aggregating multiple blacklists of dierent attack types to improve coverage of malicious sources. We further show IP addresses that were blacklisted in the past get blacklisted again, and sometimes soon after they were removed from a blacklist † Blacklist dataset and code to deploy BLAG can be found athttps://steel:isi:edu/Projects/BLAG/ 100 (Section 5.2.2). This motivates the need for inclusion of historical blacklist data to further improve coverage of malicious sources. Finally, we show that blacklisted IP addresses are often collocated within the same /24 prex, thus, expanding some IP addresses to prexes can improve attack detection (Section 5.2.3). In all cases, some addresses that appear on blacklists may be “wrongly accused”, i.e., they may be misclassied, legitimate sources or they may be previously malicious sources, which were since cleaned. We illustrate this in Section 5.2.4 to motivate the need for smart, selective aggregation and expansion only of those portions of blacklists that are unlikely to contain legitimate sources. In this section, we leverage our Blacklist dataset, whose details are given in Section 5.4.1. It consists of 157 publicly available, popular blacklists. We collected their data regularly for 11 months in 2016. We have roughly categorized each blacklist into four categories, based on the type of malicious activities they capture. Spam blacklists monitor email spam or emails that contain malicious content and Malware blacklists monitor IP addresses that host or distribute malware. Attack blacklists, on the other hand, contain IP addresses that initiate DDoS attacks, bruteforce or attacks on specic protocols such as VoIP or SSH. Finally,Reputation blacklists list IP addresses that have a low reputation, e.g., because they send unwanted trac. The algorithm to calculate this reputation is known only to blacklist maintainers. 5.2.1 FragmentedInformation Monitoring the entire Internet accurately is impossible. By necessity, each blacklist will gather data from some limited area of the Internet, and thus information about malicious sources will be fragmented over many blacklists. Figure 5.1a illustrates the coverage of individual blacklists in our Blacklist dataset and the unique contribution of blacklists over the dataset “Black24,” containing all routable /24 prexes (extracted from Routeviews [178]) that have at least one blacklisted address. On average, a blacklist reports only 3.03% of Black24. Nixspam blacklist [158] has the highest coverage of 60.7% of Black24. Some blacklists also have unique contributions, i.e., they list addresses from prexes that appear on no other blacklist. On 101 average, a blacklist contributes unique addresses that belong to 0.16% of Black24. Nixspam blacklist has the highest unique contribution – 10.9% of Black24. Previous studies observed similarly low coverage [192, 116] of blacklists. Another possible reason for fragmented information is the blacklists’ focus on a specic type of attacks. This is again by necessity as blacklists built from spam lter alerts will only see spam sources, while intrusion detection systems will only see sources of scans and network attacks. Figure 5.1b shows the overlap of blacklist categories on the y-axis with the blacklist categories on the x-axis. On average, blacklist categories have an overlap of 20.4% with other blacklist categories. The highest overlap is seen in the “attack” category (average 38.7% overlap) and the lowest overlap is seen with the “spam” category (average 2.5% overlap). Although our categorization may not be perfect, we observe that IP addresses are reported across dierent types of blacklists. Therefore, aggregating multiple blacklists across dierent attack types can increase blacklist coverage and detect sources of multiple attack types. 5.2.2 Re-oenseIsFrequent Figure 5.2a shows the percentage of blacklisted IP addresses (in any blacklist) that have been removed and then appeared again on the same blacklist. On average, 29.3% of blacklisted IP addresses re-oend. Particularly, in two blacklists,BambenekPushdo andPalevo, all IP addresses blacklisted re-oend. However, these are very small blacklists that have reported only 1 and 12 IP addresses during our monitoring period. Figure 5.2b shows the duration between each oense, that is, the number of days the IP addresses stay dormant before they are blacklisted again. On average, reoenses occur within 9 days and about 91% of reoense occurs within 30 days. This motivates the need for aggregation of historical blacklist information, especially over the recent past, to improve coverage of malicious sources. 102 Relevance Score Calculation 169.231.140.68 193.1.64.8 216.59.16.171 Blacklist 1 Blacklist 2 Blacklist m Blacklist 3 .. 169.231.140.68 193.1.64.5 193.1.64.8 216.59.0.8 243.13.0.23 MB 169.231.140.10 243.13.222.203 193.1.64.5 216.59.0.8 169.231.140.68 193.1.64.8 216.59.16.171 Blacklist 1 Blacklist 2 Blacklist m-1 Blacklist 3 .. 243.13.0.23 MB 169.231.140.10 243.13.222.203 193.1.64.5 216.59.0.8 Recommendation system Evaluate Aggregate Expand Known Legitimate sources (L t r ) Currently and historically listed IP addresses (B) Likely to be a misclassification Unlikely to be a misclassification Prune 193.1.64.0/24 216.59.0.0/24 169.231.140.68 Selective expansion BL 1 BL m .... 0.28 0.1 1 .. .. .. 0.46 .. .. 0.72 0.23 .. .. .. .. 0.32 .. .. 0.58 .. .. 0.15 .. 0.25 0.95 0.87 .. .. .. .. .. .. 0.79 0.87 .. 0.81 0.22 0.4 0.12 0.91 0.6 0.92 0.99 .. .. 0.78 .. .. 0.75 0.3 0.1 .. .. .. 0.5 .. .. 0.7 0.5 .. .. .. .. 0.04 .. .. 0.7 .. .. 0.1 .. 0.1 0.9 0.9 .. .. .. .. .. .. 0.7 1 .. 0.9 ? ? ? 1 ? 1 1 .. .. 0.8 .. .. ? Master blacklist candidates BLAG master blacklist Figure 5.3: BLAG implementation consists of assigning relevance scores to IP addresses from dierent blacklists and creating a new misclassication blacklist (MB). The MB contains all possible IP addresses, but a score is only assigned to those that are misclassications from the known-legitimate sources dataset (L tr ). Then, a recommendation system generates missing scores for IP addresses in the MB. IP addresses that have a score higher than threshold (red blocks inMB) are pruned out and the remaining ones are used for aggregation. These IP addresses will be put on a new blacklist known as “Master blacklist candidates”. Finally, we selectively expand some candidates into /24 prexes, if we estimate that this expansion will not severely increase misclassication. 5.2.3 MaliciousSourcesAreCo-located Prior research has shown that attackers tend to concentrate in a few mismanaged networks [240]. Thus blacklisting an entire network (e.g., one or a set of IP prexes) could improve coverage of malicious sources. Figure 5.1c shows the number of blacklisted IP addresses in the same /24 prex, for all /24 prexes that are present in the blacklist dataset observed during the entire monitoring period. About 57.5% of /24 prexes have at least two blacklisted IP addresses in the same /24 prexes. For about 1.7% of /24 prexes, nearly half the /24 prex (128 IP addresses) are blacklisted. Only a few /24 prexes have (0.007%) all their IP addresses in a /24 prex blacklisted. Identifying prexes that harbor more malicious sources can increase the blacklist’s coverage of malicious sources. 5.2.4 CarefulAggregationandExpansion Not all IP addresses that appear on blacklists are necessarily malicious. Some may have been malicious and got cleaned, which means that the blacklist has stale information. Others may have been misclassied 103 by the blacklisting algorithm. Since blacklists are built using proprietary algorithms, it is impossible to evaluate their accuracy and/or bias. If we were to naïvely aggregate and expand blacklisted addresses, this would amplify bias and mis- classication of legitimate sources. For example, Figure 5.2c illustrates, on the y-axis, the percentage of legitimate addresses in our three Scenario datasets (see Section 5.4 for details on datasets) that appears on a blacklist. These address sets are mostly disjoint and are collected from regular inbound trac in three dierent networks. The x-axis shows dierent blacklists in our Blacklist dataset. The order of blacklists is always the same on the x-axis. For this Figure, we usednaïvelyaggregatedhistoricaldata of each blacklist, i.e, for a given blacklist we produced the list of all addresses that were listed on it up to the time of our Scenario (shown in blue). We make several observations. First, for each Scenario, all misclassications are concentrated on a few blacklists. On average, about 0.14%, 0.17% and 0.016% of legitimate IP addresses are misclassied. Blacklists such as Cruzit [53] and Chaos Reigns [14] have high misclassications of 3.3% and 9.8% respectively. Figure 5.2c also shows that there is no single blacklist that has misclassications across all three scenarios, thus removing certain blacklists will not solve the problem. Instead, we must tailor each blacklist to the customer that is going to use it, to identify and remove portions that may lead to misclassications of the sources of customer’s inbound trac. Naïve expansion of blacklisted addresses into prexes can also lead to misclassications. We take the naïvely aggregated historical data described earlier, expand every address to its /24 prex and show the percentage of misclassication in Figure 5.2c (yellow dotted lines). We see that the percentage of misclassications further increases with naïve expansion. On average, about 0.66%, 6.6% and 1.03% of legitimate addresses are misclassied. Blacklists such as Cleantalk [44] and Chaos Reigns [14] have high misclassication of 67.2% and 22.6% respectively. Although blacklisted addresses are collocated, naïvely expanding them into /24 prexes can increase misclassications. 104 5.3 BLAGDesign We present BLAG’s design in this section and illustrate the system in Figure 5.3. BLAG collects historical data from multiple blacklists (B) and updates this dataset whenever a blacklist is updated by its main- tainers. When a customer wants to use BLAG, our system uses its historical dataset (B) and a sample of the legitimate sources that send inbound trac to the customer network (L tr ) to curate a master blacklist tailored to that customer. BLAG’s goal when producing the master blacklist is to include as much information about malicious sources from (B) as possible while ensuring that very few current (L tr ) or future (L te ) legitimate sources of the customer get blacklisted. To achieve this, we need a relatively accurate list of legitimate sources that communicate with the customer. One way a customer could build this list would be to leverage its existing, more sophisticated defenses. Many networks today run an intrusion detection system, a rewall, and a spam lter. These defenses will drop or quarantine some trac during regular operation. Let us denote the sources of dropped or quarantined trac as (D) and let (A) represent all sources that have recently sent inbound trac to the customer. The customer can create (L tr ) by starting with a set (A) and removing sources that appear in (D). BLAG’s operation proceeds in the following steps: (1)Evaluate the relevance of every address on every blacklist in (B) (Section 5.3.1), taking into account the listing’s age (similar to [222]) and re-oense history. The relevance score is the function of the address’s history on a particular blacklist. (2) Aggregate IP addresses and run the recommendation system using IP addresses in the (L tr ) set. The system predicts relevance scores of IP addresses that are not in (L tr ) but may be among legitimate sources for the customer network in the future (F L te ). This step ends with a set of addresses that are likely legitimate and likely to communicate with the customer network in the future (F). BLAG then uses 105 a threshold-based approach to prune out current and likely misclassications (all addresses from (L tr ) and most addresses from (F)) and the remaining IP addresses form the master blacklist candidates. (3)Selectivelyexpand some candidate IP addresses into prexes to increase malicious source identica- tion. During this expansion we use (L tr ) and (F) sets to estimate the likely increase in misclassication for each candidate if it were to be expanded into an IP prex. Our expansion method is selective because it bal- ances the gain in malicious source identication against potential future misclassications (Section 5.3.3). 5.3.1 RelevanceScores: EvaluatingQuality Historical blacklist data can be a valuable source to detect potential re-oenders. Earlier, we have seen that about 29% of blacklisted IP addresses re-oend and 91% of these reoenses occur within 30 days. We have also seen that blacklists of dierent attack types overlap. Existing work also agrees with our ndings. PRESTA [222], a study on three paid-for blacklists shows that recent oenders are more likely to re-oend. BLAG starts its aggregation by generating a relevance score for each addressa listed in blacklistb2 B based on the formulation used by PRESTA. BLAG denes relevance scorer a;b as: r a;b = 2 l tt out (5.1) wherel is a constant, which we set empirically (discussed in Section 5.7),t out is the most recent de-listing (removal) time ofa at blacklistb andt is the time in days when the score is calculated. The exponential factor ensures that the score decays exponentially over time, giving higher weight to recently blacklisted IP addresses. The relevance score ranges from 0 to 1, and a higher score indicates a higher chance of reoense. If the addressa is currently listed inb, we set its relevance score to 1. 106 0.7 0.6 0.3 ? 0.1 0 0.1 ? 0.2 0.1 0 ? 0.1 0 0 ? 0 0.7 0.4 1 0.27 0.12 0.04 0.11 0.11 0 0.07 0.02 0.21 0.23 0.24 0.17 0.08 0.22 0.08 0.17 0.13 0.25 0.68 0.59 0.29 0.84 0.1 0.17 0.08 0.29 0.19 0.1 0.001 0.17 0.09 0.06 0 0.12 0.61 0.68 0.39 0.99 M N M K K N 128.0.0.1 128.0.0.2 128.0.0.3 128.0.0.4 128.0.0.5 nixspam bambenek_c2 openbl MB nixspam bambenek_c2 openbl MB 128.0.0.1 128.0.0.2 128.0.0.3 128.0.0.4 128.0.0.5 Matrix R Matrix P Matrix Q Matrix R' Likely to be a misclassification Unlikely to be a misclassification Figure 5.4: Latent factorization of the score matrix R, a MN matrix, where M is the number of IP addresses and N is the number of blacklists. The cells indicate relevance scores. IP addresses not listed in a blacklist are assigned a zero score and IP addresses listed in misclassication blacklist (MB) are given a score of 1. Score matrix is factorized into two matrices ofMK andKN, and the cross product results in a new matrixR 0 , which updates the missing scores in MB. 5.3.2 RecommendationSystem: EstimatingFutureMisclassications Ideally, if we knew all legitimate IP addresses in the Internet at any given time, or if we knew which currently blacklisted addresses are no longer malicious, we could prune out misclassications during ag- gregation. However, it is impossible to know this information. At best, a customer network has limited visibility into some set of its known-legitimate sources (L tr ), which have recently communicated with the customer. We leverage this set to predict IP addresses (F L te ) that may be future legitimate trac sources for the customer network, and that would be misclassied in BLAG’s aggregation and expansion steps. When all the relevance scores are calculated, BLAG places them into a score matrix where blacklists from (B) are at the columns and all listed IP addresses (in any blacklist) are at the rows as shown in Figure 5.4 (seeEvaluate step). Each cell in the score matrix holds the relevance scorer a;b for the given row (addressa) and given column (blacklistb). BLAG adds a new, articial blacklist to this matrix, called the “misclassication blacklist” (MB column in Figure 5.4). MB contains all known-legitimate sources from the set (L tr ). BLAG assigns a relevance scorer a;MB of 1 to all IP addressesa listed in the misclassication blacklist. This high score will help us identify likely future misclassications (F). 107 The misclassication blacklist column is sparse because many addresses that exist in (B) do not ap- pear in (L tr ) and we cannot know if they are legitimate or malicious. BLAG lls the empty cells of the misclassication blacklist column by using a recommendation system. Recommendation systems are usually used to predict future product ratings by some users, given a set of past ratings of same or related products, by target users and other similar users. A well-known example is the Netix recommendation system [112], which may recommend a new movieM to userU by relying on theU’s past ratings of movies similar toM, and on ratings that users similar toU have given toM or movies similar toM. In our context, IP addresses are analogous to movies that are being evaluated, and blacklists are analogous to users assigning the rating. We view the relevance score as the rating. Two most commonly used types of recommendation systems are content-based recommendation sys- tem [164] and collaborative ltering [183]. A content-based recommendation system would require an explicit denition of features that a blacklist uses to determine if an IP address should be listed. Such features are hard to obtain since each blacklist maintainer has its private criteria for listing an address. Collaborative ltering infers information about the relationship between a blacklist and an address being listed in a blacklist, by using only the existing relevance scores. That is, it infers the relationship between blacklists and IP addresses, based on when the IP addresses were listed in blacklists and based on similarity in listing dynamics of dierent blacklists. It then uses the inferred relationships to predict relevance scores for missing cells in the score matrix. We use collaborative ltering in BLAG. Figure 5.4 illustrates the recommendation system’s operation for a customer network. LetM andN represent the set of IP addresses and blacklists, respectively. LetR be a score matrix of sizejMxNj which consists of relevance scores quantifying the relevance of an address being listed by a given blacklist. For example, in Figure 5.4, score matrixR consists of four blacklists (M = 4), and ve IP addresses (N = 5). Misclassication blacklist, curated from known-legitimate sources (L tr ) for the customer network, is added as the last column in the matrix, and in this example, 128.0.0.5 (highlighted in red) is present in (L tr ). 108 Blacklisted IP addresses have been present at various times in dierent blacklists, which is reected by the relevance score’s value. Address 128.0.0.1 listed in nixspam blacklist has a high score of 0.7 since it was the most recently listed address. Address 128.0.0.2, on the other hand, has a low score of 0.1 inopenbl blacklist, since it was listed long ago. Finally, address 128.0.0.5, has a score of zero in nixspam blacklist, where it has never been listed and has a score of 1 in MB since it is known to send legitimate trac (L tr ) to the customer network. There are latent (unknown) features of blacklists and IP addresses that lead to an address being listed in a blacklist. Let the number of latent features that inuence relevance scores of IP addresses in blacklists beK (see Section 5.7 for how we choose the value ofK). In this example, we setK = 2. Our goal is to estimate the relevance scores of IP addresses that are not present in MB, by estimating two matricesP (jMxKj) andQ(jNxKj), which are factors of the original matrixR, such that their cross product is approximately equal to known values inR. In other words, matrix factorization is used onR to obtain factor matricesP andQ such that: RPQ T =R 0 (5.2) We obtain the values of latent matricesP andQ using gradient descent [121], which randomly assigns values toP andQ and estimates how dierent the product ofP andQ is from the original score matrix R. We use root mean squared error (RMSE) to estimate the dierence. Gradient descent tries to minimize RMSE iteratively. We discuss in Section 5.7 the number of iterations required to have a small RMSE. After obtaining matricesP andQ, each row inP represents the association strength between IP addresses and latent features K, and each row in Q represents the association strength between blacklists and latent featuresK. To obtain a relevance score for an addressa in misclassication blacklist MB, the dot product of two latent vectors corresponding to addressa and MB is calculated as follows: r a,b =p T a q MB (5.3) 109 Wherep a denes the association strength of addressa with featuresK andq MB denes the association strength of MB with featuresK. Consider IP addresses 128.0.0.1 and 128.0.0.5 in Figure 5.4, where one of them is listed in the MB and the other is not. Both IP addresses have similar relevance scores in other blacklists (withbambenek_c2’s scores of 0.6 and 0.7, andopenbl’s scores of 0.3 and 0.4). Intuitively, if 128.0.0.1 were to be legitimate and listed in MB, we can expect it to have a similar relevance score as that of 128.0.0.5 which is already present in MB. The recommendation system captures this pattern and assigns a score of 0.84 to 128.0.0.1. On the other hand, address 128.0.0.4 has no similar listing pattern as that of 128.0.0.5, therefore the recommendation system assigns it a low score of 0.12 in MB. Finally, IP addresses 128.0.0.2 and 128.0.0.3 share some listings with 128.0.0.5. However, their relevance scores are not similar. This regularity is also captured by the recommendation system and assigns a score of 0.29 and 0.17 respectfully in MB. Cells in the score matrix, in the column MB, that was lled by the recommendation system contain the set of potential future sources of legitimate trac to the customer (F). After we have calculated all the missing relevance scores in the misclassication blacklist MB, we pro- ceed to construct the aggregated blacklist known asmasterblacklistcandidates. To generate the candidates, we observe relevance scores in MB and then use a threshold (choice of discussed in Section 5.7) to include all the IP addressesa for which the following holds: r a;MB . Intuitively, IP addresses, which have high scores in MB are either current legitimate sources of customer’s inbound trac (L tr ) or likely to be so in the future (F), and the thresholding excludes them from the master blacklist. 5.3.3 SelectiveExpansiontoPrexes We have discussed in Section 5.2.3 why it would be useful to identify and expand IP addresses into pre- xes. Prior works have expanded IP addresses into prexes indiscriminately [191, 86, 215] – this improves malicious source identication but greatly increases misclassications. The novelty of our approach is to 110 Master blacklist candidates 169.231.140.68 193.1.64.5 193.1.64.8 Check 1 OK OK OK Check 2 ! OK OK 193.1.64.0/24 216.59.0.0/24 169.231.140.68 Selective expansion 216.59.16.171 243.13.0.23 243.13.222.203 169.231.140.10 216.59.0.8 OK OK Known legitimate sources (L t r ) Recommended misclassifications BLAG master blacklist Figure 5.5: Selective expansion of IP addresses into prexes. selectively expand IP addresses into prexes, only when this expansion does not greatly increase misclas- sications. This is particularly useful for customers deploying BLAG under emergency scenarios. The expansion phase starts with master blacklist candidates, which are all added to the BLAG mas- ter blacklist. During expansion, we identify IP addresses, that could be expanded into their /24 prexes (see Section 5.6 for the rationale behind choosing /24 prex size). We rst generate a list of all /24 pre- xes from the master blacklist candidates. We then evaluate if each prex should be added to the master blacklist. Prexes that contain known legitimate sources (from (L tr )) are excluded (Check 1). Further, prexes that contain likely misclassications (IP addresses with high relevance scores in the misclassi- cation blacklist, i.e. set (F)) are excluded (Check 2). Remaining prexes are added to the BLAG’s master blacklist. In Figure 5.5, none of the IP addresses are present in known-legitimate sources (L tr ) and address 169.231.140.68 has another address in the same /24 prex, which is a likely misclassication (in set (F)). Therefore, 169.231.140.68 is not expanded, and the other IP addresses are expanded to their corresponding prex to be included in the master blacklist. 5.3.4 WhyBLAGWorks BLAG assigns relevance scores to capture the relevance of IP addresses being listed in a blacklist. BLAG also introduces a new articial blacklist – misclassication blacklist, which consists of known-legitimate sources (L tr ). The recommendation system used by BLAG helps during the aggregation phase by pruning 111 Scenario Duration Source Training(L tr ) Validation(L v +M v ) Testing(L te +M te ) Days IPs Days IPs Days IPs Email 6/1/16- 6/30/16 M:Mailinator - - 7 M v : 13 K 16 M te :26 K L: Ham 7 L tr : 3 K 7 L v : 2 K 16 L te : 4 K DDoS Univ 9/1/16- 9/30/16 M:Mirai - - 7 M v : 1.1 M 16 M te : 2.8 M L: Web cl. 7 L tr : 16K 7 L v : 12 K 16 L te : 33 K DDoS DNS 6/24/16- 6/25/16 M:DDoS - - - - 1 M te : 5.5 M L: DNS cl. 1 L tr : 2.7M - - 1 L te : 16 K Table 5.1: Scenario datasets used in this study. Each scenario dataset is split into three – training, validation and testing. The training dataset is collected chronologically before the validation and testing, and contains only legitimate sources (L tr ). The validation and testing datasets are collected during malicious events and contain malicious (M v and M tr ) and legitimate (L v and L tr ) sources. The validation dataset is used to tune the appropriate parameters used in BLAG. out misclassications, and also during the selective expansion phase by preventing those expansions of IP addresses into prexes that would increase future misclassications. The recommendation system helps BLAG to discover other IP addresses that are predicted to be listed in the misclassication blacklist with a high relevance score. In other words, the recommendation system predicts future misclassications based on nding IP addresses that exhibit similarities, with regard to the blacklisting process, to known legiti- mate sources. In Section 5.6.2, we quantify the contribution of the recommendation system in reducing misclassication during the aggregation and selective expansion phases. 5.4 Datasets BLAG’s fundamental goal is to nd a trade-o between identifying as many malicious sources as possible and keeping the misclassications low. In this section, we look into the blacklists used by BLAG to aggre- gate information. We also present three BLAG deployment scenarios, which we use in evaluation. These scenarios include real-world legitimate and malicious trac. We will show the performance of BLAG un- der these scenarios in Section 5.5, where BLAG achieves more than 95% specicity (5% misclassication rate), while signicantly increasing recall (high detection of malicious sources), compared to individual blacklists and their naïve aggregation. 112 Type # BlacklistMaintainers Malware 57 Emerging threats [67], Malware Bytes [31], CyberCrime [55], URLVir [212], Swiss security blog [3], Bambenek [19], NoThink [152], I-Blocklist [93], NoVirusThanks [153], DYN [66], Malc0de [132], Malware domain list [128], Botscout [26], ImproWare [10] Reputation 32 Emerging threats [67], Graphiclineweb [79], Alienvault [5], Binary Defense Systems [61], CINSscore [39], Swiss Security Blog [3], Blocklist.de [24], I-Blocklist [93], Cisco Talos [40], Bad IPs [15], Blocklist Project [144], VXVault [214], ShunList [190], GreenSnow [80] Spam 39 Spamhaus drop and edrop [196], Stop Forum Spam [194], Chaosreigns [14], Lashback [119], Nixspam [158], Project Honeypot [91], Sblam! [182], Turris [210], ImproWare [10], Malware bytes [31], Cleantalk [44], My IP [98], BadIPs [15] Attacks 29 I-Blocklist [93], Malware Bytes [31], Snort Labs [117], TrustedSec [208], Haley [84], Darklist [58], SIP blacklist [198], VoIPBL [217], DShield [95], NoThink [152], OpenBL [159], Cruzit [53], BruteforceBlocker [74], Clean MX [43], Bad IPs [15], MaxMind [137] Table 5.2: Four types of blacklists, roughly categorized by the type of malicious activities they capture. Each row gives the number of blacklists and blacklist maintainers for that type. 5.4.1 BlacklistDataset We have monitored 157 publicly available blacklists for 11 months, starting from January 2016 to Novem- ber 2016. Each blacklist is updated at a dierent frequency by its provider, ranging from 15 minutes to 7 days. We collected the update time of each blacklist manually and programmed our crawler to pull the snapshot of the blacklist when a new update was available. We have collected around 176 million black- listed IP addresses over 23,483 autonomous systems. Our monitored blacklists vary in size – on one hand, we have large blacklists (15.76%) listing more than 500,000 IP addresses and on the other, we have small blacklists (19.56%), which list fewer than 1,000 IP addresses. We do not delve into details on the various properties of blacklists, such as their volume, contribution, exclusive contribution, detection of malicious activities (refer Vector et al. [123] work on an exhaustive study on properties of blacklists). Our work fo- cuses more on identifying key properties of blacklists that make them ineective in emergency scenarios (refer Section 5.2) and presenting an improved blacklisting technique (refer Section 5.3). 113 Our blacklist dataset (B) is representative of dierent attack vectors such as spam, malware, DDoS at- tacks, ransomware, etc. Table 5.2 shows the blacklist maintainers and the number of blacklists maintained by them. Our dataset includes popular blacklists such as DShield [95], Nixspam [158], Spamhaus [196], Alienvault [5], Project Honeypot [91], Abuse.ch [3] and Emerging Threats [67]. 5.4.2 Scenarios Table 5.1 shows our three scenarios. Each scenario consists of three portions of the same dataset: training, validation and testing. The training portion contains only known-legitimate sources (L tr ) and is used to tailor BLAG to the customer network (Section 5.3.2). This portion is collected before the malicious event in each scenario. The validation and testing portions contain both the legitimate (L v and L te ) and malicious (M v and M te ) sources. The validation portion is used to calibrate BLAG’s parameters (l, and K) for testing ‡ . The testing portion is used to evaluate the performance of BLAG and competing blacklisting approaches. Our three scenarios contain sources of diverse attacks: spam, DDoS on a University network or DDoS on DNS root. This allows us to test how well BLAG could prevent these attacks if used by a customer network to lter attack trac. 5.4.2.1 MaliciousEmailCampaignorSpam In this scenario (referred to as Email), we look into a case where a University network is bombarded with spam emails. We collect malicious and legitimate IP addresses during the same time period of June 2016. Simultaneous collection is important, because an address may be malicious at one time, and cleaned afterward. We collect malicious IP addresses from Mailinator [131], a service, which allows users to redirect unwanted e-mails to a public-inbox. We lter e-mails from these public inboxes during June 2016, using SpamAsssassin [136] to obtain around 2.3 M spam e-mails, sent by around 39 K IP addresses. These IP ‡ We do not have a validation dataset for DDoS DNS. Refer Section 5.7 for further explanation 114 addresses form our malicious dataset. We trained SpamAssassin using SpamAssasin’s public corpus [195] and spam archives from Untroubled [211], to ensure we capture only malicious spam emails. We use the rst 7 days consisting of 13 K IP addresses as a validation set (M v ) and the remaining 16 days consisting of 26 K IP addresses for testing (M te ). We collect legitimate IP addresses through a human user study. This study was reviewed and approved by our IRB. We recruited 37 volunteers from our University, who allowed us automated access to their Gmail inbox, during June 2016. We scanned each participant’s Gmail account using a plugin, which we developed. Our plugin used the OAuth2 protocol to access Gmail, and it used regular expressions to extract a sender’s IP address, time and label for each e-mail. We did not extract any other information and did not record the identity of the study participants, to protect privacy. The label in Gmail can be assigned by a user or by GMail, and it is usually “spam”, “inbox” or a user-dened label like “conference”. We harvested information only from e-mails that have labels other than “spam”. Our scanning generates as output a list of {sender IP address, time} tuples, which we save. We extracted around 30 K e-mail records, sent by around 9 K IP addresses. We use the rst seven days of this dataset consisting of 3 K IP addresses for training (the known-legitimate sources set (L tr )), the next 7 days consisting of 2 K IP addresses for validation (L v ) and the remaining 16 days consisting of 4 K IP addresses for testing (L te ). 5.4.2.2 DDoSonaUniversitynetwork In this scenario (referred to asDDoS Univ ), we look into a case where web-servers at a University could be targeted by Mirai-infected hosts. The legitimate and malicious IP addresses are collected during Septem- ber 2016. We collect malicious IP addresses from Netlab’s [146] collection of Mirai-infected hosts during September 2016. There were around 3.9 M addresses, which form our malicious set. We use the rst 7 days consisting of 1.1 M IP addresses as a validation set (M v ) and the remaining 16 days consisting of 2.8 M IP addresses for testing (M te ). We collect legitimate IP addresses by identifying Web clients who 115 communicated with a set of popular web servers at a mid-size US University in September 2016. We in- cluded only those TCP connections which exchanged payload with the server (thus excluding scans). This resulted in around 61 K legitimate IP addresses. We use the rst 7 days consisting of 16 K IP addresses as the known-legitimate sources (L tr ), the next 7 days consisting of 12 K IP addresses for validation (L v ) and the remaining 16 days consisting of 33 K IP addresses as the future legitimate sources (L te ). 5.4.2.3 DDoSonDNSroot In this scenario (referred to asDDoS DNS ), we look into a case of TCP SYN ood attack on the DNS B-root server [99]. We communicated with the dataset provider to obtain non-anonymized sources of this attack, as well as non-anonymized data before the attack, which we use for training. We mine the malicious IP addresses (M te ) as those that have sent TCP SYN ood to the server for two hours on June 25, 2016. There are 5.5 M malicious IP addresses. We mine the known-legitimate sources (L tr ) as sources of DNS queries 1-day before the attack event (2.7 M IP addresses), and the future legitimate sources (L te ) as sources of DNS queries during the attack event (16 K IP addresses). 5.4.3 Limitations The blacklist dataset contains only publicly available blacklists, while many providers also oer for-pay blacklists that are usually larger and more accurate [222] (see Section 5.5 for the performance of our mon- itored blacklists). We chose to use only publicly available blacklists because: (1) these blacklists are widely used and we wanted to evaluate BLAG’s benets for a customer network that deploys such blacklists. A recent survey shows that nearly 60% of surveyed network operators use blacklists including publicly available ones [92]. We also believe that BLAG’s benets would carry over to for-pay datasets because its mechanism is generic. (2) we wanted our work to be repeatable, and using public blacklists enables us to 116 freely share our data. We plan to share the blacklist dataset and BLAG code, which could be used by any customer network to improve blacklisting. Our scenario datasets suer from several limitations. First, they only capture a small sample of le- gitimate/malicious IP addresses that were active on the Internet at a given point in time, and for a given legitimate or malicious purpose. Many other IP addresses could be legitimate or malicious at the same time, and we have no ground truth for these. We also rely on other security technologies (SpamAssassin) that may also be used by blacklist maintainers to blacklist an address. These limitations are present in other published works [238, 165, 191, 222, 86], which use similarly-sized malicious and legitimate datasets, and rely on other security technologies such as Proofpoint [166] and SpamAssassin, as we do, to establish maliciousness at a given time. A more recent study on Blacklisting has used Alexa’s top 10,000 list to eval- uate the accuracy of blacklists based on IP addresses that are present in them [123] but Alexa’s rankings are also not ideal measure of legitimacy [184, 120]. These limitations cannot be avoided, as there is no complete, 100% accurate list of legitimate and malicious IP addresses on the Internet nor in any specic network, at any given point in time. Second, our datasets are dated – captured in 2016. Ideally, we would use more recent datasets. But, it is very hard to nd data about attack events and legitimate trac, which includes non-anonymized IP addresses. It is even harder to nd such data streams that are collected simultaneously and that relate to the same type of sources (e.g., sources of legitimate email vs sources of spam). While dated, our scenarios faithfully capture sources of legitimate and malicious trac at the same time. When working with each scenario BLAG only uses blacklist data up to that point in time. Thus we faithfully simulate what BLAG’s performance would have been if it were deployed by a customer network at the time. Finally, we do not have a validation dataset for theDDoS DNS , because the attack on B-Root DNS server was observed only for a few hours. From evaluation of our parameter values on Email and DDoS Univ datasets (Section 5.7) we observe that parametersl (historical decay) andK (matrix factorization) do not 117 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Recall Max recall (a) Email 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Recall Max recall (b) DDoS Univ 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical PRESTA+L BLAG No Exp BLAG (%) Recall Max recall (c) DDoS DNS Figure 5.6: Specicity and recall of BLAG with two competing approaches on trac datasets. 0 5 10 15 20 5 10 15 20 25 30 % of BLAG IPs days Email best DDoS Univ best DDoS DNS best Email historical DDoS Univ historical DDoS DNS historical Email PRESTA DDoS Univ PRESTA DDoS DNS PRESTA Figure 5.7: Delay in reporting malicious IP addresses reported by BLAG. change with the dataset, and high values of parameter (threshold for candidate IP addresses) lead to higher specicity. Therefore, to evaluate the performance of BLAG for DDoS DNS scenario, we use the samel andK parameters as that of Email and DDoS Univ scenarios and set a high value for. 5.5 Evaluation In this section, we evaluate the performance of BLAG and several competing approaches, for dierent deployment scenarios, described in Section 5.4. We nd that BLAG achieves high specicity (95–99%), and has much better recall (3.5x–114x improvement) than competing approaches. BLAG also detects attackers 13.7 days faster than competing approaches. 118 5.5.1 EvaluationSetup We measure performance of blacklists using recall and specicity. Recall measures the percentage of ma- licious sources (out of some ground-truth set) that were blacklisted. Specicity measures the percentage of legitimate sources (out of some ground-truth set) that were not blacklisted. Higher values for both measures denote better, more accurate, blacklists. In this section, we compare several competing blacklisting approaches against BLAG. In all cases when we evaluate a blacklisting approach we “travel back in time” to the time just before the testing portion of our given Scenario. We then use past data from our (B) dataset up to the time when the testing portion starts, and refresh it as blacklists update during the testing. For example, imagine if our scenario contained two days Sep 1st, 2016 and Sep 2nd, 2016, with Sep 1st used for training and Sep 2nd for testing. Our blacklist dataset (B) overlaps this period going from Jan 8th, 2016 to Nov 30th, 2016. When we test a blacklisting approach for the given scenario we would start by including all data from (B) from Jan 8th up to, and including Sep 1st. We would then start our evaluation and keep updating the blacklist on Sep 2nd. This way we faithfully simulate the performance a given approach would have if we were to travel back in time to Sep 2nd. We compare the performance of the following approaches against BLAG: Best – the blacklist from Blacklist dataset that performs the best on a given scenario with regard to recall. Best is a hypothetical scenario, because in real deployment we could not tell which blacklist will eventually be the best. It allows us to measure the benets of aggregation over the use of a single blacklist. Historical – all IP addresses listed in any blacklist in the Blacklist dataset. This approach assumes “once malicious, always malicious” and performs naïve aggregation. PRESTA+L – the blacklist generated using techniques described in [222]. PRESTA performs spatio- temporal analysis and expansion using historical blacklist data to generate a more proactive blacklist. 119 PRESTA assigns a reputation score to all blacklisted IP addresses across three dierent spatial groups (IP address, address’s /24 prexes along with two surrounding /24 prexes and address’s corresponding au- tonomous systems). PRESTA combines all the spatial grouping into one and chooses relevant listings using a thresholding technique. We tune PRESTA such that the BLAG’s recall is equivalent to that of PRESTA for a deployment scenario, and then we compare the specicity of these two approaches. Further, to tease apart the factors that help us outperform PRESTA, we consider a variant approach, calledPRESTA+L, where we remove addresses that are already present in known-legitimate sources (L tr ) from every dataset. BLAG with and without selective expansion: We run BLAG as described in Section 5.3. We set the length of address historyl = 30 and the number of latent features for recommendation systemK = 5. We set relevance threshold = 0:8 for Email/DDos DNS datasets and = 0:6 for DDoS Univ dataset. Our choices for these variables are explained in Section 5.7. We compare the performance of BLAG with and without the selective expansion, to show the contributions of aggregation and expansion stages of BLAG. During the evaluation, for our testing dataset and each blacklisting approach (best, historical, PRESTA+L or BLAG) we simulate the dynamics of address appearance over time in the following manner. When an address appears in the blacklist dataset (B) we: (1) include the address in the best blacklist if it appeared on a blacklist, which will ultimately perform the best on the given BLAG deployment scenario, (2) include the address in the historical blacklist, (3) apply PRESTA+L algorithm on the address and all its spatial groups to determine if it is included in the PRESTA+L blacklist, (4) apply BLAG on the address to determine if it should be included in the BLAG’s aggregated master blacklist and if it should be expanded into its /24 prex. We report recall and specicity at the end of the testing datasets. 5.5.2 BLAGisMoreAccurate BLAG’s goal is to capture as many malicious sources as possible while keeping the specicity high. 120 0 20 40 60 80 100 Best Historical BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical BLAG (%) Recall (a) Email 0 20 40 60 80 100 Best Historical BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical BLAG (%) Recall (b) DDoS Univ 0 20 40 60 80 100 Best Historical BLAG (%) Specifcity 0 20 40 60 80 100 Best Historical BLAG (%) Recall (c) DDoS DNS Figure 5.8: Specicity and recall of BLAG and four competing approaches with expansion. 0 20 40 60 80 100 ASN BGP /24 (%) Specifcity 0 20 40 60 80 100 ASN BGP /24 (%) Recall (a) Email 0 20 40 60 80 100 ASN BGP /24 (%) Specifcity 0 20 40 60 80 100 ASN BGP /24 (%) Recall (b) DDoS Univ 0 20 40 60 80 100 ASN BGP /24 (%) Specifcity 0 20 40 60 80 100 ASN BGP /24 (%) Recall (c) DDoS DNS Figure 5.9: Evaluating BGP and AS expansion techniques. 121 BLAG has the best specicity/recall tradeo across the three scenarios: Figure 5.6 shows that BLAG overall has the best performance. For the Email scenario, the best blacklist has higher specicity (100%) than BLAG (95%). However, BLAG improves the best blacklist’s recall from 4.7% to 69.7%. Historical blacklist has a recall of 19.4%, which is better than the best blacklist but not better than BLAG. Naïve aggregation of IP addresses in historical blacklist lowers its specicity to 89% which is much lower than that of BLAG (95%). For the same recall, BLAG has 11.5% better specicity than PRESTA+L. We observe similar pattern with DDoS Univ and DDoS DNS scenarios. BLAG’s recall ranges between 6.4–56.1% when compared to 0.004-0.4% for best blacklist and 1.8–9.5% for historical blacklists. For the same recall, BLAG’s specicity ranges from 97.9–99.5% when compared to 53.1–84.8% of PRESTA+L. We detail in Section 5.6.2 how the contribution of recommendation systems in BLAG helps it achieve better specicity that PRESTA+L. BLAG lters more attack trac. Some IP addresses may generate more attacks than others. We evaluate the amount of malicious activity that would be ltered by BLAG, best and historical blacklists for our three scenarios. In the Email scenario, BLAG would lter 69.7% of spam, compared to 4.7% and 19.4% ltered by best and historical blacklists respectively. In the DDoS Univ scenario, BLAG would lter trac from 56.1% of infected devices, compared to only 0.4% and 9.5% ltered by best and historical blacklists. In the DDoS DNS scenario, BLAG would drop 6.4% of attack queries, compared to 0.004% and 0.1% ltered by best and historical blacklists. We pause here to address the low performance of public blacklists in general for oender identi- cation. While BLAG greatly improves performance, it can only work with what it has – existing public blacklist data. For all malicious datasets, blacklists cover addresses in only 10.7–71.7% of /24 address spaces and BLAG’s recall cannot go beyond this (shown by the red horizontal line in Figure 5.6). Overall BLAG manages to identify 6.4–69.7% of attackers and drop similar amounts of attack trac. While this is far from perfect, dropping more than half of the attack trac may be very helpful in emergency scenarios. 122 If BLAG were used to aggregate for-pay blacklists, which list many more malicious sources [222], its at- tacker coverage would likely be better. We emphasize, however, that BLAG manages to greatly improve the performance of public blacklists while limiting collateral damage to legitimate trac. BLAG lists malicious sources faster. We show in Figure 5.7 the number of days taken by other competing approaches to list malicious IP addresses after BLAG discovers them. We track competing approaches for up to 30 days. On average across all scenarios, BLAG reports malicious sources 9.4 days faster than the best blacklist, 10.3–16.1 days faster than historical blacklists and 8.8–13.4 days faster than PRESTA+L. BLAG’s aggregation helps in detecting attackers faster than the best blacklist and BLAG’s selective expansion helps in detecting attackers faster than the historical blacklist. After 30 days, the best blacklist catches up to 2.1% of malicious IP addresses discovered by BLAG for the Email scenario and does not report any malicious address for DDoS Univ and DDoS DNS scenarios. On the other hand, historical blacklists catch up to 0.2–4.1% and PRESTA+L catch up to 0.14–0.23% of malicious IP addresses discovered by BLAG. 5.6 SensitivityAnalysis In this section, we discuss the contribution of BLAG’s expansion phase. We then look into the contribution of BLAG’s recommendation system in pruning misclassications for the aggregation and expansion phase. 5.6.1 Expansionapproaches BLAG’s performance comes both from aggregation and expansion. We investigate how much of BLAG’s performance comes from its selection of high-quality data to aggregate and how much comes from expansion, by showing BLAG with and without expansion (BLAG and BLAG No Exp in Figure 5.6). Without expansion, for the Email scenario, BLAG’s recall is 19.3% and its specicity is 98.7%, while the best blacklist has 8.9% recall and 100% specicity. BLAG is thus still better than the best blacklist, even without 123 0 20 40 60 80 100 Email DDoSUniv DDoSDNS Aggregation specifcity (%) Naive aggregation BLAG aggregation 0 20 40 60 80 100 Email DDoSUniv DDoSDNS Expansion specifcity (%) Raw expansion Check 1 Check 2 (a) BLAG’s recommendation sys- tem contribution to specicity. 0 20 40 60 80 100 Email DDoSUniv DDoSDNS Aggregation recall (%) Naive aggregation BLAG aggregation 0 20 40 60 80 100 Email DDoSUniv DDoSDNS Expansion recall (%) Raw expansion Check 1 Check 2 (b) BLAG’s recommendation sys- tem contribution to recall. 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 120 140 160 specifcity/recall (%) # of blacklists specifcity recall (c) Contribution of blacklists by size. Figure 5.10: Evaluating impact on specicty/recall due to various components in BLAG and contribution of blacklist size to BLAG. expansion. The historical blacklist has 19.4% recall (vs 19.3% of BLAG without expansion), but it has 89% specicity (vs 98.7% of BLAG). Finally, PRESTA+L has 68% recall (vs 19.3% of BLAG without expansion), but it has much lower specicity (84.8% vs 98.7%). This is expected since PRESTA+L applies expansion and we compare it to BLAG without expansion. Expansion of BLAG improves recall further (to 69.7%), at a small loss of specicity (95% specicity of BLAG with expansion vs 84.8% of PRESTA+L), making BLAG the best performing approach, among the ones we tested. Similar trends hold for other scenarios we tested. BLAGoutperformsbest-expandedandhistorical-expandedblacklists. We investigate if a similar expansion approach to BLAG’s could improve the performance of the best blacklist and the historical blacklist. Instead of selective expansion here, which uses BLAG’s information, we use naïve expansion, where each candidate address is expanded into its /24 prex if there are no overlaps with the known- legitimate source (T tr ) dataset. Figure 5.8 shows that BLAG still outperforms competing approaches, due to its selection of only high-quality information to aggregate, before expansion. For the Email scenario, the best blacklist with expansion has 99.8% specicity vs 95% of BLAG. But, BLAG has 69.7% recall, while the best blacklist with expansion has only 14.4%. The historical blacklist with expansion has a comparable recall to that of BLAG (69.7% vs 70.4% respectively). However, BLAG has 95% specicity while the historical 124 blacklist with expansion has 83.8%. We observe similar trends in the DDoS Univ and DDoS DNS scenarios. In the DDoS DNS scenario, the historical blacklist with expansion achieves 98.9% specicity (vs 99.5% of BLAG) and 7.9% recall (vs 6.4% of BLAG) so they perform roughly equal. BLAG’s expansion approach using BGP prex and AS level. We investigate how BLAG’s per- formance would change if we expanded IP addresses into their full BGP prexes or entire autonomous systems, as suggested in [191]. Expanding some IP addresses into large prexes is not advisable, as this can potentially have large collateral damage, but we investigate it as a hypothetical scenario. We apply /24-prex, BGP-prex and AS-level aggregation to the master blacklist candidates, produced by BLAG for the three datasets. We then apply the selective expansion technique, but instead of using /24 prex in deciding whether to expand, use the encompassing BGP prex and the entire AS. We show the specicity and recall of BLAG with these expansion approaches in Figure 5.9. Overall, expanding IP addresses to AS or BGP prex has a higher recall than /24 expansion (7.9–23.4% higher). However, specicity varies across dierent deployment scenarios. For the Email scenario, /24 expansion has a little better specicity than AS and BGP prex expansion (2.5–3.7% better). For the DDoS Univ scenario, /24 expansion has better specicity than AS and BGP prex hijacking (3.4–17.4%). For the DNS scenario, specicity is about the same across the three expansion approaches. 5.6.2 Contributionofrecommendationsystem Figure 5.10a shows the contribution of the recommendation system in the aggregation and selective ex- pansion phase over our three scenarios. During the aggregation phase (top of Figure 5.10a), BLAG’s rec- ommendation system can prune out misclassications and improve specicity from 89–99% to 98–99.6%. BLAG is even more eective in increasing specicity during the selective expansion phase. BLAG’s recom- mendation system (shown ascheck1 in bottom half of Figure 5.10a), improves specicity from 24.5–69.7% 125 to 86.9–95.6% over naïve expansion of all IP addresses. Also, BLAG’s complete selective expansion phase (shown ascheck2), further improves specicity to 95–99.5%. On the other hand, BLAG’s recall depends on the coverage of blacklist, aggregation and selective ex- pansion phases. Figure 5.10b shows the impact of BLAG on recall. BLAG’s aggregation phase (top of Fig- ure 5.10b) uses a threshold to prune out false positives and this can also prune out malicious addresses. This phase reduces recall from 0.18–19.4% to 0.15–19.35%. BLAG’s selective expansion (bottom of Figure 5.10b) also reduces the recall. During check1 phase of selective expansion, recall further from 10.7–71.9% to 9– 70.2% and during the check2 phase of selective expansion, the recall further reduces to 6.4–69.7%. 5.6.3 ContributionofIndividualBlacklists We ran BLAG on n largest blacklists for the Email scenario, and varied n from 1 to 157 as shown in Figure 5.10c. There is a 15.7% gain in recall for the rst 106 blacklists. After this, there is a sharp increase in recall for the remaining blacklists. The next 20 largest blacklists double the recall to 32% and the next 14 largest blacklists push the recall past the 50% mark. When including all blacklists, the recall reaches 69.7%. Specicity stays at 100% for the rst 41 blacklists and then drops slightly from 100% to 99.1% for the rst 100 blacklists. The specicity drops to 95.5% for the next 40 blacklists and nally, by adding the remaining blacklists, the specicity comes down to 95%. Because we see a steady improvement in recall and a steady, slow decline in specicity, it is hard to tell how many blacklists would be enough. Instead, BLAG could let a customer choose dierent numbers of blacklists to aggregate and could produce specicity estimates (e.g., by using a validation set L v ) for every choice. 5.6.4 Contributionofknown-legitimatesources(L tr ): We ran BLAG by varying the duration of the (L tr ) available to BLAG– 0, 1, 3, 5 and 7 days for Email dataset as shown in Figure 5.11a. As the number of days is increased, the specicity improves with small reduction 126 0 20 40 60 80 100 0 days 1 days 3 days 5 days 7 days (%) Specifcity 0 20 40 60 80 100 0 days 1 days 3 days 5 days 7 days (%) Recall (a) Contribution of known- legitimate sources dataset. 0 20 40 60 80 100 Email DDoS Univ Specifcity (%) BLAG 10 BLAG 30 BLAG 50 0 20 40 60 80 100 Email DDoS Univ Recall (%) BLAG 10 BLAG 30 BLAG 50 (b) Evaluating parameterl. 80 85 90 95 100 0 0.2 0.4 0.6 0.8 1 Specifcity (%) Email DDoS Univ 0 15 30 45 60 0 0.2 0.4 0.6 0.8 1 Recall (%) Email DDoS Univ 0 10 20 30 40 50 0 0.2 0.4 0.6 0.8 1 F1 score (%) threshold Email DDoS Univ (c) Threshold. Figure 5.11: Sensitivity analysis in recall. With only 1 day of legitimate data available, the specicity improves by 36.7% with a loss of 2% of recall. As we increase the number of days the gain in specicity increases. By including all the days for training dataset, the specicity only improves by 4.5% and the recall reduces by 4.7%. 5.7 Parametertuning In this section, we discuss a methodology used to determine the parametersl, andK § . Parameterl for Historical Decay. In Section 5.3.1, l roughly controls the length of historical data (in days) that may be included in the BLAG master blacklist. Using the validation dataset (L v + M v ), we determine the impact ofl for three values (10, 30 and 50) shown in Figure 5.11b. For the two scenarios, l = 30 has the highest recall. About 1.5–8.9% and 4.3–36.4% higher recall for Email and DDoS Univ scenarios respectively. Forl = 30, the specicity is uniformly high, ranging from from 95–97.9%. Therefore,l = 30 strikes the right balance, which increases recall with minimal loss in specicity. During our evaluation, we usel = 30 for the DDoS DNS dataset. This agrees with our observations in Section 5.2.2, where 91% of blacklisted addresses that re-oend, do so within 30 days. § Scripts to generatel, andK can be found athttps://steel:isi:edu/Projects/BLAG/ 127 Parameter for choosing Master Blacklist candidates. The parameter controls the set of IP addresses, which should be considered for the expansion phase in BLAG. We show the performance in the validation dataset (L v + M v ). Figure 5.11c shows that for each scenario parameter trades accuracy of BLAG with higher coverage. For various thresholds, we plot the specicity, recall and F1-score ¶ . Network operators could use the F1-score as a metric to determine the appropriate threshold. We see that for the Email scenario, after a threshold of 0.8, the F1-score plateaus at 43%. For higher thresholds, the specicity drops from 95% to 85.9%. Similarly for DDoS Univ scenario, the F1-score plateaus after a threshold of 0.6 at 33%. For higher thresholds, the specicity again drops from 97.9% to 90.4%. Therefore, we set a threshold of 0.8 for Email and 0.6 for the DDoS Univ . Relevance scores for misclassications would typically be high since all misclassications are allocated a score of 1 in the misclassication blacklist. Therefore, for DDoS DNS scenario, we set a threshold of 0.8 to prune out misclassications. Parameter K for Matrix Factorization. Parameter K is used in non-negative matrix factorization (NMF), and denotes the number of latent features. An idealK will have the minimum error between the matrixR(MxN) andR 0 (Section 5.3.2). Brunet et al. [29] suggested using the smallestK, after which the cophenetic correlation coecient starts decreasing. For the validation datasets, we evaluate dierent values ofK and nd that the cophenetic correlation coecient starts decreasing afterK is 5. BLAG is pre-congured to run gradient descent withK = 5 until the root mean squared error (RMSE) between the original matrixR and matrixR 0 fell below 1% or the number of iterations exceeded 1,000. 5.8 RelatedWork In this section, we survey work related to blacklist analysis, blacklist improvement and aggregation. ¶ F1-score is the harmonic mean of precision (fraction of IP addresses, which are listed and are indeed malicious sources) and recall 128 Analysis of Blacklists. Kührer et al. evaluated the eectiveness of fteen publicly available malware blacklists by measuring accuracy and completeness on datasets consisting of parked domains, sinkholed IP addresses, and active malware domains [116]. Pitsillidis et al. evaluated ten dierent blacklists on purity, coverage, proportionality, and timing [165]. Purity was measured by comparing feeds to Alexa top websites and Open directory listing, whereas coverage, proportionality, and timing were obtained by comparing feeds to one another. Both Kührer et al. and Pitsillidis et al. work support our ndings that blacklists are not accurate. Zhang et al. evaluated nine blacklists using trac logs from a regional ISP [238]. They analyzed overlapping IP addresses between trac logs and blacklists. But they were unable to measure the accuracy of blacklists, as the trac in the logs was not labeled as malicious or legitimate. Vector et al. [123] analyzed 47 distinct IP address threat intelligence sources and evaluated them for volume, dierential contribution, exclusive contribution, latency, coverage, and accuracy. They used Alexa top 10 K websites as ground truth for legitimate sources and scanners captured by CAIDA darknet as ground truth for malicious sources. Vector et al. support our nding that blacklists or threat intelligence feeds have low recall (< 2%) and low misclassication as well. The main dierence between these related works and ours is twofold. First, our main focus is on distilling accurate pieces of information from blacklists and aggregating it into a master blacklist; we use data about current blacklists merely to motivate our work. Second, we use an order of magnitude more blacklists than previous works. ImprovingBlacklisting. PRESTA [222] uses historical data from three pay-for spam blacklists provided by Spamhaus [196] to infer temporal and spatial properties of IP addresses and expand IP addresses into three spatial regions. Spamhaus blacklists are highly reputable and have high recall (> 80%) and very low misclassications (< 2%). PRESTA’s technique helps to uncover 18% more spammers while keeping misclassications low. In our evaluation, too, PRESTA helped improve recall but at much higher misclas- sication cost (Section 5.5). This may be because we use public blacklists, which may not as accurate as pay-for blacklists. 129 Highly Predictive Blacklisting [237] (HPB) creates blacklists customized to a given network by a page ranking algorithm. Soldo et al. [193] extended HPB to use historical data about attack sources and destina- tions and used a recommendation system to predict possible attackers for each victim. In contrast, BLAG uses a recommendation system to remove misclassications while aggregating blacklists. Sinha et al. [191] present a new spam blacklisting technique, by monitoring email servers and spam traps to curate a spam blacklist. BLAG, on the other hand, works with existing blacklists to improve their performance and is applicable to dierent types of attacks. 5.9 Discussion We discuss possible attacks on BLAG. BLAG has no way, other than the recommendation system, to dier- entiate between low-quality and high-quality information. Thus, if an attacker could produce a blacklist that is very similar to some reputable blacklist, which does not have much overlap with MB (e.g., by copy- ing it) and if they included a few public servers in it, BLAG could conceivably propagate this information into its master blacklist. This could then lead to legitimate trac from these public servers being dropped. Current blacklists could also be polluted by the same approach. Networks today choose carefully which blacklists to use, based on their reputation in the operator community. We assume they would apply the same care to choose blacklists used by BLAG. BLAG makes any polluted information less likely to prop- agate than individual blacklists. The attacker would have to carefully craft the polluted blacklist so that the servers appear on the same blacklist as many malicious sources, and their appearances do not resem- ble appearances of the known-legitimate sources on MB. Otherwise, BLAG would be able to identify and discard low-quality information. We leave the exact handling of pollution attempts for our future work. 130 5.10 Conclusion Blacklists are widely used by network operators, but they usually miss many attacks. We have proposed BLAG– the system that can identify high-quality pieces of information from multiple blacklists, and ag- gregate them into a master blacklist, with some IP addresses expanded into /24 prexes. Such a master blacklist could be useful for an emergency response to a novel or large-scale attack. Overall, BLAG has higher recall than any single blacklist or their naïve aggregation and has 95–99% specicity. BLAG also outperforms PRESTA+L, a competing approach, by having higher specicity for the same recall. We thus believe that BLAG could signicantly improve network security, and produce higher-quality blacklists than existing approaches. 131 Chapter6 QuantifyingtheImpactofBlocklistingintheAgeofAddressReuse 6.1 Introduction Consider one user’s experience with Cloudare discussed under a trouble ticket [48]. When the user tried to access any website hosted by Cloudare, they were unnecessarily blocked and subjected to CAPTCHAs. On further inspection, the user found that their public IP address was shared with many other users via a Network Address Translation (NAT). One of the NAT users was running a spam campaign leading to the NAT’s IP address being listed in many blocklists. It is known that Cloudare uses its own IP repu- tation with the help of blocklists [45] to protect its customers. Thus users behind reused addresses will be unjustly blocked whenever they access websites hosted on Cloudare [50]. Other hosting providers have similar issues as well [71, 70]. What should legitimate users do when they are unjustly blocked? In fact, Cloudare’s best practice [49] recommends users to obtain a new IP address, by either resetting their device or by contacting their ISP. In reality, obtaining a new untainted static IP address may be impossible or too costly for many users [162, 201, 226]. This type of blocking often gets unnoticed by the network operators because currently there is no way to measure the excessive blocking for a blocking mechanism. This chapter proposes two techniques to measure unjust blocking from IP blocklisting. Blocklists are lists of identiers (most often IP addresses) that are associated with malicious activities. For a network operator, blocklists provide a simple method to quickly block malicious trac entering their network. 132 Blocklists can have unjust blocking due to two forms of addressreuse: 1) NATed addresses where several users share the same IP address at the same time and 2) dynamic addressing where the same IP address is allocated to multiple users over time. In this study, we make the following contributions: Detectingreusedaddresses: We propose two new techniques to identify reused addresses that pro- vide high-condence detection, leverage only public datasets, and can be replicated by other researchers. While extensive prior work exists on detecting NATed [176, 130, 114, 220, 21, 23, 205, 142, 138] and dynamic addresses [175, 227, 89], we nd that they either do not provide suciently accurate and ne-grained in- formation per IP address or do not publicly release the nal list of reused addresses or prexes. To detect NATed IP addresses, we implement a crawler using the BitTorrent’s Distributed Hash Table (DHT). Our crawler detects when BitTorrent users simultaneously use the same IP address (Section 6.3.1), and thus we can measure the lower bound of users behind that IP address that would be adversely aected if the address were blocklisted. To detect dynamic addresses, we use the RIPE Atlas probe measurement logs to identify probes whose IP addresses change frequently and thus determine IP prexes that are dynamically allocated (Section 6.3.2). By determining dynamic prexes, we can identify users that would be aected by address reuse when allocated a previously blocklisted IP address. Our techniques have a reasonable overlap with the blocklisted IP address space, where BitTorrent and RIPE addresses are present in 29.6% and 17.1% of autonomous systems that have blocklisted addresses. Measuringtheimpactofreusedaddresses: We apply our detection techniques to identify reused addresses in 151 publicly available IPv4 blocklists (Section 6.4). About 60% of blocklists contain at least one NATed address and about 53% of blocklists list at least one dynamic address that will lead to unjust blocking. We nd 45.1K and 30.6K listings in blocklists for each type of reuse, respectively. NATed and dynamic addresses in blocklists can have an impact on end-users by blocking as many as 78 users for as long as 44 days. 133 Finally, we survey 65 network operators (Section 6.6) on their blocklisting practices and nd that IP blocklists are often used to directly block trac. To assist network operators to avoid unjust blocking, we make our techniques publicly available and also publish a new address list that has all reused addresses we detect ∗ . 6.2 Relatedwork Existing studies identifyautonomoussystems(ASes)orIPprexes that may be reused (e.g., that use carrier- grade NATs) using heuristics. However, to estimate the impact of blocklisting reused addresses, we need to accurately identifyIPaddresses that are reused. Müeller et al. [142] use traceroutes to a dedicated server in an ISP to detect middleboxes (including NAT). Other techniques use IPid [21], OS ngerprinting [23] or UDP hole punching [205] to detect NATed addresses. Netalyzr [114] and NetPiculet [220], on the other hand, require users to install Android applications that carry out measurements from the client’s device. Though these techniques are eective in detecting NATs, they require many users to install custom appli- cations to achieve signicant coverage, and must continuously incentivize them to conduct measurements. These measurements are also no longer active. Other approaches to reused address identication use private data and cannot be replicated. Richter et al. [175] and Casado et al. [35] observed IP addresses using NAT or dynamic addressing by monitoring server connection log of a CDN. Xie et al. [227] analyzed Hotmail user-login trace to determine dynamic addresses. Metwally et al. [138], use Google’s application logs to detect NATed addresses and reduce false positives in detecting abuse trac. Cai et al. [32] present an ongoing survey by sending ICMP ECHO messages to 1% of the IPv4 address space. Based on the responses, they dene metrics on availability, volatility, and median up-time to de- termine address blocks that are potentially dynamically allocated. This work produces a public dataset, ∗ https://steel:isi:edu/members/sivaram/blocklisting_impact/ 134 which we compare against our approach in Section 6.5. However, this work has several limitations. An ICMP reply from an IP address need not uniquely identify the host using the IP address since rewalls and middleboxes can reply on behalf of hosts. Further, some networks lter outgoing ICMP trac, poten- tially leading to undercounting. Finally, this work introduces an ad-hoc estimate of dynamically allocated prexes based on the address uptime, and we cannot establish its accuracy. Foremski et al. [69] dene Entropy/IP that discovers IPv6 address structures using clustering and sta- tistical techniques on a subset of IPv6 addresses that are known to be active. Using this system, one could identify IPv6 addresses that share similar characteristics (such as a network’s address allocation strategy). Although we could use their technique to drive our measurement study to identify reused addresses, our current work focuses only on IPv4 blocklists. BitTorrent network has been used to identify carrier-grade NATs (CGNs) in autonomous systems [176, 130]. These techniques leverage the fact that CGN public-facing IP addresses are likely to appear more frequently in a time window than non-CGN addresses. However, identifying ASes that use CGN is not useful for our research, since blocklists list IP addresses and CGN’s may not be deployed across the entire AS. Thus making it hard to identify IP addresses that are using CGN. 6.3 Techniques We propose two novel techniques to identify reused IP addresses. We use a BitTorrent-based crawler to identify NATed addresses and to estimate a lower bound on the number of users behind a NATed address. To identify dynamically addressed /24 prexes, we extend Padmanabhan et al. [161]’s idea of using the RIPE Atlas measurement logs. Our priorities in designing these approaches were: (1) IP address granularity, (2) high accuracy of a positive detection, and (3) reasonable coverage. In other words, we accept some loss in coverage to achieve the rst two goals: accuracy and ne granularity of detection. Our ndings are therefore a lower bound on reused addresses. 135 Crawler Port: 2215, 12281 Port: 155, 1821 Port: 6681 IP 1 IP 2 IP 3 (a) Crawler Port: 2215, 12281 Port: 155, 1821 Port: 6681 IP 1 IP 2 IP 3 2x bt_ping 2x bt_ping (b) Crawler Port: 2215, 12281 Port: 155, 1821 Port: 6681 IP 1 IP 2 IP 3 1x reply 2x reply (c) Crawler Port: 2215, 12281 Port: 155, 1821 Port: 6681 IP 1 IP 2 IP 3 NA T (d) Figure 6.1: BitTorrent crawler to detect NATed reused addresses. In (a), the crawler identies BitTorrent users with same IP address and multiple port numbers. In (b) and (c), the crawler sends bt_ping toIP 1 andIP 2 and receives replies. In (d), the crawler determinesIP 1 is a NATed reused address. 136 6.3.1 IdentifyingNATedAddresses We crawl the BitTorrent network to identify NATed addresses among BitTorrent users. BitTorrent is a popular peer-to-peer network for content exchange. A new BitTorrent user learns about other users as it joins the network. Every user generates its own unique 160-bit node_id that is obtained by hashing the (possibly private) IP address of the user and a random number. A new user learns IP addresses and port numbers of eight other users through the BitTorrent protocol – these users become the neighbors of the new user. The protocol supports two messages – bt_ping to periodically ping active neighbors and get_nodes to get a list of neighbors of any given node. We built a crawler that uses bt_ping and get_nodes messages to crawl the BitTorrent network and identify BitTorrent users using the same IP address at the same time, indicating such users are likely behind a NAT. Initially, the crawler sends a get_nodes message to the BitTorrent bootstrap node, which returns a set of its neighbors. The crawler maintains a list of discovered BitTorrent users, and issues further get_nodes messages to the users on the list. The messages are issued in the order of discovery time. Replies to the get_nodes messages include the IP address, port number,node_id and the BitTorrent version of the node. If the crawler nds a new user with an already discovered IP address, but with a dierent port number, it tries to establish the reason behind this occurrence: (1) multiple BitTorrent users are using the same IP address (NATed address), or (2) the BitTorrent user has changed the port number and the crawler encountered stale information. We do not use node_id to determine multiple BitTorrent nodes using the same IP address, because the BitTorrent user can regenerate a new node_id every time their machine reboots. To determine if more than one active BitTorrent users share the same IP address at the same time, the crawler issues bt_ping’s to all discovered ports behind a given IP address, and waits for responses. If the crawler gets more than two responses with two dierent node_id’s and two dierent port numbers, we conclude that the IP address is shared by multiple BitTorrent users. Our technique provides an underesti- mation of NATed addresses and users, but with a high precision. 137 Figure 6.1 shows an example of how our BitTorrent crawler nds NATed addresses. The crawler en- counters two users (IP 1 andIP 2) having the same IP address, but with two dierent ports in Figure 6.1a (with ports 2215, 12281 withIP 1 and ports 155, 1821 withIP 2). To verify if multiple active BitTorrent users are using the same IP address, the crawler issues four bt_ping messages in Figure 6.1b, one for each port across two IP addresses and waits for responses. The crawler receives two replies fromIP 2 and one reply fromIP 1 (in Figure 6.1c), therefore determining thatIP 2 is shared by multiple BitTorrent users and IP 1 is not (in Figure 6.1d). BitTorrent bt_ping messages are sent over UDP, which means that they could be lost in transit. To compensate for this, the crawler sends bt_ping messages every hour for all the IP addresses that have more than one discovered port. The crawler logs all the messages (bt_ping or get_nodes) sent and all the messages received with the timestamps, which are then processed to determine NATed addresses. Once the crawler sends out a message (get_nodes orbt_ping) to all discovered ports associated with an IP address, the crawler does not contact that same IP address for the next 20 minutes. Initially, we did not restrict our BitTorrent crawler. However, the ping replies generated tremendous amount of incoming trac that was undesirable to our network administrators. Therefore, to minimize the disturbance to users we probe and reduce burden on our network due to ping replies, the crawler is rate-limited and restricted only to address spaces where blocklists are present (Section 6.4). However, we could reduce this burden and have a faster coverage by having the crawler at multiple vantage points in dierent networks. Limitations: Our crawler can only detect NATed addresses that have more than one BitTorrent user. Our NAT detection technique is certainly biased towards BitTorrent users. We will miss the NATs in networks where BitTorrent is not popular, or where BitTorrent trac is ltered. Moreover, we can only detect the NATed addresses that are reused by more than one user using BitTorrent at the same time. We are thus likely to grossly underestimate the number of users behind a NAT. 138 6.3.2 IdentifyingDynamicAddresses The RIPE NCC’s Atlas project deploys custom devices that conduct various measurement tasks. Every RIPE Atlas probe connects to a central infrastructure to get instructions on new measurement tasks and to update measurement data. All measurements are logged to include the uniqueprobeID and the IP address through which the measurement was made. Probes are usually deployed within the customer premises equipment (CPE) of the user. We use the measurement logs to infer the dynamics of IP address allocation in the covering /24 address prex. We identify three properties of measurement logs that help us to nd dynamic addresses. 1)Dynamicaddressing observedovertime: We observe RIPE Atlas probe measurement logs for 16 months to determine dynamic addressing. Observing logs for a long duration allows us to understand the frequency of IP address reallocation in probes. 2)FrequencyofIPaddresschange: We consider probes that have gone through multiple IP address allocations within the same AS during the monitoring period, to eliminate probes that experienced IP address changes rarely and to remove probes that have changed locations. Among the remaining probes, we consider probes whose average duration between every IP address change is within 1 day. This helps to estimate the risk involved in blocklisting these IP addresses, since blocklisting may be eective only for one day, and will lead to unjust blocking afterward. 3) Extent of dynamic addressing: If we detect that an IP address is dynamic, according to the criteria we described above, we consider that the entire /24 prex covering this IP address is dynamically allocated. Usually, IP address reallocation occurs from a pool of IP addresses. It is hard to determine this address pool, and network operators do not make such information public. A conservative approach is to consider the entire /24 prex as dynamic as contiguous addresses are usually administered together and /24 prexes have been previously used to identify network characteristics [89, 41, 56, 57, 175]. 139 0 2k 4k 6k 8k 10k 12k 1 10 100 1000 threshold RIPE atlas probes (#) of allocated addresses Figure 6.2: IP addresses allocated to RIPE Atlas probes. We use RIPE Atlas connection logs from 1 Jan 2019 to 11 May 2020. In this period, we observe over 15,703 RIPE Atlas probes that were allocated 311K IP addresses (referred to as RIPE addresses). About 13.1% (or 2K) of probes go through address changes but have addresses allocated across multiple au- tonomous systems. Figure 6.2 shows the remaining 13.6K probes and the number of addresses allocated to them. The majority of the probes (59% or 9.3K) did not go through any IP address change in this period and the remaining 27% (or 4.2K) of probes go through multiple address changes. We determine probes that change IP addresses more frequently than others by obtaining the knee point of this graph. The knee point indicates the number of IP address reallocation required to dierentiate probes that change IP ad- dress more frequently when compared to remaining probes. We use a technique proposed by Satopää et al. [181] to determine the knee point to be at eight addresses. This leaves us with 16.6% (2.6K) of the probes that have at least eight IP address allocations during our monitoring period, and where all reallocated IP addresses belong to the same autonomous system. These 2.6K probes cover a total of 204K IP addresses, with an average of 78 IP addresses allocated to each probe. In other words, 16.6% of all RIPE Atlas probes contributed to 65.5% of all IP addresses allocated to all RIPE Atlas probes. As a nal step to consider probes that regularly change addresses within one day, we end up with 4% (629) of probes that are in dynamically allocated address spaces. We only consider probes that 140 change IP addresses daily, since these probes are more likely to cause unjust blocking than probes that rarely change addresses. Limitations: Our detection of dynamic addresses is limited only to IP prexes that deploy a RIPE Atlas probe. RIPE Atlas probes are predominantly present only in Europe and North America. Therefore, our insights on dynamic address detection are limited to these regions. Another limitation is that our detection technique involves choosing RIPE probes that have been allocated IP addresses from the same autonomous system. However, ISPs may own several autonomous systems and can allocate addresses from multiple autonomous systems. Even within the IP address spaces covered by RIPE Atlas probes, our technique only presents a lower-bound of prexes that are dynamically allocated. For probes that change their IP addresses frequently, we consider the entire /24 prex to be potentially dynamically allocated. However, we may incorrectly gauge these boundaries. Network operators may deploy dynamic addressing within larger prexes, leading us to underestimate the extent of unjust blocking. In some cases, they may deploy dynamic addressing within smaller prexes, thereby overcounting the number of dynamic addresses. Estimating boundaries is dicult because ISPs have their own private policies for dynamic addressing. This issue is noted in previous studies as well [89, 175, 227]. 6.4 Detection Blocklists typically list IP addresses that have sent spam, DDoS attacks, dictionary attacks, or malicious scans. To quantify unjust blocking by blocklists, we use 151 IPv4 public blocklists shown in Table 6.1 taken from the BLAG dataset [172]. These lists are actively maintained and they monitor a variety of malicious activities including Spam, DDoS, malware hosting or general reputation of IP addresses. This dataset includes popular lists like DShield [95], NixSpam [158], Spamhaus [196], Alienvault [5] and Abuse.ch [3]. We collect blocklist data for 83 days over two measurement periods from 03 Aug 2019 to 10 Sep 2019 (39 141 Maintainer #ofblocklists Bad IPs [15] 44 Bambenek [19] 22 *Abuse.ch [3] 10 Normshield [150] 9 *Blocklist.de [24] 9 Malware bytes [31] 9 *Project Honeypot [91] 4 CoinBlockerLists [235] 4 NoThink [152] 3 Emerging threats [203] 2 ImproWare [10] 2 Botvrij.EU [27] 2 IP Finder [68] 1 *Cleantalk [44] 1 Sblam! [182] 1 *Nixspam [158] 1 Blocklist Project [144] 1 BruteforceBlocker [74] 1 Cruzit [53] 1 Haley [84] 1 Botscout [26] 1 My IP [98] 1 Taichung [36] 1 *Cisco Talos [40] 1 Alienvault [5] 1 Binary Defense [61] 1 GreenSnow [80] 1 Snort Labs [117] 1 GPF Comics [47] 1 Turris [210] 1 CINSscore [39] 1 Nullsecure [155] 1 DYN [66] 1 Malware domain list [128] 1 Malc0de [132] 1 URLVir [212] 1 Threatcrowd [202] 1 CyberCrime [55] 1 IBM X-Force [94] 1 VXVault [214] 1 *Stopforumspam [194] 1 Total 151 Table 6.1: Each row shows the number of blocklists provided by the blocklist maintainer. We monitor 151 blocklists to determine the number of blocklisted addresses using NAT or are dynamically allocated. Blocklists used by network operators who took our survey are marked with (*). 142 1 10 100 1000 10k 100k 0.001 0.01 0.1 1 blocklisted addresses blocklisted BitTorrent addresses blocklisted RIPE addresses (#) of ASes CDF Figure 6.3: CDF of blocklisted and reused addresses from each AS. Probes with address changes in same AS ( 34.4K) Probes with frequent address changes ( 33.1K) Probes that change address daily ( 22.7K) BitTorrent IPs ( 48.7M) NATed IPs ( 2M) NATed + blocklisted IPs ( 29.7K) NATed addresses Dynamic addresses Blocklisted addresses in RIPE prefixes ( 53.7K) Figure 6.4: Detecting NATed and dynamic addresses. days) and 29 Mar 2020 to 11 May 2020 (44 days). During our measurement periods, we observed 2.2M IP addresses and each blocklist, on average, has 30K IP addresses. Figure 6.4 shows the number of reused addresses discovered by our techniques. We ran our BitTorrent crawler at the same time as the blocklist collection period. To prevent unnecessary probing, we restrict the crawler to the blocklisted address space of 899K /24 prexes (as discussed in Section 6.3.1). During this period, the crawler sent 1.6 billion bt_ping messages and received 779M responses (48.6% response rate). The crawler identied a total of 48.7M unique IP addresses that use BitTorrent, belonging to 203M unique node_id’s. Among the discovered BitTorrent IP addresses, 2M are NATed; out of these, 29.7K are blocklisted. This overlap of blocklisted addresses with BitTorrent addresses is in agreement with previous work [62] 143 10 20 30 40 50 60 70 80 90 1 10 100 1000 10k (#) of blocklists log(#) (a) NATed addresses in blocklists. 10 20 30 40 50 60 70 1 10 100 1000 10k RIPE Cai et al. (#) of blocklists log(#) (b) Dynamic addresses in block- lists. 0 10 20 30 40 0 0.2 0.4 0.6 0.8 1 NATed addresses blocklisted addresses dynamic addresses (#) of days in blocklists CDF of IP addresses (c) Duration distribution of reused addresses. that nd devices using P2P are likely to be compromised. We use the 311K RIPE addresses obtained from Section 6.3.2, and convert them into 90.5K /24 RIPE prexes. About 53.7K blocklisted addresses are in RIPE prexes. The overlap with RIPE prexes decreases as we apply our technique. At rst, by eliminating all probes that change addresses across multiple ASes, the number of blocklisted addresses reduces to 34.4K. While considering probes that have at least eight address changes, the number of blocklisted addresses further reduces to 33.1K. Finally, after considering probes that change their addresses daily, we have 22.7K blocklisted addresses that are dynamically allocated. To determine the feasibility of our techniques to discover reused addresses, we estimate (shown in Fig- ure 6.3) the extent of overlap between the address spaces where BitTorrent or RIPE addresses are present and the address spaces where blocklisted addresses are present. The ASes in Figure 6.3 are arranged in increasing order of the number of blocklisted addresses present in them. Blocklisted addresses are present in about 26K autonomous systems, therefore, the curve for blocklisted addresses reaches upto 1. Block- listed addresses using BitTorrent are present in 7.7K (or 29.6%) ASes and blocklisted addresses among RIPE prexes are present in 1.9K (or 17.1%) ASes. Since our techniques do not cover the entire blocklisted addresses, the curves for blocklisted RIPE and BitTorrent addresses plateau at 7.7K and 1.1 ASes respec- tively. Our technique has signicant coverage in the ten most blocklisted ASes. These ASes contribute to 606K (or 27.7%) of all blocklisted addresses. Among these addresses, 38K (6.4%) use BitTorrent and 4.4K (0.7%) are among RIPE prexes. AS4134, belonging to China Telecom Backbone, has the highest number 144 of blocklisted addresses (202K or 9%). Among the blocklisted addresses from AS4134, about 3% (or 6.2K) use BitTorrent and 0.4% (or 817) are in RIPE prexes. 6.5 Analysis In this section, we estimate the potential extent of unjust blocking caused by blocklisting reused addresses. Although we identify only 29.7K blocklisted NATed addresses and 22.7K blocklisted dynamic addresses, we observe that reused addresses are present in many blocklists (up to 60%). Blocklisting reused addresses can have serious impact on users. Blocklisting NATed addresses can lead to unjust blocking of multiple users having the same IP address and dynamically allocated addresses can lead to unjust blocking of a user that has been newly allocated a previously blocklisted address. We nd that a reused address can be present in blocklists for as many as 44 days and can potentially block as many as 78 users. Reused addresses are present in many blocklists: Figure 6.5a and Figure 6.5b shows blocklists that list reused addresses. There are 61 blocklists (40% of all blocklists) that do not list any NATed addresses and 72 blocklists (47% of all blocklists) that do not list any dynamic addresses. We discover 45.1K listings † that include 29.7K IP addresses that are NATed. We also discover 30.6K listings that include 22.7K IP addresses that are dynamic addresses. On average, a blocklist lists 501 NATed IP addresses and 387 dynamic addresses. Our techniques of reused address detection is not perfect, therefore, blocklists that do not show any reused addresses can contain reused addresses but are not identied by our techniques. Someblocklistslistmorereusedaddressesthanothers: The top 10 blocklists contribute 65.9% of all listings of NATed addresses and 72.6% of all listings of dynamic addresses. This is expected, as the top 10 blocklists among NATed and dynamic addresses contribute to 53.4% and 70.3% of all blocklisted addresses. The three highest presence of NATed addresses are from spam or reputation blocklists – Stopforumspam, Nixspam andAlienvault listing about 3.3K-8.6K (or 0.01%–0.03% of blocklists) such IP addresses. Similarly, † An IP address can be present in dierent blocklists, therefore the number of listings need not be equal to the number of reused IP addresses. 145 the three highest presence of dynamic addresses are fromStopforumspam,Nixspam andBadIPs, primarily used to mitigate spam and identify IP addresses with a poor reputation. These blocklists list about 2.6K– 5.9K (or 0.01%–0.02% of blocklists) dynamic addresses. Mostreusedaddressesareremovedquickly: Figure 6.5c shows the CDF of all blocklisted addresses, blocklisted NATed/dynamic addresses and the duration in days that they were present in a blocklist. On average, blocklisted addresses are removed within nine days, NATed IP addresses are removed within ten days, and dynamic addresses are removed within three days from blocklists. Compared to all blocklisted addresses, reused addresses are removed much faster and among reused addresses, dynamic addresses are removed faster than NATed IP addresses. Within two days, 77.5% of all dynamic addresses are removed from blocklists, compared to only 60% of NATed IP addresses. On the other hand, only 42% of all blocklisted IP addresses are removed in two days. In the worst case, reused addresses are present in blocklists for the entire monitoring period of 44 days. Blocklisting NATed addresses impact many users: The BitTorrent crawler described in Sec- tion 6.3.1, helps us quantify the lower bound of active users using the same IP address that would be aected by blocklisting. Figure 6.6 shows the CDF of NATed blocklisted addresses and the lower bound of aected users. For most of these IP addresses, we detect only two active users (68.5%). 97.8% of the IP addresses have fewer than ten active users. The remaining 2.2% of the IP addresses are shared by many more users. At the maximum, we detect 78 active users behind an IP address. Exploring other techniques: As discussed in Section 6.2, there are many other techniques for de- tecting reused addresses. The only technique that can be reproduced at scale is Cai et al. [32]. We use their techniques and the datasets (IT86c andIT89w) that most closely match our monitoring periods to compare dynamic addresses. Figure 6.5b (black line) shows the number of blocklisted addresses that overlap with their study. Cai et al. have broader coverage in some blocklists compared to this study; this is likely due to the absence of RIPE Atlas probes in those address regions. We nd that the total number of listings 146 10 20 30 40 50 60 70 0.7 0.75 0.8 0.85 0.9 0.95 1 (#) of users with the same IP address CDF of IP addresses Figure 6.6: Number of users behind NATed addresses in blocklists. Question Response Blocklist usage External blocklists 85% Paid-for blocklists Avg:2 Max:39 Public blocklists Avg:10 Max:68 Active defense Directly block IPs 59% Threat intelligence system 35% Issues Dynamic addressing* 76% Carrier-grade NATs* 56% Table 6.2: Summary of survey responses on usage of blocklists. Questions in (*) were only taken by 34 out of 65 respondants. discovered by Cai et al. is roughly the same as our technique: they detect 29.8K listings compared to 30.6K listings using our technique. 6.6 Understandingblocklistsusage We survey 65 network operators, on how they use blocklists to identify malicious trac, the role blocklists play in trac ltering, and their perceptions of limitations of blocklists due to reused addresses. Questions that permitted open-ended responses are denoted with asterisks. 1. What is your company’s name and AS number if available?* 147 2. What is your position / your role in network management?* 3. What is your email address?* 4. May we reach out to you via email: to inform you once the results of this survey are publicly available 5. May we reach out to you via email: with further questions 6. What type of network do you run? (more than one choice possible) 7. How many subscribers do you connect to the Internet? 8. In what geographic region(s) do you operate? 9. Do you maintain internal blocklists? 10. How and why did you develop internal blocklists? How do they compare to third-party blocklists?* 11. How many third-party blocklists do you use? 12. Which of the following types of third-party blocklists do you use? (Please select all that apply) 13. What factors determine which third-party blocklists you use?* 14. Do you use third-party blocklists to directly block malicious activity? 15. Do you use third-party blocklists as an input to a threat intelligence system? 16. In your experience, do third-party blocklists provide accurate information on threats? 17. What are the shortcomings of any third-party blocklists you are familiar with?* 18. What are the strengths of any third-party blocklists you are familiar with?* 19. How do your ltering practices vary according to type of attack or blocklist?* 148 20. To help us map your responses to the blocklists we are monitoring, please list the third-party block- lists you use.* 21. Do you see the quality of blocklists being aected by: Dynamic addressing 22. Do you see the quality of blocklists being aected by: Carrier grade NATs 23. Do you see the quality of blocklists being aected by: Other* 24. How could blocklists be improved?* 25. Do you donate data from your network to community blocklist sources (such as Project Honeypot or DShield)? 26. Is there anything else you would like to share with us?* Our survey indicates that 85% of respondents use blocklists, and 59% of respondents use blocklists to directly block malicious trac (Table 6.2). 34 survey participants responded to direct questions about the impact of reused addresses. Of these, 56% believe that blocklists are inaccurate due to NAT and 76% believe dynamic addressing introduces inaccuracies. From these responses as well as open-ended comments made within the survey, it is evident that network operators use blocklists and are aware of unjust blocking caused by reused addresses. We make our crawler and scripts to determine reused addresses public ‡ . We believe that our lists can assist network operators in many ways. Depending on the blocklist type, a network operator could take necessary action on their incoming trac with our list. For instance, operators that use DDoS blocklists to reduce intensity of the attacker should block all trac listed in DDoS blocklists, even if there is collateral damage due to reused addresses. On the other hand, network operators using application-specic blocklists (such as spam blocklist) that require more accuracy, can use our list to implement greylisting [115], which is already built into popular spam ltering systems like Spamassassin [224] or Spamd [34]. ‡ https://steel:isi:edu/members/sivaram/blocklisting_impact/ 149 Our lists can also provide incentives to blocklist maintainers to maintain more accurate blocklists. They may identify malicious reused IP addresses in a separate greylist to their customers. Finally, our lists can assist services such as Google or Cloudare, to warn users that their IP address is reused along a compromised device. This could help users to clean up their home network, or even request help from their ISP to identify other compromised users. 6.7 Conclusion We present two techniques to identify reused addresses and analyze 151 publicly available IPv4 blocklists to quantify their impact. We nd 53–60% of blocklists list at least one reused address. We also nd 30.6K– 45.1K listings of reused addresses in blocklists. Reused addresses can be present in blocklists for as long as 44 days aecting as much as 78 users. Finally, to assist blocklist maintainers to reduce unjust blocking, we made our discovered reused addresses public. 150 Chapter7 Conclusions In this dissertation, we discussed challenges for network management in today’s evolving networks. First, monitoring all events in the network is expensive. Even among events monitored, some events are rele- vant at the microsecond granularity (e.g., microbursts) and others at a larger granularity (e.g., persistent congestion). Second, there is a lack of coordination between the victim and upstream networks that hin- ders sharing insights on attack-related events. Finally, whenever attack-related events are recorded by individual networks and shared as blocklists, they oer snapshots of unreliable attack reports, which are not useful for emergency response. To overcome this, we presented frameworks that utilize programmable switches and machine learning to improve network security and performance: 1. Frameworks that eciently monitor and react to network events: We presented SPred, an approach that allows machine learning models to be implemented in switches to predict transient events before they occur. We also presented SDProber, an algorithm which allows adaptable moni- toring of persistent delay in a network. SDProber uses dedicated probe packets to monitor links that are prone to failures than links that are stable. 151 2. Frameworkthatallowscoordinationbetweenthevictimandupstreamnetworksforbetter detection/mitigation of DDoS attacks: We presented SENSS, a framework that leverages pro- grammable switches and SDN to allow victims to query upstream networks about their own trac. This allows victims to develop their own custom algorithms for characterization of attack events, and also to devise eective attack signatures to mitigate the attack away from the victim’s network. 3. Framework to combine blocklists of dierent types to improve attack detection: We pre- sented BLAG, an appoach that uses blocklists of dierent attack types to create an aggregate block- list, suitable for emergency response. For each network that adopts BLAG, this aggregate blocklist is customized, by using a sample of the network’s legitimate trac and a recommendation system, to prune out addresses that could lead to misclassication. We also presented a technique to identify reused addresses in blocklists using a BitTorrent crawler and RIPE atlas probe measurement logs. Our frameworks also made an impact on real networks. SENSS has been experimentally deployed in three research and education networks for better detection of DDoS attacks. Our technique to identify reused addresses is being deployed at IPInfo [51], an online service that maintains a large repository of IP address related information. We have open sourced BLAG and actively collect blocklists from dierent sources daily. Finally, SPred and SDProber are patented with AT&T and SDProber has also been partially deployed at AT&T and released as open source. 152 Bibliography [1] Perrig A., Szalachowski P., Reischuk R.M., and Chuat L. The SCION Architecture. Springer, 2017. [2] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jerey Dean, Matthieu Devin, Sanjay Ghemawat, Georey Irving, Michael Isard, et al. “Tensorow: A system for large-scale machine learning”. In:12thfUSENIXgSymposiumonOperatingSystemsDesignand Implementation (fOSDIg 16). 2016, pp. 265–283. [3] Abuse.ch. Swiss Security Blog - Abuse.ch. https://www.abuse.ch/. May 2020. [4] The University of Adelaide. The Internet Topology Zoo. http://www.topology-zoo.org/. 2013. [5] Alienvault. Alienvault Reputation System. https://www.alienvault.com/. May 2020. [6] Omid Alipourfard, Jiaqi Gao, Jeremie Koenig, Chris Harshaw, Amin Vahdat, and Minlan Yu. “Risk based planning of network changes in evolving data centers”. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles. 2019, pp. 414–429. [7] Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, et al. “CONGA: Distributed congestion-aware load balancing for datacenters”. In: Proceedings of the 2014 ACM conference on SIGCOMM. 2014, pp. 503–514. [8] Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. “Data center tcp (dctcp)”. In: Proceedings of the ACM SIGCOMM 2010 conference. 2010, pp. 63–74. [9] Dana Angluin. “Learning regular sets from queries and counterexamples”. In: Information and computation 75.2 (1987), pp. 87–106. [10] Antispam. ImproWare. http://antispam.imp.ch/. (Accessed on 05/13/2020). May 2020. [11] Katerina Argyraki and David R. Cheriton. “Active Internet Trac Filtering: Real-time Response to Denial-of-service Attacks”. In: Proceedings of the Annual Conference on USENIX Annual Technical Conference. ATEC ’05. Anaheim, CA: USENIX Association, 2005, pp. 10–10.url: http://dl.acm.org/citation.cfm?id=1247360.1247370. 153 [12] Charles Arthur. Can an American judge take a British company oine? Ed. by The Guardian. Oct. 2006.url: https://www.theguardian.com/technology/2006/oct/19/guardianweeklytechnologysection3. [13] Alon Atary and Anat Bremler-Barr. “Ecient Round-Trip Time Monitoring in OpenFlow Networks”. In: Computer Communications, IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on. IEEE. 2016, pp. 1–9. [14] Automated IP reputation system for Spammers. http://www.chaosreigns.com/iprep/. Sept. 2019. [15] BadIPs. badips.com | an IP based abuse tracker. https://www.badips.com/. May 2020. [16] Wei Bai, Kai Chen, Li Chen, Changhoon Kim, and Haitao Wu. “Enabling ECN over generic packet scheduling”. In: Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies. 2016, pp. 191–204. [17] R. Bajcsy, T. Benzel, M. Bishop, et al. “Cyber Defense Technology Networking and Evaluation”. In: Commun. ACM 47.3 (2004). [18] Sean Bakley. From Comcast to Hawaiian Telcom: Tracking the top 16 residential broadband service providers in Q3 2017. FierceTelecom, https://goo.gl/otRTw2. 2017. [19] Bambenek. Bambenek Consulting Feeds. http://osint.bambenekconsulting.com/feeds/. May 2020. [20] Cristina Basescu, Raphael M. Reischuk, Pawel Szalachowski, Adrian Perrig, Yao Zhang, Hsu-Chun Hsiao, Ayumu Kubota, and Jumpei Urakawa. “SIBRA: Scalable Internet Bandwidth Reservation Architecture”. In: CoRR abs/1510.02696 (2015). [21] Steven M Bellovin. “A technique for counting NATted hosts”. In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment. ACM. 2002, pp. 267–272. [22] Ran Ben Basat, Sivaramakrishnan Ramanathan, Yuliang Li, Gianni Antichi, Minian Yu, and Michael Mitzenmacher. “PINT: probabilistic in-band network telemetry”. In: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 2020, pp. 662–680. [23] Robert Beverly. “A robust classier for passive TCP/IP ngerprinting”. In: International Workshop on Passive and Active Network Measurement. Springer. 2004, pp. 158–167. [24] Blocklist.de. Blocklist.de fail2ban reporting service. https://www.blocklist.de/en/index.html. May 2020. [25] Jean-Chrysotome Bolot. “End-to-end packet delay and loss behavior in the Internet”. In: ACM SIGCOMM Computer Communication Review 23.4 (1993), pp. 289–298. [26] Botscout. We catch bots so that you don’t have to. https://www.botscout.com. May 2020. [27] Botvrij. botvrij.eu - powered by MISP. http://www.botvrij.eu/. May 2020. 154 [28] Broadcom. In-band Telemetry. https://people.ucsc.edu/~warner/Bufs/Trident3-telemetry.pdf. [29] Jean-Philippe Brunet, Pablo Tamayo, Todd R Golub, and Jill P Mesirov. “Metagenes and molecular pattern discovery using matrix factorization”. In: Proceedings of the national academy of sciences 101.12 (2004), pp. 4164–4169. [30] Coralie Busse-Grawitz, Roland Meier, Alexander Dietmüller, Tobias Bühler, and Laurent Vanbever. “pForest: In-Network Inference with Random Forests”. In: arXiv preprint arXiv:1909.05680 (2019). [31] Malware Bytes. hpHosts - by Malware Bytes. https://hosts-file.net/. May 2020. [32] Xue Cai and John Heidemann. “Understanding block-level address usage in the visible internet”. In: Proceedings of the ACM SIGCOMM 2010 conference. 2010, pp. 99–110. [33] CAIDA. The CAIDA AS Relationships Dataset, May 01, 2017. http://www.caida.org/data/as-relationships/. 2017. [34] Calomel. Spamd tarpit and greylisting daemon. https://calomel.org/spamd_config.html. Jan. 2017. [35] Martin Casado and Michael J Freedman. “Peering through the shroud: The eect of edge opacity on IP-based client identication”. In: Proceedings of the 4th USENIX conference on Networked systems design & implementation. USENIX Association. 2007, pp. 13–13. [36] Taichung Education Center. Taichung Education Center. https://www.tc.edu.tw/net/netflow/lkout/recent/30. May 2020. [37] Xiaoqi Chen, Shir Landau Feibish, Yaron Koral, Jennifer Rexford, Ori Rottenstreich, Steven A Monetti, and Tzuu-Yi Wang. “Fine-grained queue measurement in the data plane”. In: Proceedings of the 15th International Conference on Emerging Networking Experiments And Technologies. 2019, pp. 15–29. [38] Sharad Chole, Andy Fingerhut, Sha Ma, Anirudh Sivaraman, Shay Vargaftik, Alon Berger, Gal Mendelson, Mohammad Alizadeh, Shang-Tse Chuang, Isaac Keslassy, et al. “drmt: Disaggregated programmable switching”. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 2017, pp. 1–14. [39] CIArmy. CINSscore. http://ciarmy.com/. May 2020. [40] Cisco. Cisco Talos - Additional Resources. http://www.talosintelligence.com/. May 2020. [41] Kimberly Clay, Young Hyun, Ken Keys, Marina Fomenkov, and Dmitri Krioukov. “Internet mapping: from art to science”. In: 2009 Cybersecurity Applications & Technology Conference for Homeland Security. IEEE. 2009, pp. 205–211. [42] Benoit Claise, Ganesh Sadasivan, Vamsi Valluri, and Martin Djernaes. “Cisco systems netow services export version 9”. In: (2004). 155 [43] Clean MX - Realtime DB. https://web.archive.org/web/20161102031447/http://clean-mx.com:80/. Sept. 2019. [44] Cleantalk. Cloud spam protection for forums, boards, blogs and sites. https://www.cleantalk.org. May 2020. [45] Cloudare. Understanding Cloudare Challenge Passage (Captcha). https://support.cloudflare.com/hc/en-us/articles/200170136. Feb. 2020. [46] Cloudare Inc. https://www.cloudflare.com. Sept. 2019. [47] GPF Comics. The GPF DNS Block List. https://www.gpf-comics.com/dnsbl/. May 2020. [48] Cloudare Community. Blocked IP address: Sharing IPs. https://community.cloudflare.com/t/cloudflare-blocking-my-ip/65453/57. Mar. 2019. [49] Cloudare Community. Community Tip - Best Practices For Captcha Challenges. https: //community.cloudflare.com/t/community-tip-best-practices-for-captcha-challenges/56301. Jan. 2019. [50] Cloudare Community. Getting Cloudare capcha on almost every website I visit for my home network. Help! https://community.cloudflare.com/t/getting-cloudflare-capcha-on-almost-every- website-i-visit-for-my-home-network-help/42534. Jan. 2018. [51] Comprehensive IP address data, IP geolocation API and database - IPinfo.io. [Online; accessed 4. May 2021]. May 2021.url: https://ipinfo.io. [52] CriticalStack - The secure container orchestration platform for the enterprise. https://criticalstack.com/. Sept. 2019.url: https://criticalstack.com/. [53] CruzIt. Server Blocklist / Blacklist - CruzIT.com - PHP, Linux & DNS Tools, Apache, MySQL, Postx, Web & Email Spam Prevention Information. http://www.cruzit.com/wbl.php. May 2020. [54] Andrew R Curtis, Jerey C Mogul, Jean Tourrilhes, Praveen Yalagandula, Puneet Sharma, and Sujata Banerjee. “DevoFlow: Scaling ow management for high-performance networks”. In: ACM SIGCOMM Computer Communication Review 41.4 (2011), pp. 254–265. [55] Cybercrime. CyberCrime Tracker. http://cybercrime-tracker.net/. May 2020. [56] Alberto Dainotti, Karyn Benson, Alistair King, KC Clay, Michael Kallitsis, Eduard Glatz, and Xenofontas Dimitropoulos. “Estimating internet address space usage through passive measurements”. In: ACM SIGCOMM Computer Communication Review 44.1 (2013), pp. 42–49. [57] Alberto Dainotti, Karyn Benson, Alistair King, Bradley Huaker, Eduard Glatz, Xenofontas Dimitropoulos, Philipp Richter, Alessandro Finamore, and Alex C Snoeren. “Lost in space: improving inference of IPv4 address space utilization”. In: IEEE Journal on Selected Areas in Communications 34.6 (2016), pp. 1862–1876. [58] Darklist.de | A malicious IP catcher blacklist. http://darklist.de/. Sept. 2019. 156 [59] DDoS Open Threat Signaling (dots). https://datatracker.ietf.org/wg/dots/documents/. 2018. [60] Rogério Leão Santos De Oliveira, Ailton Akira Shinoda, Christiane Marie Schweitzer, and Ligia Rodrigues Prete. “Using mininet for emulation and prototyping software-dened networks”. In: Communications and Computing (COLCOM), 2014 IEEE Colombian Conference on. IEEE. 2014, pp. 1–6. [61] Binary Defense. Binary Defense Systems | Defend. Protect. Secure. https://www.binarydefense.com/. May 2020. [62] Louis F DeKoven, Audrey Randall, Ariana Mirian, Gautam Akiwate, Ansel Blume, Lawrence K Saul, Aaron Schulman, Georey M Voelker, and Stefan Savage. “Measuring Security Practices and How They Impact Security”. In: Proceedings of the Internet Measurement Conference. 2019, pp. 36–49. [63] T Dierks and E Rescorla. “Rfc 5246: The transport layer security (tls) protocol”. In: The Internet Engineering Task Force (2008). [64] Christoph Dietzel, Anja Feldmann, and Thomas King. “Blackholing at ixps: On the eectiveness of ddos mitigation in the wild”. In: International Conference on Passive and Active Network Measurement. Springer. 2016, pp. 319–332. [65] Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, Brighten Godfrey, and Michael Schapira. “fPCCg vivace: Online-learning congestion control”. In: 15thfUSENIXg Symposium on Networked Systems Design and Implementation (fNSDIg 18). 2018, pp. 343–356. [66] DYN. Index of /pub/malware-feeds/. http://security-research.dyndns.org/pub/malware-feeds/. May 2020. [67] Emerging Threats. https://rules.emergingthreats.net/. Sept. 2019. [68] IP nder. IP Blacklist Cloud - Protect your website. https://www.ip-finder.me/. May 2020. [69] Pawel Foremski, David Plonka, and Arthur Berger. “Entropy/ip: Uncovering structure in ipv6 addresses”. In: Proceedings of the 2016 Internet Measurement Conference. 2016, pp. 167–181. [70] Comcast Forums. Dirty (blacklisted) IPs issued to Comcast Business Account holders. https://forums.businesshelp.comcast.com/t5/Connectivity/Dirty-blacklisted-IPs-issued-to- Comcast-Business-Account-holders/td-p/34297. Mar. 2018. [71] Verizon Forums. IP address blocked by SORBS, Verizon will do nothing. https://forums.verizon.com/t5/Fios-Internet/IP-address-blocked-by-SORBS-Verizon-will-do- nothing/td-p/892536. Feb. 2020. [72] Lixin Gao. “On Inferring Autonomous System Relationships in the Internet”. In: Proc. IEEE Global Internet Symposium. 2000. [73] Felix A Gers, Douglas Eck, and Jürgen Schmidhuber. “Applying LSTM to time series predictable through time-window approaches”. In: Neural Nets WIRN Vietri-01. Springer, 2002, pp. 193–200. 157 [74] Daniel Gerzo. Daniel Gerzo BruteForceBlocker. http://danger.rulez.sk/index.php/bruteforceblocker/. May 2020. [75] Soudeh Ghorbani, Zibin Yang, P Godfrey, Yashar Ganjali, and Amin Firoozshahian. “DRILL: Micro load balancing for low-latency data center networks”. In:ProceedingsoftheConferenceoftheACM Special Interest Group on Data Communication. ACM. 2017, pp. 225–238. [76] Phillipa Gill, Navendu Jain, and Nachiappan Nagappan. “Understanding Network Failures in Data Centers: Measurement, Analysis, and Implications”. In: SIGCOMM Comput. Commun. Rev. 41.4 (Aug. 2011), pp. 350–361.issn: 0146-4833.doi: 10.1145/2043164.2018477. [77] Michael T. Goodrich. “Probabilistic Packet Marking for Large-scale IP Traceback”. In: IEEE/ACM Trans. Netw. 16.1 (Feb. 2008), pp. 15–24.issn: 1063-6692.doi: 10.1109/TNET.2007.910594. [78] Prateesh Goyal, Preey Shah, Naveen Kr Sharma, Mohammad Alizadeh, and Thomas E Anderson. “Backpressure Flow Control”. In: Proceedings of the 2019 Workshop on Buer Sizing. 2019, pp. 1–3. [79] Graphiclineweb. https://graphiclineweb.wordpress.com/tech-notes/ip-blacklist/. Sept. 2019. [80] Greensnow. Greensnow Statistics. https://greensnow.co/. May 2020. [81] Tristan Groleat, Sandrine Vaton, and Matthieu Arzel. “High-speed ow-based classication on FPGA”. In: International journal of network management 24.4 (2014), pp. 253–271. [82] MAWI group. MAWI Working Group Trac Archive. http://mawi.wide.ad.jp/mawi/. 2017. [83] Chuanxiong Guo, Lihua Yuan, Dong Xiang, Yingnong Dang, Ray Huang, Dave Maltz, Zhaoyi Liu, Vin Wang, Bin Pang, Hua Chen, et al. “Pingmesh: A large-scale system for data center network latency measurement and analysis”. In: ACM SIGCOMM Computer Communication Review 45.4 (2015), pp. 139–152. [84] Charles B. Haley. SSH Dictionary Attacks. http://charles.the-haleys.org/. May 2020. [85] Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown. “I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks.” In: NSDI. Vol. 14. 2014, pp. 71–85. [86] Shuang Hao, Nadeem Ahmed Syed, Nick Feamster, Alexander G Gray, and Sven Krasser. “Detecting Spammers with SNARE: Spatio-temporal Network-level Automatic Reputation Engine”. In: USENIX security symposium. Vol. 9. 2009. [87] David Harrington, Randy Presuhn, and Bert Wijnen. RFC3411: An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks. 2002. [88] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009. 158 [89] John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. “Census and survey of the visible internet”. In: Proceedings of the 8th ACM SIGCOMM conference on Internet measurement. 2008, pp. 169–182. [90] Sepp Hochreiter and Jürgen Schmidhuber. “Long short-term memory”. In: Neural computation 9.8 (1997), pp. 1735–1780. [91] Project Honeypot. Project Honeypot. https://www.projecthoneypot.org/. May 2020. [92] Anushah Hossain, Sivaramakrishnan Ramanthan, and Sadia Afroz. Poster: Perceptions of third part blacklists. https://www.dropbox.com/s/i37ze8g4804owwz/usenix_poster.pdf?dl=0. Aug. 2019. [93] I-BlockList. https://www.iblocklist.com/. Sept. 2019. [94] IBM. IBM X-Force Exchange. https://exchange.xforce.ibmcloud.com/. May 2020. [95] SANS Institute. Internet Storm Center. https://dshield.org/about.html. Sept. 2019. [96] Intel® Tono™ 2 P4 Programmability with More Bandwidth. https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet- switch/tofino-2-series/tofino-2.html. (Accessed on 01/25/2021). [97] John Ioannidis and Steven M. Bellovin. “Pushback: Router-Based Defense Against DDoS Attacks”. In: NDSS. 2002. [98] My IP. My IP - Blacklist Checks. https://www.myip.ms/info/about. May 2020. [99] ANT ISI. https://nsnam.isi.edu/datasets/readmes/B_Root_Anomaly-20160625.README.txt. https://nsnam.isi.edu/datasets/readmes/B_Root_Anomaly-20160625.README.txt. June 2019. [100] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar. “A deep reinforcement learning perspective on internet congestion control”. In: International Conference on Machine Learning. PMLR. 2019, pp. 3050–3059. [101] Theo Jepsen, Masoud Moshref, Antonio Carzaniga, Nate Foster, and Robert Soulé. “Packet subscriptions for programmable asics”. In: Proceedings of the 17th ACM Workshop on Hot Topics in Networks. 2018, pp. 176–183. [102] Raj Joshi, Ting Qu, Mun Choon Chan, Ben Leong, and Boon Thau Loo. “BurstRadar: Practical Real-time Microburst Monitoring for Datacenter Networks”. In: (2018). [103] Michael G. Kallitsis, Stilian Stoev, Shrijita Bhattacharya, and George Michailidis. “AMON: An Open Source Architecture for Online Monitoring, Statistical Analysis and Forensics of Multi-gigabit Streams”. In: CoRR abs/1509.00268 (2015). [104] Min Suk Kang, Virgil D. Gligor, and Vyas Sekar. “Defending Against Evolving DDoS Attacks: A Case Study Using Link Flooding Incidents”. In: Security Protocols Workshop. Vol. 10368. Lecture Notes in Computer Science. Springer, 2016, pp. 47–57. 159 [105] Min Suk Kang, Virgil D. Gligor, and Vyas Sekar. “SPIFFY: Inducing Cost-Detectability Tradeos for Persistent Link-Flooding Attacks”. In: NDSS. 2016. [106] Min Suk Kang, Soo Bum Lee, and Virgil D. Gligor. “The Crossre Attack”. In: IEEE Symposium on Security and Privacy. 2013. [107] Naga Katta, Mukesh Hira, Changhoon Kim, Anirudh Sivaraman, and Jennifer Rexford. “HULA: Scalable Load Balancing Using Programmable Data Planes”. In: Proceedings of the Symposium on SDN Research. SOSR ’16. Santa Clara, CA, USA: Association for Computing Machinery, 2016. isbn: 9781450342117.doi: 10.1145/2890955.2890968. [108] Naga Katta, Mukesh Hira, Changhoon Kim, Anirudh Sivaraman, and Jennifer Rexford. “Hula: Scalable load balancing using programmable data planes”. In: Proceedings of the Symposium on SDN Research. 2016, pp. 1–12. [109] Changhoon Kim, Ed Doe, Hugh Holbrook, Anoop Ghanwani, Dan Daly, Mukesh Hira, and Bruce Davie. In-band Network Telemetry (INT). http://p4.org/wp-content/uploads/fixed/INT/INT-current-spec.pdf. 2016. [110] Y. Kim, W. C. Lau, M. C. Chuah, and H. J. Chao. “PacketScore: A Statistics-Based Packet Filtering Scheme against Distributed Denial-of-Service Attacks”. In: IEEE Trans. Depend. Secur. Comput. 3.2 (Apr. 2006). [111] Simon Knight, Hung X Nguyen, Nick Falkner, Rhys Bowden, and Matthew Roughan. “The internet topology zoo”. In: IEEE Journal on Selected Areas in Communications 29.9 (2011), pp. 1765–1775. [112] Yehuda Koren, Robert Bell, and Chris Volinsky. “Matrix Factorization Techniques for Recommender Systems”. In: Computer 42.8 (Aug. 2009), pp. 30–37.issn: 0018-9162.doi: 10.1109/MC.2009.263. [113] KrebsOnSecurity Hit With Record DDoS. https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/. Sept. 2016. [114] Christian Kreibich, Nicholas Weaver, Boris Nechaev, and Vern Paxson. “Netalyzr: illuminating the edge network”. In: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement. ACM. 2010, pp. 246–259. [115] M Kucherawy and D Crocker. Email greylisting: An applicability statement for smtp. Tech. rep. RFC 6647, June, 2012. [116] Marc Kührer, Christian Rossow, and Thorsten Holz. “Paint it black: Evaluating the eectiveness of malware blacklists”. In: International Workshop on Recent Advances in Intrusion Detection. Springer. 2014, pp. 1–21. [117] Snort Labs. Sourcere VRT Labs. https://labs.snort.org/. May 2020. [118] Kevin Lai and Mary Baker. “Measuring link bandwidths using a deterministic model of packet delay”. In: ACM SIGCOMM Computer Communication Review 30.4 (2000), pp. 283–294. 160 [119] Lashback Blacklist. http://blacklist.lashback.com/. Sept. 2019. [120] Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński, and Wouter Joosen. “Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation”. In: Proceedings of the 26th Annual Network and Distributed System Security Symposium. NDSS 2019. 2019.doi: 10.14722/ndss.2019.23386. [121] Daniel D Lee and H Sebastian Seung. “Algorithms for non-negative matrix factorization”. In: Advances in neural information processing systems. 2001, pp. 556–562. [122] Soo Bum Lee, Min Suk Kang, and Virgil D. Gligor. “CoDef: Collaborative Defense Against Large-scale Link-ooding Attacks”. In: Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies. CoNEXT ’13. Santa Barbara, California, USA: ACM, 2013, pp. 417–428.isbn: 978-1-4503-2101-3.doi: 10.1145/2535372.2535398. [123] Vector Guo Li, Matthew Dunn, Paul Pearce, Damon McCoy, Georey M. Voelker, and Stefan Savage. “Reading the Tea leaves: A Comparative Analysis of Threat Intelligence”. In: 28th USENIX Security Symposium (USENIX Security 19). Santa Clara, CA: USENIX Association, Aug. 2019, pp. 851–867.isbn: 978-1-939133-06-9.url: https://www.usenix.org/conference/usenixsecurity19/presentation/li. [124] Yuliang Li, Rui Miao, Changhoon Kim, and Minlan Yu. “Flowradar: A better netow for data centers”. In: 13thfUSENIXg Symposium on Networked Systems Design and Implementation (fNSDIg 16). 2016, pp. 311–324. [125] Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. “HPCC: High precision congestion control”. In: Proceedings of the ACM Special Interest Group on Data Communication. 2019, pp. 44–58. [126] Jieyu Lin, Rajsimman Ravichandiran, Hadi Bannazadeh, and Alberto Leon-Garcia. “Monitoring and measurement in software-dened infrastructure”. In: Integrated Network Management (IM), 2015 IFIP/IEEE International Symposium on. IEEE. 2015, pp. 742–745. [127] Linear interpolation - Wikipedia. https://en.wikipedia.org/wiki/Linear_interpolation. (Accessed on 01/27/2021). [128] Malware Domain List. Malware Domain List. http://www.malwaredomainlist.com/. May 2020. [129] Xin Liu, Xiaowei Yang, and Yanbin Lu. “To lter or to authorize: Network-layer DoS defense against multimillion-node botnets”. In: Proceedings of the ACM SIGCOMM 2008 conference on Data communication. 2008, pp. 195–206. [130] I. Livadariu, K. Benson, A. Elmokash, A. Dainotti, and A. Dhamdhere. “Inferring Carrier-Grade NAT Deployment in the Wild”. In: IEEE Conference on Computer Communications (INFOCOM). Apr. 2018. [131] Mailinator Inc. http://www.mailinator.com. Sept. 2019. [132] Malc0de. Malc0de Database. http://malc0de.com/database/. May 2020. 161 [133] MANRS. [Online; accessed 6. May 2021]. May 2021.url: https://www.manrs.org. [134] Athina Markopoulou, Gianluca Iannaccone, Supratik Bhattacharyya, Chen-Nee Chuah, Yashar Ganjali, and Christophe Diot. “Characterization of Failures in an Operational IP Backbone Network”. In: IEEE/ACM Trans. Netw. 16.4 (Aug. 2008), pp. 749–762.issn: 1063-6692.doi: 10.1109/TNET.2007.902727. [135] P Marques, N Sheth, R Raszuk, B Greene, J Mauch, and D McPherson. Dissemination of Flow Specication Rules. RFC 5575. 2009. [136] Justin Mason. “Filtering spam with spamassassin”. In: HEANet Annual Conference. 2002, p. 103. [137] MaxMind - Sample List of High Risk IP Addresses. https://www.maxmind.com/en/high-risk-ip-sample-list. Sept. 2019. [138] Ahmed Metwally and Matt Paduano. “Estimating the number of users behind ip addresses for combating abusive trac”. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. 2011, pp. 249–257. [139] Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. “Homa: A receiver-driven low-latency transport protocol using network priorities”. In: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 2018, pp. 221–235. [140] Sue B Moon, Paul Skelly, and Don Towsley. “Estimation and removal of clock skew from network delay measurements”. In: INFOCOM’99. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE. Vol. 1. IEEE. 1999, pp. 227–234. [141] David Moore, Colleen Shannon, et al. “Code-Red: a case study on the spread and victims of an Internet worm”. In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment. ACM. 2002, pp. 273–284. [142] Andreas Müller, Florian Wohlfart, and Georg Carle. “Analysis and topology-based traversal of cascaded large scale NATs”. In: Proceedings of the 2013 workshop on Hot topics in middleboxes and network function virtualization. ACM. 2013, pp. 43–48. [143] Srinivas Narayana, Anirudh Sivaraman, Vikram Nathan, Prateesh Goyal, Venkat Arun, Mohammad Alizadeh, Vimalkumar Jeyakumar, and Changhoon Kim. “Language-directed hardware design for network performance monitoring”. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 2017, pp. 85–98. [144] Blocklist NET. BlockList.net.ua. https://blocklist.net.ua/. May 2020. [145] 360.com NetLab.Aquickstatsonthe608,083MiraiIPsthathitourhoneypotsinthepast2.5months. https://goo.gl/NYWMLq. 2017. [146] Netlab: Mirai scanner feed. http://data.netlab.360.com/mirai-scanner/. Sept. 2016. [147] Network Congestion and Internet Slowdowns - Causes & Solutionsj Verizon. [Online; accessed 29. Sep. 2021]. Sept. 2021.url: https://www.verizon.com/info/internet-congestion. 162 [148] Arbor Networks. DDoS Protection by Arbor Networks APS. https://www.arbornetworks.com/ddos-protection-products/arbor-aps. 2018. [149] Barefoot Networks. Deep Insight. https://www.barefootnetworks.com/products/brief-deep-insight/. [150] Normshield. Normshield - Cyber Risk Scorecard. https://www.normshield.com/. May 2020. [151] Arman Noroozian, Jan Koenders, Eelco van Veldhuizen, Carlos H. Ganan, Sumayah Alrwais, Damon McCoy, and Michel van Eeten. “Platforms in Everything: Analyzing Ground-Truth Data on the Anatomy and Economics of Bullet-Proof Hosting”. In: 28th USENIX Security Symposium (USENIX Security 19). Santa Clara, CA: USENIX Association, Aug. 2019, pp. 1341–1356.isbn: 978-1-939133-06-9.url: https://www.usenix.org/conference/usenixsecurity19/presentation/noroozian. [152] NoThink. NoThink Individual Blacklist Maintainer. http://www.nothink.org/. May 2020. [153] NoVirusThanks: Security Software and Services. http://www.ipspamlist.com/ip-reputation-feeds/. Sept. 2019. [154] NPL – Open, High-Level language for developing feature-rich solutions for programmable networking platforms. https://nplang.org/. (Accessed on 01/25/2021). [155] Nullsecure. nullsecure. https://nullsecure.org/. May 2020. [156] George Oikonomou, Jelena Mirkovic, Peter Reiher, and Max Robinson. “A Framework for a Collaborative DDoS Defense”. In: ACSAC ’06: Proceedings of the 22nd Annual Computer Security Applications Conference. IEEE Computer Society, 2006, pp. 33–42. [157] Christian W Omlin and C Lee Giles. “Extraction of rules from discrete-time recurrent neural networks”. In: Neural networks 9.1 (1996), pp. 41–52. [158] Heise Online. NiX Spam DNSBL and blacklist for download. https://www.heise.de/ix/NiX-Spam-DNSBL-and-blacklist-for-download-499637.html. Sept. 2019. [159] OpenBL - Abuse reporting and blacklisting. https://web.archive.org/web/20170107081656/http://www.openbl.org/. Sept. 2019. [160] P4-16-v1.2.0.pdf. https://p4.org/p4-spec/docs/P4-16-v1.2.0.pdf. [161] R. Padmanabhan, A. Dhamdhere, E. Aben, k. clay k., and N. Spring. “Reasons Dynamic Addresses Change”. In: Internet Measurement Conference (IMC). Jan. 2016. [162] Spectrum Partners. Spectrum Static IP. https://partners.spectrum.com/content/spectrum/business/en/internet/staticip.html. May 2020. [163] Vern Paxson. “Bro: A System for Detecting Network Intruders in Real-Time”. In: Computer Networks (1999). 163 [164] Michael Pazzani and Daniel Billsus. “Content-based recommendation systems”. In: The adaptive web (2007), pp. 325–341. [165] Andreas Pitsillidis, Chris Kanich, Georey M Voelker, Kirill Levchenko, and Stefan Savage. “Taster’s choice: a comparative analysis of spam feeds”. In: Proceedings of the 2012 ACM conference on Internet measurement conference. ACM. 2012, pp. 427–440. [166] Proofpoint Inc. http://www.proofpoint.com. Sept. 2019. [167] Ting Qu, Raj Joshi, Mun Choon Chan, Ben Leong, Deke Guo, and Zhong Liu. “SQR: in-network packet loss recovery from link failures for highly reliable datacenter networks”. In: 2019 IEEE 27th International Conference on Network Protocols (ICNP). IEEE. 2019, pp. 1–12. [168] Sivaramakrishnan Ramanathan, Anushah Hossain, Jelena Mirkovic, Minlan Yu, and Sadia Afroz. “Quantifying the Impact of Blocklisting in the Age of Address Reuse”. In: Proceedings of the ACM Internet Measurement Conference. 2020, pp. 360–369. [169] Sivaramakrishnan Ramanathan, Yaron Kanza, and Balachander Krishnamurthy. “SDProber: A Software Dened Prober for SDN”. In: Proceedings of the Symposium on SDN Research. SOSR ’18. Los Angeles, CA, USA: ACM, 2018. [170] Sivaramakrishnan Ramanathan, Yaron Kanza, and Balachander Krishnamurthy. “SDProber: A software dened prober for SDN”. In: Proceedings of the Symposium on SDN Research. 2018, pp. 1–7. [171] Sivaramakrishnan Ramanathan, Jelena Mirkovic, and Minlan Yu. “BLAG: Improving the Accuracy of Blacklists”. In: NDSS. 2020. [172] Sivaramakrishnan Ramanathan, Jelena Mirkovic, and Minlan Yu. “BLAG: Improving the Accuracy of Blacklists”. In: 27th Annual Network and Distributed System Security Symposium, NDSS 2020, San Diego, California, USA, February 23-26, 2020. NDSS ’20. The Internet Society, 2020.doi: 10.14722/ndss.2020.24232. [173] Sivaramakrishnan Ramanathan, Jelena Mirkovic, Minlan Yu, and Ying Zhang. “SENSS against volumetric DDoS attacks”. In: Proceedings of the 34th Annual Computer Security Applications Conference. 2018, pp. 266–277. [174] Steve Ranger. GitHub hit with the largest DDoS attack ever seen. ZDNet,https: //www.zdnet.com/article/github-was-hit-with-the-largest-ddos-attack-ever-seen/. 2018. [175] Philipp Richter, Georgios Smaragdakis, David Plonka, and Arthur Berger. “Beyond Counting: New Perspectives on the Active IPv4 Address Space”. In: Proceedings of ACM IMC 2016. Santa Monica, CA, Nov. 2016. [176] Philipp Richter, Florian Wohlfart, Narseo Vallina-Rodriguez, Mark Allman, Randy Bush, Anja Feldmann, Christian Kreibich, Nicholas Weaver, and Vern Paxson. “A Multi-perspective Analysis of Carrier-Grade NAT Deployment”. In: Proceedings of ACM IMC 2016. Santa Monica, CA, Nov. 2016. 164 [177] Matthew Roughan. “Simplifying the synthesis of Internet trac matrices”. In: SIGCOMM. 2005. [178] Oregon RouteViews. “University of Oregon RouteViews project”. In: Eugene, OR.[Online]. Available: http://www. routeviews. org (). [179] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. “Inside the social network’s (datacenter) network”. In: ACM SIGCOMM Computer Communication Review. Vol. 45. 4. ACM. 2015, pp. 123–137. [180] SDN Ryu. Framework. https://osrg.github.io/ryu/. 2016. [181] Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. “Finding a" kneedle" in a haystack: Detecting knee points in system behavior”. In: 2011 31st international conference on distributed computing systems workshops. IEEE. 2011, pp. 166–171. [182] Sblam. Sblam! http://sblam.com/. May 2020. [183] J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. “Collaborative ltering recommender systems”. In: The adaptive web. Springer, 2007, pp. 291–324. [184] Quirin Scheitle, Oliver Hohlfeld, Julien Gamba, Jonas Jelten, Torsten Zimmermann, Stephen D Strowes, and Narseo Vallina-Rodriguez. “A long way to the top: signicance, structure, and stability of internet top lists”. In: Proceedings of the Internet Measurement Conference 2018. ACM. 2018, pp. 478–493. [185] Krebs on Security. memcached attack. https://krebsonsecurity.com/tag/memcached-attack/. Mar. 2018. [186] Vyas Sekar Seyed K. Fayaz Yoshiaki Tobioka and Michael Bailey. “Bohatei: Flexible and Elastic DDoS Defense”. In: USENIX. 2015. [187] Naveen Kr Sharma, Antoine Kaufmann, Thomas Anderson, Arvind Krishnamurthy, Jacob Nelson, and Simon Peter. “Evaluating the power of exible packet processing for network resource allocation”. In: 14thfUSENIXg Symposium on Networked Systems Design and Implementation (fNSDIg 17). 2017, pp. 67–82. [188] AWS Shield. Threat landscape report–q1 2020. 2020. [189] Seungwon Shin, Vinod Yegneswaran, Phillip Porras, and Guofei Gu. “Avant-guard: Scalable and vigilant switch ow management in software-dened networks”. In: Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security. 2013, pp. 413–424. [190] Shun list. https://www.autoshun.org/. Sept. 2019. [191] Sushant Sinha, Michael Bailey, and Farnam Jahanian. “Improving Spam Blacklisting Through Dynamic Thresholding and Speculative Aggregation”. In: NDSS. 2010. 165 [192] Sushant Sinha, Michael Bailey, and Farnam Jahanian. “Shades of Grey: On the eectiveness of reputation-based blacklists”. In: Malicious and Unwanted Software, 2008. MALWARE 2008. 3rd International Conference on. IEEE. 2008, pp. 57–64. [193] Fabio Soldo, Anh Le, and Athina Markopoulou. “Predictive blacklisting as an implicit recommendation system”. In: INFOCOM, 2010 Proceedings IEEE. IEEE. 2010, pp. 1–9. [194] Stop Forum Spam. Stop Forum Spam. https://stopforumspam.com/. May 2020. [195] SpamAssasin public corpus. https://spamassassin.apache.org/old/publiccorpus/. Sept. 2019. [196] Spamhaus Project. https://www.spamhaus.org/. Sept. 2019. [197] Frank Spitzer. Principles of random walk. Vol. 34. Springer Science & Business Media, 2013. [198] Stefan Goerje - SIP attacker blacklist. http://stefan.gofferje.net/it-stuff/sipfraud/sip-attacker-blacklist. Sept. 2019. [199] StrataXGS® Switch Solutions. https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs. (Accessed on 01/25/2021). [200] Tushar Swamy, Alexander Rucker, Muhammad Shahbaz, and Kunle Olukotun. “Taurus: An intelligent data plane”. In: arXiv preprint arXiv:2002.08987 (2020). [201] ARS Technica. ATT raises prices 7% by making its customers pay ATT’s property taxes. https://arstechnica.com/tech-policy/2019/10/att-raises-prices-7-by-making-its-customers- pay-atts-property-taxes/. Oct. 2020. [202] Threatcrowd.ThreatCrowd-OpenSourceThreatIntelligence. https://threatcrowd.org/. May 2020. [203] Emerging Threats. Emerging Threats Rules. https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt. May 2020. [204] Olivier Tilmans, Tobias Bühler, Stefano Vissicchio, and Laurent Vanbever. “Mille-Feuille: Putting ISP trac under the scalpel”. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks. ACM. 2016, pp. 113–119. [205] Kazuhiro Tobe, Akihiro Shimoda, and Shegeki Goto. “Extended UDP Multiple Hole Punching Method to Traverse Large Scale NATs”. In: Proceedings of the Asia-Pacic Advanced Network 30 (2010), pp. 30–36. [206] Amin Tootoonchian, Monia Ghobadi, and Yashar Ganjali. “OpenTM: trac matrix estimator for OpenFlow networks”. In: International Conference on Passive and Active Network Measurement. Springer. 2010, pp. 201–210. [207] Trident4 / BCM56880 Series. https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56880-series. (Accessed on 01/25/2021). 166 [208] TrustedSec - Information Security Consulting Services. https://www.trustedsec.com/. Sept. 2019. [209] D. Turk. Conguring BGP to Block Denial-of-Service Attacks. RFC 3882. RFC Editor, Sept. 2004. [210] Turris. Greylist :: Project:Turris. https://www.turris.cz/en/greylist. May 2020. [211] Untroubled. http://untroubled.org/spam/. Sept. 2019. [212] URLVir. URLVir: Monitor Malicious Executable Urls. http://www.urlvir.com/. (Accessed on 05/13/2020). May 2020. [213] Niels LM Van Adrichem, Christian Doerr, and Fernando A Kuipers. “Opennetmon: Network monitoring in openow software-dened networks”. In: Network Operations and Management Symposium (NOMS), 2014 IEEE. IEEE. 2014, pp. 1–8. [214] VX Vault. VX Vault ViriList. http://vxvault.net/ViriList.php. May 2020. [215] Shobha Venkataraman, Subhabrata Sen, Oliver Spatscheck, Patrick Haner, and Dawn Song. “Exploiting Network Structure for Proactive Spam Mitigation”. In: Usenix Security. 2007. [216] Jerey S Vitter. “Random sampling with a reservoir”. In: ACM Transactions on Mathematical Software (TOMS) 11.1 (1985), pp. 37–57. [217] VoIP Blacklist. http://www.voipbl.org/. Sept. 2019. [218] Michael Walsh, Mythili Vutukuru, Hari Balakrishnan, David Karger, and Scott Shenker. “DDoS Defense by Oense”. In: ACM SIGCOMM 2006. Pisa, Italy, Sept. 2006. [219] An Wang, Yang Guo, Fang Hao, T. V. Lakshman, and Songqing Chen. “UMON: Flexible and Fine Grained Trac Monitoring in Open vSwitch”. In: Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies. CoNEXT ’15. Heidelberg, Germany: ACM, 2015, 15:1–15:7.isbn: 978-1-4503-3412-9.doi: 10.1145/2716281.2836100. [220] Zhaoguang Wang, Zhiyun Qian, Qiang Xu, Zhuoqing Mao, and Ming Zhang. “An untold story of middleboxes in cellular networks”. In: ACM SIGCOMM Computer Communication Review. Vol. 41. 4. ACM. 2011, pp. 374–385. [221] Gail Weiss, Yoav Goldberg, and Eran Yahav. “Extracting automata from recurrent neural networks using queries and counterexamples”. In: arXiv preprint arXiv:1711.09576 (2017). [222] Andrew G West, Adam J Aviv, Jian Chang, and Insup Lee. “Spam mitigation using spatio-temporal reputations from blacklist history”. In: Proceedings of the 26th Annual Computer Security Applications Conference. ACM. 2010, pp. 161–170. [223] Philip Wette and Holger Karl. “Which Flows Are Hiding Behind My Wildcard Rule?: Adding Packet Sampling to Openow”. In: SIGCOMM Comput. Commun. Rev. 43.4 (Aug. 2013), pp. 541–542.issn: 0146-4833.doi: 10.1145/2534169.2491710. 167 [224] Apache Wiki. Other Trick For Blocking Spam. https: //cwiki.apache.org/confluence/display/SPAMASSASSIN/OtherTricks#OtherTricks-Greylisting. July 2019. [225] Keith Winstein and Hari Balakrishnan. “Tcp ex machina: Computer-generated congestion control”. In: ACM SIGCOMM Computer Communication Review 43.4 (2013), pp. 123–134. [226] Xnity. Business Class Internet at Home. https://www.xfinity.com/hub/business/internet-for-home-business. May 2020. [227] Yinglian Xie, Fang Yu, Kannan Achan, Eliot Gillum, Moises Goldszmidt, and Ted Wobber. “How dynamic are IP addresses?” In: ACM SIGCOMM Computer Communication Review. Vol. 37. 4. ACM. 2007, pp. 301–312. [228] Zhaoqi Xiong and Noa Zilberman. “Do switches dream of machine learning? Toward in-network classication”. In:Proceedingsofthe18thACMworkshoponhottopicsinnetworks. 2019, pp. 25–33. [229] Xiaowei Yang, David Wetherall, and Thomas Anderson. “TVA: A DoS-limiting Network Architecture”. In: IEEE/ACM Trans. Netw. 16.6 (Dec. 2008), pp. 1267–1280.issn: 1063-6692.doi: 10.1109/TNET.2007.914506. [230] Kyle York. Dyn Statement on 10/21/2016 DDoS Attack. https://dyn.com/blog/dyn-statement-on-10212016-ddos-attack/. 2016. [231] Curtis Yu, Cristian Lumezanu, Abhishek Sharma, Qiang Xu, Guofei Jiang, and Harsha V Madhyastha. “Software-dened latency monitoring in data center networks”. In: International Conference on Passive and Active Network Measurement. Springer. 2015, pp. 360–372. [232] Curtis Yu, Cristian Lumezanu, Yueping Zhang, Vishal Singh, Guofei Jiang, and Harsha V Madhyastha. “Flowsense: Monitoring network utilization with zero measurement cost”. In:InternationalConferenceonPassiveandActiveNetworkMeasurement. Springer. 2013, pp. 31–41. [233] Zenedge. Zenedge Web page. https://www.zenedge.com/. 2018. [234] Hongyi Zeng, Peyman Kazemian, George Varghese, and Nick McKeown. “Automatic Test Packet Generation”. In: Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies. CoNEXT ’12. Nice, France: ACM, 2012, pp. 241–252.isbn: 978-1-4503-1775-7. [235] ZeroDot1. CoinBlockerLists. https://gitlab.com/ZeroDot1/CoinBlockerLists. May 2020. [236] Hong Zhang, Junxue Zhang, Wei Bai, Kai Chen, and Mosharaf Chowdhury. “Resilient datacenter load balancing in the wild”. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication. ACM. 2017, pp. 253–266. [237] Jian Zhang, Phillip A Porras, and Johannes Ullrich. “Highly Predictive Blacklisting”. In: USENIX Security Symposium. 2008, pp. 107–122. 168 [238] Jing Zhang, Ari Chivukula, Michael Bailey, Manish Karir, and Mingyan Liu. “Characterization of blacklists and tainted network trac”. In: International Conference on Passive and Active Network Measurement. Springer. 2013, pp. 218–228. [239] Jing Zhang, Zakir Durumeric, Michael Bailey, Manish Karir, and Mingyan Liu. “On the Mismanagement and Maliciousness of Networks”. In: Proceedings of the 21st Annual Network & Distributed System Security Symposium (NDSS ’14). 2013. [240] Jing Zhang, Zakir Durumeric, Michael Bailey, Mingyan Liu, and Manish Karir. “On the Mismanagement and Maliciousness of Networks”. In: NDSS. 2014. [241] Jun Zhang and KF Man. “Time series prediction using RNN in multi-dimension embedding phase space”. In: SMC’98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 98CH36218). Vol. 2. IEEE. 1998, pp. 1868–1873. [242] Junxue Zhang, Wei Bai, and Kai Chen. “Enabling ECN for datacenter networks with RTT variations”. In: Proceedings of the 15th International Conference on Emerging Networking Experiments And Technologies. ACM. 2019, pp. 233–245. [243] Menghao Zhang, Guanyu Li, Shicheng Wang, Chang Liu, Ang Chen, Hongxin Hu, Guofei Gu, Qianqian Li, Mingwei Xu, and Jianping Wu. “Poseidon: Mitigating volumetric ddos attacks with programmable switches”. In: Proceedings of NDSS. 2020. [244] Qiao Zhang, Vincent Liu, Hongyi Zeng, and Arvind Krishnamurthy. “High-resolution measurement of data center microbursts”. In: Proceedings of the 2017 Internet Measurement Conference. ACM. 2017, pp. 78–85. [245] Yu Zhou, Chen Sun, Hongqiang Harry Liu, Rui Miao, Shi Bai, Bo Li, Zhilong Zheng, Lingjun Zhu, Zhen Shen, Yongqing Xi, et al. “Flow event telemetry on programmable data plane”. In: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 2020, pp. 76–89. [246] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. “Congestion control for large-scale RDMA deployments”. In: ACM SIGCOMM Computer Communication Review 45.4 (2015), pp. 523–536. [247] Yibo Zhu, Nanxi Kang, Jiaxin Cao, Albert Greenberg, Guohan Lu, Ratul Mahajan, Dave Maltz, Lihua Yuan, Ming Zhang, Ben Y. Zhao, and Haitao Zheng. “Packet-Level Telemetry in Large Datacenter Networks”. In: SIGCOMM Comput. Commun. Rev. 45.4 (Aug. 2015), pp. 479–491.issn: 0146-4833.doi: 10.1145/2829988.2787483. 169

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Scaling-out traffic management in the cloud

PDF

Protecting online services from sophisticated DDoS attacks

PDF

Supporting faithful and safe live malware analysis

PDF

Improving network security through collaborative sharing

PDF

Towards highly-available cloud and content-provider networks

PDF

Machine learning for efficient network management

PDF

Modeling information operations and diffusion on social media networks

PDF

Graph machine learning for hardware security and security of graph machine learning: attacks and defenses

PDF

Collaborative detection and filtering of DDoS attacks in ISP core networks

PDF

Diffusion network inference and analysis for disinformation mitigation

PDF

A protocol framework for attacker traceback in wireless multi-hop networks

PDF

Mitigating attacks that disrupt online services without changing existing protocols

PDF

Adaptive resource management in distributed systems

PDF

Model-driven situational awareness in large-scale, complex systems

PDF

Network reconnaissance using blind techniques

PDF

Enabling efficient service enumeration through smart selection of measurements

PDF

Improving network reliability using a formal definition of the Internet core

PDF

Performant, scalable, and efficient deployment of network function virtualization

PDF

AI-enabled DDoS attack detection in IoT systems

PDF

Studying malware behavior safely and efficiently

Asset Metadata

Creator Ramanathan, Sivaramakrishnan (author)

Core Title Leveraging programmability and machine learning for distributed network management to improve security and performance

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Degree Conferral Date 2021-12

Publication Date 10/03/2021

Defense Date 05/13/2021

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag computer networks,computer security,machine learning,measurements,OAI-PMH Harvest,programmable switches

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Mirkovic, Jelena (committee chair), Ferrara, Emilio (committee member), Govindan, Ramesh (committee member), Yu, Minlan (committee member)

Creator Email satyaman@usc.edu,sivaram@sivaram.me

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC16011908

Unique identifier UC16011908

Legacy Identifier etd-Ramanathan-10132

Document Type Dissertation

Format application/pdf (imt)

Rights Ramanathan, Sivaramakrishnan

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu