University of Southern California Dissertations and Theses

Differential verification of deep neural networks

(USC Thesis Other)

Differential verification of deep neural networks

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Dierential Verication of Deep Neural Networks by Brandon Bruce Paulsen A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulllment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) August 2022 Copyright 2022 Brandon Bruce Paulsen Acknowledgements Completing a PhD will be one of the most fullling moments in my life. Though there were countless moments that I doubted I would nish, I was able to persevere because of the support of many colleagues, friends, and family. I will always be grateful for their support. I would rst like to acknowledge my biggest advocates: my parents. My entire academic life, from undergrad until now, has been lled with struggles and achievements, doubts and condence, lows and highs. From failing my rst semester in university, to my graduation today, my parents have given me unwavering support, unconditional compassion, and their unbreakable condence in me. I could never have persisted until the end of my PhD without the foundation they provided me. I am forever grateful to them. During my PhD, I was also incredibly lucky to have a new advocate and family member entire my life: my wonderful wife Yixue Zhao. Her love and support compares only to my parent’s. No one in my life has more persistently provided me with words of encouragement, and built my condence than she. Her passion for life and positive vibes helped me overcome some of the lowest lows throughout my PhD. She has taught me many lessons, not only on how to be a better researcher, but also on how to live a happy and meaningful life. Next, I would like to acknowledge my advisor Chao Wang’s support and many contributions to this dissertation. Throughout my PhD, Chao taught me many valuable lessons about both the theory and practice of formal methods, and about doing research in general. In addition, during my many struggles, ii especially in the beginning of my PhD, he struggled with me, and did not give up. My rst project took nearly two years of eort – far longer than the average project in our group. Throughout this time, I asked many naive questions, made many mistakes, and at times I went months without making progress, but Chao was always there struggling with me. I learned many invaluable lessons working with Chao that I will carry with me throughout the rest of my career. I would also like to thank him for the many thought- provoking and productive discussions we had, which uncovered the interesting problems and solutions that make up the contributions of this dissertation. I would also like to thank those who served on my dissertation committee, namely Jyotirmoy Desh- mukh, Murali Annavaram, Nenad Medvidovic, William G.J. Halfond, and Bhaskar Krishnamachari. I’d like to thank them for their time serving on my committee, for their thought-provoking questions, and feed- back on my presentations. I’d also like to thank Jyotirmoy for his interesting discussion, ideas for future work, and for being an excellent collaborator. I hope we can continue our collaborations in the future. Next, I would especially like to thank my Master’s program advisor, Peter Peterson. Peter played a pivotal role in getting me into the PhD program at USC. I have fond memories of his computer security course, the genuine passion that went into his lectures and course material, and the one-of-a-kind learning environment he was able to create. This course helped me discover my passion for academics and learning, which motivated me to apply to a Master’s program, and eventually a PhD program. In addition, during the Master’s program, Peter was extremely supportive of my academic pursuits. He helped me nd an interesting thesis project, connected me with other researchers which lead to a summer internship, guided me through PhD application, and so much more that I can’t name them all. I also had many friends who supported my journey into the PhD program. I’d like to thank Laura Krebs for being a wonderful friend, a role model student, and for believing in me during my undergrad and Master’s program. If it were not for her pushing me to achieve excellence during my undergrad, I would never have made it into the Master’s or PhD program. I’d also like to thank the many friends I iii made during the Master’s program for the wonderful memories that made the program truly special. I remember fondly the many games of Settlers of Catan played with Jonathan Beaulieu and Xinru Yan, the countless hours spent rock climbing with Paul “The Wall” Vaynshenk, and the horribly cringy “Saviors of the Servers” costumes that Ankit Gupta and I concocted one bored Halloween night. I’d also like to thank the many friends I made during the PhD program who supported me. My labmates Jingbo Wang, Tianqin Zhao, Chungha Sung, Farmer Daniel Guo, Meng Wu, Zunchen Huang, Yannan Li, Chenggang Li, and the colleagues from my neighboring labs Adriana Seja, Marc Juarez, Paul Chiou, Nikola Lukic, Daye Nam, Yinjun Lyu, and Mian Wan helped make my PhD and enjoyable experience. Finally, I would like to thank those close friends and family who have supported me both throughout this journey, and my life as a whole. I’d like to thank my closest friends Derek Paulsen, Paul Vaynshenk, Jonathan Beaulieu, and Xinru Yan. Your friendship brings invaluable joy to my life. iv TableofContents Acknowledgements ii ListofTables ix ListofFigures x Abstract xii Chapter1: Introduction 1 1.1 Limitations of Current Verication Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Hypotheses and Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.1 Approximating the Dierence Layer-by-Layer . . . . . . . . . . . . . . . . . . . . . 7 1.3.2 Symbolic Approximations for the Dierence . . . . . . . . . . . . . . . . . . . . . . 7 1.3.3 Online Synthesis of Linear Approximations . . . . . . . . . . . . . . . . . . . . . . 8 1.3.4 Oine Synthesis of Linear Approximations . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Chapter2: RelatedWork 10 2.1 Related Work on Dierential Verication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2 Related Work on Synthesizing Linear Approximations . . . . . . . . . . . . . . . . . . . . . 11 2.3 Related Work on Verication of Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Related Work on Testing Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Chapter3: ApproximatingtheDierenceLayer-by-Layer 13 3.1 Background and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.1 Dierential Verication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.2 Existing Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.3 Our Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2.2 Interval Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2.3 Symbolic Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.4 Renement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3 Forward Interval Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.1 The Ane Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3.2 The ReLU Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 v 3.3.3 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.3.1 Opt. 1:n k;j is active,n 0 k;j is non-linear. . . . . . . . . . . . . . . . . . . . 31 3.3.3.2 Opt. 2:n k;j is non-linear,n 0 k;j is active. . . . . . . . . . . . . . . . . . . . 32 3.3.3.3 Opt. 3: bothn k;j andn 0 k;j are non-linear. . . . . . . . . . . . . . . . . . . 33 3.3.4 On the Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4 Gradient Based Renement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.1 Splitting Input Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.2 The Renement Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4.3 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5.1.1 ACAS Xu [46] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5.1.2 MNIST [62] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.5.1.3 Human Activity Recognition (HAR) [2] . . . . . . . . . . . . . . . . . . . 39 3.5.2 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.3.1 ACAS Xu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.3.2 MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.5.3.3 HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.5.4 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.7 Optimization 3 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7.1 Upper Bound of First Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7.2 Lower Bound of Second Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.7.3 Lower Bound of Third Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.7.4 Upper Bound of Third Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Chapter4: SymbolicApproximationsfortheDierence 57 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.1.1 Dierential Verication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.1.2 Limitations of Existing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.1.3 Our Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.1.3.1 Convex Approximation for the Dierence Interval . . . . . . . . . . . . . 64 4.1.3.2 Symbolic Variables for Hidden Neurons . . . . . . . . . . . . . . . . . . . 65 4.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.2.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.2.2 Symbolic Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.2.3 Convex Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3.1 Dierential Verication – Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3.2 Two Useful Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.3.3 New Convex Approximations forS( k;j ) . . . . . . . . . . . . . . . . . . . . . . . 71 4.3.3.1 Upper Bound for the General Case . . . . . . . . . . . . . . . . . . . . . . 72 4.3.3.2 Lower Bound for the General Case . . . . . . . . . . . . . . . . . . . . . 73 4.3.3.3 Tighter Bounds for Special Cases . . . . . . . . . . . . . . . . . . . . . . 75 4.3.4 Intermediate Symbolic Variables forS() . . . . . . . . . . . . . . . . . . . . . . . 77 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.4.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 vi 4.4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.4.3.1 Summary of Results on All Benchmarks . . . . . . . . . . . . . . . . . . 80 4.4.3.2 Results on ACAS Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4.3.3 Results on MNIST Networks . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.3.4 Contribution of Each Technique . . . . . . . . . . . . . . . . . . . . . . . 85 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Chapter5: OnlineSynthesisofLinearApproximations 87 5.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.1.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.1.2 Neural Network Verication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.1.3 Existing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.1.4 Limitations of Existing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.2 Synthesizing the Candidate Linear Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.2.1 Problem Statement and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.2.2 Synthesizing Candidate Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.3 Making the Bound Sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3.1 Problem Statement and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3.2 Verifying the Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.3.3 Computingv l . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.3.4 On the Correctness and Generality of LinSyn . . . . . . . . . . . . . . . . . . . . . 102 5.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.4.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.4.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Chapter6: OlineSynthesisofLinearApproximations 109 6.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.1.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.1.2 Existing Neural Network Verication Techniques and Limitations . . . . . . . . . . 114 6.2 Problem Statement and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.2.1 The Synthesis Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.2.2 Challenges and Our Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.3.1 Partitioning the Input Interval Space (I x ) . . . . . . . . . . . . . . . . . . . . . . . . 121 6.3.2 Learning the Functiong(l;u) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.3.2.1 Generating Example Coecientsa u ;b u . . . . . . . . . . . . . . . . . . . 124 6.3.2.2 Convertinga u ;b u to a Tangent Line . . . . . . . . . . . . . . . . . . . . . 125 6.3.2.3 Training on the Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.3.3 Ensuring Soundness of the Linear Approximations . . . . . . . . . . . . . . . . . . 128 6.3.4 Ecient Lookup of the Linear Bounds . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.4.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.4.1.1 Neural Networks and Datasets . . . . . . . . . . . . . . . . . . . . . . . . 130 6.4.1.2 Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.4.1.3 Robustness Verication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.4.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 vii 6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Chapter7: ConclusionandFutureWork 136 7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 7.2.1 Safety Properties Beyond Equivalence . . . . . . . . . . . . . . . . . . . . . . . . . 137 7.2.2 Other Uses for Equivalence Verication . . . . . . . . . . . . . . . . . . . . . . . . 138 7.2.3 Synthesizing Other Types of Approximations . . . . . . . . . . . . . . . . . . . . . 139 7.2.4 Improvements to Oine Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Bibliography 141 viii ListofTables 3.1 Statistics of the benchmarks: The total number of verication problems is 842. . . . . . . . 40 3.2 Accuracy of comparison of the three tools on ACAS. . . . . . . . . . . . . . . . . . . . . . 41 3.3 Eciency of ReluDiff vs. ReluVal on ACAS Xu. . . . . . . . . . . . . . . . . . . . . . . . 42 3.4 Accuracy comparison of the three tools on MNIST. . . . . . . . . . . . . . . . . . . . . . . 44 3.5 Eciency comparison of the three tools on MNIST. . . . . . . . . . . . . . . . . . . . . . . 44 3.6 Accuracy comparison of the three tools on HAR. . . . . . . . . . . . . . . . . . . . . . . . . 45 3.7 Eciency comparison of the three tools on HAR. . . . . . . . . . . . . . . . . . . . . . . . 45 4.1 Results for ACAS networks with = 0:05. . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.2 Results for ACAS networks with = 0:01. . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.3 Results of the MNIST 4x1024 pixel experiment. . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.4 Evaluating the individual contributions of convex approximation and symbolic variables using the MNIST 4x1024 global perturbation experiment. . . . . . . . . . . . . . . . . . . . 85 5.1 Comparing certied accuracy and run time of LinSyn andAutoLiRPA. . . . . . . . . . . . 106 5.2 Comparing certied accuracy and run time of LinSyn andPOPQORN. . . . . . . . . . . . 106 6.1 Denitions of activation functions used in our experiments. . . . . . . . . . . . . . . . . . 131 6.2 Comparison of the verication results of our approach andAutoLiRPA. . . . . . . . . . . 133 6.3 Comparison of the verication results of our approach andLinSyn. . . . . . . . . . . . . . 134 6.4 Statistics of the synthesis step in our method. . . . . . . . . . . . . . . . . . . . . . . . . . 134 ix ListofFigures 1.1 Composition approach for proving equivalence off(x) andf 0 (x). . . . . . . . . . . . . . . 4 3.1 Dierential verication of deep neural networks. . . . . . . . . . . . . . . . . . . . . . . . 14 3.2 The dierential verication ow of ReluDiff. . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.3 A neural network with two inputs and one output. . . . . . . . . . . . . . . . . . . . . . . . 18 3.4 Naive dierential verication of the two networks. . . . . . . . . . . . . . . . . . . . . . . 20 3.5 Forward interval analysis of a neural network. . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.6 Diagram of weight notations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.7 Example for backward renement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.8 4 max depth distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.9 -interval on MNIST 4x1024. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.1 The overall ow of NeuroDiff. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.2 Motivating example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3 The shape ofz =ReLU(n + )ReLU(n). . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.4 Bounding planes computed byReluDiff [81]. . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.5 Bounding planes computed by our new method. . . . . . . . . . . . . . . . . . . . . . . . . 63 4.6 Bounding planes computed byNeurify [116]. . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.7 Illustration of Lemmas 1 and 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.8 Tighter upper bounding plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 x 4.9 Tighter lower bounding plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.10 Comparing the execution times of NeuroDiff andReluDiff+ on all verication tasks. . . 81 4.11 Percentage of verication tasks completed on the MNIST 4x1024 network for various perturbations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.12 Accuracy comparison for a single forward pass on the MNIST 4x1024 network with perturbation of 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.1 The overall ow of LinSyn. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.2 Example of neural network verication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.3 Linear bounds for ReLU activation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.4 Bounds computed byLinSyn andAutoLiRPA forswish(x),x2 [1:5; 5:5]. . . . . . . . . 97 5.5 Candidate plane synthesis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.6 Nonlinear activation functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.7 Scatter plot comparing the nal output interval width of LinSyn andAutoLiRPA. . . . . . 107 5.8 Histogram of width ratios betweenAutoLiRPA andLinSyn. Ratio reported as AutoLiRPA LinSyn . . 107 6.1 Overview of our method for synthesizing the coecient generator function. . . . . . . . . . 111 6.2 An example of linear bounding for neural network verication. . . . . . . . . . . . . . . . 115 6.3 Approximation of AutoLiRPA (red) and our approach (green). . . . . . . . . . . . . . . . . 117 6.4 Illustration of the two-point form bound (upper dashed line) and tangent-line form bound (lower dashed line). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.5 Partition ofI x for the Swish activation function, where the blue boxes belong toI tan , and the green boxes belong toI 2pt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.6 Illustration of the sampling and linear programming procedure for computing an upper bound. Shaded green region illustrates the volume below the upper bound. . . . . . . . . . 124 6.7 Plots of the training examples, smoothed with linear interpolation. On the left: a plot of ((l;u); (t)), and on the right: a plot of ((l;u); (a u )): . . . . . . . . . . . . . . . . . . . . . . 126 6.8 Activation functions used in our experiments. . . . . . . . . . . . . . . . . . . . . . . . . . 131 xi Abstract Neural networks have become an integral component of cyber-physical systems, such as autonomous ve- hicles, automated delivery robots, and factory robots, and they have great potential in many other systems as well. However, aws in these models are frequently discovered, and thus in high-stakes applications, ensuring their safety, robustness, and reliability is crucial. While many prior works have been devoted to this problem domain, they are limited because they primarily focus on a few narrowly dened safety prop- erties, and they only focus on the most common neural network architectures and activation functions. This dissertation addresses these limitations by (1) studying a new class of safety properties – dier- ential properties – for neural networks, and (2) developing accurate algorithms for formally proving (or disproving) them that are applicable togeneral neural network architectures and activation functions. We focus on neural network equivalence as the canonical example for a dierential property, however other safety properties concerning input sensitivity and stability can be cast as dierential properties as well. This dissertation makes four key contributions towards developing accurate and general algorithms for proving dierential properties. First, we formalize the equivalence problem for neural networks, and then develop a novel technique based on interval analysis for proving equivalence of any two structurally similar feed-forward neural networks with ReLU activations. The key insight in this technique is in de- riving formulas that relate the intermediate computations of the two neural networks, which allows us to accurately bound the maximum dierence between the two networks over all inputs. Second, we develop xii a novel symbolic technique that further improves the analysis’ accuracy. We demonstrate the eective- ness of these two techniques in proving equivalence of compressed neural networks with respect to the original neural networks. Finally, we propose two novel techniques for automatically synthesizing lin- ear approximations for arbitrary nonlinear functions, thus allowing our dierential techniques to apply to architectures and activation functions beyond feed-forward ReLU networks. We demonstrate that our synthesized linear approximations signicantly improve accuracy versus the best alternative techniques. xiii Chapter1 Introduction Neural networks are being deployed in high-stakes applications due to their state-of-the-art performance for important problems such as image recognition [40], object detection [89], and other perception-related tasks. Among the most famous applications are self-driving [37], however they are being studied for aircraft collision avoidance [46], drone control [75], and even controllers in medical devices [104]. While highly performant, neural networks are “black-boxes.” Even small neural networks have thou- sands of parameters, which makes it impossible for a human to understand their behavior. In addition, they are notoriously brittle. For example, they are known to be vulnerable to adversarial examples [103], and generally can have undesirable behaviors [48]. This black-box nature combined with their brittleness raise serious concerns about deploying them in safety-critical systems, where failures could lead to catastrophe. Indeed, we have already seen catastrophic failures in self-driving cars, some of which were directly due to failures in deep learning models [84]. This motivates the question: can we ensure that a given neural network behaves safely? In fact, nding appropriate denitions of safety has proven dicult, and the main safety property considered by prior work isrobustness, i.e. proving that semantically meaningless transformations (e.g. adding a small amount of noise, rotation, scaling, etc.) to a neural network’s input do not cause a change in its output. While important, robustness hardly covers the entire space of safety properties. 1 In this dissertation, we focus on safety properties that aredierential in nature. Informally, a dierential safety property is a property of two (or more) neural networks or a property of two (or more) executions of a single network. Various important safety properties are dierential in nature, however, in this dissertation, we focus on equivalence as the canonical dierential property. Informally, proving equivalence means showing that two neural networks produce similar outputs given the same input. Equivalence is an important property because neural networks are oftencompressed [39] before being deployed. Compression is a process that modies the network’s parameters to reduce its size (in terms of bytes), energy consumption, and runtime. Several works have shown that uncompressed (i.e. “larger”) neural networks are more robust and generalize better [14, 13], thus showing equivalence between the uncompressed and compressed model is highly desirable. We dene equivalence as follows: given two neural networksf : X! Y andf 0 : X! Y trained for the same task, we aim to prove that8x2 X.jf 0 (x)f(x)j < whereXX is an input region of interest, and is some reasonably small constant. In this work, we make the assumption thatf andf 0 are structurally similar, diering only in the numerical values of their weights. This restriction is permissive enough to allow us to analyze state-of-the-art neural network compression techniques such as quantization and edge pruning [39]. To show that equivalence holds we can take one of two approaches: (1) heuristic based (e.g. testing, fuzzing, dynamic analysis), or (2) formal verication based. Along the rst line, many works have been published, both for equivalence [121, 83, 64] and other properties [63, 120, 102, 107, 77] (mostly robust- ness). While these techniques may quickly discover a violation of the equivalence property (i.e anx where jf 0 (x)f(x)j>), they cannot guarantee the absence of violations. The lack of guarantees is undesirable in general, and dangerous in safety-critical systems. This motivates us to use verication because it can provide absolute guarantees about equivalence. While there have been many works in applying verication for robustness safety properties [44, 24, 49, 2 91, 116, 99, 68, 31, 29] (or reachable set computation [43, 27], which can be formulated as a robustness property), to the best of our knowledge there is no work prior to ours on proving equivalence. Before developing our own solution, we rst attempted to adapt current verication tools to the dif- ferential setting using composition [7, 105, 6] – a standard trick in program verication to analyze dier- ential properties of programs. Specically, for showing equivalence, we can create a combined network f 00 (x) = f 0 (x)f(x) as shown in Figure 1.1 and then show thatjf 00 (x)j < using a single network verication tool. The most scalable tools use over-approximation techniques, and would compute an over- approximationY such thatff 00 (x)jx2XgY . We could then compareY with to check if equivalence holds. Unfortunately, this approach has two major limitations. First, it is not accurate. This approach com- putes an overly-conservative approximation off 00 (x) because current tools are not designed specically for proving equivalence (or any dierential property). Practically, this means that this approach either cannot prove equivalence at all, or it cannot do so in a reasonable amount of time. Second, this approach is not general because current verication tools are designed to analyze “common” types of neural net- works. Practically, this means that either their implementations do not support many state-of-the-art neural networks, or, again, they compute overly-conservative approximations, and therefore they cannot prove useful properties. This dissertation addresses these two limitations by developing accurate and general algorithms for proving or disproving dierential properties of neural networks – a problem we refer to as dierential verication. 3 -1.0 1.0 f 00 (x) =f 0 (x)f(x) n 1;2 n 1;1 n 0 1;1 n 0 1;2 n 0 2;2 n 0 2;1 n 0 3;1 n 3;1 n 2;1 n 2;2 n 0;1 n 0;2 f f 0 Figure 1.1: Composition approach for proving equivalence off(x) andf 0 (x). 1.1 LimitationsofCurrentVericationTools We rst layout in more detail the limitations of current verication tools because these lead to insights for how to address them. As previously mentioned, current tools are neither accurate nor general for proving equivalence (or dierential properties in general). Here, we highlight why this is the case. Current tools are not accurate proving equivalence because they do not have the ability to exploit the structural similarities of two closely related networks. They work by computing an over-approximation on the values of each neuron in a neural network in a forward layer-by-layer fashion. While the approxi- mations computed for the neurons in the early layers are relatively accurate, the approximations become overly conservative in the later layers as the neural network’s behavior becomes increasingly complex. Us- ing the composition approach illustrated in Figure 1.1, these tools would compute an over-approximation of f(x) and f 0 (x) independently, and then subtract them to approximate the dierence of the two net- works over all inputs. However, as previously mentioned, these approximations forf(x) andf 0 (x) will be overly conservative for deeper networks, and thus the resulting approximation on the dierence will be too conservative to verify reasonable equivalence properties. We empirically demonstrate this in Chapter 3. 4 In addition, current tools are not general because they require an expert to hand-craft alinearapprox- imation for the neural network’s activation functions in order to accurately over-approximate the values of a given neuron. While there exist techniques to compute linear approximations for arbitrary functions without the need for an expert, such as the techniques used in rigorous global optimization [16, 111] and nonlinear constraint solving [30], we show that these approaches produce overly-conservative approxi- mations. This results in poor accuracy, and thus current tools are not general. We empirically demonstrate this poor accuracy in Chapters 5 and 6. 1.2 HypothesesandInsights Based on these limitations, we propose three testable hypotheses to improve the accuracy and generality of existing tools for dierential properties, and we present insights that inspired these hypotheses. We begin with hypotheses and insights for improving the accuracy. Hypothesis1a Approximating the dierence layer-by-layer can signicantly im- prove accuracy versus existing tools for dierential properties. Insight1a This approach begins approximating the dierence before signi- cant approximation errors accumulate. Hypothesis 1a suggests a new technique for approximating the dierence between two structurally similar neural networks. Prior work independently computes approximations for the output neurons, e.g. n 3;1 andn 0 3;1 in Figure 1.1, and then subtracts them to approximate the dierence. In our new technique, we rst approximate the dierence between the neurons of the rst layers, e.g. we compute an approximation forn 0 1;1 n 1;1 andn 0 1;2 n 1;2 . Then we use this result to approximaten 0 2;1 n 2;1 andn 0 2;2 n 2;2 , and repeat until the nal output layer. Our intuition is laid out in Insight 1a. Hypothesis1b Symbolic approximations can further improve the accuracy of the layer-by-layer approximation. Insight1b Symbols capture dependency information, allowing terms to “can- cel,” and thus reducing approximation errors. 5 Hypothesis 1b suggests a technique for improving the accuracy of the technique laid out by Hypothesis 1a. Our intuition, described in Insight 1b, is inspired by program verication techniques such as symbolic execution [52] and abstract interpretation [17], which use symbols to represent inputs, resulting in more accurate approximations. Our nal hypothesis and insight aim to improve the generality of existing tools. As previously men- tioned, the key challenge to obtaining accurate approximations lies in computing linear approximations for the neural network’s activation functions. Prior to our work, an expert needed to hand-craft these linear approximations in order to be accurate, which poses signicant limitations on the generality of existing verication tools. Thus we propose the following hypothesis. Hypothesis2 Automatically synthesizing linear approximations for the activation functions can signicantly improve accuracy for general activation functions. Insight2 Both our work and prior work demonstrated signicant accuracy improvements. This hypothesis and insight follow naturally based o the results from prior work, which demonstrated that linear approximations of the neural network’s activation functions are vital for accuracy. 1.3 Contributions This dissertation tests the above hypotheses by proposing and evaluating four novel techniques foraccurate and general dierential verication of deep neural networks. The rst two techniques implement the ideas suggested by Hypotheses 1a and 1b, which aim to improve the accuracy when approximating the dierence between two networks. These techniques focus specically on feed-forward ReLU networks. The third and fourth techniques implement the idea suggested by Hypothesis 2, which aims to improve the accuracy of approximations for non-standard neural networks. For each technique, we compare with 6 the best alternative technique to test our hypotheses. Below we summarize the techniques and results, which make up the core contribution of this dissertation. 1.3.1 ApproximatingtheDierenceLayer-by-Layer The rst technique that we propose improves the accuracy when approximating the dierence between two structurally similar neural networks by approximating their dierence layer-by-layer. To implement such a technique, we make two key intellectual contributions. First, we formally derive equations that relate the intermediate computations of the two neural networks, thus allowing the layer-by-layer ap- proximation. We then use interval analysis to rigorously bound these equations, resulting in an over- approximation on the dierence between the two networks. Second, we develop a novel renement tech- nique, which can iteratively improve the accuracy of the computed approximation. We design the tech- nique for ReLU networks, and we compare with two state-of-the-art (single network) verication tools. We demonstrate one to two orders-of-magnitude improvement in both accuracy of the approximations, and runtime to prove/disprove equivalence properties, thus conrming Hypothesis 1a. 1.3.2 SymbolicApproximationsfortheDierence The second technique that we propose further improves the accuracy of the layer-by-layer approximation by using symbolic approximations. To implement this technique, we make two more intellectual contribu- tions. First, we develop a novel linear approximation for the dierence between two neurons, which allows us to maintain symbols in the intervals. Second, we propose a technique to judiciously create new sym- bolic variables that represent the intermediate neurons of the two networks. We compare this symbolic approach with our rst technique and demonstrate further improvements in both accuracy and runtime, thus conrming Hypothesis 1b. 7 1.3.3 OnlineSynthesisofLinearApproximations The third technique that we propose improves the accuracy of approximations for neural networks with non-standard activation functions (thus improving generality) by automatically synthesizing linear ap- proximations. To implement this technique, we develop a novel approach that combines ecient (un- sound) heuristics with a nonlinear SMT solver to guarantee soundness. We call this technique online because we directly call SMT solvers to synthesize the linear approximations. We compare our synthesis approach with hand-crafted approximations, and demonstrate signicant improvements in accuracy, thus conrming Hypothesis 2. 1.3.4 OlineSynthesisofLinearApproximations A key limitation of the third technique is its online nature. The use of solvers incurs a signicant runtime overhead when attempting to prove a given property. The nal technique that we propose again improves generality, but alleviates the runtime overhead by performing the synthesis oine. Whereas the third technique directly uses the SMT solver to compute the linear approximation, this technique synthesizes a static imperative program that computes linear approximations. Since the synthesized program does not call expensive solvers, it computes linear approximations several orders of magnitude faster than the online synthesis technique. While faster, the oine approach is not nearly as accurate as the online approach. However, the oine approach is still signicantly more accurate than hand-crafted approximations, thus again conrming Hypothesis 2. 1.4 Overview This dissertation is organized as follows. In Chapter 2, we discuss the most closely related work to ours, and how our work diers. The next four chapters present the main contributions of this dissertation. Chapter 3 presents our new technique for approximating the dierence between two neural network in 8 a layer-by-layer fashion, which has been published [81] in the main track of the IEEE/ACM International Conference on Software Engineering. Chapter 4 presents our symbolic technique for the layer-by-layer approximation, and has been published [82] in the main track of the IEEE/ACM International Conference on Automated Software Engineering. Chapter 5 presents our online technique for synthesizing linear approximations, and has been published [79] in the main track of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Chapter 6 presents our oine technique for synthesizing linear approximations, and has been published at the International Conference on Computer Aided Verication. Finally, in Chapter 7 we summarize the contributions of this dissertation, and present potential directions for future work. 9 Chapter2 RelatedWork In this chapter, we review related work. We begin with the work that is most closely related to the two key problems studied in our work: proving/disproving dierential properties, and synthesizing linear ap- proximations. We then review more general related work. 2.1 RelatedWorkonDierentialVerication Most closely related to analyzing dierential properties of neural networks are dierential testing tech- niques [121, 83, 64, 3]. These works focus on disproving a dierential property, but, unlike our work, cannot prove that they hold. We note that two recent works [106, 54] propose techniques targeted at proving equivalence of two arbitrary neural networks (whereas we target structurally similar networks). This work is orthogonal to ours – our technique signicantly outperforms theirs when the weights of the two DNN’s are relatively close, whereas their technique does signicantly better when there is no relation between the two neural networks’ weights. Also closely related are equivalence preserving neural network compression techniques [59, 94], also referred to as “exact neural network compression.” Here, the equivalence verication is implicit (i.e. there 10 is no equivalence verication routine), however we believe inspiration for future work could be drawn from these works. 2.2 RelatedWorkonSynthesizingLinearApproximations To the best of our knowledge, there is no work that directly addresses automatically synthesizing linear approximations, however computing linear approximations for arbitrary functions is a core component for rigorous global optimization [16] and SMT solving for nonlinear theories [30]. These approaches work by decomposing the target function into its elementary operations, and combining linear approximations for these elementary operations, to obtain a linear approximation for the target function as a whole. While this produces a sound approximation, we show in Chapters 5 and 6 that it is often overly-conservative, which results in poor accuracy. A recent neural network verication framework [122] made it possible to easily apply this decomposing-based approach, but supporting arbitrary activation functions was not the original intent. 2.3 RelatedWorkonVericationofNeuralNetworks Approximation-basedNeuralNetworkVerication. There is a large body of work on using linear approxima- tion techniques [98, 125, 95, 12, 118, 81, 82, 119, 70] and other abstract domains such as concrete intervals, symbolic intervals [117], and Zonotopes [31], for the purpose of neural network verication. All of these can be thought of as leveraging restricted versions of the polyhedral abstract domain [18, 17]. To the best of our knowledge, these approaches are the most scalable (in terms of network size) due to the use of approximations, but this also means they are less accurate than exact approaches. In addition, all these approaches have the limitation that they depend on bounds that are hand-crafted by an expert. In addi- tion, several works used semi-denite relaxations [19, 43], which essentially allow for quadratic lower and 11 upper bounds, as the approximations. These approximations allow for more expressive lower and upper bounds, and thus often outperform linear approximations in terms of accuracy. However, this approach adds a signicant amount of overhead when the lower and upper bounds need to be evaluated, thus they are signicantly less scalable. SMT solver-based Neural Network Verication There is also a large body of work on using exact con- straint solving for neural network verication. Early works include solvers specically designed for neural networks, such as Reluplex and Marabou [48, 49] and others [23], and leveraging existing solvers [24, 44, 8, 44, 5, 108, 43]. While more accurate, the reliance on an SMT solver typically limits their scalability. More recent work often uses solvers to rene the bounds computed by linear bounding [96, 99, 116, 110, 109]. Since the solvers leveraged in these approaches usually involve linear constraint solving techniques, they are usually only applicable to piece-wise linear activation functions such as ReLU and Max/Min-pooling. 2.4 RelatedWorkonTestingNeuralNetworks Other works are more geared towards single network testing, and use white-box testing techniques [63, 120, 102, 107, 77], such as neuron coverage statistics, to assess how well a network has been tested, and also report interesting behaviors. Both of these can be thought of as adapting software engineering techniques to machine learning. In addition, many works use machine learning techniques, such as gradient opti- mization, to nd interesting behaviors, such as adversarial examples[58, 65, 76, 123, 72]. These interesting behaviors can then be used to retrain the network to improve robustness [34, 87]. Again, these techniques do not provide guarantees, though we believe they could be integrated into with our techniques to quickly disprove a property. 12 Chapter3 ApproximatingtheDierenceLayer-by-Layer As deep neural networks (DNNs) make their way into safety critical systems such as aircraft collision avoid- ance [46] and autonomous driving [10], where errors may lead to catastrophes, there is a growing need for formal verication. The situation is further exacerbated byadversarialexamples [103, 34], which are secu- rity exploits created specically to cause erroneous classications [76, 123, 72, 58]. There is also a growing need for reducing the size of the neural networks deployed on energy- and computation-constrained de- vices. Consequently, compression techniques [39] have emerged to prune unnecessary edges, quantize the weights of remaining edges, and retrain the networks, but they do not provide any formal guarantee – typically the accuracy of a compressed network is only demonstrated empirically. While empirical evidence or statistical analysis may increase our condence that a network behaves as expected for most of the inputs, they cannot prove that it does so for all inputs. Similarly, while heuristic search and dynamic analysis techniques, including testing [83, 107, 63] and fuzzing [77, 120, 121], may quickly discover adversarial examples, they cannot prove the absence of such examples. At the same time, while state-of-the-art verication techniques [44, 24, 49, 91, 116, 99, 68, 31, 29], including Reluplex [48], ReluVal [117] and DeepPoly [98], can provide formal proofs, they are designed for analyzing a single network as opposed to the relationship between two networks. 13 -1.0 1.0 f 00 (x) =f 0 (x)f(x) n 1;2 n 1;1 n 0 1;1 n 0 1;2 n 0 2;2 n 0 2;1 n 0 3;1 n 3;1 n 2;1 n 2;2 n 0;1 n 0;2 f f 0 Figure 3.1: Dierential verication of deep neural networks. In this work, we focus on dierential verication of two closely related networks. In this problem domain, we assume thatf andf 0 are two neural networks trained for the same task; that is, they accept the same inputx and are expected to produce the same output. They are also structurally the same while diering only in the numerical values of edge weights (which allows us to analyze compression techniques such as quantization and edge pruning [39]). In this context, dierential verication is concerned with proving8x2 X.jf 0 (x)f(x)j < , whereX is an input region of interest and is some reasonably small bound. This problem has not received adequate attention and, as we will show in this work, existing tools are ill-suited for solving this problem. The key limitation of existing tools is that, since they are designed to analyze the behavior of a single network, they do not have the ability to exploit the structural similarities of two closely related networks. They also have diculty handling the constraint that the inputs to bothf andf 0 are identical. Typically, these tools work by computing the conservative value ranges of all neurons from input to output in a layer-by-layer style. In the early layers, they may be able to maintain relationships between the inputs, but as the functions become increasingly non-linear in subsequent layers, approximations must be made. 14 . . . x 1 x 2 y 1 y 2 . . . x 1 x 2 y 0 1 y 0 2 Parameters x 1 2X 1 ; x 2 2X 2 ; Forward -Interval Analysis Checking Safe Veried Sampling Input Space Backward Renement X 1 X 2 Violated No Violation Counterexample DNN 1 DNN 2 Interval Violation Figure 3.2: The dierential verication ow of ReluDiff. This “eager” approximation means relationships between the inputs off andf 0 are mostly lost, causing extremely large over-approximations in the output layer. In fact, state-of-the-art verication tools that we have investigated (ReluVal [117] andDeepPoly [98]) struggle to verify that two identical networks are the same. To carry out this litmus test without drastically altering these tools, we construct a combined network as shown in Figure 3.1, wheref andf 0 are actually the same network (i.e. same structure and edge weights). Since they share the same inputx, we expect f(x)f(x) to be 0 regardless of the input region forx. While our method can easily prove thatjf(x) f(x)j< for an arbitrarily small in less than a second, none of the existing tools are able to do so. In fact, DeepPoly cannot verify it no matter how much time is given (it is not a complete method) and ReluVal times out after several hours. Figure 3.2 shows the overall ow of our method,ReluDiff, whose input consists of the two networks (DNN1 and DNN2), an input region (x 1 2 X 1 and x 2 2 X 2 ), and a bound on the output dierence. There are three possible outcomes: (1) veried, meaning that the output dierence is proved to be less 15 than; (2)falsied, meaning a counterexample is found; or (3)unknown, meaning that verication remains inconclusive due to bounds on the computing resources. Internally, ReluDiff iterates through two steps: a forward pass and a backward pass. The forward pass computes over-approximated value dierences of corresponding neurons in the two networks, and propagates them layer by layer from the input to the output. If the output dierence is within the region [;], the property is veried. Otherwise, ReluDiff samples a xed number of concrete examples from the input space and tests if they violate the property. If a violation is found, the property is falsied; otherwise,ReluDiff enters the renement phase. The goal of renement is to identify an input region that should be divided into subregions. By us- ing these subregions to perform the forward pass again, some of the forced over-approximations may be avoided, thus leading to signicant accuracy increase. To identify the right input region for renement, the backward pass computes the dierence of the gradients of the two networks and uses it to nd input regions that, once divided into subregions, are more likely to result in accuracy increase. While iterative interval analysis has been used in verifying neural networks before [117], the focus has always been on a single network. In this work, we show that, by focusing on both networks simul- taneously, we can be more ecient and accurate compared to analyzing each network in isolation. Note that, in dierential verication, the two networks have identical structures and similar behaviors; there- fore, we can easily develop a correspondence between neurons inf andf 0 , thus allowing a lock-step style verication. Lock-step verication allows us to directly compute the dierences in values of neurons and propagate these dierences through edges. It also allows symbolic intervals to be used to avoid some of the approximations. Since error caused by approximation grows quickly, sometimes exponentially [116], as it is propagated through edges and neurons, this can signicantly increase accuracy. When approximation must be made, e.g., due to non-linearity of ReLU, we can handle them better by focusing on the value dierences instead of the absolute values. For example, in ReluVal [117], if 16 a symbolic expression that represents the ReLU input may be both positive and negative, the symbolic expression must be replaced by an interval with concrete upper and lower bounds, which introduces addi- tional error. In contrast, we can be more accurate: even if the input value of a neuron may be both positive and negative, in many cases we still can avoid introducing error into the dierence. We have implemented ReluDiff in a tool and evaluated it on a number of feed-forward neural net- work benchmarks, including ACAS Xu for aircraft collision avoidance [46], MNIST for hand-written digit recognition [62], and HAR for human activity recognition [2]. We also experimentally comparedReluDiff with state-of-the-art tools, including ReluVal [117] and DeepPoly [98]. Our experimental results show that, in almost all cases, ReluDiff outperforms these existing tools in both speed and accuracy. In total, we evaluate on 842 properties over our benchmark networks. ReluDiff was often one to two orders-of- magnitude faster, and was able to prove 745 out of the 842 properties whereas none of the other tools can prove more than 413 properties. To summarize, we make the following contributions: • We propose the rst iterative symbolic interval analysis for dierential verication of two neural networks. • We develop a forward pass algorithm for more accurately computing the value dierences for cor- responding neurons. • We develop a backward pass algorithm, based on gradient dierence, for computing the renement. • We implement the method and demonstrate its advantages over existing tools in terms of both speed and accuracy. The remainder of the chapter is organized as follows. First, we use examples to motivate our method in Section 3.1. Then, we review the basics of neural networks and interval analysis in Section 3.2. Next, we present our method for the forward pass in Section 3.3, followed by our method for the backward pass in 17 1.9 1.1 -2.1 1.0 2.1 -0.9 1.1 -1.0 1.0 -1.0 n 0;1 n 0;2 n 1;2 n 1;1 n 2;1 n 2;2 n 3;1 x 1 x 2 y =f(x 1 ;x 2 ) Figure 3.3: A neural network with two inputs and one output. Section 3.4. We present our experimental results in Section 3.5. Finally, we summarize the contributions of this chapter in Section 3.6. 3.1 BackgroundandOverview We illustrate the problems of existing verication tools using examples and then highlight our main con- tributions. 3.1.1 DierentialVerication Figure 3.3 shows a feed-forward neural network with one input layer, two hidden layers, and one output layer. The input layer has two nodesn 0;1 andn 0;2 , corresponding to the two input variablesx 1 andx 2 . Each hidden layer consists of two neurons,n 1;1 ;n 1;2 in one layer andn 2;1 ;n 2;2 in the other layer. Each of these neurons has two computation steps: the ane transformation and the ReLU activation. For example, insiden 1;1 , the ane transformation is 1:9x 1 2:1x 2 and the ReLU activation ismax(0; 1:9x 1 2:1x 2 ). The output layer has one node, representing the value of y = f(x 1 ;x 2 ). In general, f is a non-linear function overx 1 andx 2 . In dierential verication, we are concerned with the relationship betweenf(x 1 ;x 2 ) and another net- workf 0 (x 1 ;x 2 ). For the sake of example, we focus on a network compression technique calledquantizia- tion [39] in which the edge weights off are rounded to the nearest whole number to obtainf 0 . However, 18 we note that our method can be used onany two networks with similar structures, e.g., whenf 0 is created using other techniques including edge pruning and network retraining [39, 41, 46, 93]. These techniques, in general, raise the concern on how they aect the network’s behavior. In particular, we would like to verify that the new network produces outputs within some bound relative to the original network. Formally, letf 0 :X!Y be the second network andf :X!Y be the rst network. We would like to verify thatjf 0 (x)f(x)j < for allx2 X, whereX X is some region of importance in the input domainX. 3.1.2 ExistingApproaches Existing tools for verifying neural networks target only a single network at a time, and are often geared toward proving the absence ofadversarialexamples. That is, given an input region of interest, they decide if the output stays in a desired region. For the network in Figure 3.3, in particular, the input region may bex 1 2 [4; 6] andx 2 2 [1; 5], and the desired output may bef(x 1 ;x 2 ) < 15. However, these tools are not designed for verifying the relationship between two networks. While we could try and re-use them for our purpose, they lack the ability to exploit the similarities of the two networks. For example, we could use the existing tool ReluVal [117] on bothf andf 0 to compare the concrete output intervals it computes for an input region of interest, e.g., [f low ;f up ] and [f 0 low ;f 0 up ]. In order to conservatively estimate the dierence betweenf andf 0 , we must assume the maximum dierence falls in the interval [f 0 low f up ;f 0 up f low ]. In Figure 3.3, the interval dierence would be [25:76; 22:93], which is too large to be useful. Even though ReluVal could tighten the interval by rening the input intervals, this naive approach cannot even verify that two identical networks always produce the same output, since the output intervals do not capture that the corresponding inputs tof andf 0 (i.e., values ofx 1 andx 2 ) are always the same. 19 1.9 -2.1 1.1 1.0 2.1 -0.9 1.1 -1.0 1.0 -1.0 2 -1 1 -1 1 -1 2 1 -2 1 n 0;1 n 0;2 n 1;1 n 1;2 n 2;1 n 2;2 n 3;1 n 0 1;1 n 0 1;2 n 0 2;1 n 0 2;2 n 0 3;1 -1 1 x 1 x 2 f 0 f Figure 3.4: Naive dierential verication of the two networks. To compensate, we could encode the constraint that values of the corresponding inputs are always the same by composingf andf 0 into a single feed-forward network equivalent tof 0 f, as shown in Figure 3.4. In theory, a sound and complete technique would be able to verify, eventually, that the output dierence is bounded by an arbitrarily small, but with a caveat. That is, to maintain the relationships between the input variables and the dierence in the outputs of the two networks, each neuron must remain in a linear state across the entire input region; otherwise, approximation must be made to maintain the soundness of the interval analysis. However, approximation inevitably loses some of the relationships between the inputs and the outputs. Indeed, we constructed some merged networks in the same way as in Figure 3.4 and then fed them to existing tools. Unfortunately, they all exhibit the “worst-case” value range blowup in the output. The key reason is that existing tools such as ReluVal are forced to approximate ReLU activations by concretizing, which is then followed by interval subtractions, thus causing error introduced by these approximations to be quickly amplied. The forward pass overf computes an output interval of [1:2x 1 1:1x 2 ;1:1x 1 x 2 + 19:53], and forf 0 it computes [x 1 x 2 ;x 1 x 2 + 20]. Although the equations 20 are symbolic, the dierence [21:36; 19:13], computed conservatively by ReluVal, is still too large to be useful. 3.1.3 OurMethod Existing tools cannot exploit structural and behavioral similarities of the two networks in dierential ver- ication. Our insight is to leverage such similarities to drastically improve both the eciency and the accuracy of the verication tool. Specically, in this work, we pair neurons and edges of the rst network with those of the second network and then perform a lock-step verication. This allows us to focus on the value dierences of the corresponding neurons as opposed to their absolute values. The benet is that doing so results in both fewer and tighter approximations and more error reduction due to the use of symbolic intervals. We also perform better renement by focusing on inputs that have the greatest inuence on the output dierence, rather than the absolute output values. While focusing on thedierence as opposed toabsolutevaluesseems to be a straightforward idea, there are many technical challenges. For example, there will be signicantly more complex ReLU activation patterns to consider since we have to handle both networks simultaneously, instead of one network at a time. Approximating symbolic intervals when considering the output dierence of two ReLU activations (i.e., ReLU(x 0 ) - ReLU(x)) has yet to be studied and is non-trivial. Furthermore, how to determine which input neuron to rene when the goal is to reduce error in the output dierence between two networks has not been considered either. In this work, we develop solutions to overcome these challenges. During forward interval analysis, we carefully consider the ReLU activation patterns, and propose a technique for handling each pattern soundly while minimizing the approximation error. During the renement, we compute the dierence 21 n 0;2 n 0;1 n 1;2 n 1;1 n 2;1 n 2;2 n 3;1 1.9 1.1 -2.1 1.0 2.1 -0.9 1.1 -1.0 1.0 -1.0 S(n 1;1 ) = [2:2; 9:3]S(n 2;1 ) = [4:7; 11:8] S(n 1;2 ) = [5:4; 11:6] S(n 2;2 ) = [:2; 2:6] x 1 2 [4; 6] x 2 2 [1; 5] (x 1 ) = [0; 0] (x 2 ) = [0; 0] (n 1;1 ) = [0; 1:1] (n 1;2 ) = [:6;:4] (n 2;1 ) = [:53; 3:02] (n 2;2 ) = [3:8; 0] (n 3;1 ) = [:53; 6:81] Figure 3.5: Forward interval analysis of a neural network. between gradients of the two networks, and use it to identify the input neuron most likely to increase the accuracy of the dierential verication result. As a result, our method can solve the dierential verication problems much more eciently. Consider the litmus test of verifying the equivalence of two identical networks. Our method can obtain a formal proof (thatjf 0 fj<) after performing the forward interval analysis once; in contrast, all other existing tools have failed to do so. For the example in Figure 3.5, we can prove the output dierence(n 3;1 ) is bounded by [0:53; 6:81] after only the rst pass. It also outperforms existing tools on other verication problems wheref 0 is obtained fromf through quantization; details of the experimental comparisons are in Section 3.5. 3.2 Preliminaries First, we review the basics of interval analysis for neural networks. 3.2.1 NeuralNetworks We consider a neural network as a non-linear function that takes some value inR n as input and returns some value inR m as output, where n is the number of input variables and m is the number of output 22 variables. Let the networkf be denotedf :X!Y, whereXR n is the input domain andYR m is the output domain. In image recognition applications, for instance,X may be a vector of pixels representing an image andY may be a vector of probabilities for class labels. In aircraft collision detection, on the other hand,X may be sensor data andY may be a set of actions to take. In this work, we consider fully-connected feed-forward networks with rectied linear unit (ReLU) acti- vations, which are the most popular in practical hardware/software implementations. Thus,y =f(x) is a series of ane transformations (e.g.,xW 1 = i x i w 1;i ) followed by point-wise ReLU (e.g.,ReLU (xW 1 ) = max(0;xW 1 )). LetW k , where 1 k l, be the weight matrix associated with thek-th layer, andl be the number of layers; the ane transformation in thek-th layer is a standard matrix multiplication, followed by the point-wise application of ReLU. Formally,f = f l (f l1 (:::f 2 (f 1 (xW 1 )W 2 )):::W l1 ), where eachf k , 1 k l, is a point-wise ReLU. For the network in Figure 3.3, in particular, the input is a vectorx =fx 1 ;x 2 g, the weight matrix W 3 =f1:0;1:0g T , andxW 1 =f1:9x 1 2:1x 2 ; 1:1x 1 + 1:0x 2 g. For ease of presentation, we denote the weight of the edge from thei-th neuron of layerk 1 to the j-th neuron of layerk asW k [i;j]. We also denote thej-th neuron of layerk asn k;j . 3.2.2 IntervalAnalysis To ensure that our analysis is over-approximated, we use interval analysis [71], which can be viewed as a specic instantiation of the general abstract interpretation [17] framework. Interval analysis is well- suited for analyzing ReLU neural networks as it has well-dened transformers over addition, subtraction, and scaling (i.e., multiplication by a constant). Interval addition as denoted [a;b] + [c;d] = [a +c;b +d] does not lead to loss of accuracy. Scaling as denoted [a;b]c = [ac;bc] whenc 0, or [a;b]c = [bc;ac] whenc < 0, does not lead to 23 loss of accuracy either. Interval subtraction as denoted [a;b] [c;d] = [ad;bc], however, may lead to accuracy loss. To illustrate, considerf(x) = 2:1x andf 0 (x) = 2x, and say we want to approximate their dierence for the input regionx2 [1; 1]. Using interval arithmetic, we would computef([1; 1])f 0 ([1; 1]) = [2:1; 2:1][2; 2] = [4:1; 4:1]. Clearly this is far from the exact interval off(x)f 0 (x) = 2:1x2x = 0:1x overx2 [1; 1], which is [0:1; 0:1]. The reason for such loss of accuracy is that, during interval arithmetic, the relationship between values of 2:1x and 2x (i.e., they are for the same value ofx) is lost. 3.2.3 SymbolicInterval One way to overcome the accuracy loss is usingsymbolicintervals [117], which can encode the constraint that inputs tof andf 0 are actually related. With this technique, we would use the symbolx with the con- straintx2 [1; 1] to initialize the input intervals. Then, the computation becomesf([x;x])f 0 ([x;x]) = [2:1x; 2:1x] [2x; 2x] = [0:1x; 0:1x]. Finally, we would compute the upper and lower bounds forx2 [1; 1] and return the precise interval [0:1; 0:1]. Unfortunately, symbolic intervals depend onf andf 0 being linear in the entire input region in order to be sound. Indeed, if we add ReLU to the functions, i.e.,ReLU (f(x)) =max(0; 2:1x) andReLU (f 0 (x)) = max(0; 2x), wherex2 [1; 1], then the lower and upper bounds are no longer precise nor sound. The reason is because max(0; 2:1x) is non-linear in x2 [1; 1]. Thus, we have to approximate using the concrete interval [0; 2:1]. Similarly, max(0; 2x) is approximated using [0; 2]. Thus, [0; 2:1] [0; 2] = [2; 2:1]. 3.2.4 Renement To improve the accuracy of the symbolic interval analysis, we need to divide the input region into subre- gions. The intuition is that, within a smaller subregion, the ReLU is less likely to exhibit non-linear behavior 24 and force the analyzer to over-approximate. Consider ReLU (2:1x) = max(0; 2:1x), wherex2 [1; 1]. After the input region is divided into subregionsx2 [1; 0][ [0; 1], we havemax(0; 2:1x) = [0; 0] for x2 [1; 0] andmax(0; 2:1x) = [2:1x; 2:1x] forx2 [0; 1]. In both cases, the intervals are precise – there is no approximation at all. When we only have one input variable, we do not have a choice on which variable to rene. However, neural networks have many inputs, and rening some of them will not always yield benet. Thus, we have to identify the right input to split. Considerf(x 1 ;x 2 ) = ReLU (5x 1 ) + ReLU (2x 2 ) ReLU (x 2 ), wherex 1 2 [1; 3] andx 2 2 [1; 1]. The initial analysis is not very accurate due to approximations caused by the ReLU: ReLU (5 [1; 3]) + ReLU (2 [1; 1])+ReLU ([1; 1]) = ReLU ([5; 15])+ReLU ([2; 2])+ReLU ([1; 1]) = [5; 15]+[0; 2] [0; 1] = [4; 17]. If we splitx 1 2 [1; 3] intox 1 2 [1; 2][ [2; 3] and perform interval analysis for both subregions, the output would be [4; 12][ [9; 17] = [4; 17], which does not improve over the initial result. In contrast, if we splitx 2 2 [1; 1] intox 2 2 [1; 0][[0; 1], the accuracy would improve signicantly. Since the ReLU is always activated for x 2 2 [0; 1], ReLU (2x 2 ) and ReLU (x 2 ) can be represented by [2x 2 ; 2x 2 ] and [x 2 ;x 2 ], respectively, and ReLU (2x 2 ) ReLU (x 2 ) = [x 2 ;x 2 ] = [0; 1]. Since the ReLU is always de-activated forx 2 2 [1; 0], we have ReLU (2x 2 ) ReLU (x 2 ) = [0; 0]. Thus, the combined output [5; 15][ [5; 16] = [5; 16] is more accurate than the initial approximation [4; 17]. While how to analyze non-linear activation functions such as ReLU has been studied in prior work [116, 98, 49], none of the existing techniques touch upon the complex scenarios arising from dierential veri- cation of two closely related networks. Our work lls the gap. Specically, we propose a more accurate forward pass for the interval analysis (Section 3.3) and a more accurate backward pass for the renement (Section 3.4). 25 . . . n k1;1 n k;j n k1;i W k [1;j] W k [i;j] layerk 1 layerk Figure 3.6: Diagram of weight notations. 3.3 ForwardIntervalAnalysis In this section, we describe our forward pass for computing the value dierences between neurons in the two networks. Recall that networkf hasl layers and weight matricesW k , 1kl, andn k;j is thej-th node in thek-th layer. Furthermore,W k [i;j] is the weight of the edge fromn k1;i ton k;j . We illustrate these notations in Figure 3.6. Similarly, network f 0 has weight matrices W 0 k , nodes n 0 k;j , and weights W 0 k [i;j] . LetW k [i;j] be the weight dierence, i.eW k [i;j] =W 0 k [i;j]W k [i;j]. We now dene notations for the interval values of neurons. Since each neuron n k;j has an ane transformation (multiplying by the incoming weights) and a ReLU, we denote the input interval to the neuron (after applying the ane transform) asS in (n k;j ), and we denote the output interval of the neuron (after applying the ReLU) asS(n k;j ). We denote the interval bound on the dierence between the inputs to n 0 k;j andn k;j as in (n k;j ), and we denote the interval dierence between the outputs as(n k;j ). Finally, we denote the symbolic upper and lower bound of any value using the notationUB andLB . For example, UB S(n k;j ) andLB S(n k;j ) denote the symbolic upper and lower bound for the output of neuronn k;j . With this notation, our forward pass is shown in Algorithms 1 and 2. The input consists of the two networks,f andf 0 , and the input region of interestX, which denes an interval for each input neuron. After initializing the input intervals, the algorithm iteratively computes eachS(n k;j );S(n 0 k;j ), and(n k;j ) of the subsequent layer by applying the ane transformation followed by the ReLU transformation. The algorithm iterates until the output layer is reached. In addition, it computes the gradient masks for the neurons of f and f 0 , denoted as R and R 0 , which record the state of each neuron in the forward pass 26 Input: networkf, networkf 0 , input regionX Result: dierencef(n l;j )g for output InitializefS(n0;j)g;fS(n 0 0;j )g to input regionX andf(n0;j)g to0 for kin 1..NLayer do // Ane transformer for jin 1..layerSize[k] do S in (n k;j ) P i S(n k1;i )W k [i;j] S in (n 0 k;j ) P i S(n 0 k1;i )W 0 k [i;j] in (n k;j ) P i (S(n k1;i )W k [i;j]+(n k1;i )W 0 k [i;j]) end if k =NLayer then return { in (n k;j )} end // ReLU transformer for jin 1..layerSize[k] do hS(n k;j );S(n 0 k;j );(n k;j )i ReLuTransform(S in (n k;j );S in (n 0 k;j ); in (n k;j )) end end Algorithm1: Forward symbolic interval analysis. ([0; 0] is inactive, [1; 1] is active, and [0; 1] is both). These are used in the renement phase (Section 3.4) to determine which input neuron to rene. We omit discussion of computingS in (n k;j ) andS(n k;j ) because it has been studied in previous work [117]. We focus on computing in (n k;j ) and(n k;j ) in the following section. 3.3.1 TheAneTransformer Computing in (n k;j ) involves two steps. First, we compute(W k [i;j]) for each incoming edge ton k;j . Here,(W k [i;j]) is the dierence in values produced by the edges fromn k1;i ton k;j and fromn 0 k1;i to n 0 k;j . Second, we sum them to obtain in (n k;j ). In the rst step, there are two components to consider when computing(W k [i;j]). First, there is the “new quantity” introduced by the dierence in edge weights, which formally isS(n k1;i )W k [i;j]. In English, this is the interval of neuronn k1;i in the previous layer multiplied by the edge weight dierence in the current layer. Second, there is the “old quantity” accumulated in previous layers being scaled by 27 the edge weight in the current layer. Formally this is(n k1;i )W 0 k [i;j]. Below we write out the formal derivation: (W k [i;j]) =W 0 k [i;j]S(n 0 k1;i )W k [i;j]S(n k1;i ) =W 0 k [i;j]S(n 0 k1;i )W k [i;j]S(n k1;i )+ (W 0 k [i;j]S(n k1;i )W 0 k [i;j]S(n k1;i )) = (W 0 k [i;j]S(n 0 k1;i )W 0 k [i;j]S(n k1;i ))+ (W 0 k [i;j]S(n k1;i )W k [i;j]S(n k1;i )) =(n k1;i )W 0 k [i;j] +S(n k1;i )W k [i;j]: In the second step, we sum together each incoming(W k [i;j]) term to obtain in (n k;j ), which is the dierence of the valuesS in (n k;j ) andS in (n 0 k;j ). That is, in (n k;j ) = X i (W k [i;j]): We demonstrate the computation on the example in Figure 3.5. First, we compute (W 1 [1; 1]) = 0:1 [4; 6] + 2 [0; 0] = [0:4; 0:6] and(W 1 [2; 1]) = 0:1 [1; 5] + 2 [0; 0] = [0:1; 0:5]. Then, we compute (W 1 [1; 2]) = [0:6;0:4] and(W 1 [2; 2]) = [0; 0]. Next, we compute in (n 1;1 ) =(W 1 [1; 1]) +(W 1 [2; 1]) = [0:5; 1:1] and in (n 1;2 ) =(W 1 [1; 2]) +(W 1 [2; 2]) = [0:6;0:4]. 28 3.3.2 TheReLUTransformer Next, we apply the ReLU activation to in (n k;j ) to obtain(n k;j ). We consider nine cases based on whether the ReLUs ofn k;j andn 0 k;j are always activated, always deactivated, or non-linear. In the remainder of the section, we discuss how to soundly over-approximate. Algorithm 2 shows the details. First, we consider the three cases when the ReLU inn k;j is always deactivated. (1) If the ReLU inn 0 k;j is alsoalwaysdeactivated, the outputs of both ReLUs are 0, and the dierence is 0. (2) If the ReLU inn 0 k;j is always activated, the output dierence will beS in (n 0 k;j ) [0; 0] =S in (n 0 k;j ). Note that we can maintain symbolic equations here. (3) If the ReLU inn 0 k;j is non-linear, the dierence will be [0;UB S in (n 0 k;j ) ]. While UB S in (n 0 k;j ) is the symbolic upper bound of S in (n 0 k;j ), UB S in (n 0 k;j ) is the concrete upper bound. Note that sincen 0 k;j is non-linear, we must concretize to be sound. Next, we consider the three cases when the ReLU in n k;j is always activated. (1) If the ReLU in n 0 k;j is always deactivated, the dierence is [0; 0]S in (n k;j ). Again, we can soundly maintain symbolic equations. (2) If the ReLU in n 0 k;j is always activated, then the dierence is the same as in (n k;j ). (3) If the ReLU inn 0 k;j is non-linear, the dierence is [0;UB S in (n 0 k;j ) ]S in (n k;j ), which is the same as [UB S in (n k;j ) ;UB S in (n 0 k;j ) LB S in (n k;j ) ]. Again, we concretize to ensure soundness. Third, we consider the three cases where the ReLU in n k;j is non-linear. (1) If the ReLU in n 0 k;j is alwaysdeactivated, the dierence is [0; 0] [0;UB S in (n k;j ) ]. (2) If the ReLU inn 0 k;j isalwaysactivated, the dierence isS in (n 0 k;j ) [0;UB S in (n k;j ) ], which is the same as [LB S in (n 0 k;j ) UB S in (n k;j ) ; UB S in (n 0 k;j ) ]. (3) If the ReLU in n 0 k;j is also non-linear, then the dierence is [0;UB S in (n 0 k;j ) ] [0;UB S in (n k;j ) ], which is [UB S in (n k;j ) ;UB S in (n 0 k;j ) ]. 29 Input: valueS in (n k;j ), valueS in (n 0 k;j ), dierence in (n k;j ) Result: valueS(n k;j ), valueS(n 0 k;j ), dierence(n k;j ) ifUB S in (n k;j ) 0then R[k][j] = [0; 0] S(n k;j ) = [0; 0] if UB S in (n 0 k;j ) 0then R 0 [k][j] = [0; 0] S(n 0 k;j ) = [0; 0] (n k;j ) = [0; 0] elseifLB S in (n 0 k;j ) > 0then R 0 [k][j] = [1; 1] S(n 0 k;j ) =S in (n 0 k;j ) (n k;j ) =S in (n 0 k;j ) else R 0 [k][j] = [0; 1] S(n 0 k;j ) = [0;UB S in (n 0 k;j ) ] (n 0 k;j ) = [0;UB S in (n 0 k;j ) ] elseifLB S in (n k;j ) > 0then R[k][j] = [1; 1] S(n k;j ) =S in (n k;j ) if UB S in (n 0 k;j ) 0then R 0 [k][j] = [0; 0] S(n 0 k;j ) = [0; 0] (n k;j ) =S in (n k;j ) elseifLB S in (n 0 k;j ) > 0then R 0 [k][j] = [1; 1] S(n 0 k;j ) =S in (n 0 k;j ) (n k;j ) = in (n k;j ) else R 0 [k][j] = [0; 1] S(n 0 k;j ) = [0;UB S in (n 0 k;j ) ] (n k;j ) = [UB S in (n k;j ) ;UB S in (n 0 k;j ) LB S in (n k;j ) ] Opt. 1: (n k;j ) = [max UB S in (n k;j ) ;LB in (n k;j ) , max LB S in (n k;j ) ;UB in (n k;j ) ] else R[k][j] = [0; 1] S(n k;j ) = [0;UB S in (n k;j ) ] if UB S in (n 0 k;j ) 0then R 0 [k][j] = [0; 0] S(n 0 k;j ) = [0; 0] (n k;j ) = [UB S in (n k;j ) ; 0] elseifLB S in (n 0 k;j ) > 0then R 0 [k][j] = [1; 1] S(n 0 k;j ) =S in (n 0 k;j ) (n k;j ) = [LB S in (n 0 k;j ) UB S in (n k;j ) ;UB S in (n 0 k;j ) ] Opt. 2: (n k;j ) = [min LB S in (n 0 k;j ) ;LB in (n k;j ) ; min UB in (n k;j ) ;UB S in (n 0 k;j ) ] else R 0 [k][j] = [0; 1] S(n 0 k;j ) = [0;UB S in (n 0 k;j ) ] (n k;j ) = [UB S in (n k;j ) ;UB S in (n 0 k;j ) ] Opt. 3: ifLB in (n k;j ) 0then (n k;j ) = [0;UB in (n k;j ) ] elseifUB in (n k;j ) 0then (n k;j ) = [max LB in (n k;j ) ;UB S in (n k;j ) ; 0] else (n k;j ) = [max LB in (n k;j ) ;UB S in (n k;j ) , UB S in (n 0 k;j ) ] Algorithm2: Over-approximating ReLU activation function. 30 3.3.3 Optimization The most important optimization we make in the forward pass when computing(n k;j ) is in shifting from bounding the equation ReLU S in (n 0 k;j ) ReLU S in (n k;j ) to bounding one the following equivalent equations ReLU S in (n k;j ) + in (n k;j ) ReLU S in (n k;j ) (3.1) ReLU S in (n 0 k;j ) ReLU S in (n 0 k;j ) in (n k;j ) (3.2) Equation 3.1 says that for any concreten k;j 2S in (n k;j ) , the mostn 0 k;j can change is bounded by in (n k;j ), and similarly for Equation 3.2. As shown in Algorithm 2, we have identied three optimization opportuni- ties, marked asOpt.1-3. We note that, even though we widen the dierence interval in some of these cases, the interval is almost always tighter than if we subtract the bounds ofn 0 k;j andn k;j , even when they are symbolic. Below, we give formal proofs for most of the bounds, and the remaining proofs can be found in the appendix of our arXiv paper [80]. 3.3.3.1 Opt. 1:n k;j isactive,n 0 k;j isnon-linear. Using Equation 3.1, we can potentially tighten the lower bound. To reduce the notation complexity, we rewrite Equation 3.1 as the function: (n;d) = ReLU (n +d)ReLU (n) (3.3) = ReLU (n +d)n (3.4) 31 wheren2S in (n k;j ) andd2 in (n k;j ), and we can simplify from Equation 3.3 to 3.4 becausen k;j is active. Now, computing(n k;j ) amounts to nding the upper and lower bounds on (n;d). Observe that ifn +d 0 then (n;d) = d because ReLU(n +d) simplies ton +d, and the like terms cancel. Otherwise (n;d) =n becauseReLU(n +d) = 0. Observing thatn +d 0 =dn, this means (n;d) is equivalent to: (n;d) = 8 > > > > < > > > > : d dn n d<n =max(n;d) max() is well-dened for intervals. Specically, for two intervals [a;b]; [c;d], we have: max([a;b]; [c;d]) = [max(a;c);max(b;d)]: Now plugging in the the bounds ofn andd we get: (n;d) =max [UB S in (n k;j ) ;LB S in (n k;j ) ]; [LB in (n k;j ) ;UB in (n k;j ) ] = [max UB S in (n k;j ) ;LB in (n k;j ) ; max LB S in (n k;j ) ;UB in (n k;j ) ] 3.3.3.2 Opt. 2:n k;j isnon-linear,n 0 k;j isactive. Using Equation 3.2, we can tighten the upper bound. We rst rewrite Equation 3.2 as: 0 (n 0 ;d) =n 0 ReLU n 0 d : (3.5) 32 Just like Equation 3.4, Equation 3.5 can be broken into two cases based on the inequalityn 0 d 0 = n 0 d, which gives us the piece-wise equation: 0 (n 0 ;d) = 8 > > > > < > > > > : d n 0 d n 0 n 0 <d =min(n 0 ;d) For two intervals [a;b]; [c;d], we have min([a;b]; [c;d]) = [min(a;c);min(b;d)]: Replacingn 0 andd with the proper bounds gives us themin() function in Algorithm 2. 3.3.3.3 Opt. 3: bothn k;j andn 0 k;j arenon-linear. We consider three cases. First, letLB in (n k;j ) 0. This means thatn 0 k;j n k;j before applyingReLU, and then we can derive 0 as a lower bound as follows: n 0 k;j n k;j =) ReLU n 0 k;j ReLU n k;j = ReLU n 0 k;j ReLU n k;j 0 In addition, UB in (n k;j ) can be derived as an upper bound ∗ from Equation 3.2 [80]. This is the case in our motivating example, so(n 1;1 ) = [0; 1:1]. Second, we considerUB in (n k;j ) 0. This means n 0 k;j n k;j before ReLU, which allows us to derive an upper bound of 0 in a symmetric manner to the rst case. The lower bound shown in Algorithm 2 can be derived from Equation 3.1 [80]. In the third case ∗ In fact, the tighter upper boundmin(UB in (n k;j ) ;UB S in (n 0 k;j ) ) can be derived, however we had not yet proved this at the time of submission. 33 whereLB in (n k;j ) < 0<UB in (n k;j ) , the lower bound and the upper bound shown in Algorithm 2 can be derived from Equations 3.1 and 3.2, respectively [80] (also see Footnote∗). 3.3.4 OntheCorrectness The operations of the ane transformation are soundly dened for intervals as described in Section 3.2.2. For the ReLU transformation, we give formal explanations to show that they over-approximate. Since composing over-approximations also results in an over-approximation, the forward analysis is itself a sound over-approximation. 3.4 GradientBasedRenement After performing the forward pass, the computed dierence may not be tight enough to prove the desired property. In this section, we discuss how we can improve the analysis result. 3.4.1 SplittingInputIntervals As mentioned in Section 3.2, a common way to improve the result of interval analysis is dividing an in- put interval into disjoint sub-intervals, and then performing interval analysis on the sub-intervals. After unioning the output intervals, the result will beatleast as good as the original result [71]. Prior work [117] also shows a nice property of ReLU networks: after a nite number of such splits, the result of the interval analysis can be arbitrarily accurate. However, determining the optimal order of renement is dicult and, so far, the best algorithms are all heuristic based. For example, the method used in ReluVal chooses to split the input interval that has the most inuence on the output value. The intuition is that splitting such an input interval reduces the approximation error of the output interval. 34 However, the approach is not suitable in our case because we focus on the dierence between the two networks: the input interval with the most inuence on the absolute value of the output may not have the most inuence on the output dierence. To account for this dierence, we develop a method for determining which input interval to split. 3.4.2 TheRenementAlgorithm Our idea is to compute the dierence of the gradients for the two networks, denotedr . Toward this end, we compute the gradient of the rst network (r) and the gradient of the second network (r 0 ). Then, we use them to compute the dierencer . Formally,r =f@f=@x 1 ;:::;@f=@x n g is a vector whosei-th element,r i = @f=@x i , is the partial derivative of the outputf with respect to the inputx i . Similarly,r 0 =f@f 0 =@x 1 ;:::;@f 0 =@x n g. The dierence isr =f@(f 0 f)=@x 1 ;:::;@(f 0 f)=@x n g =r 0 r. That is, the derivative of a dierence of functions is the dierence of their derivatives. During interval analysis, the accurate gradient is dicult to compute. Therefore, we compute the approximated gradient, where each elementr i is represented by a concrete interval. Algorithm 3 shows our gradient computation procedure. In addition to the network, which may be eitherf orf 0 , it also takes the mask matrixR as input. Recall that bothR[k][j] andR 0 [k][j] have been computed by Algorithm 2 during the forward pass. R[k][i] may be [0; 0], [1; 1], or [0; 1], indicating if the ReLU inn k;j is always de-activated, always activated, or non-linear, respectively. It can be understood as the gradient interval of the ReLU. The gradient computation is performed backwardly beginning at the output layer and then moving through the previous layers. In each layer, the computation has two steps. First we apply ReLU to the current gradient and update the upper and lower bounds of the gradient if needed. Then, we scale the gradient interval by the weights of the previous layer. 35 Input: networkf, mask matrixR Result: gradientr // Initialize to edge weights in the output layer UB r1 =LB r1 =1 fork =l1::2do r New =f[0;0];:::;[0;0]g forj =1::layerSize[k]do // Perform ReLU for noden k;j ifR[k][j]==[0;0]then LB rj =UB rj =0 elseifR[k][j]=[0;1]then LB rj =min(0;LB rj ) UB rj =max(0;UB rj ) // Multiply by weights of incoming edges to noden k;j fori=1::layerSize[k1]do ifW k [i;j]0then UB r New i +=W k [i;j]UB rj LB r New i +=W k [i;j]LB rj else UB r New i +=W k [i;j]LB rj LB r New i +=W k [i;j]UB rj end end r=r NEW end returnr Algorithm3: Computing the gradient of a network. After computingr andr 0 by invoking Algorithm 3 onf andf 0 , respectively, we compute the gradient dierencer . Then, we use the gradient dierence to determine which input has the most inuence on the output dierence. Note that the gradient itself is not sucient to act as an indicator of inuence. For example, while an input’s gradient may be large, but the width of its input interval is small, splitting it will not have much impact on the output interval. Thus, we split the input interval with the maximumsmear value [51, 50]. The smear value of an inputx i is dened as the width of its input intervaljUB x i LB x i j scaled by the upper bound of its corresponding gradient dierenceUB r i . 3.4.3 AnExample We now walk through the gradient computation in Algorithm 3 for the example in Figure 3.7, where blue weights are for network f, and green weights are for network f 0 . We focus on the gradient of f rst. 36 1.1 0.5 1 -1 x 1 x 2 [0; 1:5] [0:5; 0:5] n 1 n 2 n 3 1.0 1.0 1 -1 Figure 3.7: Example for backward renement. After performing the forward pass, we know thatn 1 is in a linear state, i.e.,R[1][0] = [1; 1], andn 2 is in a non-linear state, i.e.,R[1][1] = [0; 1]. We initialize the gradient to the weights of the nal layer; that is, UB r 1 = LB r 1 = 1 and UB r 2 = LB r 2 =1. Next, we apply ReLU. Sincen 1 is in the always activated mode, we leave its gradient unchanged. However,n 2 is in thenon-linear mode, meaning the gradient could be 0, and hence we must ensure that 0 is in the gradient interval. We updateLB r 2 = 0. Then, we scale the gradient interval by weights of the incoming edges, which gives us the gradient intervals for input variables:r 1 = [1:1; 1:1] forx 1 andr 2 = [0:5; 0] forx 2 . Here, we point out a problem with ReluVal’s renement method. It would compute the smear value ofx 1 andx 2 to bej(1:5 0) 1:1j = 1:65 andj(0:5 (0:5))0:5j = 0:5, respectively, which means it would split onx 1 . However, this is not appropriate for dierential verication, since the two networks dier the most in the weights of the outgoing edge ofx 2 . Our method, instead, would compute the gradient dierencer = rr 0 . Therefore, we have r 1 = [0:1;0:1] forx 1 andr 2 = [0:5; 1] forx 2 . Based on the new smear values, we would choose to split the input interval ofx 2 . 37 3.5 Experiments We have implemented ReluDiff and compared it experimentally with state-of-the-art neural network verication tools. Like ReluVal, ReluDiff is written in C using OpenBLAS [126] as the library for ma- trix multiplications. We also note that we implement outward-rounding to soundly handle oating point arithmetic. Symbolic interval arithmetic is implemented using matrix multiplication. ReluDiff takes two networksf andf 0 together with a small and input regionX as input, and then decides whether8x2X. jf 0 (x)f(x)j < for the target label’s value. Since ReluDiff is the only tool currently available for dierential verication of neural networks, to facilitate the experimental comparison with existing tools, we developed a tool to mergef andf 0 into a combined networkf 00 , as shown in Figure 3.4, before feeding f 00 to these existing tools as input. 3.5.1 Benchmarks Our benchmarks are 49 feed-forward neural networks from three applications: aircraft collision detection, image recognition, and human activity recognition. We producef 0 by truncating each network’s weights from 32-bit oats to 16-bit oats. 3.5.1.1 ACASXu[46] ACAS Xu is a set of 45 neural networks commonly used in evaluating neural network verication tools. They are designed to be used in an aircraft to advise the pilot of what action to take in the presence of an intruder aircraft. They each take ve inputs: distance between self and the intruder, angle of self relative to the intruder, angle of intruder relative to self, speed of self, and speed of intruder. They output a score in the range [0:5; 0:5] for ve dierent actions: clear-of-conict, weak left, weak right, strong left, and strong right. The action with the minimum score is the action advised. In addition to the input and output 38 layers, each network has 6 hidden layers of 50 neurons each, for a total of 300 neurons. For dierential verication, we use the same input ranges as in [48, 117] as our regions of interest. 3.5.1.2 MNIST[62] MNIST is a data set of labeled images of hand-written digits that are often used as a benchmark to test image classiers. The images are 28x28 pixels, and each pixel has a gray-scale value in the range [0; 255]. Neural networks trained on this data set take in 784 inputs (one per pixel) each in the range [0; 255], and output 10 scores, typically in the range of [10; 10] for our networks, for each of the 10 digits. The digit with the highest score is the chosen classication. We use three neural networks trained on the MNIST data set with architectures of 3x100, 2x512, and 4x1024; that is, the networks have 3, 2, and 4 hidden layers, with layer size 100, 512, and 1024 neurons, respectively. Thus, in addition to the input and output layers, these networks have 300, 1024, and 4096 hidden neurons, respectively. Empirical analysis shows that each network has > 95% accuracy on hold-out test data. 3.5.1.3 HumanActivityRecognition(HAR)[2] HAR is a labeled data set used to train models to recognize specic human activities based on input from a smartphone’s accelerometer and gyroscope. Input examples in this data set are labeled with one of six activities: walking, walking upstairs, walking downstairs, sitting, standing, and laying down. The input data for the model are statistics computed from a smartphone’s accelerometer and gyroscope sensor, such as mean, median, min, max, etc. In total, 561 input statistics are computed from these two sensors. Inputs to the network are normalized to be in the range of [1; 1]. We use a network trained on this data set with an architecture of 1x500, meaning there is a hidden layer with 500 neurons. The network takes the 561 inputs, and produces a score in the range of [20; 20] for each of the 6 outputs, one per activity. The output with the maximum score is the classication. 39 Table 3.1: Statistics of the benchmarks: The total number of verication problems is 842. Name # in each network # input out NN’s # in # out # hidden # neurons region ACAS- 1 - 2 45 5 5 6 * 50 300 1 0.05 ACAS- 3 42 5 5 6 * 50 300 1 0.05 ACAS- 4 42 5 5 6 * 50 300 1 0.05 ACAS- 5 - 13 1 5 5 6 * 50 300 1 0.05 ACAS- 14 2 5 5 6 * 50 300 1 0.05 ACAS- 15 2 5 5 6 * 50 300 1 0.05 MNIST 3x100 1 784 10 3 * 100 300 200 1 MNIST 2x512 1 784 10 2 * 512 1,024 200 1 MNIST 4x1024 1 784 10 4 * 1024 4,096 200 1 HAR 1x500 1 561 6 1 * 500 500 100 0.25 Table 3.1 shows the statistics of these benchmarks, including the number of input neurons, the number of output neurons, the number of hidden layers, as well as the total number of neurons in these hidden layers. The last two columns list the experimental parameters we used, namely the number of “regions of interest” in the verication problems and the output we attempt to verify. 3.5.2 ExperimentalEvaluation We want to answer the following research questions: 1. Is ReluDiff more ecient than existing methods in dierential verication of neural networks in that it can both verify properties faster and verify more properties in general? 2. IsReluDiff more accurate than existing methods in the forward pass? Toward this end, we directly comparedReluDiff to two state-of-the-art verication tools: ReluVal [117] andDeepPoly [98]. Both are designed to formally verify the absence of adversarial examples. A comparison with Reluplex [48] was not possible since it does not support ane hidden layers, which are necessary for analyzing the combined networkf 00 (x) as shown in Figure 3.4, however we note that ReluVal previously has been shown to signicantly outperform Reluplex on all ACAS Xu bench- marks [117]. DeepPoly, a followup of AI2 [31], has also been shown to outperformAI2. 40 Table 3.2: Accuracy of comparison of the three tools on ACAS. Benchmark Verif. ReluDiff (new) ReluVal DeepPoly problems proved undet. proved undet. proved undet. ACAS 1 - 2 45 28 17 7 38 0 45 ACAS 3 42 42 0 24 18 6 36 ACAS 4 42 42 0 34 8 2 40 ACAS 5 1 1 0 0 1 0 1 ACAS 6 1 1 0 1 0 0 1 ACAS 7 1 0 1 0 1 0 1 ACAS 8 1 0 1 0 1 0 1 ACAS 9 1 1 0 0 1 0 1 ACAS 10 1 1 0 1 0 0 1 ACAS 11 1 1 0 0 1 0 1 ACAS 12 1 0 1 0 1 0 1 ACAS 13 1 1 0 1 0 0 1 ACAS 14 2 2 0 0 2 0 2 ACAS 15 2 2 0 0 2 0 2 Total 142 123 19 69 73 8 134 We ran all experiments on a Linux server running Ubuntu 16.04, an Intel Xeon CPU E5-2620, and 124 GB memory. Timeout is set at 30 minutes for each verication problem. When available, we enable parallelization in all tools and congure them to allow up to 10 threads at a time. 3.5.3 Results To evaluate eciency and accuracy, we truncate each network’s weights from 32-bit oats to 16-bit oats, and attempt to verify the shown in Table 3.1. We measure the number of properties veried and the execution time to verify each property. 3.5.3.1 ACASXu The results for ACAS Xu are shown in Tables 3.2 and 3.3. In Table 3.2, columns 1 and 2 show the input property used, and the number of networks we veried the property on, which are taken from [48, 117]. Columns 3-5 show the number of neural networks for which the property was veried and undetermined 41 Table 3.3: Eciency of ReluDiff vs. ReluVal on ACAS Xu. Benchmark Verif. Total Time (s) problems ReluDiff (new) ReluVal Avg. Speedup ACAS 12 45 40595.6 69167.5 1:7 ACAS 3 42 175.4 38414.2 219 ACAS 4 42 46.8 22159.2 473:4 ACAS 5 1 9.6 1800.0 187:5 ACAS 6 1 11.0 50.8 4:6 ACAS 7 1 1800.0 1800.0 1:00 ACAS 8 1 1800.0 1800.0 1:00 ACAS 9 1 52.3 1800.0 34:4 ACAS 10 1 31.0 53.3 1:6 ACAS 11 1 10.2 1800.0 177:3 ACAS 12 1 1800.0 1800.0 1:0 ACAS 13 1 157.9 999.2 6:3 ACAS 14 2 859.0 3600.0 4:2 ACAS 15 2 453.8 3600.0 7:9 for each tool. Undetermined means that either the tool reported it could not verify the problem (due to over-approximation), or the timeout of 30 minutes was reached. In Table 3.3, Columns 3-4 show the time taken byReluDiff andReluVal for all problems veried. The last column shows the time of ReluVal divided by the time of ReluDiff. For timeouts, we add 30 minutes to the total, which is why we display that the speedup is greater than or equal toX for some properties. We omit the timing data forDeepPoly since it cannot verify most properties. These results emphasize the improvement that ReluDiff can obtain in both speed and accuracy. It achieves orders of magnitude speedups over ReluVal. For example, ReluDiff nishes the 42 networks for 4 in 48.6 seconds, whereas ReluVal takes at least more than 6 hours. Overall ReluDiff veries 54 more problems for whichReluVal times out, and 115 more problems for whichDeepPoly is too inaccurate to verify. To understand whyReluDiff performs better, we plot the distribution of the depth at which each sub- interval was nally able to be veried for 4 , shown in Figure 3.8. We can see thatReluDiff consistently veries sub-intervals at much shallower split depths. We point out that the number of sub problems grows 42 Figure 3.8: 4 max depth distribution. Figure 3.9: -interval on MNIST 4x1024. exponentially as the split depth increases. Indeed, even though the dierence between the average depths does not seem large (about 14 for ReluDiff and 29 for ReluVal), ReluVal had to verify > 66 million sub-intervals for 4 , whereasReluDiff only had to verify 66K. 3.5.3.2 MNIST While in ACAS Xu the input region to verify is dened by the property, for MNIST, we must generate the input region ourselves. We generate 200 input regions for MNIST using two methods. The rst method is based on global perturbation [98]. We take 100 test images, and for each one, we allow each of the pixels to be perturbed by +/-3 gray scale units. The second method is based on targeted pixel perturbation [36, 35]. We take the same 100 test images, and for each one, we set the range of 3 random pixels to [0; 255], while the remaining 781 remain xed. We can again see in Tables 3.4 and 3.5 thatReluDiff is signicantly more accurate and ecient than bothReluVal andDeepPoly. Both competing techniques struggle to handle global perturbations even on the small 3x100 network, let alone the larger 2x512 and 4x1024 networks. On the other hand, ReluDiff can easily handle both the 3x100 and 2x512 networks, achieving at least 3 orders of magnitude speedup 43 Table 3.4: Accuracy comparison of the three tools on MNIST. Benchmark Verif. ReluDiff (new) ReluVal DeepPoly problems proved undet. proved undet. proved undet. 3x100-global 100 100 0 47 53 34 66 2x512-global 100 100 0 0 100 0 100 4x1024-global 100 22 78 0 100 0 100 3x100-3-pixel 100 100 0 100 0 100 0 2x512-3-pixel 100 100 0 100 0 80 20 4x1024-3-pixel 100 100 0 97 3 100 0 Table 3.5: Eciency comparison of the three tools on MNIST. Benchmark Verif. Total Time (s) problems ReluDiff (new) ReluVal DeepPoly 3x100-global 100 29.47 95458.32 118823.09 2x512-global 100 77.83 180000.00 180000.0 4x1024-global 100 141604.53 180000.00 180000.0 3x100-3-pixel 100 23.90 32.60 163.75 2x512-3-pixel 100 79.24 715.16 37674.40 4x1024-3-pixel 100 296.59 92100.10 49042.98 on these networks. We also see a three orders of magnitude speedup on the two largest networks for our targeted-pixel perturbation experiments. Even though ReluDiff begins to reach its limit in the global perturbation experiment on the largest 4x1024 network, we point out thatReluDiff is signicantly outperforming bothDeepPoly andReluVal in the accuracy of their forward passes. Figure 3.9 compares the output bound veried on the rst, single forward pass of each technique. The comparison is presented as a scatter plot, where the x-axis is the bound veried byReluDiff, and the y-axis is that of the competing technique. The graph shows that ReluDiff is nearly two orders of magnitude more accurate than ReluVal and one order of magnitude more thanDeepPoly. The improvement overDeepPoly especially emphasizes the promise of ReluDiff’s approach. This is becauseReluDiff is already outperformingDeepPoly, yet it uses a simpler concretization approach during the forward pass, whereas DeepPoly uses a more sophisticated linearrelaxation. We believe thatReluDiff can be extended to use more accurate techniques such as linear relaxation which would further improve the accuracy, however we leave this as future work. 44 Table 3.6: Accuracy comparison of the three tools on HAR. Benchmark Verif. ReluDiff (new) ReluVal DeepPoly problems proved undet. proved undet. proved undet. 1x500 100 100 0 0 100 0 100 Table 3.7: Eciency comparison of the three tools on HAR. Benchmark Verif. Total Time (s) problems ReluDiff (new) ReluVal DeepPoly 1x500 100 28.79 180000.00 180000.00 3.5.3.3 HAR For HAR, we also created our verication problems using input perturbation. We take 100 concrete test inputs, and for each one, we allow a global perturbation of +/-0.1. The results are summarized in Tables 3.6 and 3.7. Again, the experimental comparison shows that ReluDiff is signicantly more accurate and ecient. 3.5.4 ThreatstoValidity Our method is designed for verifying neural networks typically found in control applications, where the number of input signals is not large. In this context, dividing the input region turns out to be a very eective way of increasing the accuracy of interval analysis. However, neural networks in dierent application domains may have dierent characteristics. Therefore, it remains an open problem whether bi-section of individual input intervals is always an eective way of performing renement. Our method is designed for feed-forward ReLU networks. Although there is no signicant technical hurdle for it to be extended to convolutional neural networks or other activation functions, such as sigmoid, tanh and max-pool as shown recently by Singh et al. [98], we have not evaluated the eectiveness. Specif- ically, linear relaxation can be used to handle these features when it comes to approximating non-linear behavior. While we use concretization in ReluDiff, extending it with linear relaxation is possible [116]. However, we leave these extensions for future work. 45 At a higher level, our method relies on symbolic interval analysis, which can be viewed as a specic form of abstract interpretation [17]. While the abstract interpretation framework allows approximations to be performed in a more general way, e.g., using relational abstract domains [67] such as the octagon [66] and polyhedral [18] domains, so far, it has not be adequately explored. We plan to explore the use of these abstract domains as part of the future work. Finally, the term dierential verication has been used in the context of verifying a new version of a program with respect to a previous version, which is treated as an “oracle” [60]. In a sense, the truncated network is a “new version” of the original network, and the original network can be thought of as an oracle. 3.6 Summary We have presented a new method, named ReluDiff, for dierential verication of two closely related neural networks. It is capable of formally proving the accuracy of a compressed network with respect to the original network. Internally,ReluDiff relies on symbolic interval analysis to more accurately compute and propagate dierences in the values of neurons of the two networks from the input to the output, and then relies on the gradient dierence to more accurately compute the renement. Our experimental comparison of ReluDiff with state-of-the-art formal verication techniques shows that it can often achieve two orders of magnitude speedup and produce many more proofs. 46 3.7 Optimization3Proofs Here we give formal proofs for the un-proven bounds in Section 3.3.3. As in section 3.3.3, we rewrite Equations 3.1 and 3.2 as (n;d) = ReLU n +d ReLU n 0 (n 0 ;d) = ReLU n 0 ReLU n 0 d : wheren2S in (n k;j );n 0 2S in (n 0 k;j ); andd2 in (n k;j ). 3.7.1 UpperBoundofFirstCase Recall this case isLB in (n k;j ) 0. We derive the bound from Equation 3.2 by dividing into two cases, and then combining their result to get the lower bound. Case1:n 0 d> 0 =n 0 >d Based on the above constraint, we can simplify 0 (n 0 ;d) to 0 (n 0 ;d) = ReLU n 0 (n 0 d): In addition, since we are considering the case whereLB in (n k;j ) 0, we haved 0. Combining this with our case 1 constraintn 0 >d, we have n 0 >d^d 0 =) n 0 > 0: 47 Thus, in case 1, we have that 0 (n 0 ;d) is just 0 (n 0 ;d) =n 0 (n 0 d) =d: We also note thatn 0 > 0 means that n 0 >d () max(0;n 0 )>d becausen 0 > 0 means thatmax(0;n 0 ) =n 0 . We use this fact when we combine the two cases. Case2:n 0 d 0 =n 0 d The above constraint allows us to simplify 0 (n 0 ;d) to 0 (n 0 ;d) = ReLU n 0 0 = ReLU n 0 =max(0;n 0 ): We also note that under our constraints we have n 0 d () max(0;n 0 )d; which we will use when we combine the cases. To prove this, rst observe that 0d^n 0 d =) max(0;n 0 )d: That is, ifd is greater-than-or-equal to both 0 andn 0 , then clearly it is greater-than-or-equal to the max of the two. And for the other way, max(0;n 0 )d =) n 0 d: 48 That is, ifd is greater-than-or-equal to the max of 0 andn 0 , then clearly it is greater-than-or-equal ton 0 . CombingtheTwoCases Combining our two cases, we get the function 0 (n 0 ;d) = 8 > > > > < > > > > : d n 0 >d max(0;n 0 ) n 0 d and then substituting with the equations we derived at the end of case 1 and 2, we get 0 (n 0 ;d) = 8 > > > > < > > > > : d max(0;n 0 )>d max(0;n 0 ) max(0;n 0 )d =min(max(0;n 0 );d) Since the upper bound of bothmin() andmax() occur when we take the upper bounds of their input variables, we get that the upper bound of 0 (n 0 ;d) is min(max(0;UB S in (n 0 k;j ) );UB in (n k;j ) ) =min(UB S in (n 0 k;j ) ;UB in (n k;j ) ): We can remove themax() function becausen 0 k;j is non-linear, soUB S in (n 0 k;j ) > 0. 3.7.2 LowerBoundofSecondCase Recall this case isUB in (n k;j ) 0. We derive the lower bound from Equation 3.1 by dividing into two cases. The proof is symmetric to the previous case. 49 Case1:n +d> 0 =n>d =d>n In this case, we can immediately simplify (n;d) =n +dReLU n : Then, sinceUB in (n k;j ) 0 impliesd 0 =d 0, we can use our case 1 constraintn >d to derive n>d^d 0 =) n> 0: Thus we can further simplify (n;d) =n +dn =d: We also note thatn> 0 =n< 0 =) min(0;n) =n, so we also have d>n () d>min(0;n): We use this fact when combining the two cases. Case2:n +d 0 =dn In this case, we can immediately simplify (n;d) = 0ReLU n =max(0;n) =min(0;n): We also not here that dn^d 0 () dmin(0;n): We use this fact when combining the two cases. 50 CombiningtheCases Combing our two cases, we get the function (n;d) = 8 > > > > < > > > > : d d>n min(0;n) dn and rewriting this equation using the inequalities we derived at the end of case 1 and 2 we get (n;d) = 8 > > > > < > > > > : d d>min(0;n) min(0;n) dmin(0;n) =max(d;min(0;n)): Since the lower bound of bothmin() andmax() occur when we minimize their inputs, we get the lower bound of (n;d) is max(LB in (n k;j ) ;min(0;UB S in (n k;j ) )) =max(LB in (n k;j ) ;UB S in (n k;j ) ): We can remove themin() function becausen k;j is non-linear soUB S in (n k;j ) < 0. 3.7.3 LowerBoundofThirdCase Recall this case isLB in (n k;j ) < 0<UB in (n k;j ) . We derive the lower bound from Equation 3.1. We divide into two cases and then combine them as done previously. 51 Case1:n +d> 0 =d>n (n;d) simplies to (n;d) =n +dReLU n : We further divide into two sub-cases. Case1.1:n> 0 (n;d) further simplies to (n;d) =n +dn =d: Since we only care about the lower bound, observe that the lower bound of case 1.1 occurs when we take the minimum value ofd, which is always less than 0. Case1.2:n 0 (n;d) further simplies to (n;d) =n +d 0 =n +d: Observe that the lower bound cannot be less than 0 in case 1.2 because of the case 1 constraint. This means the lower bound always occurs in case 1.1, so we can safely ignore case 1.2. (But we emphasize this only applies when evaluating (n;d) for the lower bound). Case2:n +d 0 =dn (n;d) further simplies to (n;d) = 0ReLU n =ReLU n : 52 We consider the two cases of this function. Case2.1:n> 0 (n;d) becomes (n;d) =n: The minimum value here occurs at the upper bound of n, which is always less than 0 because n k;j is non-linear. Case2.2:n 0 (n;d) = 0 in this case. Since the lower bound of case 2.1 is always less than 0, the minimum value will never occur in this case, so we can safely ignore it. CombiningtheCases We’ve shown that the minimum value occurs in either case 1.1 or case 2.1, which gives us the function (however only for the minimum value and not the maximum) (n;d) = 8 > > > > < > > > > : d d>n n dn =max(n;d): Evaluating this function for its lower bounds gives us max(UB S in (n k;j ) ;LB in (n k;j ) ): 53 3.7.4 UpperBoundofThirdCase Recall this case isLB in (n k;j ) < 0 < UB in (n k;j ) . We derive the upper bound from Equation 3.2. We divide into two cases and then combine them as done previously. This proof mirrors the proof for the lower bound of the third case. Case1:n 0 d> 0 =n 0 >d 0 (n 0 ;d) simplies to 0 (n 0 ;d) = ReLU n 0 n 0 +d: We further divide into two sub-cases. Case1.1:n 0 > 0 0 (n 0 ;d) further simplies to 0 (n 0 ;d) =n 0 n 0 +d =d: Since we only care about the upper bound, observe that the upper bound of case 1.1 occurs when we take the maximum value ofd, which is always greater than 0. Case1.2:n 0 0 0 (n 0 ;d) further simplies to 0 (n 0 ;d) = 0n 0 +d =n 0 +d: Our case 1 constraintn 0 d > 0 = 0 >n 0 +d implies the upper bound in case 1.2 can be no greater than 0. This means the upper bound 0 (n 0 ;d) always occurs in case 1.1, so we can ignore case 1.2. 54 Case2:n 0 d 0 =n 0 d 0 (n 0 ;d) further simplies to 0 (n 0 ;d) = ReLU n 0 0 = ReLU n 0 : We consider the two cases of this function. Case2.1:n 0 > 0 0 (n 0 ;d) becomes 0 (n 0 ;d) =n 0 ; which has its maximum value at the upper bound ofn 0 . Case2.2:n 0 0 (n;d) = 0 in this case. Since the upper bound ofn 0 is greater than 0, the maximum value will never occur in this case, so we can safely ignore it. CombiningtheCases We’ve shown that the maximum value occurs in either case 1.1 or case 2.1, which gives us the function (only for the maximum value and not the minimum) 0 (n 0 ;d) = 8 > > > > < > > > > : d n 0 >d n 0 n 0 d =min(n 0 ;d): Evaluating this function for its upper bound gives us 55 min(UB S in (n 0 k;j ) ;UB in (n k;j ) ): 56 Chapter4 SymbolicApproximationsfortheDierence There is a growing need for rigorous analysis techniques that can compare the behaviors of two or more neural networks trained for the same task. For example, such techniques have applications in better un- derstanding the representations learned by dierent networks [115], and nding inputs where networks disagree [121]. The need is further motivated by the increasing use of neural network compression [39] – a technique that alters the network’s parameters to reduce its energy and computational cost – where weexpect the compressed network to be functionally equivalent to the original network. In safety-critical systems where a single instance of misbehavior can lead to catastrophe, having formal guarantees on the equivalence of the original and compressed networks is highly desirable. Unfortunately, most work aimed at verifying or testing neural networks does not provide formal guar- antees on their equivalence. For example, testing techniques geared toward refutation can provide inputs where a single network misbehaves [63, 120, 102, 107, 77] or multiple networks disagree [121, 83, 64], but they do not guarantee the absence of misbehaviors or disagreements. While techniques geared toward verication can prove safety or robustness properties of a single network [44, 24, 49, 91, 116, 99, 68, 31, 29], they lack crucial information needed to prove the equivalence of multiple networks. One exception is the ReluDiff tool of Paulsen et al. [81], which computes a sound approximation of the dierence of two neural networks, a problem known asdierentialverication. WhileReluDiff performs better than other 57 Inputs NeuroDi Forward Analysis Convex Approximation Intermediate Variables Check Yes Veried Partition X No f f 0 XR n X 1 X 2 Proven? Figure 4.1: The overall ow of NeuroDiff. techniques, the overly conservative approximation it computes often causes both accuracy and eciency to suer. To overcome these problems, we proposeNeuroDiff, a newsymbolic andne-grained approximation technique that signicantly increases the accuracy of dierential verication while achieving many orders- of-magnitude speedup. NeuroDiff has two key contributions. The rst contribution is the development of convex approximations, a ne-grained approximation technique for bounding the output dierence of neurons for all possible inputs, which drastically improves over the coarse-grained concretizations used byReluDiff. The second contribution is judiciously introducing symbolic variables to represent neurons in hidden layers whose dierence bounds have accumulated signicant approximation error. These two techniques are also complementary, i.e., when combined, the benet is signicantly greater than the sum of their individual benets. The overall ow of NeuroDiff is shown in Figure 4.1, where it takes as input two neural networks f andf 0 , a set of inputs to the neural networksX dened by box intervals, and a small constant that quanties the tolerance for disagreement. We assume thatf andf 0 have the same network topology and only dier in the numerical values of their weights. In practice, f 0 could be the compressed version of f, or they could be networks constructed using the same network topology but slightly dierent training data. We also note that this assumption can support compression techniques such as weight pruning [39] 58 (by setting edges’ weights to 0) and even neuron removal [32] (by setting all of a neuron’s incoming edge weights to 0). NeuroDiff then aims to prove8x2 X:jf 0 (x)f(x)j < . It can return (1) veried if a proof can be found, or (2) undetermined if a specied timeout is reached. Internally, NeuroDiff rst performs a forward analysis using symbolic interval arithmetic to bound both the absolute value ranges of all neurons, as in single network verication, and the dierence between the neurons of the two networks. NeuroDiff then checks if the dierence between the output neurons satises, and if so returns veried. Otherwise,NeuroDiff uses a gradient-based renement to partition X into two disjoint sub regionsX 1 andX 2 , and attempts the analysis again on the individual regions. Since X 1 and X 2 form independent sub-problems, we can do these analyses in parallel, hence gaining signicant speedup. The new convex approximations used inNeuroDiff are signicantly more accurate than not only the coarse-grained concretizations in ReluDiff [81] but also the standard convex approximations in single- network verication tools [98, 96, 116, 125]. While these (standard) convex approximations aim to bound the absolute value range ofy =ReLU(x), wherex is the input of therectiedlinearunit (ReLU) activation function, our new convex approximations aim to bound the dierencez = ReLU(x + )ReLU(x), wherex andx + are ReLU inputs of two corresponding neurons. This is signicantly more challenging because it involves the search of bounding planes in a three-dimensional space (dened byx, andz) as opposed to a two-dimensional space as in the prior work. The symbolic variables we judiciously add to represent values of neurons in hidden layers should not be confused with the symbolic inputs used by existing tools either. While the use of symbolic inputs is well understood, e.g., both in single-network verication [98, 96, 116, 125] and dierential verication [81], this is the rst time that symbolic variables are used to substitute values of hidden neurons during dierential verication. While the impact of symbolic inputs often diminishes after the rst few layers of neurons, the impact of these new symbolic variables, when judiciously added, can be maintained in any hidden layer. 59 We have implemented the proposedNeuroDiff in a tool and evaluated it on a large set of dierential verication tasks. Our benchmarks consists of 49 networks, from applications such as aircraft collision avoidance, image classication, and human activity recognition. We have experimentally compared with ReluDiff [81], the state-of-the-art tool which has also been shown to be superior to ReluVal [117] and DeepPoly [98] for dierential verication. Our results show thatNeuroDiff is up to 1,000X faster and 5X more accurate. In addition, NeuroDiff is able to prove many of the same properties as ReluDiff while considering much larger input regions. To summarize, this paper makes the following contributions: • We propose new convex approximations to more accurately bound the dierence between corre- sponding neurons of two structurally similar neural networks. • We propose a method for judiciously introducing symbolic variables to neurons in hidden layers to mitigate the propagation of approximation error. • We implement and evaluate the proposed technique on a large number of dierential verication tasks and demonstrate its signicant speed and accuracy gains. The remainder of this chapter is organized as follows. First, we provide a brief overview of our method in Section 4.1. Then, we provide the technical background in Section 4.2. Next, we present the detailed al- gorithms in Section 4.3 and the experimental results in Section 4.4. Finally, we summarize the contributions of this chapter in Section 4.5. 4.1 Overview In this section, we highlight our main contributions and illustrate the shortcomings of previous work on a motivating example. 60 1.9 1.1 -1.9 1.0 2.1 0.9 1.1 -1.0 1.0 -1.0 n 0;1 n 0;2 n 1;2 n 1;1 n 2;1 n 2;2 n 3;1 x 1 2 [2; 2] x 2 2 [2; 2] -2.0 1.0 2.0 1.0 -1.0 1.0 2.0 1.0 1.0 -1.0 Figure 4.2: Motivating example. 4.1.1 DierentialVerication We use the neural network in Figure 4.2 as a running example. The network has two input nodesn 0;1 ;n 0;2 , two hidden layers with two neurons each (n 1;1 ;n 1;2 andn 2;1 ;n 2;2 ), and one output noden 3;1 . Each neu- ron in the hidden layer performs a summation of their inputs, followed by a rectied linear unit (ReLU) activation function, dened asy =max(0;x), wherex is the input to the ReLU activation function, and y is the output. Let this entire network bef, and the value of the output node ben 3;1 = f(x 1 ;x 2 ), wherex 1 andx 2 are the values of input nodesn 0;1 andn 0;2 , respectively. The network can be evaluated on a specic input by performing a series matrix multiplications (i.e., ane transformations) followed by element-wise ReLU transformations. For example, the output of the neurons of the rst hidden layer is 2 6 6 4 n 1;1 n 1;2 3 7 7 5 =ReLU 2 6 6 4 1:9 1:9 1:0 1:1 3 7 7 5 2 6 6 4 x 1 x 2 3 7 7 5 ! = 2 6 6 4 ReLU(1:9x 1 1:9x 2 ) ReLU(1:1x 1 + 1:0x 2 ) 3 7 7 5 Dierential verication aims to comparef to another networkf 0 that is structurally similar. For our example,f 0 is obtained by rounding the edge weights off to the nearest whole numbers, a network com- pression technique known asweightquantization. Thus,f 0 ,n 0 k;j andn 0 3;1 =f 0 (x 1 ;x 2 ) are counterparts of 61 f,n k;j andn 3;1 =f(x 1 ;x 2 ) for 0k 2 and 1j 2. Our goal is to prove thatjf 0 (x 1 ;x 2 )f(x 1 ;x 2 )j is less than some reasonably small for all inputs dened by the intervalsx 1 2 [2; 2] andx 2 2 [2; 2]. For ease of understanding, we show the edge weights off in black, andf 0 in light blue in Figure 4.2. 4.1.2 LimitationsofExistingMethods Naively, one could adapt any state-of-the-art, single-network verication tool for our task, includingDeep- Poly [98] and Neurify [116]. Neurify, in particular, takes a neural network and an input region of the network, and uses interval arithmetic [71, 117] to produce sound symbolic lower and upper bounds for each output node. Typically,Neurify would then use the computed bounds to certify the absence ofadversarial examples [103] for the network. However, for our task, the bounds must be computed for both networksf andf 0 . Then, we subtract them, and concretize to compute lower and upper bounds onf 0 (x 1 ;x 2 )f(x 1 ;x 2 ). In our example, the individual bounds would be (approximately, due to rounding) [LB(f);UB(f)] = [0:94x 1 0:62x 2 6:51; 0:71x 1 2:35x 2 + 7:98] and [LB(f 0 );UB(f 0 )] = [0:94x 1 0:44x 2 6:75; 0:75x 1 2:25x 2 + 8:00] for nodesn 3;1 andn 0 3;1 , respectively. After the subtraction, we would obtain the bounds [LB(f 0 ) UB(f);UB(f 0 )LB(f)] = [1:65x 1 + 1:9x 2 14:73; 1:68x 1 1:63x 2 + 14:5]. After concretization, we would obtain the bounds [21:83; 21:12]. Unfortunately, the bounds are far from being accurate. The ReluDiff method of Paulsen et al. [81] showed that, by directly computing a dierence interval layer-by-layer, the accuracy can be greatly improved. For the running example, ReluDiff would rst compute bounds on the dierence between the neuronsn 1;1 andn 0 1;1 , which is [0; 1:1], and then similarly compute bounds on the dierence between outputs ofn 1;2 andn 0 1;2 . Then, the results would be used to compute dierence bounds of the subsequent layer. The reason it is more accurate is because it begins computing part of the dierence bound before errors have accumulated, whereas the naive approach rst 62 Figure 4.3: The shape ofz = ReLU(n + ) ReLU(n). Figure 4.4: Bounding planes computed by Re- luDiff [81]. Figure 4.5: Bounding planes computed by our new method. LB (n) UB (n) UB(ReLU(n)) LB(ReLU(n)) Figure 4.6: Bounding planes computed byNeu- rify [116]. accumulates signicant errors at each neuron, and then computes the dierence bound. In our running example,ReluDiff [81] would compute the tighter bounds [3:1101; 2:5600]. WhileReluDiff improves over the naive approach, in many cases, it usesconcrete values for the upper and lower bounds. In practice, this approach can suer from severe error-explosion. Specically, whenever a neuron of either network is in anunstable state – i.e., when a ReLU’s input interval contains the value 0 – it has to concretize the symbolic expressions. 4.1.3 OurMethod The key contribution inNeuroDiff, our new method, is asymbolic and ne-grained approximation tech- nique that both reduces the approximation error introduced when a neuron is in an unstable state, and mitigates the explosion of such approximation error after it is introduced. 63 4.1.3.1 ConvexApproximationfortheDierenceInterval Our rst contribution is developing convex approximations to directly bound the dierence between two neurons after these ReLU activations. Specically, for a neuronn inf and corresponding neuronn 0 inf 0 , we want to bound the value ofReLU(n 0 )ReLU(n). We illustrate the various choices using Figures 4.3, 4.4, and 4.5. The naive way to bound this dierence is to rst compute approximations ofy =ReLU(n) andy 0 = ReLU(n 0 ) separately, and then subtract them. Since each of these functions has a single variable, convex approximation is simple and is already used by single-network verication tools [98, 116, 118]. Figure 4.6 shows the functiony =ReLU(n) and its bounding planes (shown as dashed-lines) in a two-dimensional space (details in Section 4.2). However, as we have already mentioned, approximation errors would be accumulated in the bounds ofReLU(n) andReLU(n 0 ) and then amplied by the interval subtraction. This is precisely why the naive approach performs poorly. The ReluDiff method of Paulsen et al. [81] improves upon the new approximation by computing an interval bound on n 0 n, denoted , then rewriting z = ReLU(n 0 )ReLU(n) as z = ReLU(n + ) ReLU(n), and nally bounding this new function instead. Figure 4.3 shows the shape of z = ReLU(n+)ReLU(n) in a three-dimensional space. Note that it has four piece-wise linear subregions, dened by values of the input variablesn and . While the bounds computed byReluDiff [81], shown as the (horizontal) yellow planes in Figure 4.4, are sound, in practice they tend to be loose because the upper and lower bounds are both concrete values. Such eager concretization eliminates symbolic information that contained before applying the ReLU activation. In contrast, our method computes a convex approximation ofz, shown by the (tilted) yellow planes in Figure 4.5. Since these tilted bounding planes are in a three-dimensional space, they are signicantly more challenging to compute than the standard two-dimensional convex approximations (shown in Fig- ure 4.6) used by single network verication tools. Our approximations have the advantage of introducing 64 signicantly less error than the horizontal planes used in ReluDiff [81], while maintaining some of the symbolic information for before applying the ReLU activation. We will show through experimental evaluation (Section 4.4) that our convex approximation can dras- tically improve the accuracy of the dierence bounds, and are particularly eective when the input region being considered is large. Furthermore, the tilted planes shown in Figure 4.5 are for the general case. For certain special cases, we obtain even tighter bounding planes (details in Section 4.3). In the running example, using our new convex approximations would improve the nal bounds to [1:97; 1:42]. 4.1.3.2 SymbolicVariablesforHiddenNeurons Our second contribution is introducing symbolic variables to represent the output values of some unstable neurons, with the goal of limiting the propagation of approximation errors after they are introduced. In the running example, since bothn 1;1 andn 0 1;1 are in unstable states, i.e., the input intervals of the ReLUs contain the value 0, we may introduce a new symbolx 3 =ReLU(n 0 1;1 )ReLU(n 1;1 ). In all subsequent layers, whenever the value ofReLU(n 0 1;1 )ReLU(n 1;1 ) is needed, we use the bounds [x 3 ;x 3 ] instead of the actual bounds. The reason why usingx 3 can lead to more accurate results is because, even though our convex ap- proximations reduce the error introduced, there is inevitably some error that accumulates. Introducingx 3 allows this error to partially cancel in the subsequent layers. In our running example, introducing the new symbolic variablex 3 would be able to improve the nal bounds to [1:65; 1:18]. While creating x 3 improved the result in this case, carelessly introducing new variables for all the unstable neurons can actually reduce the overall benet (see Section 4.3). In addition, the computational cost of introducing new variables is not negligible. Therefore, in practice, we must introduce these symbolic variables judiciously, to maximize the benet. Part of our contribution in NeuroDiff is in developing heuristics to automatically determine when to create new symbolic variables (details in Section 4.3). 65 4.2 Background In this section, we review the technical background and then introduce notations that we use throughout the paper. 4.2.1 NeuralNetworks We focus on feed-forward neural networks, which we dene as a functionf that takes ann-dimensional vector of real values x2 X, whereX R n , and maps it to an m-dimensional vector y 2 Y, where YR m . We denote this function asf :X!Y. Typically, each dimension ofy represents a score, such as a probability, that the inputx belongs to classi, where 1im. A network withl layers hasl weight matrices, each of which is denotedW k ; for 1kl. For each weight matrix, we haveW k 2R l k1 l k wherel k1 is the number of neurons in layer (k 1) and likewise forl k , andl 0 = n. Each element inW k represents the weight of an edge from a neuron in layer (k 1) to one in layer k. Let n k;j denote the j th neuron of layer k, and n k1;i denote the i th neuron of layer (k 1). We useW k [i;j] to denote the edge weight fromn k1;i ton k;j . In our motivating example, we haveW 1 [1; 1] = 1:9 andW 1 [1; 2] = 1:1. Mathematically, the entire neural network can be represented byf(x) =f l (W l f l1 (W l1 :::f 1 (W 1 x):::)), wheref k is the activation function of thek th layer and 1 k l. We focus on neural networks with ReLU activations because they are the most widely implemented in practice, but our method can be extended to other activation functions, such assigmoid andtanh, and other layer types, such as convo- lutional and max-pooling. We leave this as future work. 4.2.2 SymbolicIntervals To compute approximations of the output nodes that are sound for all input values, we leverage interval arithmetic [71], which can be viewed as an instance of the abstract interpretation framework [17]. It is 66 well-suited to the verication task because interval arithmetic is soundly dened for basic operations of the network such as addition, subtraction, and scaling. LetI = [LB(I);UB(I)] be an interval with lower boundLB(I) and upper boundUB(I). Then, for intervalsI 1 ;I 2 , we have addition and subtraction dened asI 1 +I 2 = [LB(I 1 )+LB(I 2 );UB(I 1 )+UB(I 2 )] andI 1 I 2 = [LB(I 1 )UB(I 2 );UB(I 1 )LB(I 2 )], respectively. For a constantc, scaling is dened as cI 1 = [cLB(I 1 );cUB(I 1 ) whenc> 0, andcI 1 = [cUB(I 1 );cLB(I 1 )] otherwise. While interval arithmetic is a sound over-approximation, it is not always accurate. To illustrate, let f(x) = 3xx, and say we are interested in boundingf(x) whenx2 [1; 1]. One way to boundf is by evaluatingf(I) whereI = [1; 1]. Doing so yields 3 [1; 1] [1; 1] = [4; 4]. Unfortunately, the most accurate bounds are [2; 2]. There are (at least) two ways we can improve the accuracy. First, we can soundly rene the result by dividing the input intervals into disjoint partitions, performing the analysis independently on each partition, and then unioning the resulting output intervals together. Previous work has shown the result will be at least as precise [117], and often better. For example, if we partitionx2 [1; 1] intox2 [1; 0] andx2 [0; 1], and perform the analysis for each partition, the resulting bounds improve to [3; 3]. Second, the dependence between the two intervals are not leveraged when we subtract them, i.e., that they were both x terms and hence could partially cancel out. To capture the dependence, we can use symbolic lower and upper bounds [117], which are expressions in terms of the input variable, i.e., I = [x;x]. Evaluatingf(I) then yields the intervalI f = [2x; 2x], forx2 [1; 1]. When using symbolic bounds, eventually, we must concretize the lower and upper bound equations. We denote concretization ofLB(I f ) = 2x andUB(I f ) = 2x asLB(I f ) =2 andUB(I f ) = 2, respectively. Compared to the naive solution, [4; 4], this is a signicant improvement. When approximating the output of a given functionf : X! Y over an input intervalX X, one may prove soundness by showing that the evaluation of the lower and upper bounds on any inputx2X 67 are always greater than and less than, respectively, to the true value off(x). Formally, for an intervalI, letLB(I)(x) be the evaluation of the lower bound equation on inputx, and similarly forUB(I)(x). Then, the approximation is considered sound if8x2X, we haveLB(I)(x)f(x)UB(I)(x). 4.2.3 ConvexApproximations While symbolic intervals are exact for linear operations (i.e. they do not introduce error), this is not the case for non-linear operations, such as the ReLU activation. This is because, for eciency reasons, the symbolic lower and upper bounds must be kept linear. Thus, developing linear approximations for non-linear activa- tion functions has become a signifciant area of research for single neural network verication [116, 98, 118, 125]. We review the basics below, but caution that they are dierent from our new convex approximations inNeuroDiff. We denote the input to the ReLU of a neuronn k;j asS in (n k;j ) and the output asS(n k;j ). The approach used by existing single-network verication tools is to apply an ane transformation to the upper bound of S in (n k;j ) such that UB(S in (n k;j ))(x) 0, where x2 X, and X is the input region for the entire network. For the lower bound, there exist several possible transformations, including the one used by Neurify [116], shown in Figure 4.6, wheren = S in (n k;j ) and the dashed lines are the upper and lower bounds. We illustrate the upper bound transformation forn 1;1 of our motivating example. After computing the upper bound of the ReLU input UB(S in (n 1;1 )) = 1:9x 1 1:9x 2 , where x 1 2 [2; 2] and x 2 2 [2; 2], it computes the concrete lower and upper bounds. We denote these as UB(S in (n 1;1 )) =7:6 andUB(S in (n 1;1 )) = 7:6. We refer to them asl andu, respectively, for short hand. Then, it computes the line that passes through (u;u) and (0;l). Lettingy = UB(S in (n 1;1 )) be the upper bound equation of the ReLU input, it computes the upper bound of the ReLU output as UB(S(n 1;1 )) = u ul (yl) = 0:95x 1 0:95x 2 + 3:81. 68 When considering a single ReLU of a single network, convex approximation is simple because there are only three states that the neuron can be in, namely active, inactive, and unstable. Furthermore, in only one of these states, convex approximation is needed. In contrast, dierential verication has to consider a pair of neurons, which has up to nine states to consider between the two ReLUs. Furthermore, dierent states may result in dierent linear approximations, and some states can even have multiple linear ap- proximations depending on the dierence bound of =n 0 n. As we will show in Section 4.3, there are signicantly more considerations in our problem domain. 4.3 OurApproach We rst present our baseline procedure for dierential verication of feed-forward neural networks (Sec- tion 4.3.1), and then present our algorithms for computing convex approximations (Section 4.3.3) and in- troducing symbolic variables (Section 4.3.4). 4.3.1 DierentialVerication–Baseline We build o the work of Paulsen et al. [81], so in this section we review the relevant pieces. We assume that the input to NeuroDiff consists of two networksf andf 0 , each withl layers of the same size. Let n 0 k;j inf 0 be the neuron paired withn k;j inf. This implicitly creates a pairing of the edge weights between the two networks. We rst introduce additional notation. • We denote the dierence between a pair of neurons as k;j =n 0 k;j n k;j . For example, 1;1 = 0:1 under the inputx 1 = 2;x 2 = 1 in our motivating example shown in Figure 4.2. • We denote the dierence in a pair of edge weights asW k [i;j] =W 0 k [i;j]W k [i;j]. For example, W 1 [1; 1] = 2:0 1:9 = 0:1. 69 • We extend the symbolic interval notation to these terms. That is,S in ( k;j ) denotes the interval that boundsn 0 k;j n k;j before applying ReLU, andS( k;j ) denotes the interval after applying ReLU. Given that we have computedS(n k1;i ),S(n 0 k1;i ),S( k1;i ) for every neuron in the layerk1, now, we compute a singleS( k;j ) in the subsequent layerk in two steps (and then repeat for each 1jl k ). First, we computeS in ( k;j ) by propagating the output intervals from the previous layer through the edges connecting to the target neuron. This is dened as S in ( k;j ) = X i S( k1;i )W 0 k [i;j] +S(n k1;i )W k1 [i;j] We illustrate this computation on node 1;1 in our example. First, we initializeS( 0;1 ) = [0; 0] ,S( 0;2 ) = [0; 0]. Then we compute S in ( 1;1 ) = [0; 0] 2:0+[x 1 ;x 1 ] 0:1+[0; 0]2:0+ [x 2 ;x 2 ]0:1 = [0:1x 1 0:1x 2 ; 0:1x 1 0:1x 2 ]. For the second step, we apply ReLU toS in ( k;j ) to obtainS( k;j ). This is where we apply the new convex approximations (Section 4.3.3) to obtain tighter bounds. Toward this end, we will focus on the following two equations: z 1 =ReLU(n k;j + k;j )ReLU(n k;j ) (4.1) z 2 =ReLU(n 0 k;j )ReLU(n 0 k;j k;j ) (4.2) While Paulsen et al. [81] also compute bounds of these two equations, they use concretizations instead of linear approximations, thus throwing away all the symbolic information. For the running example, their method would result in the bounds ofS( 1;1 ) = [:4;:4]. In contrast, our method will be able to maintain some or all of the symbolic information, thus improving the accuracy. 70 (xl) ul 0 ul +l 0 (xu) u 0 l ul +u 0 l u Figure 4.7: Illustration of Lemmas 1 and 2. 4.3.2 TwoUsefulLemmas Before presenting our new linear approximations, we introduce two useful lemmas, which will simplify our presentation as well as our soundness proofs. Lemma 1. Letx be a variable such thatl x u for constantsl 0 and 0 u. For a constantl 0 such thatll 0 0, we havex (xl) ul 0 ul +l 0 l 0 . Lemma 2. Letx be a variable such thatl x u for constantsl 0 and 0 u. For a constantu 0 such that 0u 0 u, we haveu 0 (xu) u 0 l ul +u 0 x. We illustrate these lemmas in Figure 4.7. The solid blue line shows the equationy = x for the input intervallxu. The upper dashed line illustrates the transformation of Lemma 1, and the lower dashed line illustrates Lemma 2. Specically, Lemma 1 shows a transformation applied tox whose result is always greater than bothl 0 andx. Similarly, Lemma 2 shows a transformation applied tox whose result is always less than bothu 0 andx. These lemmas will be useful in bounding Equations 4.1 and 4.2. 4.3.3 NewConvexApproximationsforS( k;j ) Now, we are ready to present our new approximations, which are linear symbolic expressions derived from Equations 4.1 and 4.2. 71 We rst assume thatn k;j andn 0 k;j could both be unstable, i.e., they could take values both greater than and less than 0. This yields bounds for the general case in that they are sound in all states ofn k;j andn 0 k;j (Sections 4.3.3.1 and 4.3.3.2). Then, we consider special cases ofn k;j andn 0 k;j , in which even tighter upper and lower bounds are derived (Section 4.3.3.3). To simplify notation, we let n;n 0 ; and stand in for n k;j ;n 0 k;j ; and k;j in the remainder of this section. 4.3.3.1 UpperBoundfortheGeneralCase Letl =UB(S in ()) andu =UB(S in ()). The upper bound approximation is: UB(S()) = 8 > > > > > > > > > < > > > > > > > > > : UB(S in ()) UB(S in ()) 0 0 UB(S in ()) 0 (UB(S in ())l) u ul otherwise That is, when the input’s (delta) upper bound is greater than 0 for all x2 X, we can use the input’s upper bound unchanged. When the upper bound is always less than 0, the new output’s upper bound is then 0. Otherwise, we apply a linear transformation to the upper bound, which results in the upper plane illustrated in Figure 4.5. We prove all three cases sound. Proof. We consider each case above separately. In the following, we use Equation 4.1 to derive the bounds, but we note a symmetric proof using Equation 4.2 exists and produces the same bounds. Case1:UB(S in ()) 0. We rst show that, according to Equation 4.1, when 0 we havez 1 . This then implies that, ifUB(S in ()) 0, thenz 1 UB(S in ())(x) for allx2 X, and hence it is a valid upper bound for the output interval. 72 Assume 0 . We consider two cases ofn. First, consider 0 n. Observe 0 n^ 0 =) 0 n + . Thus, the ReLU’s of Equation 4.1 simplify toz 1 = n + n = =) z 1 . When n< 0, Equation 4.1 simplies toz 1 =ReLU(n + ). Sincen< 0, we haven + ^ 0 =) ReLU(n + ) . Thus,z 1 =ReLU(n + ) , so the approximation is sound. Case2:UB(S in ()) 0. This case was previously proven [81], but we restate it here.UB(S in ()) 0 () n 0 n =) ReLU(n 0 )ReLU(n) () ReLU(n 0 )ReLU(n) 0. Case3. By case 1, anyUB(S()) that satisesUB(S())(x) 0 andUB(S())(x)UB(S in ())(x) for allx2 X is sound. Both inequalities hold by Lemma 1, withx = UB(S in ()), l = UB(S in ()), u =UB(S in ()) andl 0 = 0. We illustrate the upper bound computation on node n 1;1 of our motivating example. Recall that UB(S in (n 1;1 )) = 0:1x 1 0:1x 2 . SinceUB(S in (n 1;1 )) =0:4 andUB(S in (n 1;1 )) = 0:4, we are in the third case of our linear approximation above. Thus, we haveUB(S in (n 1;1 )) =(0:1x 1 0:1x 2 (0:4)) 0:4 0:4(0:4) =0:5x 1 0:5x 2 + 0:2. This is the upper bounding plane illustrated in Figure 4.5. The volume under this plane is 50% less than the upper bounding plane of ReluDiff shown in Figure 4.4. 4.3.3.2 LowerBoundfortheGeneralCase Letl =LB(S in ()) andu =LB(S in ()), the lower bound approximation is: LB(S()) = 8 > > > > > > > > > < > > > > > > > > > : LB(S in ()) LB(S in ()) 0 0 LB(S in ()) 0 (LB(S in ())u) l ul otherwise 73 That is, when the input lower bound is always less than 0, we can leave it unchanged. When it is always greater than 0, the new lower bound is then 0. Otherwise, we apply a transformation to the lower bound, which results in the lower plane illustrated in Figure 4.5. We prove all three cases sound. Proof. We consider each case above separately. In the following, we use Equation 4.1 to derive the bounds, but we note a symmetric proof using Equation 4.2 exists and produces the same bounds. Case1:LB(S in ()) 0. We rst show that according to Equation 4.1, when 0 we have z 1 . This then implies that, ifLB(S in ()) 0, we haveLB(S in ())(x)z 1 for allx2X, and hence it is a valid lower bound for the output interval. Assume 0. We consider two cases ofn + . First, let 0n + . Observe 0n + ^ 0 =) 0n, so we can simplify Equation 4.1 toz 1 =n + n = =) z 1 . Second, letn + < 0 () <n. Then, Equation 4.1 simplies toz 1 =ReLU(n) =max(0;n) =min(0;n). Now observe <n^ < 0 =) <min(0;n) =z 1 . Case2:LB(S in ()) 0. This case was previously proven sound [81], but we restate it here. LB(S in ()) 0 () n 0 n =) ReLU(n 0 )ReLU(n) () ReLU(n 0 )ReLU(n) 0. Case3. By case 1, anyLB(S()) that satisesLB(S())(x) 0 andLB(S())(x)LB(S in ())(x) for all x 2 X will be valid. Both inequalities hold by Lemma 2, with x = LB(S in ());u 0 = 0;l = LB(S in ()); andu =LB(S in ()). We illustrate the lower bound computation on node n 1;1 of our motivating example. Recall that LB(S in (n 1;1 )) = 0:1x 1 0:1x 2 . SinceLB(S in (n 1;1 )) =0:4 andLB(S in (n 1;1 )) = 0:4, we are in the third case of our linear approximation. Thus, we haveLB(S(n 1;1 )) = (0:1x 1 0:1x 2 (0:4)) (0:4) 0:4(0:4) = 74 Figure 4.8: Tighter upper bounding plane. Figure 4.9: Tighter lower bounding plane. 0:05x 1 0:05x 2 0:2. This is the lower bounding plane illustrated in Figure 4.5. The volume above this plane is 50% less than the lower bounding plane of ReluDiff shown in Figure 4.4. 4.3.3.3 TighterBoundsforSpecialCases While the bounds presented so far apply in all states ofn andn 0 , under certain conditions, we are able to tighten these bounds even further. Toward this end, we restate the following two lemmas proved by Paulsen et al. [81], which will come in handy. They are related to properties of Equations 4.1 and 4.2, respectively. Lemma3. ReLU(n + )nmax(n; ) Lemma4. n 0 ReLU(n 0 )min(n 0 ; ) These lemmas provide bounds whenn andn 0 are proved to be linear based on the absolute bounds that we compute. TighterUpperBoundWhenn 0 IsLinear. In this case, we haveUB(S()) =UB(S in ()), which is an improvement for the second or third case of our general upper bound. Proof. By our case assumption, Equation 4.2 simplies to the one in Lemma 4. Thus, z 2 =min(n 0 ; ) =) z 2 . 75 TighterUpperBoundWhennIsLinear,UB(S in ())LB(S in (n))UB(S in ()). We illustrate thez 1 plane under these constraints in Figure 4.8. Letl = UB(S in ()), and letu = UB(S in ()), and l 0 =LB(S in (n)), we use Lemma 1 to deriveUB(S()) = (UB(S in ())l) ul 0 ul +l 0 . This results in the upper plane of Figure 4.8. This improves over the third case in our general upper bound because it allows the lower bound ofUB(S()) to be less than 0. Proof. By our case assumption, Equation 4.1 simplies to the one in Lemma 3. By Lemma 1, we have for allx2X,UB(S())(x)LB(S in (n)) andUB(S())(x) UB(S in ())(x). These two inequalities implyUB(S())max(n; ). Tighter Lower Bound Whenn Is Linear. We can use the approximationLB(S()) = LB(S in ()). This improves over the second and third cases of our general lower bound. Proof. By our case assumption, Equation 4.1 simplies to the one in Lemma 3. Thus,z 1 = max(n; ) =) z 1 . Tighter Lower Bound whenn 0 is Linear,LB(S in ()) LB(S in (n 0 )) LB(S in ()). We illustrate thez 2 plane under these constraints in Figure 4.9. Here, lettingl = LB(S in ()),u = LB(S in ()), and u 0 = LB(S in (n 0 )), we can use Lemma 2 to derive the approximationLB(S()) = (LB(S in ())u) ul 0 ul +u 0 . This results in the lower plane of Figure 4.9. This improves over the third case, since it allows the upper bound to be greater than 0. Proof. By our case assumption, Equation 4.2 simplies to the one shown in Lemma 4. By Lemma 2, we have for allx2X,LB(S())(x)LB(S in ())(x) andLB(S())(x)LB(S in (n 0 )). These two inequalities implyLB(S())(x)min(n 0 ; ). 76 4.3.4 IntermediateSymbolicVariablesforS() While convex approximations reduce the error introduced by ReLU, even small errors tend to be ampli- ed signicantly after a few layers. To combat the error explosion, we introduce new symbolic terms to represent the output values of unstable neurons, which allow their accumulated errors to cancel out. We illustrate the impact of symbolic variables on n 1;1 of our motivating example. Recall we have S( 1;1 ) =[0:05x 1 0:05x 2 0:2; 0:05x 1 0:05x 2 + 0:2]. After applying the convex approximation, we introduce a new variablex 3 such thatx 3 = [0:05x 1 0:05x 2 0:2; 0:05x 1 0:05x 2 + 0:2]. Then we setS( 1;1 ) = [x 3 ;x 3 ], and propagate this interval as before. After propagating throughn 2;1 andn 2;2 and combining them atn 3;1 , thex 3 terms partially cancel out, resulting in the tighter nal output interval [1:65; 1:18]. In principle, symbolic variables may be introduced at any unstable neurons that introduce approxima- tion errors, however there are eciency vs. accuracy tradeos when introducing these symbolic variables. One consideration is how to deal with intermediate variables referencing other intermediate variables. For example, if we decide to introduce a variablex 4 forn 2;1 , thenx 4 will have anx 3 term in its equation. Then, when we are evaluating a symbolic bound that contains anx 4 term, which will be the case forn 3;1 , we will have to recursively substitute the bounds of the previous intermediate variables, such asx 3 . This be- comes expensive, especially when it is used together with our bisection-based renement [117, 81]. Thus, in practice, we rst remove any back-references to intermediate variables by substituting in their lower bounds and upper bounds into the new intermediate variable’s lower and upper bounds, respectively. Given that we do not allow back-references, there are two additional considerations. First, we must consider that introducing a new intermediate variable wipes out all the other intermediate variables. For example, introducing a new variable atn 2;1 wipes out references tox 3 , thus preventing anyx 3 terms from canceling atn 3;1 . Second, the runtime cost of introducing symbolic variables is not negligible. The bulk of computation time in NeuroDiff is spent multiplying the network’s weight matrices by the neuron’s 77 symbolic bound equations, which is implemented using matrix multiplication. Since adding variables increases the matrix size, this increases the matrix multiplication cost. Based on these considerations, we have developed heuristics for adding new variables judiciously. First, since the errors introduced by unstable neurons in the earliest layers are the most prone to explode, and hence benet the most when we create variables for them, we rank them higher when choosing where to add symbolic variables. Second, we bound the total number of symbolic variables that may be added, since our experience shows that introducing symbolic variables for the earliestN unstable neurons gives drastic improvements in both run time and accuracy. In practice,N is set to a number proportional to the weighted sum of unstable neurons in all layers. Formally,N = L k=1 k N k , whereN k is the number of unstable neurons in layerk and k = 1 k is the discount factor. 4.4 Experiments We have implementedNeuroDiff and compared it withReluDiff [81], the state-of-the-art tool for dier- ential verication of neural networks. NeuroDiff builds upon the codebase of ReluDiff [78], which was also used by single-network verication tools such as ReluVal [117] and Neurify [116]. All use Open- BLAS [126] to optimize the symbolic interval arithmetic (namely in applying the weight matrices to the symbolic intervals). We note that NeuroDiff uses the algorithm from Neurify to computeS(n k;j ) and S(n 0 k;j ), whereas ReluDiff uses the algorithm of ReluVal. Since Neurify is known to compute tighter bounds thanReluVal [116], we compare to bothReluDiff, and an upgraded version of ReluDiff which uses the bounds fromNeurify to ensure that any performance gain is due to our optimizations and not due to using Neurify’s bounds. We use the name ReluDiff+ to refer to ReluDiff upgraded with Neurify’s bounds. 78 4.4.1 Benchmarks Our benchmarks consist of the 49 feed-forward neural networks used by Paulsen et al. [81], taken from three applications: aircraft collision avoidance, image classication, and human activity recognition. We briey describe them here. As in Paulsen et al. [81], the second networkf 0 is generated by truncating the edge weights off from 32 bit to 16 bit oats. ACAS Xu [46] ACAS (aircraft collision avoidance system) Xu is a set of forty-ve neural networks, each with ve inputs, six hidden layers of 50 units each, and ve outputs, designed to advise a pilot (the ownship) how to steer an aircraft in the presence of an intruder aircraft. The inputs describe the position and speed of the intruder relative to the ownship, and the outputs represent scores for dierent actions that the ownship should take. The scores range from [0:5; 0:5]. We use the input regions dened by the properties of previous work [48, 117]. MNIST[62] MNIST is a standard image classication task, where the goal is to correctly classify 2828 pixel greyscale images of handwritten digits. Neural networks trained for this task take 784 inputs (one for each pixel) each in the range [0; 255], and compute ten outputs – one score for each of the ten possible digits. We use three networks of size 3x100 (three hidden layers of 100 neurons each), 2x512, and 4x1024 taken from Weng et al. [118] and Wang et al.[116]. All achieve at least 95% accuracy on holdout test data. Human Activity Recognition (HAR) [2] The goal for this task is to classify the current activity of human (e.g. walking, sitting, laying down) based on statistics from a smartphone’s gyroscopic sensors. Networks trained on this task take 561 statistics computed from the sensors and output six scores for six dierent activities. We use a 1x500 network. 79 4.4.2 ExperimentalSetup Our experiments aim to answer the following research questions: 1. IsNeuroDiff signicantly faster than state-of-the-art? 2. IsNeuroDiff’s forward pass signicantly more accurate? 3. CanNeuroDiff handle signicantly larger input regions? 4. How much does each technique contribute to the overall improvement? To answer these questions, we run bothNeuroDiff andReluDiff/ReluDiff+ on all benchmarks and compare their results. BothNeuroDiff andReluDiff/ReluDiff+ can be parallelized to use multithread- ing, so we congure a maximum of 12 threads for all experiments. Our experiments are run on a computer with an AMD Ryzen Threadripper 2950X 16-core processor, with a 30-minute timeout per dierential verication task. While we could try and adapt a single-network verication tool to our task as done previously [81], we note that ReluDiff has been shown to signicantly outperform (by several orders of magnitude) this naive approach. 4.4.3 Results In the remainder of this section, we present our experimental results in two steps. First, we present the overall verication results on all benchmarks. Then, we focus on the detailed verication results on the more dicult verication tasks. 4.4.3.1 SummaryofResultsonAllBenchmarks Our experimental results show that, on all benchmarks, the improvedReluDiff+ slightly but consistently outperforms the original ReluDiff due to its use of the more accurate component from Neurify instead 80 Figure 4.10: Comparing the execution times of NeuroDiff andReluDiff+ on all verication tasks. of ReluVal for bounding the absolute values of individual neurons. Thus, to save space, we will only show the results that compareNeuroDiff (our method) andReluDiff+. We summarize the comparison betweenNeuroDiff andReluDiff+ using a scatter plot in Figure 4.10, where each point represents a dierential verication task: the x-axis is the execution time of NeuroDiff in seconds, and the y-axis the execution time of ReluDiff+ in seconds. Thus, points on the diagonal line are ties, while points above the diagonal line are wins forNeuroDiff. The results show that NeuroDiff outperformed ReluDiff+ for most verication tasks. Since the execution time is in logrithmic scale the speedups of NeuroDiff are more than 1000X for many of these verication tasks. While there are cases whereNeuroDiff is slower thanReluDiff+, due to the overhead of adding symbolic variables, the dierences are on the order of seconds. Since they are all on the small MNIST networks and the HAR network that are very easy for both tools, we omit an in-depth analysis of them. In the remainder of this section, we present an in-depth analysis of the more dicult verication tasks. 81 Table 4.1: Results for ACAS networks with = 0:05. Property NeuroDiff (new) ReluDiff+ Speedup proved undet. time (s) proved undet. time (s) ' 1 45 0 522.6 44 1 4800.6 9.2 ' 3 42 0 2.3 42 0 4.1 1.8 ' 4 42 0 1.7 42 0 2.8 1.7 ' 5 1 0 0.2 1 0 0.2 1.4 ' 6 2 0 0.6 2 0 0.4 0.7 ' 7 1 0 1404.4 0 1 1800.0 1.3 ' 8 1 0 132.2 1 0 361.8 2.7 ' 9 1 0 0.6 1 0 2.3 3.7 ' 10 1 0 0.9 1 0 0.7 0.8 ' 11 1 0 0.2 1 0 0.3 1.6 ' 12 1 0 2.8 1 0 360.9 129.4 ' 13 1 0 5.8 1 0 5.1 0.9 ' 14 2 0 0.5 2 0 95.9 196.2 ' 15 2 0 0.6 2 0 65.0 113.2 4.4.3.2 ResultsonACASNetworks For ACAS networks, we consider two dierent sets of properties, namely the original properties from Paulsen et al. [81] where = 0:05, and the same properties but with = 0:01. We emphasize that, while verifying = 0:05 is useful, this means that the output value can vary by up to 10%. Considering = 0:01 means that the output value can vary by up to 2%, which is much more useful. Our results are shown in Tables 4.1 and 4.2, where the rst column shows the property, which denes the input space considered. The next three columns show the results for NeuroDiff, specically the number of veried networks (out of the 45 networks), the number of unveried networks, and the total run time across all networks. The next three show the same results, but forReluDiff+. The nal column shows the average speed up of NeuroDiff. The results show thatNeuroDiff makes signicant gains in both speed and accuracy. Specically, the speedups are up to two and three orders of magnitude for = 0:05 and 0:01, respectively. In addition, at the more accurate = 0:01 level, NeuroDiff is able to complete 53 more verication tasks, out of the total 142 verication tasks. 82 Table 4.2: Results for ACAS networks with = 0:01. Property NeuroDiff (new) ReluDiff+ Speedup proved undet. time (s) proved undet. time (s) ' 1 41 4 11400.1 15 30 55778.6 4.9 ' 3 42 0 14.3 35 7 13642.2 957.2 ' 4 42 0 3.8 37 5 9115.0 2390.1 ' 5 1 0 0.3 0 1 1800.0 5520.5 ' 16 2 0 1.0 2 0 0.8 0.8 ' 7 0 1 1800.0 0 1 1800.0 1.0 ' 8 1 0 1115.9 0 1 1800.0 1.6 ' 9 1 0 2.4 0 1 1800.0 738.2 ' 10 1 0 1.6 1 0 1.1 0.7 ' 11 1 0 0.3 0 1 1800.0 5673.8 ' 12 1 0 132.2 0 1 1800.0 13.6 ' 13 1 0 15.9 1 0 14.8 0.9 ' 14 2 0 1589.3 0 2 3600.0 2.3 ' 15 2 0 579.4 0 2 3600.0 6.2 4.4.3.3 ResultsonMNISTNetworks For MNIST, we focus on the 4x1024 network, which is the largest network considered by Paulsen et al. [81]. In contrast, since the smaller networks, namely 3x100 and 2x512 networks, were handled easily by both tools, we omit their results. In the MNIST-related verication tasks, the goal is to verify = 1 for the given input region. We consider the two types of input regions from the previous work, namely global perturbations and targeted pixel perturbations, however we use input regions that are hundreds of orders of magnitude larger. First, we look at the global perturbation. For these, the input space is created by taking an input image and then allowing a perturbation of +/- p greyscale units to all of its pixels. In the previous work, the largest perturbation wasp = 3. Figure 4.11 compares NeuroDiff and ReluDiff+ onp = 3 all the way up to 8, where the x-axis is the perturbation applied, and the y-axis is the percentage of verication tasks (out of 100) that each can handle. The results show that NeuroDiff can handle perturbations up to +/- 6 units, whereas ReluDiff+ begins to struggle at 4. While the dierence between 4 and 6, may seem small, the volume of input space 83 Figure 4.11: Percentage of verication tasks completed on the MNIST 4x1024 network for various perturbations. Figure 4.12: Accuracy comparison for a single forward pass on the MNIST 4x1024 network with perturbation of 8. Table 4.3: Results of the MNIST 4x1024 pixel experiment. Num. Pixels NeuroDiff (new) ReluDiff+ Speedup proved undet. time (s) proved undet. time (s) 15 100 0 236.5 100 0 1610.2 6.8 18 100 0 540.8 88 12 34505.8 63.8 21 100 0 1004.0 30 70 145064.5 144.5 24 99 1 7860.1 1 99 179715.9 22.9 27 83 17 49824.0 0 100 180000.0 3.6 for a perturbation of 6 is 6 784 =4 784 1:1 10 138 times larger than 4, or in other words, 138 orders of magnitude larger. Next, we show a comparison of the epsilon veried by a single forward pass for a perturbation of 8 on the MNIST 4x1024 network in Figure 4.12. Points above the blue line indicate NeuroDiff performed better. Overall,NeuroDiff is between two and three times more accurate thanReluDiff+. Finally, we look at the targeted pixel perturbation properties. For these, the input space is created by taking an image, randomly choosingn pixels, and setting there bounds to [0; 255], i.e., allowing arbitrary changes to the chosen pixels. We again use the 4x1024 MNIST network. The results are summarized in Table 4.3. The rst column shows the number of randomly perturbed pixels. We can again see very large speedups, and a signicant increase in the size of the input region thatNeuroDiff can handle. 84 Table 4.4: Evaluating the individual contributions of convex approximation and symbolic variables using the MNIST 4x1024 global perturbation experiment. Perturb Average Veried ReluDiff+ Conv. Approx. Int. Vars. NeuroDiff 3 0.59 0.42 (+1.39x) 0.43 (+1.38x) 0.20 (+2.93x) 4 1.02 0.70 (+1.46x) 0.87 (+1.18x) 0.36 (+2.85x) 5 1.60 1.06 (+1.52x) 1.47 (+1.09x) 0.56 (+2.87x) 6 2.29 1.47 (+1.55x) 2.19 (+1.04x) 0.79 (+2.90x) 7 3.02 1.92 (+1.58x) 2.96 (+1.02x) 1.04 (+2.91x) 8 3.80 2.39 (+1.59x) 3.77 (+1.01x) 1.30 (+2.93x) 4.4.3.4 ContributionofEachTechnique Here, we analyze the contribution of individual techniques, namely convex approximations and symbolic variables, to the overall performance improvement. In Table 4.4, we present the average that was able to be veried after a single forward pass on the 4x1024 MNIST network for each of the four techniques: ReluDiff+ (baseline), NeuroDiff with only convex approximations,NeuroDiff with only intermediate variables, and the fullNeuroDiff. Overall, the individual benets of the two proposed approximation techniques are obvious. While con- vex approximation (alone) consistently provides benet as perturbation increases, the benet of symbolic variables (alone) tends to decrease. In addition, combining the two provides much greater benet than the sum of their individual contributions. With perturbation of 8, for example, convex approximations alone are 1.59 times more accurate thanReluDiff+, and intermediate variables alone are 1.01 times more accurate. However, together they are 2.93 times more accurate. The results suggest two things. First, intermediate symbolic variables perform well when a signicant portion of the network is already in the stable state. We conrm, by manually inspecting the experimental results, that it is indeed the case when we use a perturbation of 3 and 8 in the MNIST experiments. Second, the convex approximations provide the most benet when the pre-ReLU delta intervals are (1) signicantly wide, and (2) still contain a signicant amount of symbolic information. This is also conrmed by manually inspecting our MNIST results: increasing the perturbation increases the overall width of the delta intervals. 85 4.5 Summary We have presented NeuroDiff, a scalable dierential verication technique for soundly bounding the dierence between two feed-forward neural networks. NeuroDiff leverages novel convex approxima- tions, which reduce the overall approximation error, and intermediate symbolic variables, which control the error explosion, to signicantly improve eciency and accuracy of the analysis. Our experimental evaluation shows thatNeuroDiff can achieve up to 1000X speedup and is up to ve times as accurate. 86 Chapter5 OnlineSynthesisofLinearApproximations Prior work has shown that neural networks are vulnerable to various types of (adversarial) perturbations, such as smalll-norm bounded perturbations [103], geometric transformations [25, 47], and word substi- tutions [1]. Such perturbations can often cause a misclassication for any given input, which may have serious consequences, especially in safety critical systems. Certifying robustness to these perturbations has become an important problem as it can show the network does not exhibit these misclassications, and furthermore previous work has shown that a given input feature’s certied robustness can be a useful indicator to determine the feature’s importance in the network’s decision [95, 55]. Indeed, many approaches have been proposed for certifying the robustness of inputs to these pertur- bations. Previous work typically leverages two types of techniques: (1) fast and scalable, but approximate techniques [98, 31, 118, 95, 55], and (2) expensive but exact techniques that leverage some type of constraint solver [48, 49, 108]. Several works have also combined the two [99, 96, 116, 110]. The most successful ap- proaches, in terms of scalability in practice, are built on top of the approximate techniques, which all depend on computing linear bounds for the non-linear activation functions. However, a key limitation is that the linear bounds must be handcrafted and proven sound by ex- perts. Not only is this process dicult, but also ensuring the tightness of the crafted bounds presents 87 an additional challenge. Unfortunately, prior work has only crafted bounds for the most common activa- tion functions and architectures, namely ReLU [116], sigmoid, tanh [98, 125, 119], the exp function [95], and some 2-dimensional activations found in LSTM networks [55]. As a result, existing tools for neu- ral network verication cannot handle a large number of activation functions that are frequently used in practice. Examples include theGELU function [42], which is currently the activation function used in Ope- nAI’s GPT [86], and the Swish function which has been shown to outperform the standard ReLU function in some applications [88] and, in particular, can reduce over-tting in adversarial training [100]. In ad- dition, these recently introduced activation functions are often signicantly more complex than previous activation functions, e.g., we have gelu(x) = 0:5x(1 + tanh [ p 2=(x + 0:044715x 3 )]). In this work, we study the problem ofeciently andautomatically synthesizingsound andtight linear bounds for any arbitrary activation function. By arbitrary activation function, we mean any (non-linear) computable functionz = (x 1 ;:::;x d ) used inside a neural network withd input variables. By sound we mean, given an interval bound on each variable x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;x d 2 [l d ;u d ], the problem is to eciently compute lower bound coecientsc l 1 ;c l 2 ;:::;c l d+1 , and upper bound coecients c u 1 ;c u 2 ;:::;c u d+1 such that the following holds: 8x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;x d 2 [l d ;u d ] c l 1 x 1 +c l 2 x 2 + +c l d+1 (x 1 ;:::;x d )c u 1 x 1 +c u 2 x 2 + +c u d+1 (5.1) Byautomatically, we mean that the above is done using only the denition of the activation function itself. Finally, bytight, we mean that some formal measure, such as the volume above/below the linear bound, is minimized/maximized. We have developed a new method, named LinSyn, that can automatically synthesize tight linear bounds for any arbitrary non-linear activation function (). We illustrate the ow of our method on the left-hand side of Fig. 5.1. As shown, LinSyn takes two inputs: a denition of the activation function, 88 Input Region Analyzer AutoLiRPA, ERAN, etc) Over-approx. of Output Region Synthesize Candidate Coefficients Verify Adjust Fail Proved LinSyn LinSyn Inputs LinSyn Outputs Bound Mods. ReLU Tanh . . . Figure 5.1: The overall ow of LinSyn. and an interval for each of its inputs. LinSyn outputs linear coecients such that Equation 5.1 holds. In- ternally,LinSyn uses sampling and an LP (linear programming) solver to synthesize candidate lower and upper bound coecients. Next, it uses an ecient local minimizer to compute a good estimate of the oset needed to ensure soundness of the linear bounds. Since the candidate bounding functions constructed in this manner may still be unsound, nally, we use a highly optimized branch-and-bound nonlinear SMT solver, named dReal [30], to verify the soundness of the linear bounds. Even though our new method involves the use of solvers and optimizers, the entire process typically takes less than 1/100th of a second per pair of bounds. Fig. 5.1 also illustrates howLinSyn ts in with existing neural network verication frameworks, such as ERAN [26], andAutoLiRPA [122]. These tools take as input a neural network, and a region of the neural networks input space, and compute an over-approximation of the neural network’s outputs. Internally, these frameworks have modules that compute linear bounds for a specic activation functions.LinSyn is a one-size-ts-all drop-in replacement for these modules that are invoked at runtime whenever a linear bound of a non-linear activation function is needed. 89 Our method diers from these existing frameworks because a user (usually an expert in neural net- work verication) must provide hand-crafted, sound linear bounds for the activation functions of a neural network. However, to date, they only support the previously mentioned activation functions. We note however that the recent frameworkAutoLiRPA supports binary operations (namely addition, subtraction, multiplication, and division) as “activation functions”. Thus, while it’s not explicitly designed to handle complex activations, it has the ability to by decomposing, e.g., gelu(x) into operations that it supports, and then combining them. In contrast, LinSyn bounds the activation function as a whole, which we will show produces much tighter linear bounds. We have implemented our method in tool called LinSyn, and evaluated it on benchmarks in com- puter vision and natural language processing (NLP). Our evaluation shows that we can obtain nal output bounds often 2-5X tighter than the most general tool [122], thus allowing us to drastically increase certied robustness. In addition, our tool achieves accuracy equal to or better than the handcrafted LSTM bounds of Popqorn [55], which is currently the most accurate tool for analyzing LSTM-based NLP models, at a comparable runtime. To summarize, this paper makes the following contributions: • We propose the rst method for automatically synthesizing tight linear bounds for arbitrary activa- tion functions. • We implement our approach in a tool called LinSyn, and integrate it as a bounding module into theAutoLiRPA framework, thus producing a neural network verication tool that can theoretically compute tight linear bounds for any arbitrary activation function. • We extensively evaluate our approach and show it outperforms state-of-the-art tools in terms of accuracy and certied robustness by a large margin. 90 The rest of this chapter is organized as follows. First, we provide the technical background in Sec- tion 5.1. Then, we present our method for synthesizing the linear bounds in Section 5.2 and our method for verifying the linear bounds in Section 5.3. Next, we present the experimental results in Section 5.4. Finally, we summarize the contributions of this chapter in Section 5.5. 5.1 Preliminaries In this section, we dene the neural network verication problem, and illustrate both how state-of-the-art verication techniques work, and their limitations. 5.1.1 NeuralNetworks Following conventional notation, we refer to matrices with capital bold letters (e.g.W2R nm ), vectors as lower case bold letters (e.g. x2 R n ), and scalars or variables with lower case letters (e.g. x2 R). Slightly deviating from the convention, we refer to a set of elements with capital letters (e.g.XR n ). We consider two types of networks in our work: feed-forward and recurrent. We consider a feed- forward neural network to be a (highly) non-linear functionf :X!Y, whereXR n andYR m . We focus on neural networkclassiers. For an inputx2X, each element in the outputf(x) represents a score for a particular class, and the class associated with the largest element is the chosen class. For example, in image classication,X would be the set of all images, each element of an inputx2X represents a pixel’s value, and each element inY is associated with a particular object that the image might contain. In feed-forward neural networks the outputf(x) is computed by performing a series of ane trans- formations, i.e., multiplying by a weight matrix, followed by application of an activation function(). Formally, a neural network withl layers hasl two-dimensional weight matrices andl one-dimensional bias vectorsW i ;b i ; wherei2 1::l, and thus we havef(x) =W l (W l1 (W 1 x+b 1 )+b l1 )+b l , 91 where() is the activation function applied element-wise to the input vector. The default choice of acti- vation is typically the sigmoid(x) = 1=(1 +e x ), tanh , or ReLU function(x) =max(0;x), however recent work [42, 88, 86] has shown that functions such asgelu(x) and swish(x) = x sigmoid(x) can have better performance and desirable theoretical properties. Unlike feed-forward neural networks, recurrent neural networks receive a sequence of inputs [x (1) ; :::; x (t) ], and the nal output of f on x t is used to perform the classication of the whole sequence. Recurrent neural networks are state-ful, meaning they maintain a state vector that contains information about inputs previously given tof, which also gets updated on each call tof. In particular, we focus onlong short-termmemory (LSTM) networks, which have seen wide adoption in natural language processing (NLP) tasks due to their sequential nature. For LSTMs trained for NLP tasks, the network receives a sequence of word embeddings. A word embedding is ann-dimensional vector that is associated with a particular word in a (natural) language. The distance between word embeddings carries semantic signicance – two word embeddings that are close to each other inR n typically have similar meanings or carry a semantic relatedness (e.g. dog and cat or king and queen), whereas unrelated words typically are farther apart. LSTM networks further dier from feed-forward networks in that their internal activation functions are two-dimensional. Specically, we have the following two activation patterns: 1 (x) 2 (y) andx 1 (y). The default choices are 1 (x) = sigmoid(x), and 2 (x) = tanh(x). However, we can swap 1 with any function with output range bounded by [0; 1], and swap 2 with any function with output range bounded by [1; 1]. Indeed, prior work [33] has shown that 1 (x) = 1e e x can achieve better results in some applications. 5.1.2 NeuralNetworkVerication A large number of problems in neural network verication can be phrased as the following: given an input regionXX, compute an over-approximationY , such thatff(x)jx2XgY Y. TypicallyX and 92 Figure 5.2: Example of neural network verication. Figure 5.3: Linear bounds for ReLU activation. Y are hyper-boxes represented by an interval for each of their elements. A common problem is to prove that a pointx2X isrobust, meaning that small perturbations will not cause an incorrect classication. In this case,X is the set of all perturbed versions ofx, and to prove robustness, we check that the element of the correct class inY has a lower bound that is greater than the upper bound of all other elements. We illustrate a simple verication problem on the neural network shown in Fig. 5.2. The network has two inputs,x 1 ;x 2 , and two outputsx 7 ;x 8 which represent scores for two dierent classes. We refer to the remaining hidden neurons asx i ;i2 3::6. Following prior work [98], we break the ane transformation and application of the activation function into two separate neurons, and the neurons are assumed to be ordered such that, ifx i is in a layer beforex j , theni < j. For simplicity, in this motivating example, we let(x) =max(0;x) (the ReLU function). We are interested in proving that the regionx 1 2 [1; 1];x 2 2 [1; 1] always maps to the rst class, or in other words, we want to show that the lower bound ofx 7 is greater than the upper boundx 8 . 5.1.3 ExistingMethods The most scalable approaches (to date) for neural network verication are based on linear bounding and back-substitution [122], also referred to as abstract interpretation in the polyhedral abstract domain [98] or symbolic interval analysis [116] in prior work. 93 For each neuronx j in the network, these approaches compute a concrete lower and upper boundl j ;u j , and a linear lower and upper bound in terms of the previous layer’s neurons. The linear bounds (regardless of the choice of()) have the following form: P j1 i=0 x i c l i +c l j x j P j1 i=0 x i c u i +c u j . The bounds are computed in a forward, layer-by-layer fashion which guarantees that any referenced neurons will already have a bound computed when back-substitution is performed. To obtain the concrete bounds l j ;u j for a neuron x j , the bounds of any non-input neurons are re- cursively substituted into the linear bounds of x j until only input nodes x 1 ;:::;x n remain. Finally, the concrete input intervals are substituted into the bound to obtainl j ;u j . Example We illustrate on the two-layer network in Fig. 5.2 for the previously dened property. We trivially havel 1 = l 2 =1,u 1 = u 2 = 1,1 x 1 1, and1 x 2 1. We then compute linear bounds forx 3 ;x 4 in terms of previous layer’s neuronsx 1 ;x 2 . We multiplyx 1 ;x 2 by the edge weights, obtainingx 1 +x 2 as the lower and upper bound for both ofx 3 andx 4 . Since this bound is already in terms of the input variables, we substitute the concrete bounds into this equation and obtainl 3 =l 4 =2 andu 3 =u 4 = 2. Next, we need to compute the linear bounds for x 5 = (x 3 ) and x 6 = (x 4 ) after applying the activation function. Solving this challenge has been the focus of many prior works. There are two require- ments. First, they need to be sound. For example, forx 5 we need to nd coecientsc l 1 ;c l 2 ;c u 1 ;c u 2 such thatc l 1 x 3 +c l 2 (x 3 ) c u 1 x 3 +c u 2 for allx 3 2 [l 3 ;u 3 ], and similarly forx 6 . Second, we want them to be tight. Generally, this means that volume below the upper bound is minimized, and volume below the lower bound is maximized. 94 As an example, prior work [98, 125] has proposed the following sound and tight bound for(x) = max(0;x): 8x i 2 [l i ;u i ]: u i u i l i x i + l i u i u i l i (x i ) 8 > > > > < > > > > : 0 l i u i x i l i u 8 to verify the property, which fails in this case. 5.1.4 LimitationsofExistingMethods Current approaches only support a limited number of activation functions, and designing linear bounds for new activation functions often requires a signicant amount of eort even for a domain expert. For example, handcrafted sound and tight linear bounds for activation functions such as ReLU, sigmoid, and tanh [98, 118, 125, 119, 117, 116], convolution layers and pooling operations [12], the two-dimensional activations found in LSTMs [55, 92], and those in transformer networks [95] are worthy of publication. Furthermore, even bounds that are hand-crafted by experts are not always tight. For example, a recent work [119] was able to nearly triple the precision of previous state-of-the-art sigmoid and tanh linear bounds simply by improving tightness. To the best of our knowledge, AutoLiRPA [122] is the only tool that has the ability to handle more complex activation functions, though it was not originally designed for this. It can do so by decomposing them into simpler operations, and then composing the bounds together. We illustrate withswish(x) =x sigmoid(x), wherex2 [1:5; 5:5].AutoLiRPA would rst boundsigmoid(x) over the region [1:5; 5:5], resulting in the bound:11x +:35sigmoid(x):22x +:51. For the left-hand side of the function, we trivially havexxx. AutoLiRPA would then bound a multiplicationyz, where in this casey =x andz =sigmoid(x), resulting in the nal bound:15x:495xsigmoid(x) 0:825x +:96. We 95 illustrate this bound in Fig. 5.4, and we provide bounds computed byLinSyn as a comparison point.LinSyn provides a slightly better upper bound, and a signicantly better lower bound. The reason for the looseness is because when AutoLiRPA boundssigmoid(x), it necessarily accumulates some approximation error because it is approximating the behavior of a non-linear function with linear bounds. The approximation error eectively “loses some information” about about its input variable x. Then, when bounding the multiplication operation, it has partially lost the information thaty andz are related (i.e. they are both derived fromx). In contrast,LinSyn overcomes this issue by consideringswish(x) as a whole. We explain how in the following sections. 5.2 SynthesizingtheCandidateLinearBounds In this section, we describe our method for synthesizing candidate, possibly unsound linear bounds. 5.2.1 ProblemStatementandChallenges We assume we are given a d-dimensional activation function z = (x 1 ;:::;x d ), and an input interval x i 2 [l i ;u i ] for eachi2f1::dg. Our goal is to synthesize linear coecientsc l i ;c u i , wherei2f1::d + 1g that are sound, meaning that the following condition holds: 8x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;x d 2 [l d ;u d ] c l 1 x 1 +c l 2 x 2 + +c l d+1 (x 1 ;x 2 ;::: )c u 1 x 1 +c u 2 x 2 + +c u d+1 (5.2) In addition, we want to ensure that the bounds aretight. The ideal denition of tightness would choose linear bounds that maximize the precision of the overall analysis, for example minimizing the width of the output neuron’s intervals. Unfortunately, such a measure would involve all of the neurons of the network, and so is impractical to compute. Instead, the common practice is to settle for tightness that’s local to the specic neuron we are bounding. 96 -4 -2 2 4 6 0 2 4 6 AutoLiRPA Ours Figure 5.4: Bounds computed by LinSyn and Au- toLiRPA forswish(x),x2 [1:5; 5:5]. min shift Z 3:5 1 c u 1 x 1 +c u 2 dx 4 2 0 2 4 0 0:2 0:4 0:6 0:8 1 x 1 Figure 5.5: Candidate plane synthesis. Informally, we say a bound is tight if the volume below the upper bound is minimized, and volume below the lower bound is maximized. Prior work [125, 98, 55] has found this to be a good heuristic ∗ . Formally, volume is dened as the following integral: R u 1 l 1 R u d l d P d i=1 c u i x i +c u d+1 dx 1 :::dx d which, for the upper bound, should be minimized subject to Equation 5.2. This integral has the following closed-form solution: d X i=0 2 4 1 2 c i d Y j=0 u 1+1 i=j i l 1+1 i=j i 3 5 +c d+1 d Y i=0 (u i l i ) (5.3) where1 i=j is the (pseudo Boolean) indicator function that returns 1 when its predicate is true. We omit the proof, but note that the above expression can be derived inductively ond. Also note that, since each l i ;u i are concrete, the above expression is linear in terms of the coecients, which will be advantageous in our approach below. While recent approaches in solving non-linear optimization problems [56, 16] could directly minimize Equation 5.3 subject to Equation 5.2 in one step, we nd the runtime to be very slow. Instead, we adopt ∗ We also experimented with minimizing the volume between the linear bound and the activation function, which gave almost identical results. 97 a two-step approach that rst uses ecient procedures for computing candidate coecients that are al- most sound (explained in this section), and second, only calls an SMT solver when necessary to verify Equation 5.2 (explained in the next section). We illustrate the approach on a concrete example. 5.2.2 SynthesizingCandidateBounds The rst step in our approach computes candidate coecients for the linear bound. In this step we focus on satisfying the tightness requirement, while making a best eort for soundness. We draw inspiration from prior work [92, 4] that leverages sampling to estimate the curvature of a particular function, and then uses a linear programming (LP) solver to compute a plane that is sound. However, unlike prior work which targeted a xed function, we target arbitrary (activation) functions, and thus these are special cases of our approach. The constraints of the LP are determined by a set of sample pointsS R d . For the upper bound, we minimize Equation 5.3, subject to the constraint that the linear bound is above() at the points inS. Usings i to refer to thei th element of the vectors2S, the linear program we solve is: minimize Equation (5:3) subject to ^ s2S c 1 s 1 +c 2 s 2 + +c d+1 (s) (5.4) We generateS by sampling uniformly-spaced points over the input intervals. Example We demonstrate our approach on the running example illustrated in Fig. 5.5. For the example, let(x 1 ) = 1 1+e x 1 (the sigmoid function, shown as the blue curve), wherex 1 2 [1; 3:5]. We focus only on the upper bound, but the lower bound is computed analogously. Plugging in the variables into Equation 5.3, the objective of the LP that we minimize is: Z 3:5 1 c u 1 x 1 + c u 2 dx 1 = 6:625c u 1 + 4:5c u 2 which is shown as the shaded region in Fig. 5.5. 98 We sample the pointsS =f1; 0:25; 1:5; 2:75g, resulting in the following four constraints:c 1 +c 2 (1)^ 0:25c 1 +c 2 (0:25)^ 1:5c 1 +c s (1:5)^ 2:75c 1 +c 2 (2:75). Solving the LP program results inc 1 = 0:104;c 2 = 0:649, which is illustrated by the green line in Fig. 5.5. 5.3 MakingtheBoundSound In this section, we present our method for obtaining soundness because the candidate bounds synthesized in the previous section may not be sound. Here, we focus only on making the upper bound sound, but note the procedure for the lower bound is similar. 5.3.1 ProblemStatementandChallenges We are given the activation function(), the input intervalsx i 2 [l i ;u i ], and the candidate coecients c 1 ;c 2 ;:::;c d+1 . The goal is to compute an upward shift, if needed, to make the upper bound sound. First, we dene the violation of the upper bound as: v(x 1 ;x 2 ;:::;x d ) :=c u 1 x 1 +c u 2 x 2 + +c u d+1 (x 1 ;x 2 ;:::;x d ) (5.5) A negative value indicates the upper bound is not sound. We then need to compute a lower bound onv(), which we termv l . Then the equation we pass to the verier is: 8x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;x d 2 [l d ;u d ] v(x 1 ;x 2 ;:::;x d ) + (v l ) 0 (5.6) Expandingv() with its denition in the above equation results in the soundness denition of Equation 5.2. Thus, if the verier proves Equation 5.6, then shifting the upper bound upward byv l ensures its sound- ness. For our running example, the quantityv l is shown by the red line in Fig. 5.5. 99 This problem is non-trivial because nding a solution forv l requires a search for a sound global mini- mum/maximum of a function involving(), which may be highly non-linear. State-of-the-art SMT solvers such as Z3 do not support all non-linear operations, and furthermore, since we assume arbitrary(), the problem may even be (computationally) undecidable. 5.3.2 VerifyingtheBound We rst assume we have a candidate (possibly unsound)v l , and explain our verication method. To ensure decidability and tractability, we leverage the-decision procedure implemented bydReal [30]. To the best of our knowledge this is is the only framework that is decidable for all computable functions. In this context, instead of verifying Equation 5.6, the formula is rst negated thus changing it into an existentially quantied one, and then applying a-relaxation. Formally, the formula dReal attempts to solve is: 9x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;x d 2 [l d ;u d ] v(x 1 ;x 2 ;::: ) + (v l ) (5.7) where is a small constant (e.g. 10 5 ), which we explain in a moment. The above is formulated such that Equation 5.6 holds if (but not only if) there does not exist a solution to Equation 5.7. Internally,dReal performs interval constraint propagation (ICP) on the left-hand side of Equation 5.7 over the intervals dened by each [l i ;u i ] to compute an upper bound, and compares this upper bound with . If the upper bound is less than, then no solution exists (i.e., Equation 5.7 is unsatisable, and we have proven the original Equation 5.6 holds). Otherwise a solution may exist. In this case, dReal iteratively partitions the input space dened by the [l i ;u i ] and repeats this process on each partition separately. dReal stops partitioning either when it proves all partitions do not have solutions , or when a partition whose intervals all have width less than some is found. Here, is proportional to (i.e., smaller means smaller). In the latter case,dReal returns this partition as a “solution”. 100 While Equation 5.6 holds if there does not exist a solution to Equation 5.7, the converse does not hold true both because of the error inherent in ICP, and because we “relaxed” the right-hand side of Equation 5.7. This means that controls the precision of the analysis. controls both the size of the false solution space, and determines how many times we will sub-divide the input space before giving up on proving Equation 5.7 to be unsatisable. Practically, this has two implications for our approach. The rst one is that our approach naturally inherits a degree of looseness in the linear bounds dened by . Specically, we must shift our plane upward by in addition to the truev l , so that dReal can verify the bound. The second is that we have to make a trade-o between computation and precision. While smaller will allow us to verify a tighter bound, it generally will also mean a longer verication time. In our experiments, we nd that = 10 7 gives tight bounds at an acceptable runtime, though we may be able to achieve a shorter runtime with a larger. 5.3.3 Computingv l Now that we have dened how we can verify a candidate bound, we explain our approach for computing v l . The implementation is outlined in Algorithm 4. Since failed calls to the verier can be expensive, at lines 1-2, we rst use a relatively cheap (and unsound) local optimization procedure to estimate the truev l . While local optimization may get stuck in local minima, neural network activation functions typically do not have many local minima, so neither willv(). We use L-BFGS-B [15], the bounded version of L-BFGS, to perform the optimization. At a high-level, L-BFGS-B takes as inputv(), the input boundsx i 2 [l i ;u i ], and an initial guess g2 R d at the location of the local minimum. It then uses the Jacobian matrix (i.e., derivatives) ofv() to iteratively move towards the local minimum (the Jacobian can be estimated using the nite dierences method or provided explicitly – we use Mathematica [45] to obtain it). We nd that sampling points uniformly inv() can usually nd a goodg, and thus L-BFGS-B often converges in a small 101 number of iterations. L-BFGS-B typically produces an estimate within 10 8 of the true value. To account for estimation error we add an additional 10 6 , plus 2 to account for the-relaxation (line 3). Finally, we iteratively decreasev l by a small amount (10 6 ) untildReal veries it (lines 4-9). Going back to our motivating example, we would estimate v l with a local minimizer, and then use dReal to verify the following: 8x 1 2 [1; 3:5]:(x 1 )c u 1 x 1 +c u 2 + (v l ) + 2 + 10 6 If verication fails, we iteratively decrease the value of v l by 10 6 , and call dReal until the bound is veried. The nal value ofc u 1 x 1 +c u 2 + (v l ) + 2 + 10 6 is the nal sound upper bound. 5.3.4 OntheCorrectnessandGeneralityof LinSyn The full LinSyn procedure is shown in Algorithm 5. The correctness (i.e. soundness) of the synthesized bounds is guaranteed if thev l returned by Algorithm 4 is a true lower bound onv(). Since Algorithm 4 does not return untildReal veriesv l at line 6, the correctness is guaranteed. Both our procedure in Section 5.2 and L-BFGS-B require only black-box access to(), so the only po- tential limit to the arbitrariness of our approach lies in what elementary operations are supported bydReal. During our investigation, we did not nd activations that use operations unsupported bydReal, however if an unsupported operation is encountered, one would only need to dene an interval extension [71] for the operation, which can be done for any computable function. 5.4 Evaluation We have implemented our method in a module calledLinSyn, and integrated it into theAutoLiRPA neu- ral network verication framework [122]. A user instantiates LinSyn with a denition of an activation 102 Input: Activation(x 1 ;x 2 ;::: ), Candidate Coecientsc u 1 ;c u 2 ;:::;c u d+1 , Input Bounds x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];::: , Jacobianrv (optional) Output: Lower Bound on Violationv l g sample points onv(x 1 ;x 2 ;::: ) and take minimum; v l L-BFGS-B(v(x 1 ;x 2 ;::: );x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;g;rv) ; v l v l 10 6 2; whileTruedo // Call dReal if Equation 5.2 holdsthen returnv l ; end v l v l 10 6 ; end Algorithm4: BoundViolation Input: Activation(x 1 ;x 2 ;::: ), Input Boundsx 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];::: , Jacobianrv (optional) Output: Sound Coecientsc u 1 ;c u 2 ;:::;c u d+1 c u 1 ;c u 2 ;:::;c u d+1 Sampling and LP procedure on(x) over Input Bounds; v l BoundViolation(c u 1 ;c u 2 ;:::;c u d+1 ;x 1 2 [l 1 ;u 1 ];x 2 2 [l 2 ;u 2 ];:::;rv); c u d+1 c u d+1 + (v l ); returnc u 1 ;c u 2 ;:::;c u d+1 ; Algorithm5: SynthesizeUpperBoundCoecients function, which results in an executable software module capable of computing the sound linear lower and upper bounds for the activation function over a given input region. LinSyn uses Gurobi [38] to solve the LP problem described in Section 5.2, anddReal [30] as the verier described in 5.3. In total,LinSyn is implemented in about 1200 lines of Python code. 5.4.1 Benchmarks NeuralNetworks Our benchmarks are nine deep neural networks trained on the three dierent datasets shown below. In the following, a neuron is a node in the neural network where a linear bound must be computed, and thus the neuron counts indicate the number of calls toLinSyn that must be made. • MNIST: MNIST is a dataset of hand-written integers labeled with the corresponding integer in the image. The images have 28x28 pixels, with each pixel taking a gray-scale value between 0 to 255. We trained three variants of a 4-layer CNN (convolutional neural network). Each takes as input a 28x28 103 = 784-dimensional input vector and outputs 10 scores, one for each class. In total, each network has 2,608 neurons – 1568, 784, and 256 in the rst, second, and third layers, respectively. • CIFAR: CIFAR is a dataset of RGB images from 10 dierent classes. The images have 32x32 pixels, with each pixel having an R, G, and B value in the range 0 to 255. We trained three variants of a 5-layer CNN. Each takes a 32x32x3 = 3072-dimensional input vector and outputs 10 scores, one for each class. In total, each network has 5376 neurons, 2048, 2048, 1024, and 256 neurons in the rst, second, third, and fourth layers, respectively. • SST-2: The Stanford Sentiment Treebank (SST) dataset consists of sentences taken from movie re- views that are human annotated with either positive or negative, indicating the sentiment expressed in the sentence. We trained three dierent variants of the standard LSTM architecture. These net- works take as input a sequence 64-dimensional word embeddings and output 2 scores, one for pos- itive and one for negative. Each network has a hidden size of 64, which works out to 384 neurons per input in the input sequence. 0:5x(1 + tanh ( p 2=(x + 0:044715x 3 ))) (GeLU) min(1;max(x;1)) (Hard Tanh) 1e e x (Log-Log) x(x) (Swish) Figure 5.6: Nonlinear activation functions. ActivationFunctions We experimented with the four activation functions as shown in Fig. 5.6. GELU and Swish were recently proposed alternatives to the standard ReLU function due to their desirable theo- retical properties [42] such as reduced overtting [100], and they have seen use in OpenAI’s GPT [86] and 104 very deep feed forward networks [88]. Similarly, Hard-Tanh is an optimized version of the common tanh function, while the Log-Log function [33] is a sigmoid-like function used in forecasting. The Verication Problem The verication problem we consider is to certify that an input is robust to bounded perturbations of magnitude, where is a small number. Certifying means proving that the classication result of the neural network does not change in the presence of perturbations. We focus on l 1 robustness, where we take an inputx2R n and allow a bounded perturbation of += to each element inx. For each network, we take 100 random test inputs, lter out those that are incorrectly classied, apply an bounded perturbation to the correctly classied inputs, and then attempt to prove the classication remains correct. We choose values common in prior work. For MNIST networks, in particular, we choose = 8=255. For CIFAR networks, we choose = 1=255. For SST-2 networks, we choose = 0:04, and we only apply it to the rst word embedding in the input sequence. 5.4.2 ExperimentalResults Our experiments were designed to answer the following two questions: (1) How doLinSyn’s linear bounds compare with handcrafted bounds? (2) How does the runtime of LinSyn compare to state-of-the-art linear bounding techniques? To answer these questions, we compare the eectiveness of LinSyn’s linear bounds with the state-of-the-art linear bounding technique implemented inAutoLiRPA. To the best of our knowl- edge this is the only tool that can handle the activation functions we use in our benchmarks. As another comparison point, we also compare withPOPQORN, a state-of-the-art linear bounding technique for LSTM networks. POPQORN tackles the challenge of computing tight linear bounds forsigmoid(x)tanh(y) andxsigmoid(y) using an expensive gradient descent based approach, and thus makes a good com- parison point for runtime and accuracy. Our experiments were conducted on a computer with an Intel 2.6 GHz i7-6700 8-core CPU and 32GB RAM. BothAutoLiRPA andLinSyn are engineered to bound individual neurons in parallel. We congure each method to use up to 6 threads. 105 Table 5.1: Comparing certied accuracy and run time of LinSyn andAutoLiRPA. Network Architecture AutoLiRPA [122] Our Method (new) % certied time (s) % certied time(s) MNIST 4-Layer CNN with Swish 0.34 15 0.76 796 4-Layer CNN with Gelu 0.01 359 0.72 814 4-Layer CNN with Log Log 0.00 38 0.24 867 CIFAR 5-Layer CNN with Swish 0.03 69 0.35 1,077 5-Layer CNN with Gelu 0.00 1,217 0.31 1,163 5-Layer CNN with Log Log 0.59 98 0.69 717 SST-2 LSTM with sig tanh 0.93 37 0.91 1,074 LSTM with hard tanh - - 0.64 2300 LSTM with log log 0.16 1,072 0.82 2,859 Table 5.2: Comparing certied accuracy and run time of LinSyn andPOPQORN. Network Architecture POPQORN [55] Our Method (new) % certied time (s) % certied time(s) SST-2 LSTM with sig tanh 0.93 1517 0.90 1,074 Overall Comparison First, we compare the overall performance of our new method and the default linear bounding technique in AutoLiRPA. The results are shown in Table 5.1. Here, Columns 1 and 2 show the name of the dataset and the type of neural networks. Columns 3 and 4 show the results of the defaultAutoLiRPA, including the percentage of inputs certied and the analysis time in seconds. Similarly, Columns 5 and 6 show the results of our new method. The results in Table 5.1 show that, in terms of the analysis time, our method is slower, primarily due to the use of constraint solvers (namelydReal and the LP solver) but overall, the analysis speed is comparable to AutoLiRPA. However, in terms of accuracy, our method signicantly outperforms AutoLiRPA. In al- most all cases, our method was able to certify a much higher percentage of the inputs. For example,LinSyn more than quadruples the certied robustness of theLSTMwithloglog benchmark, and handles very well the relatively complex GeLU function. As for SST-2: LSTM with hard tanh, AutoLiRPA does not support the generalmax(x;y) operation, so a comparison is not possible without signicant engineering work. The only exception to the improvement is SST-2: LSTM with sig tanh, for which the results are sim- ilar (.93 versus .91). In this case, there is likely little to be gained over the default, decomposition-based approach of AutoLiRPA in terms of tightness because the inputs to sigmoid(x) tanh(y) and x 106 10 2 10 0 10 2 10 4 LinSyn 10 2 10 0 10 2 10 4 AutoLiRPA Figure 5.7: Scatter plot comparing the nal output interval width of LinSyn and Au- toLiRPA. 0.5 1.0 2.0 5.0 10.0 Ratio 0 200 400 600 800 Figure 5.8: Histogram of width ratios between Au- toLiRPA andLinSyn. Ratio reported as AutoLiRPA LinSyn . sigmoid(y) are not related, i.e.,x andy are two separate variables. This is in contrast to, e.g.,swish(x) = xsigmoid(x), where the left-hand side and right-hand side of the multiplication are related. In Table 5.2, we show a comparison between LinSyn and POPQORN. The result shows that our ap- proach achieves similar certied robustness and runtime, even thoughPOPQORN was designed to specif- ically target this particular type of LSTM architecture, whileLinSyn is entirely generic. Detailed Comparison Next, we perform a more in depth comparison of accuracy by comparing the widths of the nal output neuron’s intervals that are computed by AutoLiRPA and LinSyn. The results are shown in the scatter plot in Fig. 5.7 and the histogram in Fig. 5.8. Each point in the scatter plot represents a single output neuronx i for a single verication problem. Thex-axis is the width of the interval of the output neuronx i (i.e.u i l i ) computed byLinSyn, and they-axis is the width computed byAutoLiRPA. A point above the diagonal line indicates thatLinSyn computed a tighter (smaller) nal output interval. In the histogram, we further illustrate the accuracy gain as the width ratio, measured as AutoLiRPA LinSyn . Overall, the results show thatLinSyn is more accurate in nearly all cases, andLinSyn often produces nal output bounds 2-5X tighter thanAutoLiRPA. 107 5.5 Summary We have presented LinSyn, a method for synthesizing linear bounds for arbitrary activation functions. The key advantage of LinSyn is that it can handle complex activation functions, such as Swish, GELU, and Log Log as a whole, allowing it to synthesize much tighter linear bounds than existing tools. Our experimental results show this increased tightness leads to drastically increased certied robustness, and tighter nal output bounds. 108 Chapter6 OlineSynthesisofLinearApproximations Neural networks have become a popular model choice in machine learning due to their performance across a wide variety of tasks ranging from image classication [40], natural language processing [113], and con- trol [43, 46]. However, they are also known to misclassify inputs in the presence of both small amounts of input noise and seemingly insignicant perturbations to the inputs [103]. Indeed, many works have shown they are vulnerable to a variety of seemingly benign input transformations [25, 47, 1], which raises concerns about their deployment in safety-critical systems. As a result, a large number of works have pro- posed verication techniques to prove that a neural network is not vulnerable to these perturbations [98, 117, 118], or in general satises some specication [48, 49, 43]. Crucial to the precision and scalability of these verication techniques are linear approximations of the network’s activation functions. In essence, given some arbitrary activation function(x), a linear approximation is a coecient gen- erator functionG : (l;u)!ha l ;b l ;a u ;b u i, wherel;u2R are real values that correspond to the interval [l;u], anda l ;b l ;a u ;b u 2R are real-valued coecients in the linear lower and upper bounds such that the following condition holds: 8x2 [l;u]: a l x +b l (x)a u x +b u (6.1) 109 Indeed, a key contribution in many seminal works on neural network verication was a hand-crafted G(l;u) [98, 117, 116, 4, 22, 55, 125, 118, 119, 92, 95] and follow-up work built o these hand-crafted approx- imations [49, 108, 99, 96]. Furthermore, linear approximations have applications beyond neural network verication, such as rigorous global optimization and verication [61, 111]. However, craftingG(l;u) is tedious, error-prone, and requires an expert. Unfortunately, in the case of neural network activation functions, experts have only crafted approximations for the most common functions, namely ReLU, sigmoid, tanh, max-pooling, and those in vanilla LSTMs. As a result, existing tech- niques cannot handle new and cutting-edge activation functions, such as Swish [88], GELU [42], Mish [69], and LiSHT [90]. In this work, we consider the problem of automatically synthesizing the coecient generator function G(l;u), which can alternatively be viewed as four individual functionsG a l (l;u),G b l (l;u),G au (l;u), and G bu (l;u), one for each coecient. However, synthesizing the generator functions is a challenging task because (1) the search space for each function is very large (in fact, technically innite), (2) the optimal generator functions are highly nonlinear for all activation functions considered both in our work and prior work, and (3) to prove soundness of the synthesized generator functions, we must show: 8[l;u]2IR;x2 [l;u]: (G a l (l;u)x +G b l (l;u))(x) (G au (l;u)x +G bu (l;u)) (6.2) whereIR =f[l;u]jl;u2R;l ug is the set of all real intervals. The above equation has highly non- linear constraints, which cannot be directly handled by standard verication tools, such as the Z3 [20] SMT solver. To solve the problem, we propose a novel example-guided synthesis and verication approach, which is applicable to any dierentiable, Lipschitz-continuous activation function(x). (We note that activation functions are typically required to be dierentiable and Lipschitz-continuous in order to be trained by 110 Partition Example Generation Subsets Template Assigning Unfilled Templates Train ML Components Unsound Templates Bound Max Viol. Spec. Inputs Design Time AutoLiRPA, DeepPoly, ... Lookup Template Template Sound Templates Application Time Verification Result Figure 6.1: Overview of our method for synthesizing the coecient generator function. gradient descent, thus our approach applies to anypractical activation function). To tackle the potentially innite search space ofG(l;u), we rst propose twotemplates forG(l;u), which are inspired by the hand- crafted coecient functions of prior work. The “holes” in each template are lled by a machine learning model, in our case a small neural network or linear regression model. Then, the rst step is to partition the input space ofG(l;u), and then assign a single template to each subset in the partition. The second step is to ll in the holes of each template. Our approach leverages an example-generation procedure to produce a large number of training examples of the form ((l;u); (a l ;b l ;a u ;b u )), which can then be used to train the machine learning component in the template. However, a template instantiated with a trained model may still violate Equation 6.2, specically the lower bound (resp. upper bound) may be above (resp. below) the activation function over some interval [l;u]. To ensure soundness, the nal step is to bound the maximumviolation of a particular template instance using a rigorous global optimization technique based on interval analysis, which is implemented by the tool IbexOpt [16]. We then use the computed maximum violation to adjust the template to ensure Equation 6.2 always holds. The overall ow of our method is shown in Figure 6.1. It takes as input the activation function(x), and the set of input intervalsI x IR for whichG(l;u) will be valid. Duringdesigntime, we follow the pre- viously described approach, which outputs a set of sound, instantiated templates which make upG(l;u). 111 Then the synthesizedG(l;u) is integrated into an existing verication tool such as AutoLiRPA [122] or DeepPoly [98]. These tools take as input a neural network and a specication, and output the veri- cation result (proved, counterexample, or unknown). At application time (i.e., when attempting to verify the input specication), when these tools need a linear approximation for(x) over the interval [l;u], we lookup the appropriate template instance, and use it to compute the linear approximation (a l ;b l ;a u ;b u ), and return it to the tool. To the best of our knowledge, our method is the rst to synthesize a linear approximation generator functionG(l;u) for any given activation function(x). Our approach is fundamentally dierent from the ones used by state-of-the-art neural network verication tools such asAutoLiRPA andDeepPoly, which require an expert to hand-craft the approximations. We note that, whileAutoLiRPA can handle activations that it does not explicitly support bydecomposing(x) into elementary operations for which it has (hand- crafted) linear approximations, and then combining them, the resulting bounds are often not tight. In contrast, our method synthesizes linear approximations for(x) as a whole, and we show experimentally that our synthesized approximations signicantly outperformAutoLiRPA. We have implemented our approach and evaluated it on popular neural network verication problems (specically, robustness verication problems in the presence of input perturbations). Compared against state-of-the-art linear approximation based verication tools, our synthesized linear approximations can drastically outperform these existing tools in terms of the number of problems veried on recently pub- lished activation functions such as Swish [88], GELU [42], Mish [69], and LiSHT [90]. To summarize, we make the following contributions: • We propose the rst method for synthesizing the linear approximation generator functionG(l;u) for any given activation function. 112 • We implement our method, use it to synthesize linear approximations for several novel activation functions, and integrate these approximations into a state-of-the-art neural network verication tool. • We evaluate our method on a large number of neural network verication problems, and show that our synthesized approximations signicantly outperform the state-of-the-art tools. 6.1 Preliminaries In this section, we discuss background knowledge necessary to understand our work. Throughout the paper, we will use the following notations: for variables or scalars we use lower case letters (e.g.,x2R), for vectors we use bold lower case letters (e.g.,x2 R n ) and for matrices we use bold upper case letters (e.g.,W2R nm ). In addition, we use standard interval notation: we let [l;u] =fx2Rjlxug be a real-valued interval, we denote the set of all real intervals asIR =f[l;u]jl;u2R;lug, and nally we dene the set ofn-dimensional intervals asIR n =f n i=1 [l i ;u i ]j [l i ;u i ]2IRg, where is the Cartesian product. 6.1.1 NeuralNetworks We consider a neural network to be a function f : X R n ! Y R m , which has n inputs and m outputs. For ease of presentation, we focus the discussion onfeed-forward,fully-connected neural networks (although the bounds synthesized by our method apply to all neural network architectures). Forx2 X, such networks computef(x) by performing an alternating series of matrix multiplications followed by the element-wise application of an activation function(x). Formally, anl-layer neural network withk i neurons in each layer (and lettingk 0 =n;k l =m) hasl weight matrices and bias vectorsW i 2R k i1 k i andb i 2R k i fori2f1::lg. The input of the network is 113 f 0 =x T , and the output of layeri is given by the function:f i =(f i1 W i +b i ) which can be applied recursively until the output layer of the network is reached. Initially, common choices for the activation function(x) wereReLU(x) =max(0;x),sigmoid(x) = e x e x +1 , andtanh(x) = e x e x e x +e x , however the eld has advanced rapidly in recent years and, as a result, automatically discovering novel activations has become a research subeld of its own [88]. Many recently proposed activations, such as Swish and GELU [88, 42], have been shown to outperform the common choices in important machine learning tasks. 6.1.2 ExistingNeuralNetworkVericationTechniquesandLimitations We consider neural network verication problems of the following form: given a neural networkf :X! Y and an input setXX, compute an over-approximationY such thatff(x)jx2XgY Y. The most scalable approaches to neural network verication (where scale is measured by number of neurons in the network) use linear bounding techniques to computeY , which require a linear approximation of the network’s activation function. This is an extension of interval analysis [71] (e.g., intervals with linear lower/upper bounds [98, 122]) to computeY , and thusX andY are represented as elements ofIR n and IR m , respectively. We use Figure 6.2 to illustrate a typical neural network verication problem. The network has input neurons x 1 ;x 2 , output neurons x 7 ;x 8 and a single hidden layer. We assume the activation function is swish(x) = xsigmoid(x), which is shown by the blue line in Figure 6.3. Our input space is X = [1; 1] [1; 1] (i.e.,x 1 ;x 2 2 [1; 1]), and we want to provex 7 > x 8 , which can be accomplished by rst computing the boundsx 7 2 [l 7 ;u 7 ];x 8 2 [l 8 ;u 8 ], and then showingl 7 > u 8 . Following the prior work [98] and for simplicity, we split the ane transformation and application of activation function in the hidden layer into two steps, and we assume the neuronsx i , wherei2f1::8g, are ordered such that i<j implies thatx i is in either the same layer asx j , or a layer prior tox j . 114 Figure 6.2: An example of linear bounding for neural network verication. Linear bounding based neural network verication techniques work as follows. For each neuronx i , they compute the concrete lower and upper boundsl i andu i , together with symbolic lower and upper bounds. The symbolic lower and upper bounds are linear constraints P i1 j=0 c l j x j +c l i x i P i1 j=0 c u j x j +c u i , where each ofc l i ;c u i is a constant. Both bounds are computed in a forward layer-by-layer fashion, using the result of the previous layers to compute bounds for the current layer. We illustrate the computation in Figure 6.2. In the beginning, we havex 1 2 [1; 1] as the concrete bounds, and1x 1 1 as the symbolic bounds, and similarly forx 2 . To obtain bounds forx 3 ;x 4 , we multiplyx 1 ;x 2 by the edge weights, which forx 3 gives the linear boundsx 1 +x 2 x 3 x 1 +x 2 . Then, to computel 3 andu 3 , we minimize and maximize the linear lower and upper bounds, respectively, overx 1 ;x 2 2 [1; 1]. Doing so results inl 3 =2;u 3 = 2. We obtain the same result forx 4 . However, we encounter a key challenge when attempting to boundx 5 , as we need a linear approx- imation of(x 3 ) over [l 3 ;u 3 ] when boundingx 5 , and similarly forx 6 . Here, a linear approximation for x 5 can be regarded as a set of coecientsa l ;b l ;a u ;b u such that the following soundness condition holds: 8x 3 2 [l 3 ;u 3 ]:a l x 3 +b l (x 3 )a u x 3 +b u . In addition, a sub goal for the bounds istightness, which typically means the volume between the bounds and(x) is minimized. Crafting a function to generate these coecients has been the subject of many prior works. Many seminal papers on neural network 115 verication have focused on solving this problem alone. Broadly speaking, they fall into the following categories. Hand-crafted Approximation Techniques The rst category of techniques use hand-crafted func- tions for generatinga l ;b l ;a u ;b u . Hand-crafted functions are generally fast because they are static, and tight because an expert designed them. Unfortunately, current works in this category are not general – they only considered the most common activation functions, and thus cannot currently handle our mo- tivating example or any recent, novel activation functions. For these works to apply to our motivating example, an expert would need to hand-craft an approximation for the activation function, which is both dicult and error-prone. Expensive Solver-Aided Techniques The second category use expensive solvers and optimization tools to compute sound and tight bounds in a general way, but at the cost of runtime. Recent works include DiRNN [70] and POPQORN [55]. The former uses (unsound) optimization to synthesize candidate coecients and then uses an SMT solver to verify soundness of the bounds. The latter uses constrained- gradient descent to compute coecients. We note that, while these works do not explicitly target an arbitrary activation function(x), their techniques can be naturally extended. Their high runtime and computational cost are undesirable and, in general, make them less scalable than the rst category. Decomposing Based Techniques The third category combine hand-crafted approximations with a decomposing based technique to obtain generality and eciency, but at the cost of tightness. Interest- ingly, this is similar to the approach used by nonlinear SMT solvers and optimizers such as dReal [30] and Ibex [16]. To the best of our knowledge, only one work AutoLiRPA [122] implements this approach for neural network verication. Illustrating on our example,AutoLiRPA does not have a static linear ap- proximation for(x 3 ) = x 3 sigmoid(x 3 ), but it has static approximations forsigmoid(x 3 ) andx 3 y. 116 -2 2 0 Figure 6.3: Approximation of AutoLiRPA (red) and our approach (green). Thus we can boundsigmoid(x 3 ) overx 3 2 [2; 2], and then, lettingy = sigmoid(x 3 ), boundx 3 y. Doing so results in the approximation shown as red lines in Figure 6.3. While useful, they are suboptimal because they do not minimize the area between the two bounding lines. This suboptimality occurs due to the decomposing, i.e., the static approximations used here were not designed forswish(x) as a whole, but designed for the individual elementary operations. Our Work: Synthesizing Static Approximations Our work overcomes the limitation of prior work by automatically synthesizing a static function specically for any given activation function(x) with- out decomposing. Since the synthesis is automatic, and results in a bound generator function, we obtain generality and eciency, and since the synthesis targets(x) specically, we usually (demonstrated em- pirically) obtain tightness. In Figure 6.3, for example, the bounds computed by our method are represented by the green lines. The synthesized bound generator function can then be integrated to state-of-the-art neural network verication tools, includingAutoLiRPA. Wrapping Up the Example For our running example, using AutoLiRPA’s linear approximation, we would add the linear bounds for x 5 shown in Figure 6.2. To compute l 5 ;u 5 , we would substitute the linear bounds forx 3 intox 5 ’s linear bounds, resulting in linear bounds with onlyx 1 ;x 2 terms that can be 117 minimized/maximized forl 5 ;l 6 respectively. We do the same forx 6 , and then we repeat the entire process until the output layer is reached. 6.2 ProblemStatementandChallenges In this section, we formally dene the synthesis problem and then explain the technical challenges. During the discussion, we focus on synthesizing the generator functions for the upper bound. We note that we can synthesize lower bound generator functions analogously. 6.2.1 TheSynthesisProblem Given an activation function(x) and an input universex2 [l x ;u x ], we dene the set of all intervals over x in this universe asI x =f [l;u]j [l;u]2IR;l;u2 [l x ;u x ]g. (In our experiments, for instance, we use l x =10 andu x = 10). Our goal is to synthesize a generator functionG : (l;u)!ha u ;b u i, or equivalently, two generator functionsG au (l;u) andG bu (l;u) such that8[l;u]2 I x ;x2 R, the conditionx2 [l;u] =) (x) G au (l;u)x +G bu (l;u) holds. This is the same as requiring that the following condition does not hold (i.e., the formula is unsatisable): 9[l;u]2I x ;x2R:x2 [l;u]^(x)>G au (l;u)x +G bu (l;u) The formula above expresses the search for a counterexample, i.e., an input interval [l;u] such thatG au (l;u) x+G bu (l;u) is not a sound upper bound of(x) over the interval [l;u]. Thus, if the above formula is unsatisable, the soundness of the coecient functionsG au ;G bu is proved. In addition tosoundness, we want the bound to betight, which in our context has two complementary goals. For a given [l;u]2I x we should have (1)(z) =G au (l;u)z +G bu (l;u) for at least onez2 [l;u] 118 0 Figure 6.4: Illustration of the two-point form bound (upper dashed line) and tangent-line form bound (lower dashed line). (i.e., the bound touches(x) at some pointz), and (2) the volume belowG au (l;u)x +G bu (l;u) should be minimized (which we note is equivalent to minimizing the volume between the upper bound and(x) since(x) is xed). We will illustrate the volume by the shaded green region below the dashed bounding line in Figure 6.6. The rst goal is intuitive: if the bound does not touch(x), then it can be shifted downward by some constant. The second goal is a heuristic taken from prior work that has been shown to yield a precise approximation of the neural network’s output set. 6.2.2 ChallengesandOurSolution We face three challenges in searching for the generator functionsG au andG bu . First, we must restrict the search space so that a candidate can be found in a reasonable amount of time (i.e., the search is tractable). The second challenge, which is at odds with the rst, is that we must have a large enough search space such that it permits candidates that represent tight bounds. Finally, the third challenge, which is at odds with the second, is that we must be able to formally verifyG au ;G bu to be sound. While more complex geneator functions (G au ;G bu ) will likely produce tighter bounds, they will be more dicult (if not impractical) to verify. 119 We tackle these challenges by proposing two templates forG au ;G bu and then developing an approach for selecting the appropriate template. We observe that prior work has always expressed the linear bound for(x) over an intervalx2 [l;u] as either the line connecting the points (l;(l)); (u;(u)), referred to as the two-point form, or as the line tangent to(x) at a pointt, referred to as tangent-line form. We illustrate both forms in Figure 6.4. Assuming that 0 (x) is the derivative of(x), the two templates for G au andG bu as follows: G au (l;u) = (u)(l) ul G bu (l;u) =G au (l;u)l +(l) + two-point form template (6.3) G au (l;u) = 0 (g(l;u)) G bu (l;u) =G au (l;u)g(l;u) +(g(l;u)) + tangent-line form template (6.4) In these templates, there are two holes to ll during synthesis: and g(l;u). Here, is a real-valued constant upward (positive) shift that ensures soundness of the linear bounds computed by both templates. We compute when we verify the soundness of the template (discussed in Section 6.3.3). In addition to, for the tangent-line template, we must synthesize a functiong(l;u) =t, which takes the interval [l;u] as input and returns the tangent pointt as output. These two templates, together, address the previously mentioned three challenges. For the rst chal- lenge, the two-point form actually does not have a search space, and thus can be computed eciently, and for the tangent-line form, we only need to synthesize the functiong(l;u). In Section 6.3.2, we will show empirically thatg(l;u) tends to be much easier to learn than a function that directly predicts the coe- cientsa u ;b u . For the second challenge, if the two-point form is sound, then it is also tight since the bound touches(x) by construction. Similarly, the tangent-line form touches(x) att. For the third challenge, 120 we will show empirically that these templates can be veried to be sound in a reasonable amount of time (on the order of an hour). At a high level, our approach contains three steps. The rst step is to partitionI x into subsets, and then for each subset we assign a xed template – either the two-point form template or tangent-line form template. The advantage of partitioning is two-fold. First, no single template is a good t for the entire I x , and thus partitioning results in overall tighter bounds. And second, if the nal veried template for a particular subset has a large violation (which results in a large upward shift and thus less tight bounds) the eect is localized to that subset only. Once we have assigned a template to each subset ofI x , the second step is to learn ag(l;u) for each subset that was assigned a tangent-line template. We use an example- generation procedure to generate training examples, which are then used to train a machine learning model. After learning eachg(l;u), the third step is to compute for all of the templates. We phrase the search for a sound as a nonlinear global optimization problem, and then use the interval-based solver IbexOpt [16] to bound. 6.3 OurApproach In this section, we rst present our method for partitioningI x , the input interval space, into disjoint subsets and then assigning a template to each subset. Then, we present the method for synthesizing the bounds- generating function for a subset in the partition ofI x (see Section 6.2.1). Next, we present the method for making the bounds-generating functions sound. Finally, we present the method for eciently looking up the appropriate template at runtime. 6.3.1 PartitioningtheInputIntervalSpace(I x ) A key consideration when partitioningI x is how to represent each disjoint subset of input intervals. While we could use a highly expressive representation such as polytope or even use non-linear constraints, for 121 10 5 0 5 10 l 10 5 0 5 10 u Figure 6.5: Partition ofI x for the Swish activation function, where the blue boxes belong toI tan , and the green boxes belong toI 2pt . eciency reasons, we represent each subset (of input intervals) as a box. Since a subset uses either the two-point form template or the tangent-line form template, the input interval space can be divided into I x =I 2pt [I tan . Each ofI 2pt andI tan is a set of boxes. At a high-level, our approach rst partitionsI x into uniformly sized disjoint boxes, and then assigns each box to either I 2pt or I tan . In Figure 6.5, we illustrate the partition computed for swish(x) = x sigmoid(x). Thex-axis andy-axis represent the lower boundl and the upper boundu, respectively, and thus a point (l;u) on this graph represents the interval [l;u], and a box on this graph denotes the set of intervals represented by the points contained within it. We give details on computing the partition below. DeningtheBoxes We rst dene a constant parameterc s , which is the width and height of each box in the partition ofI x . In Figure 6.5,c s = 1. The benets of using a smallerc s value is two-fold. First, it allows us to more accurately choose the proper template (two-point or tangent) for a given interval [l;u]. Second, as mentioned previously, the negative impact of a template with a large violation (i.e., large) is localized to a smaller set of input intervals. Assuming that (u x l x ) can be divided byc s , then we have ( uxlx cs ) 2 disjoint boxes in the partition of I x , which we represent byI i;j wherei;j2f1:: uxlx cs g.I i;j represents the box whose lower-left corner is 122 located at (l x +ic s ;l x +jc s ), or alternatively we haveI i;j =f[l;u]jl2 [l x +ic s ;l x +ic s +c s ];u2 [l x +jc s ;l x +jc s +c s ]g. To determine which boxesI i;j belong to the subsetI 2pt , we uniformly sample intervals [l;u]2 I i;j . Then, for each sampled interval [l;u], we compute the two-point form for [l;u], and attempt to search for a counter-example to the equation(x)G au (l;u)x +G bu (l;u) by samplingx2 [l;u]. If a counter- example is not found for more than half of the sampled [l;u]2I i;j , we add the boxI i;j toI 2pt , otherwise we add the box toI tan . We note that more sophisticated (probably more expensive) strategies for assigning templates exist. We use this strategy simply because it is ecient. We also note that some boxes in the partition may contain invalid intervals (i.e., we have [l;u]2 I i;j whereu < l). These invalid intervals are ltered out during the nal verication step described in Section 6.3.3, and thus do not aect the soundness of our algorithm. 6.3.2 LearningtheFunctiong(l;u) In this step, for each boxI i;j 2I tan , we want to learn a functiong(l;u) =t that returns the tangent point for any given interval [l;u]2I i;j , wheret will be used to compute the tangent-line form upper bound as dened in Equation 6.4. This process is done for all boxes inI tan , resulting in a separateg(l;u) for each boxI i;j . A sub-goal when learningg(l;u) is to maximize the tightness of the resulting upper bound, which in our case means minimizing the volume below the tangent line. We leverage machine learning techniques (specically linear regression or a small neural network with ReLU activation) to learng(l;u), which means we need a procedure to generate training examples. The examples must have the form ((l;u);t). To generate the training examples, we (uniformly) sample [l;u]2 I i;j , and for each sampled [l;u], we attempt to nd a tangent pointt whose tangent line represents a tight 123 0 Figure 6.6: Illustration of the sampling and linear programming procedure for computing an upper bound. Shaded green region illustrates the volume below the upper bound. upper bound of(x). Then, given the training examples, we use standard machine learning techniques to learng(l;u). The crux of our approach is generating the training examples. To generate a single example for a xed [l;u], we follows two steps: (1) generate upper bound coecientsa u ;b u , and then (2) nd a tangent point t whose tangent line is close toa u ;b u . In the following paragraphs, we describe the process for a xed [l;u], and then discuss the machine learning procedure. 6.3.2.1 GeneratingExampleCoecientsa u ;b u Given a xed [l;u], we aim to generate upper bound coecientsa u ;b u . A good generation procedure has three criteria: (1) the coecients should be tight for the input interval [l;u], (2) the coecients should be sound, and (3) the generation should be fast. The rst two criteria are intuitive: good training examples will result in a good learned model. The third is to ensure that we can generate a large number of examples in a reasonable amount of time. Unfortunately, the second and third criteria are at odds, because proving soundness is inherently expensive. To ensure a reasonable runtime, we relax the second criteria toprobably sound. Thus our nal goal is to minimize volume belowa u ;b u such that(x)a u x +b u probably holds forx2 [l;u]. 124 Our approach is inspired by a prior work [92, 4], which formulates the goal of a non-linear optimization problem as a linear program that can be solved eciently. Our approach samples points (s i ;(s i )) on the activation function fors i 2 [l;u], which are used to to convert the nonlinear constraint(x)a u x +b u into a linear one, and then uses volume as the objective (which is linear). For a setS of sample points s i 2 [l;u], the linear program we solve is: minimize : volume below a u x +b u subj: to : ^ s i 2S (s i )a u s i +b u We illustrate this in Figure 6.6. Solving the above problem results ina u ;b u , and the prior work [92, 4] proved that the solution (theoretically) approaches the optimal and sounda u ;b u as the number of samples goes to innity. We use Gurobi [38] to solve the linear program. 6.3.2.2 Convertinga u ;b u toaTangentLine To use the generateda u ;b u in the tangent-line form template, we must nd a pointt whose tangent line is close toa u ;b u . That is, we require that the following condition (almost) holds: ( 0 (t) =a u )^ ( 0 (t)t +(t) =b u ) To solve the above problem, we use local optimization techniques (specically a modied Powell’s method [85] implemented in SciPy [114], but most common techniques would work) to nd a solution to 0 (t) = a u . We then check that the right side of the above formula almost holds (specically, we check (j( 0 (t) t +(t))b u j 0:01). If the local optimization does not converge (i.e., it does not nd at such that 0 (t) =a u ), or the check onb u fails, we throw away the example and do not use it in training. 125 Figure 6.7: Plots of the training examples, smoothed with linear interpolation. On the left: a plot of ((l;u); (t)), and on the right: a plot of ((l;u); (a u )): One may ask the question: could we simply train a model to directly predict the coecientsa u and b u , instead of predicting a tangent point and then converting it to the tangent line? The answer is yes, however this approach has two caveats. The rst caveat is that we will lose the inherent tightness that we gain with the tangent-line form – we no longer have a guarantee that the computed linear bound will touch(x) at any point. The second caveat is that the relationship betweenl;u andt tends to be close to linear, and thus easier to learn, whereas the relationship betweenl;u anda u , or betweenl;u andb u , is highly nonlinear. We illustrate these relationships as plots in Figure 6.7. The left graph plots the generated training examples ((l;u);t), converted to a smooth function using linear interpolation. We can see most regions are linear, as shown by the at sections. The right plot shows ((l;u);a u ), where we can see the center region is highly nonlinear. 6.3.2.3 TrainingontheExamples Using the procedure presented so far, we sample [l;u] uniformly fromI i;j and generate the corresponding t for each of them. This results in a training dataset ofr examplesD train =f((l i ;u i );t i )ji2f1::rgg. We then choose between one of two models – a linear regression model or a 2-layer, 50-hidden-neuron, ReLU 126 network – to become the nal functiong(l;u). To decide, we train both model types, and choose the one with the lowest error, where error is measured as the mean absolute error. We give details below. A linear regression model is a functiong(l;u) = c 1 l +c 2 u +c 3 , wherec i 2 R are coecients learned by minimizing the squared error, which formally is: X ((l i ;u i );t i )2D train (g(l i ;u i )t i ) 2 (6.5) Finding the coecientsc i that minimize the above constraint has a closed-form solution, thus convergence is guaranteed and optimal, which is desirable. However, sometimes the relationship between (l;u) and t is nonlinear, and thus using a linear re- gression model may result in a poor-performingg(l;u), even though the solution is optimal. To capture more complex relationships, we also consider a 2-layer ReLU network whereW 0 2R 250 ,W 1 2R 501 , b 0 2R 50 ,b 1 2R, and we haveg(l;u) = ReLU(hl;ui T W 0 +b 0 )W 1 +b 1 . The weights and biases are initialized randomly, and then we minimize the squared error (Equation 6.5) using gradient descent. While convergence to the optimal weights is not guaranteed in theory, we nd in practice it usually converges. We choose these two models because they can capture a diverse set of g(l;u) functions. While we could use other prediction models, such as polynomial regression, generally, a neural network will be equally (if not more) expressive. However, we believe exploring other model types or architectures of neural networks would be an interesting direction to explore. 127 6.3.3 EnsuringSoundnessoftheLinearApproximations For a given I i;j , we must ensure that its corresponding coecient generator functionsG au (l;u) and G bu (l;u) are sound, or in other words, that the following condition doesnot hold: 9[l;u]2I i;j ; x2 [l;u]:(x)>G au (l;u)x +G bu (l;u) (6.6) We ensure the above condition does not hold (the formula is unsatisable) by bounding the maximum violation on the clause(x)>G au (l;u)x +G bu (l;u), which we formally dene as (l;u;x) =(x) (G au (l;u)x +G bu (l;u)). is positive when the previous clause holds. Thus, if we can compute an upper bound u , we can set the term inG bu (l;u) to u to ensure the clause does not hold, thus making the coecient generator functions sound. To compute u , we solve (i.e., bound) the following optimization problem: for : l;u;x2 [l i;j ;u i;j ] maximize : (l;u;x) subj: to : lu i ) for alli6=j andi2f1::mg. For each network, we take 100 random test images, and following prior work [31], we lter out mis- classied images. We then take the remaining images, and create a robustness verication problem for each one. Again following prior work, we usep = 8=255 for MNIST networks andp = 1=255 for CIFAR networks. 131 6.4.2 ExperimentalResults Our experiments were designed to answer the following two questions: (1) How do our synthesized lin- ear approximations compare with other state-of-the-art, hand-crafted linear approximation techniques on novel activation functions? and (2) how do our synthesized linear approximations compare with the online synthesis approach of LinSyn? To the best of our knowledge,AutoLiRPA [122] is the only neural network verication tool capable of handling the activation functions we considered here using static, hand-crafted approximations. We primarily focus on comparing the number of verication problems solved. We caution against directly comparing the runtime of our approach against AutoLiRPA, as the latter is highly engi- neered for parallel computation, whereas our approach is not currently engineered to take advantage of parallel computation (although it could be). To perform a fair runtime comparison between our approach and LinSyn, we limit LinSyn to synthesize linear approximations sequentially (as opposed to in parallel, as was done in Chapter 5)). We conducted all experiments on an 8-core 2.7GHz processor with 32GB of RAM. We present the comparison of our approach to AutoLiRPA in Table 6.2. The rst column shows the dataset and architecture. The next two columns show the percentage of the total number of verication problems solved (i.e., robustness proved) and the total runtime in seconds for AutoLiRPA. The next two columns show the same statistics for our approach. The nal column compares the output set sizes of Au- toLiRPA and our approach. We rst denejYj as the volume of the (hyper)boxY . Then lettingY auto andY ours be the output set computed byAutoLiRPA and our approach, respectively, jYoursj jYautoj measures the reduction in output set size. In general,jY ours j<jY auto j indicates our approach is better because it implies that our approach has more accurately approximated the true output set, and thus jYoursj jYautoj < 1 indicates our approach is more accurate. ∗ AutoLiRPA does not have an approximation for tan 1 . 132 Table 6.2: Comparison of the verication results of our approach andAutoLiRPA. Network Architecture AutoLiPRA [122] Our Approach jYoursj jYautoj % certied time (s) % certied time (s) MNIST 4-Layer CNN with Swish 0.34 15 0.74 195 0.59 4-Layer CNN with Gelu 0.01 359 0.70 289 0.22 4-Layer CNN with Log Log 0.00 38 0.00 174 21.29 4-Layer CNN with Mish 0.00 50 0.28 236 0.29 4-Layer CNN with LiSHT 0.00 15 0.11 289 0.32 4-Layer CNN with AtanSq ∗ - - 0.16 233 - CIFAR 5-Layer CNN with Swish 0.03 69 0.35 300 0.42 5-Layer CNN with Gelu 0.00 1,217 0.29 419 0.21 5-Layer CNN with Log Log 0.59 97 0.10 168 32.31 5-Layer CNN with Mish 0.00 202 0.29 363 0.17 5-Layer CNN with LiSHT 0.00 68 0.00 303 0.09 5-Layer CNN with AtanSq ∗ - - 0.22 347 - We point out three trends in the results. First, our automatically synthesized linear approximations almost always result in more verication problems solved (except for the Log Log activation). This is because our approach synthesizes a linear approximation specically for (x), which results in tighter bounds. Second, AutoLiRPA takes longer on more complex activations such as GELU and Mish, which have more elementary operations than Swish and LiSHT. This occurs becauseAutoLiRPA has more linear approximations to compute (it must compute one for every elementary operation before composing the results together). On the other hand, our approach computes the linear approximation in one step, and thus does not have the additional overhead for the more complex activation functions. Third, our approach almost always computes a much smaller output set, in the range of 2-10X smaller, which again is a reection of the tighter linear bounds. We present the comparison of our approach toLinSyn in Table 6.3. We can see that our new approach allows us to compute the linear approximations 10-20X faster than LinSyn. In addition, our approach either matches or comes close to the accuracy of LinSyn on several of the activation functions (Swish, Mish, GELU, and Atansq). However, the results also suggest there is potential for improvement, as shown 133 Table 6.3: Comparison of the verication results of our approach andLinSyn. Network Architecture LinSyn Our Approach jYoursj jY linsyn j % certied time (s) % certied time (s) MNIST 4-Layer CNN with Swish 0.76 2653 0.74 195 1.03 4-Layer CNN with Gelu 0.72 2832 0.70 289 1.05 4-Layer CNN with Log Log 0.24 2963 0.00 174 72 4-Layer CNN with Mish 0.47 3193 0.28 236 1.33 4-Layer CNN with LiSHT 0.79 2884 0.11 289 2.09 4-Layer CNN with AtanSq 0.18 2653 0.16 233 1.03 CIFAR 5-Layer CNN with Swish 0.35 3441 0.35 300 0.99 5-Layer CNN with Gelu 0.31 3755 0.29 419 1.06 5-Layer CNN with Log Log 0.69 2205 0.10 168 44.43 5-Layer CNN with Mish 0.29 4142 0.29 363 0.99 5-Layer CNN with LiSHT 0.00 4049 0.00 303 1.18 5-Layer CNN with AtanSq 0.23 3696 0.22 347 1.16 Table 6.4: Statistics of the synthesis step in our method. Activation(x) Partition Time (s) Learning Time (s) Verication Time (s) jI 2ptj jIxj Swish 81 1,762 20,815 0.45 GELU 104 2,113 45,504 0.57 Mish 96 2,052 38,156 0.45 LiSHT 83 1,650 61,910 0.46 AtanSq 85 1,701 18,251 0.38 by the accuracy achieved for the Log Log and LiSHT activations. We discuss potential improvements in Chapter 7. SynthesisResults We also report some key metrics about the synthesis procedure. Results are shown in Table 6.4. The rst three columns show the total CPU time for the three steps in our synthesis proce- dure. We note that all three steps can be heavily parallelized, thus the wall clock time is roughly 1/8 the reported times on our 8-core machine. The nal column shows the percentage of boxes in the partition that were assigned a two-point template (we can take the complement to get the percentage of tangent-line templates). 134 6.5 Summary We have presented the rst method for statically synthesizing a function that can generate tight and sound linear approximations for neural network activation functions. Our approach is example-guided, in that we rst generate example linear approximations, and then use these approximations to train a predic- tion model for linear approximations at run time. We leverage nonlinear global optimization techniques to ensure the soundness of the synthesized approximations. Our evaluation on popular neural network verication tasks shows that our approach signicantly outperforms state-of-the-art verication tools. 135 Chapter7 ConclusionandFutureWork In this chapter, we summarize the main contributions of this dissertation, and discuss potentially fruitful directions for future work. 7.1 Summary As neural networks make their way into safety-critical systems, providing formal guarantees about their safety has become paramount. This dissertation focuses on dierential safety properties, such as neural network equivalence, and it proposesaccurate andgeneral algorithms for proving or disproving them. We proposed four novel techniques towards this end. In Chapter 3, we proposed a novel technique for over-approximating the dierence between two struc- turally similar neural networks. Our technique approximates the dierence between the two networks layer-by-layer, as opposed to existing techniques, which analyze the two networks independently. To im- plement this technique, we formally derived equations that relate the intermediate computations of the two networks, and then used interval analysis to bound the dierence. In addition, we proposed an ap- proach to iteratively rene the approximation. We demonstrated that this new technique signicantly improves both the accuracy of the approximations and runtime to prove equivalence. 136 In Chapter 4, we extended the layer-by-layer approximation technique with novel symbolic approxi- mations. To implement this technique, we proposed novel linear approximations for the dierence between two (ReLU) neurons, and an approach to introducing new symbolic variables to represent the intermediate neurons of the two networks. We demonstrated that this new symbolic technique further improves both the accuracy and runtime for proving equivalence. In Chapter 5, we solved a key challenge in making dierential verication general, by proposing a technique for automatically synthesizing linear approximations for arbitrary functions. To implement this technique, we designed an ecient synthesis algorithm that combines inexpensive heuristic techniques with an SMT solver. We demonstrated that our synthesized linear approximations are more accurate than the best alternative approach that uses hand-crafted approximations. In Chapter 6, we proposed an oine technique for synthesizing linear approximations to eliminate the overhead incurred by the use of solvers in the online technique. This approach synthesizes a static imperative program (that does not call solvers) that computes the linear approximation, thus eliminating the overhead from expensive solvers. We demonstrate that this approach again improves the accuracy over hand-crafted approximations, though it trades accuracy for eciency compared to the online synthesis technique. 7.2 FutureWork Here we lay out potential extensions of our work. 7.2.1 SafetyPropertiesBeyondEquivalence Bounding Input Sensitivity. Another important dierential safety property is to prove a bound on the sen- sitivity of a neural network’s inputs. Informally, this means proving that “small changes to the inputs only 137 cause small changes to the outputs”. Formally, for a neural networkf(x), an input spaceX, a sensitivity bound, and a perturbationr, we would aim to provejf(x)f(x +r)j for allx2X. Hyperproperties. Related to dierential properties are hyperproperties. Here we give an example of a hyperproperty, which we believe has not been considered by prior work in the context of neural network verication. We rst give an informal denition of the property. Say we have a neural network that takes sensor inputs from an autonomous vehicle and outputs a steering direction. A property we may want to prove is: “if in the current time step, the neural network outputs a right turn, then in the following time step the neural network should continue to output a right turn.” This property ensures that the car’s driving path is smooth, and does not unnecessarily make sharp turns. A more formal denition follows. Letf(x) be the neural network controller that outputs a steering angle for an autonomous vehicle. Here, x 2 R n is a real-valued vector of sensor inputs, and assume f(x)2 [30; 30] is the rotation of the steering wheel, so that f(x) < 0 is a left turn, f(x) > 0 is a right turn, andf(x) = 0 goes straight. Furthermore, letx t be the input given to the neural network at time-stept. The property we wish to prove is thatf(x t ) > 0 =) f(x t+1 ) 0). Here, x t+1 would be dened by a function ofx t and the system dynamics. A similar property we may be interested in is jf(x t )f(x t+1 )j < , where is some reasonably small constant, which says that the steering wheel rotation should not change too quickly. See e.g. [43] for examples of potential benchmarks where we may want to prove these properties. 7.2.2 OtherUsesforEquivalenceVerication In our work, we only proved equivalence of compressed networks with respect to their original networks, but there are other practical use cases for proving equivalence. A recent example is neural network re- pair [124, 112, 101, 21]. Typically, the repairs made by prior work do not come with formal guarantees (they only test the repairs on a restricted set of inputs). Our equivalence verication could provide such 138 guarantees on the repairs made by these techniques. In addition, prior work typically aims at minimally modifying the neural network’s weights, which is a case where our verication technique excels. Yet an- other application would be avoidingcatastrophicforgetting [53] when performing online training of neural networks. 7.2.3 SynthesizingOtherTypesofApproximations Besides linear approximations, many types approximations have been used in neural network verication. In all cases, experts hand-crafted the approximations, and thus automatically synthesizing these approxi- mations would be interesting as well. We highlight other types of approximations used in prior work. Zonotopes. Zonotopes [127] have been used extensively in neural network verication [97, 11, 74]. Zonotopes are essentially a restricted linear approximation, with the restriction being that the slopes of the lower and upper bound must be the same. Zonotopes could be synthesized both online and oine. For online synthesis, one would only need to devise a suitable linear program for synthesizing the candidate zonotope, as in Section 5.2. Verifying that the zonotope is a sound approximation is straightforward. After devising an online approach, this procedure could then be used to generate examples for oine synthesis. A key challenge for oine synthesis is that the templates developed in Chapter 6 would likely not be applicable to zonotopes. One would either need to train a neural network to directly predict the zonotope’s parameters, or devise new templates. Multi-NeuronApproximations. A recent fruitful direction for improving accuracy of approximations in the context of neural network verication has been multi-neuron approximations [96, 73]. Whereas our online and oine synthesis techniques consider only a single neuron in isolation, prior work has shown that computing approximations for multiple neurons jointly can signicantly improve accuracy. However, these approximations are hand-crafted by experts, and thus could benet from synthesis. 139 Semi-DeniteRelaxations. Similar to linear approximations are semi-denite relaxations, which permit quadratic lower and upper bounds. The added expressiveness of quadratic bounds versus linear bounds has been shown to result in signicantly more accurate approximations [19], though at the cost of ad- ditional runtime because a quadratic program must be solved as opposed to a linear program. Again, these techniques currently require and expert to handcraft the approximation, and thus could benet from synthesis. 7.2.4 ImprovementstoOlineSynthesis As previously mentioned, the oine synthesis technique sacrices accuracy for eciency, however we believe we can achieve a better trade-o. In Section 6.3.3, we solve Equation 6.3.3 to compute the max- imum violation, and adjust all linear approximations by this amount. In many cases, we found that this adjustment is relatively large, which means that often the computed linear approximations will be looser than necessary. A potential improvement would be to use counter-examples to rene the synthesized approximation when the maximum violation is above some threshold. The underlying solver returns an input (i.e. a counter-example) that maximizes Equation 6.3.3, which provides a hint at the inputs that are causing the violation to be large. We could then take this counter-example and attempt to reduce the violation for this input (and nearby inputs), either by retraining the neural network, or re-partitioning the input interval space. This could be done repeatedly until the maximum violation is suciently reduced. 140 Bibliography [1] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. “Generating natural language adversarial examples”. In: arXiv preprint arXiv:1804.07998 (2018). [2] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L. Reyes-Ortiz. “A Public Domain Dataset for Human Activity Recognition Using Smartphones”. In: 21st European Symposium on Articial Neural Networks, Computational Intelligence and Machine Learning (2013). [3] Muhammad Hilmi Asyro, Zhou Yang, Jicke Shi, Chu Wei Quan, and David Lo. “Can dierential testing improve automatic speech recognition systems?” In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE. 2021, pp. 674–678. [4] Mislav Balunović, Maximilian Baader, Gagandeep Singh, Timon Gehr, and Martin Vechev. “Certifying geometric robustness of neural networks”. In: Advances in Neural Information Processing Systems 32 (2019). [5] Teodora Baluta, Shiqi Shen, Shweta Shinde, Kuldeep S Meel, and Prateek Saxena. “Quantitative verication of neural networks and its security applications”. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019, pp. 1249–1264. [6] Gilles Barthe, Juan Manuel Crespo, and César Kunz. “Relational verication using product programs”. In: International Symposium on Formal Methods. Springer. 2011, pp. 200–214. [7] Gilles Barthe, Pedro R D’argenio, and Tamara Rezk. “Secure information ow by self-composition”. In: Mathematical Structures in Computer Science 21.6 (2011), pp. 1207–1252. [8] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. “Measuring Neural Net Robustness with Constraints”. In: Annual Conference on Neural Information Processing Systems. 2016, pp. 2613–2621. [9] Frédéric Benhamou and Laurent Granvilliers. “Continuous and interval constraints”. In: Foundations of Articial Intelligence 2 (2006), pp. 571–603. [10] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. “End to end learning for self-driving cars”. In: arXiv preprint arXiv:1604.07316 (2016). 141 [11] Gregory Bonaert, Dimitar I Dimitrov, Maximilian Baader, and Martin Vechev. “Fast and precise certication of transformers”. In: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 2021, pp. 466–481. [12] Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel. “Cnn-cert: An ecient framework for certifying robustness of convolutional neural networks”. In: Proceedings of the AAAI Conference on Articial Intelligence. Vol. 33. 01. 2019, pp. 3240–3247. [13] Alon Brutzkus and Amir Globerson. “Why do larger models generalize better? A theoretical perspective via the XOR problem”. In: International Conference on Machine Learning. PMLR. 2019, pp. 822–830. [14] Sébastien Bubeck and Mark Sellke. “A universal law of robustness via isoperimetry”. In: Advances in Neural Information Processing Systems 34 (2021). [15] Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. “A limited memory algorithm for bound constrained optimization”. In: SIAM Journal on scientic computing 16.5 (1995), pp. 1190–1208. [16] Gilles Chabert and Luc Jaulin. “Contractor programming”. In: Articial Intelligence 173.11 (2009), pp. 1079–1100. [17] Patrick Cousot and Radhia Cousot. “Abstract Interpretation: A Unied Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints”. In: ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages. 1977, pp. 238–252. [18] Patrick Cousot and Nicolas Halbwachs. “Automatic Discovery of Linear Restraints Among Variables of a Program”. In: ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages. 1978, pp. 84–96. [19] Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy R Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy S Liang, et al. “Enabling certication of verication-agnostic networks via memory-ecient semidenite programming”. In: Advances in Neural Information Processing Systems 33 (2020), pp. 5318–5331. [20] Leonardo De Moura and Nikolaj Bjørner. “Z3: An ecient SMT solver”. In: International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer. 2008, pp. 337–340. [21] Guoliang Dong, Jun Sun, Xingen Wang, Xinyu Wang, and Ting Dai. “Towards repairing neural networks correctly”. In: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). IEEE. 2021, pp. 714–725. [22] Tianyu Du, Shouling Ji, Lujia Shen, Yao Zhang, Jinfeng Li, Jie Shi, Chengfang Fang, Jianwei Yin, Raheem Beyah, and Ting Wang. “Cert-RNN: Towards Certifying the Robustness of Recurrent Neural Networks”. In: (2021). 142 [23] Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy A. Mann, and Pushmeet Kohli. “A Dual Approach to Scalable Verication of Deep Networks”. In: International Conference on Uncertainty in Articial Intelligence. 2018, pp. 550–559. [24] Rüdiger Ehlers. “Formal Verication of Piece-Wise Linear Feed-Forward Neural Networks”. In: Automated Technology for Verication and Analysis - 15th International Symposium, ATVA 2017, Pune, India, October 3-6, 2017, Proceedings. 2017, pp. 269–286. [25] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. “Exploring the landscape of spatial robustness”. In: International Conference on Machine Learning. PMLR. 2019, pp. 1802–1811. [26] ERAN. https://github.com/eth-sri/eran. 2021. [27] Michael Everett, Golnaz Habibi, and Jonathan P. How. “Ecient Reachability Analysis of Closed-Loop Systems with Neural Network Controllers”. In:2021 IEEE International Conferenceon Robotics and Automation (ICRA). 2021, pp. 4384–4390.doi: 10.1109/ICRA48506.2021.9561348. [28] Raphael A Finkel and Jon Louis Bentley. “Quad trees a data structure for retrieval on composite keys”. In: Acta informatica 4.1 (1974), pp. 1–9. [29] Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin T. Vechev. “DL2: Training and Querying Neural Networks with Logic”. In: International Conference on Machine Learning. 2019, pp. 1931–1941. [30] Sicun Gao, Soonho Kong, and Edmund M Clarke. “dReal: An SMT solver for nonlinear theories over the reals”. In: International conference on automated deduction. Springer. 2013, pp. 208–214. [31] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin T. Vechev. “AI2: Safety and Robustness Certication of Neural Networks with Abstract Interpretation”. In: IEEE Symposium on Security and Privacy. 2018, pp. 3–18. [32] Sumathi Gokulanathan, Alexander Feldsher, Adi Malca, Clark Barrett, and Guy Katz. “Simplifying Neural Networks with the Marabou Verication Engine”. In: arXiv preprint arXiv:1910.12396 (2019). [33] Gecynalda S da S Gomes and Teresa B Ludermir. “Complementary log-log and probit: activation functions implemented in articial neural networks”. In: 2008 Eighth International Conference on Hybrid Intelligent Systems. IEEE. 2008, pp. 939–942. [34] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and Harnessing Adversarial Examples”. In: International Conference on Learning Representations. 2015. [35] Divya Gopinath, Guy Katz, Corina S. Pasareanu, and Clark W. Barrett. “DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks”. In: Automated Technology for Verication and Analysis - 16th International Symposium, ATVA 2018, Los Angeles, CA, USA, October 7-10, 2018, Proceedings. 2018, pp. 3–19. 143 [36] Divya Gopinath, Corina S. Pasareanu, Kaiyuan Wang, Mengshi Zhang, and Sarfraz Khurshid. “Symbolic execution for attribution and attack synthesis in neural networks”. In: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. 2019, pp. 282–283. [37] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. “A survey of deep learning techniques for autonomous driving”. In:JournalofFieldRobotics 37.3 (2020), pp. 362–386. [38] Gurobi Optimization, LLC.GurobiOptimizerReferenceManual. 2021.url: https://www.gurobi.com. [39] Song Han, Huizi Mao, and William J. Dally. “Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Human Coding”. In: International Conference on Learning Representations. 2016. [40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep residual learning for image recognition”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770–778. [41] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. “AMC: AutoML for Model Compression and Acceleration on Mobile Devices”. In: European Conference on Computer Vision. 2018, pp. 815–832. [42] Dan Hendrycks and Kevin Gimpel. “Gaussian error linear units (gelus)”. In: arXiv preprint arXiv:1606.08415 (2016). [43] Haimin Hu, Mahyar Fazlyab, Manfred Morari, and George J Pappas. “Reach-sdp: Reachability analysis of closed-loop systems with neural network controllers via semidenite programming”. In: 2020 59th IEEE Conference on Decision and Control (CDC). IEEE. 2020, pp. 5929–5934. [44] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. “Safety Verication of Deep Neural Networks”. In: International Conference on Computer Aided Verication. 2017, pp. 3–29. [45] Wolfram Research Inc. Mathematica, Version 12.3.1. Champaign, IL, 2021.url: https://www.wolfram.com/mathematica. [46] Kyle D. Julian, Mykel J. Kochenderfer, and Michael P. Owen. “Deep Neural Network Compression for Aircraft Collision Avoidance Systems”. In: CoRR abs/1810.04240 (2018). arXiv: 1810.04240.url: http://arxiv.org/abs/1810.04240. [47] Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. “Geometric robustness of deep networks: analysis and improvement”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, pp. 4441–4449. [48] Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. “Reluplex: An Ecient SMT Solver for Verifying Deep Neural Networks”. In: International Conference on Computer Aided Verication. 2017, pp. 97–117. 144 [49] Guy Katz, Derek A. Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljic, David L. Dill, Mykel J. Kochenderfer, and Clark W. Barrett. “The Marabou Framework for Verication and Analysis of Deep Neural Networks”. In: International Conference on Computer Aided Verication. 2019, pp. 443–452. [50] R Baker Kearfott.Rigorousglobalsearch:continuousproblems. Vol. 13. Springer Science & Business Media, 2013. [51] R Baker Kearfott and Manuel Novoa III. “Algorithm 681: INTBIS, a portable interval Newton/bisection package”. In: ACM Transactions on Mathematical Software (TOMS) 16.2 (1990), pp. 152–157. [52] James C King. “Symbolic execution and program testing”. In: Communications of the ACM 19.7 (1976), pp. 385–394. [53] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. “Overcoming catastrophic forgetting in neural networks”. In: Proceedings of the national academy of sciences 114.13 (2017), pp. 3521–3526. [54] Marko Kleine Büning, Philipp Kern, and Carsten Sinz. “Verifying equivalence properties of neural networks with relu activation functions”. In: International Conference on Principles and Practice of Constraint Programming. Springer. 2020, pp. 868–884. [55] Ching-Yun Ko, Zhaoyang Lyu, Lily Weng, Luca Daniel, Ngai Wong, and Dahua Lin. “POPQORN: Quantifying robustness of recurrent neural networks”. In: International Conference on Machine Learning. PMLR. 2019, pp. 3468–3477. [56] Soonho Kong, Armando Solar-Lezama, and Sicun Gao. “Delta-decision procedures for exists-forall problems over the reals”. In: International Conference on Computer Aided Verication. Springer. 2018, pp. 219–235. [57] Alex Krizhevsky, Georey Hinton, et al. “Learning multiple layers of features from tiny images”. In: (2009). [58] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. “Adversarial examples in the physical world”. In: International Conference on Learning Representations. 2017. [59] Ori Lahav and Guy Katz. “Pruning and Slicing Neural Networks using Formal Verication”. In: 2021 Formal Methods in Computer Aided Design (FMCAD). IEEE. 2021, pp. 1–10. [60] Shuvendu K. Lahiri, Kenneth L. McMillan, Rahul Sharma, and Chris Hawblitzel. “Dierential Assertion Checking”. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ESEC/FSE 2013. Saint Petersburg, Russia: Association for Computing Machinery, 2013, pp. 345–355.isbn: 9781450322379.doi: 10.1145/2491411.2491452. 145 [61] Yahia Lebbah, Claude Michel, and Michel Rueher. “An ecient and safe framework for solving optimization problems”. In: Journal of computational and applied mathematics 199.2 (2007), pp. 372–377. [62] Yann Lecun, Leon Bottou, Yoshua Bengio, and Patrick Haner. “Gradient-based learning applied to document recognition”. In: Proceedings of the IEEE 86.11 (1998), pp. 2278–2324. [63] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al. “Deepgauge: Multi-granularity testing criteria for deep learning systems”. In: IEEE/ACM International Conference On Automated Software Engineering. ACM. 2018, pp. 120–131. [64] Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Ananth Grama. “MODE: automated neural network model debugging via state dierential analysis and input selection”. In: Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018. 2018, pp. 175–186. [65] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. “Towards deep learning models resistant to adversarial attacks”. In: International Conference on Learning Representations (2018). [66] A. Miné. “The Octagon Abstract Domain”. In: Analysis, Slicing, and Transformation. Oct. 2001, pp. 310–319. [67] A. Miné. Weakly Relational Numerical Abstract Domains. Ph.D. Thesis. Computer Science Department, ENS, France, Dec. 2004. [68] Matthew Mirman, Timon Gehr, and Martin T. Vechev. “Dierentiable Abstract Interpretation for Provably Robust Neural Networks”. In: International Conference on Machine Learning. 2018, pp. 3575–3583. [69] Diganta Misra. “Mish: A self regularized non-monotonic neural activation function”. In: arXiv preprint arXiv:1908.08681 4 (2019), p. 2. [70] Sara Mohammadinejad, Brandon Paulsen, Chao Wang, and Jyotirmoy V Deshmukh. “DiRNN: Dierential Verication of Recurrent Neural Networks”. In:arXivpreprintarXiv:2007.10135 (2020). [71] Ramon E Moore, R Baker Kearfott, and Michael J Cloud. Introduction to interval analysis. Vol. 110. Siam, 2009. [72] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. “DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks”. In: IEEE Conference on Computer Vision and Pattern Recognition. 2016, pp. 2574–2582. [73] Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, and Martin Vechev. “Prima: Precise and general neural network certication via multi-neuron convex relaxations”. In: arXiv preprint arXiv:2103.03638 (2021). 146 [74] Mark Niklas Müller, Robin Staab, Marc Fischer, and Martin Vechev. “Eective Certication of Monotone Deep Equilibrium Models”. In: arXiv preprint arXiv:2110.08260 (2021). [75] Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. “Regularizing Action Policies for Smooth Control with Reinforcement Learning”. In: 2021. [76] Anh Mai Nguyen, Jason Yosinski, and Je Clune. “Deep neural networks are easily fooled: High condence predictions for unrecognizable images”. In: IEEE Conference on Computer Vision and Pattern Recognition. 2015, pp. 427–436. [77] Augustus Odena and Ian Goodfellow. “Tensorfuzz: Debugging neural networks with coverage-guided fuzzing”. In: arXiv preprint arXiv:1807.10875 (2018). [78] Brandon Paulsen. ReluDi-ICSE2020-Artifact. https://github.com/pauls658/ReluDiff-ICSE2020-Artifact. 2020. [79] Brandon Paulsen and Chao Wang. “LinSyn: Synthesizing Tight Linear Bounds for Arbitrary Neural Network Activation Functions”. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer. 2022, pp. 357–376. [80] Brandon Paulsen, Jingbo Wang, and Chao Wang. ReluDi: Dierential Verication of Deep Neural Networks. 2020. arXiv: 2001.03662[cs.LG].url: https://arxiv.org/abs/2001.03662. [81] Brandon Paulsen, Jingbo Wang, and Chao Wang. “Reludi: Dierential verication of deep neural networks”. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. 2020, pp. 714–726. [82] Brandon Paulsen, Jingbo Wang, Jiawei Wang, and Chao Wang. “NeuroDi: scalable dierential verication of neural networks using ne-grained approximation”. In: 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE. 2020, pp. 784–796. [83] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. “DeepXplore: Automated Whitebox Testing of Deep Learning Systems”. In: ACM symposium on Operating Systems Principles. 2017, pp. 1–18. [84] Phil Phil McCausland. Self-driving uber car that hit and killed woman did not recognize that pedestrians jaywalk. Nov. 2019.url: https://www.nbcnews.com/tech/tech-news/self-driving-uber- car-hit-killed-woman-did-not-recognize-n1079281. [85] Michael JD Powell. “An ecient method for nding the minimum of a function of several variables without calculating derivatives”. In: The computer journal 7.2 (1964), pp. 155–162. [86] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. “Improving language understanding by generative pre-training”. In: (2018). [87] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. “Certied Defenses against Adversarial Examples”. In: International Conference on Learning Representations. 2018. [88] Prajit Ramachandran, Barret Zoph, and Quoc V Le. “Searching for activation functions”. In: arXiv preprint arXiv:1710.05941 (2017). 147 [89] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. “You only look once: Unied, real-time object detection”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 779–788. [90] Swalpa Kumar Roy, Suvojit Manna, Shiv Ram Dubey, and Bidyut Baran Chaudhuri. “LiSHT: Non-parametric linearly scaled hyperbolic tangent activation function for neural networks”. In: arXiv preprint arXiv:1901.05894 (2019). [91] Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. “Reachability Analysis of Deep Neural Networks with Provable Guarantees”. In: International Joint Conference on Articial Intelligence. 2018, pp. 2651–2659. [92] Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. “Scalable Polyhedral Verication of Recurrent Neural Networks”. In: International Conference on Computer Aided Verication. Springer. 2021, pp. 225–248. [93] Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. “Towards Compact and Robust Deep Neural Networks”. In: CoRR abs/1906.06110 (2019). arXiv: 1906.06110.url: http://arxiv.org/abs/1906.06110. [94] Thiago Serra, Xin Yu, Abhinav Kumar, and Srikumar Ramalingam. “Scaling Up Exact Neural Network Compression by ReLU Stability”. In: Advances in Neural Information Processing Systems 34 (2021). [95] Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. “Robustness verication for transformers”. In: International Conference on Learning Representations (2020). [96] Gagandeep Singh, Rupanshu Ganvir, Markus Püschel, and Martin Vechev. “Beyond the Single Neuron Convex Barrier for Neural Network Certication”. In: Advances in Neural Information Processing Systems (NeurIPS). 2019. [97] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. “Fast and eective robustness certication”. In: Advances in neural information processing systems 31 (2018). [98] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin T. Vechev. “An abstract domain for certifying neural networks”. In: ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (2019), 41:1–41:30. [99] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin T. Vechev. “Boosting Robustness Certication of Neural Networks”. In: International Conference on Learning Representations. 2019. [100] Vasu Singla, Sahil Singla, Soheil Feizi, and David Jacobs. “Low curvature activations reduce overtting in adversarial training”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 16423–16433. [101] Matthew Sotoudeh and Aditya V Thakur. “Provable repair of deep neural networks”. In: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 2021, pp. 588–603. 148 [102] Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. “Concolic testing for deep neural networks”. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018. 2018, pp. 109–119. [103] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. “Intriguing properties of neural networks”. In: arXiv preprint arXiv:1312.6199 (2013). [104] Liang Tan, Keping Yu, Ali Kashif Bashir, Xiaofan Cheng, Fangpeng Ming, Liang Zhao, and Xiaokang Zhou. “Toward real-time and ecient cardiovascular monitoring for COVID-19 patients by 5G-enabled wearable medical devices: a deep learning approach”. In: Neural Computing and Applications (2021), pp. 1–14. [105] Tachio Terauchi and Alex Aiken. “Secure information ow as a safety problem”. In: International Static Analysis Symposium. Springer. 2005, pp. 352–367. [106] Samuel Teuber, Marko Kleine Büning, Philipp Kern, and Carsten Sinz. “Geometric Path Enumeration for Equivalence Verication of Neural Networks”. In: 2021 IEEE 33rd International Conference on Tools with Articial Intelligence (ICTAI). IEEE. 2021, pp. 200–208. [107] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. “DeepTest: Automated testing of deep-neural-network-driven autonomous cars”. In: International Conference on Software Engineering. 2018, pp. 303–314. [108] Vincent Tjeng, Kai Xiao, and Russ Tedrake. “Evaluating robustness of neural networks with mixed integer programming”. In: International Conference on Learning Representations (2019). [109] Hoang-Dung Tran, Stanley Bak, Weiming Xiang, and Taylor T Johnson. “Verication of deep convolutional neural networks using imagestars”. In: International Conference on Computer Aided Verication. Springer. 2020, pp. 18–42. [110] Hoang-Dung Tran, Diago Manzanas Lopez, Patrick Musau, Xiaodong Yang, Luan Viet Nguyen, Weiming Xiang, and Taylor T Johnson. “Star-based reachability analysis of deep neural networks”. In: International Symposium on Formal Methods. Springer. 2019, pp. 670–686. [111] Gilles Trombettoni, Ignacio Araya, Bertrand Neveu, and Gilles Chabert. “Inner regions and interval linearizations for global optimization”. In: Proceedings of the AAAI Conference on Articial Intelligence. Vol. 25. 1. 2011. [112] Muhammad Usman, Divya Gopinath, Youcheng Sun, Yannic Noller, and Corina S Păsăreanu. “NN repair: Constraint-Based Repair of Neural Network Classiers”. In: International Conference on Computer Aided Verication. Springer. 2021, pp. 3–25. [113] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need”. In:Advancesinneuralinformation processing systems. 2017, pp. 5998–6008. 149 [114] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. “SciPy 1.0: Fundamental Algorithms for Scientic Computing in Python”. In:NatureMethods 17 (2020), pp. 261–272.doi: 10.1038/s41592-019-0686-2. [115] Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. “Towards understanding learning representations: To what extent do dierent neural networks learn the same representation”. In: Advances in Neural Information Processing Systems. 2018, pp. 9584–9593. [116] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. “Ecient Formal Safety Analysis of Neural Networks”. In: Annual Conference on Neural Information Processing Systems. 2018, pp. 6369–6379. [117] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. “Formal Security Analysis of Neural Networks using Symbolic Intervals”. In: USENIX Security Symposium. 2018, pp. 1599–1614. [118] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane S. Boning, and Inderjit S. Dhillon. “Towards Fast Computation of Certied Robustness for ReLU Networks”. In: International Conference on Machine Learning. 2018, pp. 5273–5282. [119] Yiting Wu and Min Zhang. “Tightening robustness verication of convolutional neural networks with ne-grained linear approximation”. In: Proceedings of the AAAI Conference on Articial Intelligence. Vol. 35. 13. 2021, pp. 11674–11681. [120] Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. “DeepHunter: a coverage-guided fuzz testing framework for deep neural networks”. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2019, pp. 146–157. [121] Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, and Xiaohong Li. “Dichaser: Detecting disagreements for deep neural networks”. In:Proceedingsof the28th InternationalJoint Conference on Articial Intelligence. AAAI Press. 2019, pp. 5772–5778. [122] Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. “Automatic Perturbation Analysis for Scalable Certied Robustness and Beyond”. In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 1129–1141.url: https://proceedings.neurips.cc/paper/2020/file/0cbc5671ae26f67871cb914d81ef8fc1-Paper.pdf. [123] Weilin Xu, Yanjun Qi, and David Evans. “Automatically Evading Classiers: A Case Study on PDF Malware Classiers”. In: Network and Distributed System Security Symposium. 2016. 150 [124] Xiaodong Yang, Tom Yamaguchi, Hoang-Dung Tran, Bardh Hoxha, Taylor T Johnson, and Danil Prokhorov. “Neural Network Repair with Reachability Analysis”. In: arXiv preprint arXiv:2108.04214 (2021). [125] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. “Ecient neural network robustness certication with general activation functions”. In: Advances in neural information processing systems. 2018, pp. 4939–4948. [126] Xianyi Zhang, Qian Wang, and Yunquan Zhang. “Model-driven Level 3 BLAS Performance Optimization on Loongson 3A Processor”. In: 18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012, Singapore, December 17-19, 2012. 2012, pp. 684–691. [127] Günter M Ziegler. Lectures on polytopes. Vol. 152. Springer Science & Business Media, 2012. 151

Abstract (if available)

Abstract Neural networks have become an integral component of cyber-physical systems, such as autonomous vehicles, automated delivery robots, and factory robots, and they have great potential in many other systems as well. However, flaws in these models are frequently discovered, and thus in high-stakes applications, ensuring their safety, robustness, and reliability is crucial. While many prior works have been devoted to this problem domain, they are limited because they primarily focus on a few narrowly defined safety properties, and they only focus on the most common neural network architectures and activation functions. This dissertation addresses these limitations by (1) studying a new class of safety properties - differential properties - for neural networks, and (2) developing accurate algorithms for formally proving (or disproving) them that are applicable to general neural network architectures and activation functions. We focus on neural network equivalence as the canonical example for a differential property, however other safety properties concerning input sensitivity and stability can be cast as differential properties as well. This dissertation makes four key contributions towards developing accurate and general algorithms for proving differential properties. First, we formalize the equivalence problem for neural networks, and then develop a novel technique based on interval analysis for proving equivalence of any two structurally similar feed-forward neural networks with ReLU activations. The key insight in this technique is in deriving formulas that relate the intermediate computations of the two neural networks, which allows us to accurately bound the maximum difference between the two networks over all inputs. Second, we develop a novel symbolic technique that further improves the analysis' accuracy. We demonstrate the effectiveness of these two techniques in proving equivalence of compressed neural networks with respect to the original neural networks. Finally, we propose two novel techniques for automatically synthesizing linear approximations for arbitrary nonlinear functions, thus allowing our differential techniques to apply to architectures and activation functions beyond feed-forward ReLU networks. We demonstrate that our synthesized linear approximations significantly improve accuracy versus the best alternative techniques.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Learning logical abstractions from sequential data

PDF

Data-driven and logic-based analysis of learning-enabled cyber-physical systems

PDF

Formal analysis of data poisoning robustness of K-nearest neighbors

PDF

Verification, learning and control in cyber-physical systems

PDF

Experimental analysis and feedforward design of neural networks

PDF

Machine learning for efficient network management

PDF

Side-channel security enabled by program analysis and synthesis

PDF

Sample-efficient and robust neurosymbolic learning from demonstrations

PDF

Towards the efficient and flexible leveraging of distributed memories

PDF

Neural networks for narrative continuation

PDF

Detection, localization, and repair of internationalization presentation failures in web applications

PDF

Automatic detection and optimization of energy optimizable UIs in Android applications using program analysis

PDF

Exploring complexity reduction in deep learning

PDF

Deep learning for subsurface characterization and forecasting

PDF

Physics-aware graph networks for spatiotemporal physical systems

PDF

Towards highly-available cloud and content-provider networks

PDF

Dynamical representation learning for multiscale brain activity

PDF

Deep generative models for time series counterfactual inference

PDF

Automated repair of presentation failures in Web applications using search-based techniques

PDF

Building straggler-resilient and private machine learning systems in the cloud

Asset Metadata

Creator Paulsen, Brandon Bruce (author)

Core Title Differential verification of deep neural networks

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Degree Conferral Date 2022-08

Publication Date 07/18/2022

Defense Date 07/17/2022

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag AI safety,automated reasoning,formal methods,formal verification,machine learning,neural networks,OAI-PMH Harvest

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Wang, Chao (committee chair), Annavaram, Murali (committee member), Deshmukh, Jyotirmoy (committee member)

Creator Email bpaulsen@usc.edu,brandonpaulsen8@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC111372150

Unique identifier UC111372150

Legacy Identifier etd-PaulsenBra-10844

Document Type Dissertation

Format application/pdf (imt)

Rights Paulsen, Brandon Bruce

Type texts

Source 20220718-usctheses-batch-954 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu