University of Southern California Dissertations and Theses

Software security economics and threat modeling based on attack path analysis; a stakeholder value driven approach

(USC Thesis Other)

Software security economics and threat modeling based on attack path analysis; a stakeholder value driven approach

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content SOFTWARE SECURITY ECONOMICS AND THREAT MODELING BASED ON ATTACK PATH ANALYSIS; A STAKEHOLDER VALUE DRIVEN APPROACH by Yue Chen A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2007 Copyright 2007 Yue Chen ii Dedication To my family iii Acknowledgements Small seeds can grow into big trees, but not all of them. Often, preliminary research ideas and sparks are just like the small seeds. I would like to take this opportunity to thank all the people who helped me to grow my preliminary ideas into this dissertation. I want to say my first “thank you” to my father and passed away mother for encouraging me to follow my curiosity in science and engineering, as well as teaching me to love, appreciate and give. I would also thank my dearest wife, Yinghua, for her constant support and understanding through the years. I express my deepest “thank you” to my advisor, Dr. Barry W. Boehm, for the wonderful opportunities and experience that I had in his research group. He is a great live example for me as a knowledgeable, disciplined and respectful scientist. I sincerely appreciate his patient guidance that helped me not only on software engineering knowledge, but also on the scientific attitude, selecting appropriate research approach, and making connections with other researchers. Without Barry’s help, I could never have made it. I also thank my other committee professors: Professor Clifford Neuman, who kindly helped me to polish and baseline my work through carefully examining the methodology, limitations, as well as pointing me to the important related work; Professor Mingdeh Huang, who inspired me most on exploring better algorithms and helped me review the correctness of the algorithms; Professor Neno Medvidovic, who helped me carefully define the research hypothesis and make validation plans; and Professor Bert Steece from USC Marshall Business School, who educated me on the statistical regression, and gave me constructive feedbacks on research validation methodology. iv Research grows along with inspiring discussions and sharing perspectives. I am especially grateful to Professor Mary Shaw at Carnegie Mellon University for her time, insightful feedback, notes, and interest on this research. I also highly appreciate the very valuable discussions with Dr. Rick Kazman and Mr. Robert C. Seacord from Carnegie Mellon University, Dr. Dan Port from University of Hawaii, Ms. Elizabeth Fong from NIST, and Mr. Tyler Moore from Cambridge University. No scientific research can stand without validation. I am very grateful to Mr. Luke Sheppard at USC Information Technology Services, Mr. Mike Oppenheim at Manual Art High School, Ms. Dan Wu and Dr. Jesal Bhuta at CSSE for helping me on the case studies and sharing their insights and experiences. I would take this opportunity to thank Mr. Winsor Brown who introduced me to the CSSE research group; Dr. Ye Yang, Ms. Jo Ann Lane, Dr. Ricardo Valerdi, Mr. Ed Colbert, Dr. Zhihao Chen and Mr. Dan Ingold for our research discussions; Ms. Julie Sanchez for her great help in proof-reading this dissertation. Also, I will miss all my CSSE friends with whom I spent many happy hours at Starbucks, Trojan football games, camping trips and dinners: Vesile Evrim, Steve Meyers, Gustavo Perez, Ye Yang, Dan Wu, Alex Lam, Liguo Huang, Tom Tan, Da Yang, Di Wu, Vu Nguyen, Supannika Koolmanojwong, Monvorath Phongpaibul, and Apurva Jain. Last but the most important, I am grateful to the US NSF Grant 0438931 (Value-Based Science of Design) and CSSE affiliates who funded this research. v Table of Contents Dedication............................................................................................................................... ii Acknowledgements ............................................................................................................... iii List of Tables........................................................................................................................ vii List of Figures ....................................................................................................................... ix List of Acronyms................................................................................................................... xi Abstract ................................................................................................................................ xii Chapter 1 Introduction ......................................................................................................... 1 1.1 Motivation................................................................................................................ 1 1.2 Research Statement and Hypothesis ........................................................................ 6 1.3 Contributions ........................................................................................................... 8 1.4 Dissertation Roadmap............................................................................................ 10 1.5 Terminologies........................................................................................................ 11 Chapter 2 Background and Related Work........................................................................ 13 2.1 Value Based Software Engineering Philosophy .................................................... 13 2.2 Dependency Analysis ............................................................................................ 15 2.3 Stakeholder Utilities and Decision Making ........................................................... 19 2.4 Security Economics ............................................................................................... 21 2.5 Security Threat Modeling ...................................................................................... 23 2.6 COTS Vulnerability Analysis................................................................................ 25 2.7 Security Vulnerability Scanners and Tools............................................................ 30 Chapter 3 Nature of the Problem....................................................................................... 32 Chapter 4 The T-MAP Framework ................................................................................... 35 4.1 Structured Attack Graph ........................................................................................ 35 4.2 UML Model of Structured Attack Path.................................................................. 39 4.3 Value Driven Evaluation of Attack Scenarios ....................................................... 41 4.4 T-MAP Weighting System..................................................................................... 49 vi Chapter 5 ITS Case Study .................................................................................................. 57 Chapter 6 Tool Support – Tiramisu................................................................................... 64 Chapter 7 Validation and Results ...................................................................................... 68 7.1 Validation Methodology........................................................................................ 68 7.2 Case Study Results and Evaluation ....................................................................... 73 7.3 Testing Hypotheses ................................................................................................ 83 7.4 Lessons Learned: Top Reasons for Ranking Mismatches ..................................... 88 7.5 Threats to Validity ................................................................................................. 89 Chapter 8 Using T-MAP over Software Life-cycles ......................................................... 92 Chapter 9 Value Sensitive Firewall Rule Generation....................................................... 98 9.1 The Problem........................................................................................................... 98 9.2 Observation: Security Risk Distribution Over Network Ports............................... 99 9.3 Stakeholder Value Driven Automated Firewall Rule Generation ....................... 101 9.4 Preliminary Implementation Results ................................................................... 108 9.5 Summary.............................................................................................................. 114 Chapter 10 Conclusions and Future Work ..................................................................... 116 10.1 Conclusions.......................................................................................................... 116 10.2 Limitations........................................................................................................... 119 10.3 Feedbacks ............................................................................................................ 120 10.4 Future Work......................................................................................................... 121 References........................................................................................................................... 123 Appendix Initial t test Results .......................................................................................... 129 vii List of Tables Table 1 Attack Path Attribute Description (Partial) .............................................................. 40 Table 2 Security Cost Estimation of the Case Study ............................................................. 43 Table 3 USC ITS Case Study Stakeholder Value Propositions............................................. 44 Table 4 Pair-wise Comparison across ITS Values................................................................. 45 Table 5 Security Breach Scenario Evaluation Score ............................................................. 46 Table 6 Pair-wise Comparison of Attacker Group Size......................................................... 47 Table 7 Ratings of A.GS, A.SL, and A.MT........................................................................... 48 Table 8 Vulnerability Exposure to Attackers ........................................................................ 48 Table 9 Vulnerability Technical Attribute Ratings (Partial).................................................. 49 Table 10 COTS Software Installed on Server X.................................................................... 58 Table 11 Suppressing Matrix*............................................................................................... 60 Table 12 MASH Server Case Study Stakeholder Value Propositions................................... 76 Table 13 MASH Case Study Security Breach Scenario AHP Evaluation Score................... 77 Table 14 COTS Software Installed on MASH Server........................................................... 77 Table 15 CSSE Greenbay Case Study Stakeholder Value Propositions................................ 80 Table 16 GreenBay Case Study Security Breach Scenario AHP Evaluation Score .............. 81 Table 17 COTS Software Installed on CSSE Greenbay Server ............................................ 81 Table 18 Prioritization Inaccuracy Comparison across Approaches ..................................... 86 viii Table 19 Security Professional Profiles................................................................................. 91 Table 20 Using T-MAP in Boehm's Spiral Model ................................................................ 93 Table 21 Using T-MAP in the ISO 15288 System Engineering Life-cycle Model............... 94 Table 22 Using T-MAP in Royce’s Waterfall Model............................................................ 95 Table 23 Using T-MAP in the Incremental Commitment Model (ICM)............................... 96 Table 24 T Test Analysis between T-MAP and Mainstream Approaches .......................... 129 ix List of Figures Figure 1 Increasing Trend of Using COTS Based Applications.............................................. 2 Figure 2 Increasing Trend of OTS Vulnerabilities .................................................................. 3 Figure 3 The "4+1" Theory of VBSE: Overall Structure ...................................................... 14 Figure 4 Basic Scheiner Attack Tree ..................................................................................... 16 Figure 5 Result Chain Example............................................................................................. 18 Figure 6 The Effect of Vulnerability Disclosure and Patching.............................................. 29 Figure 7 Nature of The Problem............................................................................................ 33 Figure 8 Structured Attack Graph.......................................................................................... 36 Figure 9 UML Model of A Structured Attack Path ............................................................... 39 Figure 10 ThreatKey Calculation for a Three-Layer-Attack-Graph...................................... 53 Figure 11 Screenshot of Load Vulnerability Database.......................................................... 57 Figure 12 Screenshot of Attack Path Calculation Results ..................................................... 59 Figure 13 Screenshot of Security Practice Simulator ............................................................ 61 Figure 14 Optimal Security Practice Plan.............................................................................. 62 Figure 15 Sweet Spots of Investment .................................................................................... 63 Figure 16 Tiramisu Component Model ................................................................................. 65 Figure 17 Tiramisu Software Architecture ........................................................................... 66 x Figure 18 Prioritization Clash Counting................................................................................ 69 Figure 19 Vulnerability Ranking Comparison: Security Manager vs. T-MAP ..................... 74 Figure 20 Vulnerability Ranking Comparison: System Manager vs. T-MAP....................... 78 Figure 21 Vulnerability Ranking Comparison: System Manager vs. T-MAP....................... 82 Figure 22 ITS Case Study Result Comparison: TMAP vs. Value Neutral Approaches........ 83 Figure 23 MASH Case Study Result Comparison: TMAP vs. Value Neutral Approaches .. 84 Figure 24 Greenbay Study Result Comparison: TMAP vs. Value Neutral Approaches ....... 85 Figure 25 Inaccuracy Comparison - Box Plots...................................................................... 86 Figure 26 The Composable CBA Assessment Process Element ........................................... 97 Figure 27 COTS Vulnerability Distribution over Network Ports (partial) .......................... 100 Figure 28 Effect of Firewall Rules ...................................................................................... 102 Figure 29 Structured Attack Graph with Network Port Information................................... 103 Figure 30 Firewall Rule Generation - System Input and Output......................................... 108 Figure 31 Example of Netflow Data Format ....................................................................... 109 Figure 32 Screenshot of Specifying Enforced Port Control ................................................ 110 Figure 33 Example Output of Firewall Rule Generation..................................................... 111 Figure 34 Number of Firewall Rules vs. Number of Exploitable Attack Paths.................. 112 Figure 35 Number of Packets vs. Number of Exploitable Attack Paths.............................. 113 xi List of Acronyms ASCII American Standard Code for Information Interchange CBA Commercial-Off-The-Shelf Based Applications CBS Commercial-Off-The-Shelf Based Systems CERT Computer Emergency Response Team CIA Confidentially, Integrity, and Availability CISSP Certified Information System Security Professional COTS Commercial Off-The-Shelf COCOMO Constructive Cost Estimation Model CSSE Center for Systems and Software Engineering (previously CSE) CVSS Common Vulnerability Scoring System IOC Initial Operational Capability ISS Internet Security Systems LCO Life Cycle Objective LCA Life Cycle Architecture NIST National Institute of Standard Technology NVD National Vulnerability Database OTS Off-The-Shelf ROTS Research Off-The-Shelf SEI Software Engineering Institute T-MAP Threat Modeling based on Attack Path analysis TRR Transition Readiness Review USC University of Southern California VBSE Value Based Software Engineering xii Abstract The thesis presents the Threat Modeling Method Based on Attack Path Analysis (T-MAP) which quantifies security threats by calculating the total severity weights of relevant attacking paths for Commercial Off The Shelf (COTS) based systems. Further security economic analysis enabled by T-MAP is demonstrated. Compared to existing approaches, T-MAP is sensitive to system stakeholder value priorities and organizational IT environment. It distills the technical details of thousands of relevant software vulnerabilities into management-friendly numbers at a high-level; it systematically establishes the traceability and consistency from management-level organizational value propositions to technical-level security threats and corresponding mitigation strategies. In its trial usage in a large IT organization, T- MAP demonstrated promising strength in prioritizing and estimating security investment cost-effectiveness, as well as in evaluating the security performance of COTS systems. The steps of using T-MAP to optimize the cost-effectiveness of software patching, user account control and firewall are demonstrated through case studies. A system stakeholder value sensitive Firewall rule generation method based on T-MAP is introduced. In addition, a software tool that automates T-MAP has been developed. 1 Chapter 1 Introduction 1.1 Motivation In the past three decades, the information revolution has deeply changed almost every corner of our life. The tremendous potentials enabled by computers and software have significantly reshaped the technology landscapes and system capabilities in modern financial systems, communication systems, healthcare systems, military systems, aerospace systems, entertainment systems, and much more. As software becomes a key infrastructure in many domains, software security failure can cause severe damages and losses to system key stakeholders. Driven by the strong need for security assurance, the recent legislations such as the Sarbanes- Oxley Act, HIPAA, and Gramm-Leach-Bliley have enforced security as a must practice to a wide range of organizations. Trends Previous investigations have identified two striking trends in the past decade: (1) the trend of increasing usage of Commercial Off The Shelf (COTS) software; (2) meanwhile, the trend of more and more security vulnerabilities have been found in COTS software. 2 For example, the IT infrastructures that most business processes rely upon are usually not built from scratch. Instead, they are typically built upon COTS systems such as Operating Systems, Network Systems, Database Systems, Web Services, etc. As shown in Figure 1, the empirical results at the Center for Systems and Software Engineering (CSSE) revealed that the percentage of COTS Based Applications (CBA) in CSSE e-Services projects increased from 28% in 1997 to 70% in 2002 [B. Boehm, 2003]. This observation closely matches with the Standish Group’s survey result for the IT field at large in 2000 [StandishGroup, 2001]. These evidences well illustrated the increasing dependencies on COTS and third party software in general. Figure 1 Increasing Trend of Using COTS Based Applications However, COTS components are not always secure and reliable. COTS security has become a major concern for many large organizations whose daily business heavily relies upon healthy IT infrastructures. According to the US Computer Emergency Response Team statistics (US CERT), the number of annual published COTS product vulnerabilities increased significantly from 417 in 1997 to 5990 in 2005 [US-CERT, DB]. In the 2006 CSI/FBI Security Criminal Survey, Gordon et al reported a total loss of $52,494,290 from 313 S&P 500 organizations [L. Gordon, 3 2006]. As a live example, a recent security incident on a USC sensitive database made USC ITS send out more than 200,000 mails to inform existing and former students and faculties whose personal information were affected by the victim computer [Sci-tech-today]. Figure 2 Increasing Trend of OTS Vulnerabilities Challenges As one of the most prominent challenges, the needs for security have been often competing with very limited IT resources and the fast-changing internet threat patterns. According to the CSI/FBI 2006 Survey, about 47% of the S&P 500 organizations who participated in the survey spent only 2% of their IT budget on security [L. Gordon, 2006]. The numbers unveiled the fact that the ability to prioritize security vulnerabilities correctly and fix them efficiently has become a critical factor for every modern organization whose daily business heavily relies upon healthy IT infrastructures. 4 Obviously, performing effective cost-benefit analysis is the key to effective vulnerability prioritization. In previous work, Gordon and Loeb presented their findings that the ability to estimate security benefit is a key driver to perform effective security economics analysis based on an empirical study across 212 S&P 500 firms [L. Gordon, Jan 2006]. However, other researchers reported that it was very difficult to measure the benefit of security investment [Butler, May 2002], because security incidents often involve damages to none-monetary stakeholder utilities such as reputation, productivity and privacy. Furthermore, regardless of the complex and subjective nature of stakeholder utility, usually it is a key driver to prioritize security threats in many situations. Currently, the most influential COTS vulnerability rankings are generated by authority organizations such CERT, NIST, Microsoft, and Symantec. In most of these ranking systems, the security vulnerabilities are statically rated in terms of Critical/High, Moderate/Medium, or Low in a general sense by security experts. However, these ratings are static and do not take the system stakeholder utilities into account. In practice, extra investigations by security professionals are generally needed to determine the criticalness of vulnerabilities under specific stakeholder value context. Usually, because of the large volume of COTS vulnerabilities reported everyday, it is a very effort-consuming task even for small IT systems. To date, it is still very difficult to measure COTS security threats and prioritize vulnerabilities efficiently due to lack of effective metrics to measure stakeholder utilities, lack of firm historical data, and the complex and sensitive nature of security. Just as Butler 5 pointed out, many security managers have to make decisions based on their experience, judgment, and best knowledge [Butler, May 2002]. In order to scope the focus of this dissertation, we summarize the major challenges as follows: • Lack of effective methods/metrics to measure and prioritize security threats from software vulnerabilities under specific stakeholder value/utility context. • Most current vulnerability rating systems by authority organizations such as CERT, Microsoft, NIST, and Symantec are value neutral, static, and not sensitive to stakeholder utility functions. • The challenge to measure the “composition of forces” of possible thousands of COTS vulnerabilities in IT systems and the effectiveness of security investment, which are critical for security budgeting. • Describe and report security assessment in a management friendly language. These difficulties can further cause: • Mis-identifying critical security vulnerabilities and result in business loss. • The allocation of security resources does not match the system stakeholder utility priorities and result in resource abuse. • Mis-communication between technical and management, which makes security investment decisions based on inaccurate perceptions. 6 1.2 Research Statement and Hypothesis Insight #1: As stated in the NIST IT Security Guideline, the security impacts of all vulnerabilities can be specified in terms of Confidentiality, Integrity, and Availability (CIA) [G. Stoneburner, 2002]. These security attributes may have very different business indications under different application context. Therefore, vulnerabilities associated with different type of CIA impacts should be treated differently based on system stakeholder value context. Furthermore, traceability should be established between the stakeholder value propositions and the treatments to vulnerabilities that involve different CIA impacts Insight #2: Most current COTS vulnerability ranking systems by authority organizations such as NIST, Cisco, ISS, Microsoft, SANS, CERT, Symantec/BugTraq, and Qualys are value-neutral and static 1 . Often, the coarse resolution of ranking scales such as “High/Medium/Low” or “Critical/Medium/Moderate/Low” are used to differentiate more than 27,400 known COTS vulnerabilities. Research Statement In this dissertation, we devised a stakeholder value sensitive approach based on previous work to prioritize COTS security vulnerabilities and measure the security 1 The CVSS v2.0 that was published recently (in July 2007) made promising initial steps to allow user set the requirements on Confidentiality, Integrity and Availability in its “Environmental Metrics”. But the relationship between security and stakeholder value proposition was not explicitly specified in this approach. By default, CIA are treated equally and given identical weights in its base score system. 7 threat associated with COTS based systems. Furthermore, we tested if T-MAP can better prioritize COTS security vulnerabilities than existing value-neutral approaches. Hypotheses The fundamental question that motivated our research is “compared to existing value-neutral approaches, can stakeholder value injection improve vulnerability prioritization?” The overall hypothesis is stated as follows: No stakeholder-value-sensitive COTS vulnerability prioritization approach can be devised so that its performance will significantly differ from the existing value neutral ones. It is a null hypothesis and we aim to disprove it through empirical case studies, analyses, and tool demos. In order to test this hypothesis: first, we examine that if a stakeholder value driven framework can be devised which carries the desired attributes as discussed in Insight #1; second, if such framework can be devised, we test whether or not the devised framework can generate better results than the current state of the art value-neutral approaches through at least three empirical case studies. Here, “better” is measured by accuracy/inaccuracy in vulnerability rankings. The inaccuracy metric is presented in detail in Chapter 7.1. In order to make the overall hypothesis more explicitly testable, we elaborate the overall hypothesis into Sub-hypothesis #1 as follows: 8 For a COTS based system whose Confidentiality, Integrity and Availability have different indications to the system stakeholder values, the accuracy of T- MAP rankings, measured by inaccuracy (the ratio of the number of clashes between vulnerability priorities and stakeholder value priorities and the total number of comparisons made as presented in detail in Chapter 7.1), will not differ from the existing representative value-neutral approaches. 1.3 Contributions In this dissertation, we present a stakeholder value driven approach, namely the Security Threat-Modeling method based on Attacking Path analysis (T-MAP), to inject stakeholder incentives into COTS vulnerability prioritization and measure the cost effectiveness of common security protections such as Firewall, data encryption, and patching for COTS based systems. The contributions are summarized as follows: (1) Devised a systematic framework that captures the stakeholder value priorities through AHP analysis and injected the value propositions into COTS vulnerability ranking. (2) Provided empirical data points/case studies that value-driven method (T- MAP) significantly outperformed mainstream value-neutral methods in COTS security vulnerability prioritization. (3) Enabled distilling the technical details of thousands of software 9 vulnerabilities into executive-friendly language at a high-level. (4) Systematically established the traceability and consistency from organizational-level value propositions to technical-level COTS security vulnerabilities and corresponding mitigation strategies. (5) Developed an O(n) algorithm to calculate the associated threat weight (ThreatKey) of vulnerabilities, which considerably increased the method scalability to large COTS based systems. This scalability is achieved at the tradeoff on limiting the scope of analyses to single-step-exploits. (6) Designed a metric to measure and compare the accuracy/inaccuracy of security vulnerability ranking systems. (7) Automated the security threat evaluation process with the Tiramisu tool. (8) Devised a preliminary stakeholder value sensitive Firewall rule generation method based on T-MAP. This method can be automated by the Tiramisu tool. The output Firewall rule set was proven to have the minimal risk to stakeholder values among all the other possible rule sets that have the same number of rules. Here the risk associated with a Firewall rule set is measured by ThreatKey as presented in Chapter 4, 5, and 9. (9) Established a comprehensive COTS security vulnerability database across current authority resources such as NIST, CERT, Microsoft, Symantec, and FrSIRT. 10 (10) Developed an automated tool which scrawls Internet resources across authority organizations such as NIST National Vulnerability Database(NVD), ISS, Microsoft, SANS, CERT, Symantec/BugTraq and FrSIRT to collect and update the vulnerability database continuously. 1.4 Dissertation Roadmap The structure of this dissertation is arranged as follows: Chapter 2 reviews the related works; Chapter 3 presents the nature of the problem; Chapter 4 presents the T-MAP framework and demonstrates T-MAP with a real-life ITS case study; Chapter 5 presents the ITS case study results with a focus on economics analyses; Chapter 6 introduces the T-MAP tool (project code Tiramisu) ; Chapter 7 presents the other two case studies, method validation, and threats to validation; Chapter 8 discusses how T-MAP may be integrated into mainstream software/system processes; Chapter 9 presents a stakeholder value sensitive Firewall rule generation method based on T-MAP; Chapter 10 summarizes research conclusions, limitations, and future work. 11 1.5 Terminologies Commercial Off The Shelf (COTS) Product “A product that is: 1) Sold, leased, or licensed to the general public; 2) Offered by a vendor trying to profit from it; 3) Supported and evolved by the vendor, who retains the intellectual property rights; 4) Available in multiple identical copies; 5) Used without source code modification.” [L. Brownsword, August 2000] It also includes other Off The Shelf software such as open source software, and other possible third party software from which vulnerability information are publicly available. Common Vulnerabilities and Exposures (CVE) “CVE is a list of standardized names for known vulnerabilities and other information security exposures. ” [MITRE] Exposure “An exposure is a state in a computer system (or set of systems) which is not a universal vulnerability, but either: 1) allows an attacker to conduct information gathering activities; 2) allows an attacker to hide activities; 3) includes a capability that behaves as expected, but can be easily compromised; 4) is a primary point of entry that an attacker may attempt to use to gain access to the system or data; 5) is considered a problem according to some reasonable security policy.” [MITRE] 12 Universal Vulnerability “A universal vulnerability is a state in a computing system (or set of systems) which either 1) allows an attacker to execute commands as another user; 2) allows an attacker to access data that is contrary to the specified access restrictions for that data; 3) allows an attacker to pose as another entity; 4) allows an attacker to conduct a Denial of Service attack.” [MITRE] Utility Function “Primarily used in trying to characterize the nature of a function relating a stakeholder’s degree of preference for alternative (often multi-dimensional) outcomes.” [B. Boehm, 2005] Value Proposition “Primarily used as a generic term encompassing both win conditions and utility function.” [B. Boehm, 2005] Win Condition “Primarily used in the context of stakeholders negotiating mutually satisfactory or win-win agreements.” [B. Boehm, 2005] 13 Chapter 2 Background and Related Work Our work is primarily relevant to the research areas in Value Based Software Engineering, Security Economics, Threat Modeling, COTS System Vulnerability studies, Attack Tree, and Software Economics. 2.1 Value Based Software Engineering Philosophy The first cornerstone work that T-MAP replies upon is the Value Based Software Engineering framework. In the initial theory of Value Based Software Engineering, Boehm et al unveiled the fact that “your enterprise will succeed if and only if it makes winners of your success-critical stakeholders”. The authors traced many difficulties and failures that caused because of technical-value mismatches in current software engineering practices, and pointed out “the fundamental goal of all good design and engineering is to create maximal value added for any given investment” [B. Boehm, 2000; B. Boehm, 2005; Boehm, 1981]. In the 2005 VBSE book, Boehm argued that it is hard for a value-neutral approach to provide guidance for making its products useful to people, because the most critical success factors of software projects lie in the value domain. The authors further pointed out that maximizing key stakeholder values should drive the essential purpose of software activities. The authors presented a “4+1” theory framework of VBSE, including the Theory W, 14 Utility Theory, Dependency Theory, Decision Theory, and Control Theory, as shown in Figure 3. This model provided a general framework to guide the value based practices in software engineering. Figure 3 The "4+1" Theory of VBSE: Overall Structure [Boehm, 2005] As a special type of software activities, the COTS vulnerability management shares the common nature with many other software practices. Clearly, stakeholder values and utilities often have strong influences on security investment decision making. Thus, the question “how much security is enough?” should be answered under system stakeholder value/utility context. A good example of how utility function can affect decision making was presented in Boehm’s 81 Software Engineering Economics book [Boehm, 1981] (Page 284). In such example, although the expected value of two decision options are exactly the same, the manager always prefer one of the options because it involves significant less risk to the manager’s future career. Similar situations widely exist in COTS security vulnerability management. For example, the confidentiality, integrity and availability (CIA) may have different impacts to system stakeholder values. Thus, COTS vulnerabilities that involve 15 different types of CIA impacts should be treated differently to reflect their potential impacts to system stakeholder values. T-MAP adopted the Value Based Software Engineering philosophy and explicitly injected stakeholder value propositions into security threat evaluation, attempting to achieve consistency and traceability from high level stakeholder value propositions to low level security technical strategies. Specifically, we integrated the Theory W for identifying key stakeholders and their win conditions; employed the Analytical Hierarchy Process (AHP) to measure and prioritize stakeholder value propositions (Utility Theory); leveraged the attack tree method to analyze how stakeholder value propositions may be affected by system COTS vulnerabilities (Dependency Theory); and employed the Figure of Merit method for decision making (Decision Theory). Because T-MAP is a threat modeling method and emphasizes on the decision making process, control theory is beyond the discussion scope of this dissertation. 2.2 Dependency Analysis Attack Tree [Amoroso, 1994; B. Schneier, Dec. 1999; O. Sheyner, May 2002] and Results Chain [Thorp, 1998] are the major tools used in T-MAP to analyze how key stakeholder values can be impacted by COTS security vulnerabilities. 16 Attack Tree Analysis Attack Tree is also known as “And/Or Trees” [Amoroso, 1994; B. Schneier, Dec. 1999; O. Sheyner, April 2004; O. Sheyner, May 2002]. The history of Attack Tree can be traced back to the Fault Tree developed by Bell Labs and the U.S. Air Force in 1960s [N. H. Roberts, January 1981]. The software related application of Fault Tree is also known as Threat Tree, which is an excellent tool for attack scenario analyses [Amoroso, 1994; B. Schneier, Dec. 1999]. A typical Scheiner’s Attack Tree is illustrated as follows: Figure 4 Basic Scheiner Attack Tree [B. Schneier, Dec. 1999] In Figure 4, all the possible ways that an attacker can take to achieve his goal of “Open Safe” are illustrated through the nodes and connections between nodes. On each node, the actions that the attacker may take are clearly noted; the possible/impossible next-step scenarios that are associated with each node are also specified by the connecting nodes in next layer. Intuitively, if a defender can block 17 all the possible attack paths on the tree, he will be safe, assuming the Attack Tree is complete. In previous work, Sheyner and Wing at CMU established the first formal Failure Scenario Graphs based on B űchi Automata. They devised two algorithms to produce scenario graphs with tools implemented [O. Sheyner, April 2004; O. Sheyner, May 2002]. In order to improve the scalability problem that Sheyner’s approach encountered [X. Ou, Oct. 2006], Qu et al proposed a formal framework based on “dependency graph” and reduced the size of the graph to polynomial to the network size [S. Noel, Dec. 2003; X. Ou, Oct. 2006]. Currently, a significant challenge to bring the attack tree analyses into practice is the difficulties in collecting accurate information that is necessary for building practical attack graph, for example, the exploit pre-condition and post-condition of large volume of software vulnerabilities. In addition, other researchers argued that “there is no standard way of building, simulating and analyzing attack trees” and “the tool is only as good as the analyst using it” [G. C. Dalton II, 2006]. T-MAP employs a special type of Attack Tree, the Structured Attack Tree, to model the stakeholder value impact path from COTS vulnerability exploits. We extend the Attack Tree by customizing it with the classic IT risk management framework [G. Stoneburner, 2002], as presented in detail in Chapter 4. The key difference is that stakeholder value/utility is explicitly incorporated into the tree structure. Also, the layered structure of the T-MAP attack tree reduced the 18 complexity in graph analysis so that it enabled a fast algorithm to analyze the attack graph in O(n) time complexity. Result Chain Analysis In the USC eServices projects, the Result Chain method was found to be an effective tool to reason how project “Initiatives” can benefit the system key stakeholders as demonstrated in Boehm’s Initial Theory of Value-Based Software Engineering [B. Boehm, 2005]. In general, the Result Chain can illustrate how project initiatives can make contributions and added values under certain assumptions [B. Boehm, 2005]. An example of the Result Chain in The Information Paradox [Thorp, 1998] is shown as follows in Figure 5. This example demonstrates how the “Initiative” of “Implement a new order entry system” can generate stakeholder values in terms of “Increase sales”. Figure 5 Result Chain Example [Thorp, 1998] 19 This method can be used to reason how COTS system vulnerabilities may affect key stakeholder values as well. For example, in Figure 5, if the confidentiality of the order entry system is compromised, the desired outcome of “Increase sales” might not happen because customers may be scared away from using the system due to privacy concerns. Thus, by comparing the result chain with perfect and imperfect security assumptions, result chain can provide an intuitive method to reason how much the stakeholder values is relying upon system security attributes such as CIA, and further determine how much security investment is reasonable. 2.3 Stakeholder Utilities and Decision Making T-MAP employs the Figure of Merit and AHP to quantify the stakeholder value priorities into normalized numeric weights and evaluate security threats against stakeholder value criteria. In previous work, Bodin et al, Butler, and Boehm demonstrated successful examples that used the attribute-rating method (also called Figure of Merit Analysis) to make good security/software investment decisions [Boehm, 1981; Butler, May 2002]. As a relevant approach, AHP is a multiple criteria decision making framework invented by Dr. Saaty at the Business School of Pennsylvania University in 1970s. It captures decision maker’s utility preferences and priorities through pair-wise comparisons. Kontio proposed a framework based on AHP to select COTS components [Kontio, 1996]. Recently, Bodin et al proposed 20 a promising approach that uses Analytical Hierarchy Process (AHP) to evaluate security solution alternatives [L. D. Bodin, Feb. 2005]. The examples of using AHP for decision making can be found in [L. D. Bodin, Feb. 2005]. The author revealed that the nature of software security investment is a multi-criteria decision making problem that may involve both quantitative and qualitative criteria. As one of the classic decision making methods, the Figure of Merit has unique strength here because of its capability to handle multiple criteria as well as not requiring accurate values of risk- relevant probabilities and frequencies, which are usually impractical to obtain. But, it is important to note that the AHP has limitations when the method involves large number of evaluation criteria, because the number of pair-wise comparison increases by n 2 complexity where n is the number of criteria involved in. In practice, we found appropriate grouping on criteria can significantly reduce the number of pair-wise comparisons needed. Also, it is important to carefully select an appropriate evaluation function in the Figure of Merit decision method. An example of an inappropriate evaluation function that could lead to wrong selection of technologies was presented in Boehm’s previous work [Boehm, 1981], Chapter 15. 21 2.4 Security Economics Our work is also closely related to a faster growing research area on security economics. As one of the pioneer works, Anderson pointed out that information security is also driven by economic incentives besides technical factors [R. Anderson, 2001; R. Anderson, October 27, 2006]. Gordan and Loeb presented a quantitative economic model to determine the optimal amount to invest in security to protect a given set of information [L. Gordon, 2002]. The authors argued that “the loss associated with given vulnerability affect the optimal level of resources that should be devoted to securing the information”, and “the optimal amount to spend on information security is an increasing function of the level of vulnerability of such information” [L. Gordon, 2002]. These observations demonstrated that the existing COTS security vulnerability ranking systems that purely based on technical analysis are inadequate. Instead, economic factors and stakeholder value propositions should be taken into consideration to achieve effective vulnerability management. As one of the most promising approaches, Butler presented the multiple-attribute risk assessment in SAEM framework to reason the cost-benefit of security investments based on the traditional risk management methods and concepts [Butler, May 2002]. This framework provided a systematic way to help security managers to make security investment decisions based on an extensible multiple-attribute risk driven cost-benefit evaluation framework. Meanwhile, the author also reported difficulties in accurately measuring the benefit of security investment. 22 The traditional risk management method plays an important role in security economic analysis [AS/NZS 4360, 1999; CMMS, 2002; ISO 15288, 2002; G. Stoneburner, 2002; US-GAO, 1999]. In an early pioneer work, a risk management approach to answer the question “how much security is enough” was proposed by Hoo [Hoo, 2000]. As one of the most influential approaches, in the NIST Risk Management Guide for IT Systems, Stoneburner et al pointed out the adverse impact of security incidents can be described in terms of loss or degrade of Confidentiality, Integrity, and Availability [G. Stoneburner, 2002], which greatly simplified and standardized the impact analysis of security events. In such guideline, the risk was calculated by the multiplicity of Size of Loss and Probability; manual templates are provided to estimate vulnerability risks. Risk values are used to determine which vulnerabilities are more important. Some other approaches are based on game theory. For example, Cavusoglu et al proposed a quantitative model to evaluate security investments which can reflect the technical features of specific security systems such as the Intrusion Detection Systems [H. Cavusoglu, July 2004]. Generally speaking, the existing security economics research scoped a rich and sound theoretical framework to reason the cost-effectiveness of security investment. However, the difficulties to accurately estimate the probabilities, frequencies, size of losses and other similar parameters generally exist in these approaches. Also, because of the large volume of COTS vulnerabilities that pop out every day, it is almost impossible to manually perform risk analysis on every COTS vulnerability 23 through paper templates. Another non-trivial challenge is to measure the overall vulnerability risks as a compound. 2.5 Security Threat Modeling The Microsoft “STRIDE/DREAD” Model The Microsoft “STRIDE/DREAD” model [M. Howard] helps security managers identify how organization business values can be affected by Spoofing, Tampering, Repudiation, Information Disclosure, Deny of Service, and Elevation of Privileges through data flow analysis, and further calculation of risks. Comparing to T-MAP, this method targets on more general software threat modeling rather than COTS based system vulnerabilities. Trike Trike is an open source approach that defined a “unified conceptual framework for security auditing from a risk management perspective through the generation of threat models in a reliable, repeatable manner.” It provided a common language that enables “completely and accurately describing the security characteristics of a system from its high-level architecture to its low-level implementation details.” [P. Saitta, B. Larcom, M. Eddington, 2005] It leverages attach tree for threat modeling. A tool set has been developed to automatically generate attach graphs. Different from T-MAP that quantifies stakeholder value and threat priorities as a heuristic of 24 risks through AHP, the Strike’s risk calculation relies upon user’s estimations on probabilities and asset values. According to its FAQ, the authors mentioned the system has constraints in handling systems that involve large number of actors and assets. [P. Saitta, B. Larcom, M. Eddington, 2005] Other researchers reported that “to date much of the Trike methodology is still in the experimental stage, and has not been fully exercised again real systems.” [K. M. Goertzel, T. Winograd, H. L. McKinley, P. Holley, B. A. Hamilton, 2006] Skybox Skybox is another interesting commercial system that generates attack paths based on network topologies and identify critical vulnerabilities [Skybox]. Unfortunately, the information about this tool is very much limited and we are unaware of any scientific validation of this system. According to its whitepapers and a related patent, it has its own vulnerability database and uses a special network monitoring engine to generate attack paths [R. Lippmann, March 2005; Skybox]. The security risk is calculated through multiplying the probability of an attack to the size of loss of such an attack. This approach has clear credits in automating risk assessment through attack tree analyses. Meanwhile, some other researchers reported it may have scalability problems [R. Lippmann, March 2005]. Other Security Threat Modeling Approaches There are a few other important but less relevant security threat modeling methods include: (1) The Practical Threat Analysis (PTA) Calculative Threat Modeling 25 Methodology (CTMM) that aims at improving the Microsoft Threat Modeling v1 [PTA Technologies]; (2) The Consultative Objective Risk Analysis System (CORAS) that provides a tool-supported methodology for model-based risk analysis of security critical systems [CORAS]; (3) The Secure Information System (SECURIS) that extends CORAS and “targets the development of secure IT systems through the establishment of toll-supported methodology for model-driven specification, analysis and development of secure IT systems” [SECURIS]. 2.6 COTS Vulnerability Analysis Value-Neutral COTS Vulnerability Ranking Systems In the COTS Software Vulnerability sector, previous efforts have made it possible to look up the COTS software vulnerabilities from a comprehensive database by name, version and vendor [MITRE]. Third party COTS and open source vulnerabilities have been published and discussed real-time-ly by authority organizations such as NIST, MITRE, CERT, Cisco, FrSIRT, OVAL, Microsoft, and SANS. Comprehensive software vulnerability databases have been made accessible to the public [FiSIRT; ISS; Microsoft; MITRE; NIST; SANS; US-CERT]; The Common Vulnerability Exposures (CVE) naming standard has effectively uniformed the vulnerability names across vulnerability reporting sources [MITRE]. Security scanners are tools such as the Kuang tool [Baldwin, 1988], the Computer Oracle and Password System (COPS) [D. Farmer, Sept. 1991], Nessus [Nessus], SATAN 26 [SATAN], ISS [ISS], are developed to detect system security vulnerabilities and mis- configurations. As one of the fundamental works that T-MAP relies upon, NIST maintained a comprehensive COTS vulnerability database indexed by the CVE naming standard, namely the National Vulnerability Database (NVD). This work formulated the presentation of vulnerability technical features into a vector [NIST], and has been widely supported across software vendors, users, and security researchers. In NVD, the vulnerability information is stored in XML format so that further analysis on COTS vulnerability can be automated. Surprisingly, we found most current vulnerability ranking systems are stakeholder/value neutral. For example, the current leading COTS software vulnerabilities rating systems [Microsoft; M. Schiffman; Symantec; US-CERT, Metrics] are strongly technical-focused. As a representative leading value-neutral approach, NIST Common Vulnerability Scoring System (CVSS) is a vulnerability ranking system that brought in-depth-evaluation on comprehensive technical details of vulnerabilities. Comparing to most High/Medium/Low ranking systems, CVSS established a mechanism to interpret the technical features of a vulnerability into a numeric score which ranges from 1 to 10, which greatly improved the ranking granularity to differentiate vulnerabilities. However, missing the system stakeholder value context, the rating generated by CVSS can be misleading to security managers sometimes. For example, the vulnerability that can only compromise availability has a max rating of 3.3 out of 10 (low) in the NIST Common Vulnerability Scoring 27 System (CVSS) Version 1.0 (i.e., the CVE-2006-3468). But for many production servers, wherein availability can be mission-critical, this rating suggests this vulnerability is only at a “Low” priority. CVSS v2.0 Published in July 2007, CVSS v2.0 is probably the most advanced COTS vulnerability ranking system by the time this thesis is written. In CVSS v2.0, the metrics of COTS vulnerability severity is calculated based on the vulnerability attributes of: • Access Vector, characterizing if a vulnerability could be exploited remotely or locally. • Access Complexity, characterizing how difficult it is for an attacker to exploit a vulnerability. • Authentication, characterizing how many times an attacker needs to authenticate to the system to exploit the vulnerability. • CIA Impact, in terms of None, Partial, and Complete. Besides the above technical attributes, CVSS v2.0 also introduced other Temporal Metrics characterizing the patch availabilities and Environmental Metrics characterizing user environment properties. 28 Comparing to the CVSS v1.0, CVSS v2.0 gave more relative weights to those vulnerabilities that may completely compromise the system CIA. Although in its base metrics the CIA attributes are still treated equally, it made initial steps to enable users to specify the requirement level of these attributes in its environmental metrics. However, the stakeholder values and utilities are still implicit in CVSS 2.0. There is no mechanism to systematically ensure the traceability from organizational business values to the vulnerability ratings. The settings of the requirement level on CIA will largely depend on the individual vision and experience of the user. In practice, the CVSS base score are the most popularly used. Empirical Studies on COTS Vulnerability Patch Impacts Another relevant work examined the life-cycle of the COTS vulnerabilities from discovery, publishing, and patching. Ashish Arora, Ramayya Krishnan, Anand Nandkumar, Rahul Telang, and Yubao Yang at CMU are the pioneers in this field. In their 2004 paper, an empirical study was conducted to analyze the frequency of exploits observed from vulnerabilities in a honey pot. The results show that “disclosing a vulnerability increases the frequency of attacks, while the release of a patch for a known vulnerability decreases the number of attacks.” They also found that “open source vendors are quicker to patch when vulnerability is found in their products.” [A. Arora, June 2004] These results indicate that the known vulnerabilities are more frequently exploited than those unknown, and those vulnerabilities has patches available are less frequently exploited than those without patches available. 29 Figure 6 The Effect of Vulnerability Disclosure and Patching [A. Arora, June 2004] Arora’s findings are supportive to one of the most important assumptions that T- MAP makes: most of the attacks are associated with known vulnerabilities instead of unknowns. 30 2.7 Security Vulnerability Scanners and Tools Security scanners have been widely used to detect security vulnerabilities in IT systems. In one of the previous works, Baldwin developed the first rule based security vulnerability checker, the Kuang tool developed at MIT lab, for UNIX system [Baldwin, 1988]. It examines the abnormal user behaviors and privileges based on a pre-defined knowledge base to discover possible tricks that attackers are playing. The Computer Oracle and Password System COPS developed at Purdue University [D. Farmer, Sept. 1991] extended the Kuang tool by adding other types of security checks such as weak passwords, check sum of system files and binaries, files permissions, etc. Other successful security scanners include Nessues [Nessus, ], SATAN [SATAN], ISS [ISS], etc. The scanners can effectively identify system/network vulnerabilities for existing systems, networks and services. However, they cannot scan the blueprint of COTS system in early project life-cycle. Also, most existing scanners assign stakeholder value neutral and static ranks to identified system vulnerabilities. There are some other relevant IT security risk management systems. Stellar is a system that can construct security attack scenarios from IDS alerts and then prioritizes the scenarios based on pre-defined rules and keywords. Its attack scenarios are derived from categorizing and grouping IDS alerts [S. Boyer, Mar. 2005]. Stellar made clear contribution in prioritizing the IDS alerts and security events based on the alert characteristics such as the source IP addresses. However, it 31 does not have a mechanism to ensure the consistency between the Stellar rules and the organizational value propositions, although the accuracy of its prioritization output largely depends on how good the rules are. In addition, since Stellar takes the IDS output as input to generate attack scenarios, its results may be affected by the IDS false alarms and false negatives. 32 Chapter 3 Nature of the Problem The T-MAP framework is based upon the observations of: (1) the more security holes left open for an IT system, the less secure it is; (2) different IT servers might have different levels of importance in terms of supporting the business’s core values; (3) the more vulnerabilities are exposed to motivated attackers, the more risk is involved. As a value driven approach, T-MAP uses the Attack Path concept to characterize possible scenarios wherein an attacker can jeopardize organizational values. T-MAP calculates the severity weight of each Attack Path (attack scenario) based on not only the technical severity of the relevant system vulnerability, but also its value impacts. T-MAP then quantifies the IT system threat with the total weight of all possible Attack Paths. A typical scenario of how a successful attack can affect organizational values is illustrated in Figure 1. The core with a caption of “Org. Values” represents the key values that are important to an organization, for example, reputation and productivity; the second layer from inside to outside with a caption of “IT Hosts” stands for the IT servers and hosts in organization’s IT infrastructure. The third layer with a caption of “Software Applications, COTS” stands for the commercial off the shelf software installed on the IT servers. The outer-most layer with a caption of “Firewall Wrapper” presents the Firewall or other types of security wrappers 33 protecting the IT infrastructure. On the edge of the “Software Applications, COTS” layer, the small solid dots represent the software vulnerabilities that make attacks possible, while the small circles stands for the vulnerabilities that are made inaccessible to attackers by the firewall. Figure 7 Nature of The Problem In a typical successful attack, for example, the attacker may first find and get access to the victim host, then exploit the vulnerabilities of the COTS software installed on the victim host, thus compromise the Confidentiality, Integrity or Availability (CIA) of the victim host, and result in further damages to stakeholder values. We liken the COTS system security to the defense of a castle. Castle Defense Analog: measure the security of a castle by the value of treasures inside the castle, the number of holes on the walls, as well as the size of the holes. 34 In brief, T-MAP involves the following steps: (1) identify key stakeholders and value propositions (the treasures in the castle); (2) establish a set of security evaluation criteria based on stakeholder value propositions; (3) enumerate and analyze attack paths based on a comprehensive COTS vulnerability database containing 27,400 vulnerability information (the holes); (4) evaluate the severity of each scenario in terms of numeric ratings against the evaluation criteria established in Step 2 (the size of the holes); (5) the security threat of each vulnerability is quantified with the total severity ratings of all attack paths that are relevant to this vulnerability; (6) system total threat is quantified with the total severity ratings of all attack paths. Step 3~6 are tool automated. 35 Chapter 4 The T-MAP Framework 4.1 Structured Attack Graph T-MAP defines a formal framework to enumerate and evaluate attack scenarios based on attack graph analysis. We propose a Structured Attack Graph approach that incorporates the Schneier’s attack tree and the classic IT risk management framework of attacker, asset, vulnerability, and impact to enumerate the possible attack scenarios [B. Schneier, Dec. 1999; G. Stoneburner, 2002]. The new attack tree nodes are structured into five layers corresponding to each layer in Figure 1: the first layer nodes represent stakeholder values, for example, productivity, privacy, or reputation. The second layer nodes represent IT hosts that the stakeholder values rely upon. The third layer nodes represent the COTS software that is installed on the IT hosts. The fourth layer nodes represent software vulnerabilities that reside in the COTS software. The fifth layer nodes represent possible attackers, for example, insiders, external hackers, etc. The associations between fifth layer nodes and fourth layer nodes represent that the associated attacker has adequate privilege and access to the vulnerabilities. Under this framework, we found most nodes in the attack tree are “OR nodes” in real world. For example, if a COTS software has multiple vulnerabilities, then the 36 software can be compromised upon the exploit of any of the associated vulnerability. Thus, the new attack tree can be simplified into a Structured Attack Graph as illustrated in Figure 8. In practice, the Structured Attack Graph can be derived from analyzing the stakeholder value result chains [B. Boehm, 2005], data flow of organizational IT operations [P. Torr, Sept. 2005], and the system use/abuse cases [D. Verdon, July 2004]. Perceivably, T-MAP can be extended to analyze more complex multi-step-exploit attack graph by chaining the exploit steps between the layers of “IT hosts”, “COTS Software”, and “Vulnerability” in the Structured Attack Graph, if the connectability between exploits is known. Figure 8 Structured Attack Graph Definition 1: (Structured Attack Graph) A Structured Attack Graph G=<Va, Vv, Vc, Vh, Vs, Eav, Evc, Ech, Ehs> consists of five finite nonempty sets of vertices Va, Vv, Vc, Vh, Vs objects together with four (possibly empty) sets Eav, Evc, Ech, Ehs of 37 ordered pairs of distinct vertices of G called edges. Va, Vv, Vc, Vh, and Vs represent the sets of attacker nodes, vulnerability nodes, COTS software nodes, IT host nodes, and Stakeholder Value nodes, respectively. Eav=<a, v> only contains ordered pairs of vertices where a∈Va and v∈Vv; Evc=<v, c> only contains ordered pairs of vertices where v∈Vv and c∈Vc; Ech=<c, h> only contains ordered pairs of vertices where c∈Vc and h∈Vh; and Ehs=<h, s> only contains ordered pairs of vertices where h∈Vh and s∈Vs; Definition 2 (Structured Attack Path) For a given attack graph G, A Structured Attack Path is a 5-tuple P=<A, V, C, H, S>, where A∈Va, V∈Vv, C∈Vc, H∈Vh, S∈Vs, and the associations (edges) between elements from adjacent layers must exist. Clearly, P characterizes an attack scenario in terms of attacker, vulnerability, COTS software, IT Host, and Stakeholder Values, respectively. 38 Algorithm 1 is a brutal force algorithm that enumerates all Structured Attack Paths for a given Structured Attack Graph G. Algorithm 1 List EnumerateAttackPath (Structured_Attack_Graph G) Return: The complete set of Structured Attack Paths associated with G 1: list attackPathList; 2: StructuredAttackPath p; 3: for(each sv in G.Vs) { //s stands for stakeholder value 4: // find the associated host to this value 5: generate es1:=subset of G.Ehs where e1.s==sv, for ∀ e1∈es1; 6: for (each ehs in es1) { 7: // find the COTS associated with this host 8: generate es2:=subset of G.Ech where e2.h==ehs.h for ∀ e2∈es2; 9: for (each ech in es2) { 10: // find vulnerability that affect the COTS software 11: generate es3:=subset of G.Evc where e3.c == ech.c for ∀ e3 ∈ es3; 12: for(each evc in es3) { 13: // Find attackers have privilege/access to the vulnerability 14: generate es4:=subset of G.Eav where e4.v == evc.v for ∀ e4 ∈ es4; 15: for(each eav in es4) { 16: p Å <eav.a, evc.v, ech.c, ehs.h, sv>; 17: append p to attackPathList; 18: }}}}} 19: return attackPathList; Algorithm 1 Enumerate Attack Paths 39 4.2 UML Model of Structured Attack Path Figure 9 UML Model of A Structured Attack Path In order to enable further fine-grained threat assessment of attack scenarios, T- MAP defines a set of threat-relevant attributes for each compositing elements of P, as illustrated in the Unified Modeling Language (UML) [OMG] model of P in Figure 4. Obviously, by definition, the class of Structured Attack Path has 1:1 composition associations with classes of A, V, C, H, S, respectively. These class attributes can be classified into three categories: probability-relevant, size-of-loss relevant and descriptive. The selection of the class attributes are primarily based on, but not 40 limited to the NIST IT Risk Guide and the emerging national standard CVSS [M. Schiffman], a joint effort across CERT, Cisco, MITRE, eBay, ISS, Microsoft, Qualys, and Symantec. Assuming P is a Structured Attack Path in Structured Attack Graph G, the description of the attributes of P is summarized in Table 1. Because of the page limit, we elaborate some of the important attributes as follows. Table 1 Attack Path Attribute Description (Partial) S.VN Affected stakeholder value. Clearly, S∈G.Vs; S.IW Level of importance of the stakeholder value S.VN in terms of a ratings value between 0 and 1. Details of obtaining the weight is presented in Section 4.4 H.HN Affected host in the IT infrastructure. Clearly, H∈G.Vh; H.AA The estimated motivation of attacker P.A to attack host P.H terms of a ratings value between 0 and 1. Details of obtaining the weight is presented in Section 3.5 C.AP If automated patching service is provided by vendor V.VN The Common Vulnerability Name (CVE) of the vulnerability that involved in this attack path [NIST]. Clearly, V∈G.Vv; V.TI The impact of the vulnerability V can cause on victim host H in terms of confidentiality, integrity, and availability [NIST] V.AC Specifies if the associated vulnerability requires victim activities to enable such an attack, for example, opening an email attachment [NIST]; V.ER Specifies if the vulnerability associated with the attacking path can be exploited remotely or locally [NIST]; V.AN Specifies if the associated vulnerability requires valid user account on the victim computer [NIST]; V.TF A Specifies if the fixes of the associated vulnerability are publicly available in terms of Patch, Advisory, or None [NIST]; V.GP How popular the vulnerability is exploited in a general sense, for example, if listed as top 20 by authority organizations such as SANS. V.NP The network port(s) that the vulnerability uses, if there is any; 41 Table 1: Continued V.TS The level of compromise to the victim host if exploited in terms of None, Partially, or Completely to confidentiality, integrity, or availability [NIST]; A.TN The type of potential attackers, for example, insiders, hackers from internet, etc. A.AL The level of authentication that the attackers of A.TN have on host H. A.R If the attackers of A.TN have Local or Remote access to host H. A.SL The estimated skill level of the attacker in terms of Low, Medium, High A.GS The group size of this type of possible attackers A.MT The level of motivation of this type of attackers These attributes are practically obtainable. We have established a vulnerability database that contains 27,400 published COTS vulnerabilities affecting 31713 COTS software based on the comprehensive NIST National Vulnerability Database (NVD) [NIST]. It extends NVD by adding severity ratings, recommended solutions, and network ports and protocol information from other sources such as Microsoft, SecurityFocus, Frsirt, SANS, and CERT. In addition, security managers usually are aware of the stakeholder value propositions, the value dependencies on the IT infrastructure, the installed COTS software, as well as the most-likely attackers. 4.3 Value Driven Evaluation of Attack Scenarios As a descriptive model of attack scenarios, the adverse impact of an attack path can be characterized in terms of confidentiality, integrity and availability (CIA) [G. Stoneburner, 2002]. However, CIA is a value neutral concept and does not reflect the utility function of stakeholders. In order to capture the stakeholders’ value perception 42 of the attack path impact, we firstly identify the system relevant key stakeholders/values. Then we establish a set of evaluation criteria based on the identified key stakeholder values. Finally, we evaluate the system CIA against each criterion. Clearly, some of the stakeholder values can be quantified in terms of tangible units such as dollars. However, some of them are not, for example, the organizational reputation. Therefor, we propose an evaluation method based on Figure of Merit method [Boehm, 1981] and AHP [T. L. Saaty, 1980] to evaluate the impact severity of Attack Paths. Traditionally used in Decision Sciences as well as tradeoff analysis in System Engineering, the Figure of Merit method has unique strength in multiple criteria evaluation that involves quantitative and qualitative criteria. An example of applying AHP in security investment evaluation can be found in a recent work of Bodin et al [L. D. Bodin, Feb. 2005]. Case Study Server X is an important development server that communicates with a sensitive USC database. Recently, a hacker crime against a USC database-driven web application was reported by Sci-Tech-Today on April 21, 2006 [sci-tech-today]. In order to improve system security, the security manager identified three alternative plans: 1) patch server X and harden its Operating System; 2) build a firewall segment for server X to deny all access from remote non-USC IPs; 3) tighten user accounts. Given a budget around $6,000, the security manager is wondering what plan could produce the best cost-effectiveness for improving the security of Server X. Assuming the labor cost is $125 per hour, the estimated costs are: 43 Table 2 Security Cost Estimation of the Case Study Plan Practice Cost 1 Apply software patch and work around (manually) $250 per patch 2 Deploy and configure a firewall segment $6,884 3 Tighten user account control (might affect other systems) $750 Step 1: Identify Key Stakeholders/Values and Establish Evaluation Criteria In the critical first step to evaluate the severity of attack paths, we identify stakeholder values that are dependent on IT system security. Furthermore, we establish a set of criteria to evaluate the value-criticality of attack paths. For example, Table 3 summarizes the top-level USC stakeholders, stakeholder value propositions, and corresponding criteria weights that are relevant to Server X. 44 Table 3 USC ITS Case Study Stakeholder Value Propositions Stakeholders Weight Criteria S1 * S2 * S3 * Organizational Value Description 0.095 1. Productivity + + ++ Help faculties, students and staff create, acquire, and disseminate knowledge. 0.738 2. Regulation + ++ Comply with applicable federal, state, and local laws 0.167 3. Privacy 0.648 a. Student 0.230 b. Faculty 0.122 c. Staff + ++ ++ ++ ++ Protect the privacy of software, files, and materials stored on or transmitted by university computer equipment * S1: Students; S2: Faculties; S3: ISD Staff +: relevant; ++: highly relevant The weight values in the first column are determined through AHP [L. D. Bodin, Feb. 2005; T. L. Saaty, 1980]. For example, the calculation of top-level criteria weights is illustrated in Table 4. The number in each cell represents the value pair- wise relative importance: number of 1, 3, 5, 7, or 9 in row i and column j stands for that the stakeholder value in row i is equally, moderately, strongly, very strongly, and extremely strongly more important than the stakeholder value in column j, respectively. By definition, cell(i,j) equals 1/cell(j,i). In order to calculate weight, each cell is divided by the sum of its column, and then averaged by each row [L. D. Bodin, Feb. 2005]. The results of the final averaged weight are listed in the bolded Weight column in Table 4. The sum of weights equals 1. Similar process is used to determine the weights for sub-criteria. 45 Table 4 Pair-wise Comparison across ITS Values Productiv ity Regulati on Privacy Weig ht Productiv ity 1 1/7 1/2 0.095 Regulatio n 7 1 6 0.738 Privacy 2 0.2 1 0.167 Step 2: Evaluate Compromise Scenarios for Each Host Given the adverse impact of an attack path can be specified in terms of CIA of the victim host [G. Stoneburner, 2002], we evaluate the relative severity of loss of confidentiality, integrity and availability against the evaluation criteria established in step 1. The example results of our case study are summarized in Table 5. Cell(i,j) stands for the evaluation score of compromise condition j against evaluation criteria i. The weights of each row are derived through AHP pair-wise comparison. The sum of the weights of each row equals 1. In our case study, since the stakeholder values are additive, we used weight of sum as the evaluation function. The confidentiality and integrity have high score because they might result in regulation violation that has a high penalty. Thus, for a given Attack Path, we can look up its value impact score by S.VN (stakeholder value name) and V.TI (type of impact) in Table 5. Obviously, the score values reflect the stakeholder value priorities. 46 It is important to note that selecting an appropriate evaluation function based on the stakeholder values is a critical step toward generating meaningful evaluation scores. An example of an inappropriate evaluation function that could lead to wrong selection of technologies was presented in Boehm’s previous work [Boehm, 1981], Chapter 15, Pp 223-242. Table 5 Security Breach Scenario Evaluation Score Server X Weight Criteria Confidentiality Integrity Availability 0.095 1. Productivity 0.12 0.65 0.23 0.738 2. Regulation 0.63 0.26 0.11 0.167 3. Privacy 0.648 a. Student 0.230 b. Faculty 0.122 c. Staff 0.78 0.54 0.60 0.16 0.30 0.20 0.07 0.16 0.20 Evaluation Score 0.593 0.287 0.121 Step 3 Modeling Attackers The attacker is another significant factor that drives the severity of security incidents [G. Stoneburner, 2002]. T-MAP models attacker groups with attributes of skill level, group size, and the motivation, represented by A.SL, A.GS and A.MT, respectively. Often, the accurate values of these attributes are difficult to estimate, for example, the group size of hackers from internet. Thus, in order to compare and prioritize attacker groups, we employ AHP to determine the relative weights of these attributes. 47 In our case study, the security manager identified three top attacker groups: AG1: the skilled internet hackers who know how to escalate user privileges on the victim; AG2: less skilled internet hackers who don’t know how to escalate user privileges; AG3: insiders who usually have user accounts. The relative ratings of the A.GS, A.SL, and A.MT for each group were calculated through AHP as summarized in Table 7. For example, the relative group size ratings (A.GS) are calculated through AHP pair-wise comparison as illustrated in Table 6. Furthermore, not all vulnerabilities are exploitable for all attackers. For example, some vulnerability requires attackers to have valid user account on the victim host, thus they are not vulnerable to hackers who do not have such privileges. Some vulnerability requires attackers to have local access to the victim host, so they are not vulnerable to hackers who do not have local access. T-MAP reflects these differences in Attack Path generation based on the values of Authentication Level and Remote/Local Access, which are represented by A.AL and A.R, respectively. The information of Attacker Groups of AG1, AG2, and AG3 in our case study is summarized in Table 8 as follows. Table 6 Pair-wise Comparison of Attacker Group Size AG1 AG2 AG3 Weight AG1 1 1/9 1/3 0.07 AG2 9 1 7 0.78 AG3 3 1/7 1 0.15 48 Table 7 Ratings of A.GS, A.SL, and A.MT Description AG1 AG2 AG3 A.GS Group Size 0.07 0.78 0.16 A.SL Skill Level 0.63 0.26 0.11 A.MT Motivation 0.72 0.22 0.06 Table 8 Vulnerability Exposure to Attackers AG1 AG2 AG3 A.R (Access) Remote Remote Local A.AL (Authentication Level) User None Admin Step 4 Assess Difficulty Level of Exploiting Attack Path We model the difficulty level of exploiting an attack path based on the pioneer work of CVSS [M. Schiffman]. The ratings of all relevant attributes are between 0 and 1. Because of page limit, only part of the table is shown for illustration: 49 Table 9 Vulnerability Technical Attribute Ratings (Partial) Attribute Rating Rating alue Remote 1.0 V.ER Exploit Range (Access Vector) * Local 0.7 Official Fix 0.85 Temporal Fix 0.9 Work around 0.95 V.TFA Type of Fix Available (Remediation Level) * None 1.0 Required 0.6 V.AN Authentication Needed (Authentication) * Not Required 1.0 Listed as Top 1.0 V.GP Vuln. General Popularity Not Listed as Top 0.8 Attractive 1.0 Neutral 0.8 H.AA Attractiveness of Asset Computer Not Attractive 0.7 [Note] Fields marked with * are referenced from CVSS [M. Schiffman] 4.4 T-MAP Weighting System After all the Attack Path attribute-ratings determined through the above value driven evaluations, we establish the T-MAP weighting system to reason and quantify the threat associated with given COTS system. Furthermore, we propose a stakeholder value sensitive method to assess the effectiveness of common security practices such as patching and firewalls. 50 A. Weight of Attack Paths T-MAP scores the severity of the Attack Path by a numeric weight. Based on the classic risk calculation formula: Risk = Probability * Size of Loss, we calculate the weight of each Attack Path by multiplying its threat relevant attribute ratings together. Definition 3: Weight of Attack Path. For given Attack Path P, define: ∏ = i i Attribute P Rating P Weight ) . ( ) ( where P.Attribute i enumerates once each of the Probability Relevant and the Size-of- Loss Relevant attributes of P. The Probability Relevant Attributes include H.AA, V.ER, V.AN, V.AC, V.TFA, V.GP, A.GS, A.MT, and A.SL; The Size of Loss Relevant Attributes, including S.IW, and V.TS; Other attributes are Descriptive, including S.VN, H.HN, C.CN, V.VN, V.TI, A.TN, A.AL, and A.AR. (Attributes descriptions are in Table 1.) By nature, the weight serves as a heuristic or proxy of the risk associated with the Attack Path. It is a rating number ranges between 0 and 1. 51 B. Total Threat of IT System Under the assumption that the more attacking scenarios (or attacking paths) exist, the more an organization’s core values are vulnerable, we use the total weight of all attacking paths to quantify the overall security threat to stakeholder values. Definition 4: Total Threat. For given Structured Attack Graph G, define: ∑ = i i AttackPath Weight G t TotalThrea ) ( ) ( , where i varies from 1 to the total number of attacking paths of G and AttackPath i represents the i-th Attack Path of G. C. ThreatKey As illustrated in Figure 2 in Section 4.2, each node of the Structured Attack Graph is associated with a set of attack paths that pass through that node. Thus, by comparing the total weight of all attacking paths passing through the node, we can determine which entities are most vulnerable. 52 Definition 5: ThreatKey. For a given node N in a Structured Attack Grraph G, define: ∑ = i i AttackPath Weight N ThreatKey ) ( ) ( , where i varies from 1 to the total number of attacking paths that go through node N, and AttackPath i enumerates all the Attack Paths that go through N. Therefore, the greater the value of the ThreatKey of a node, the more vulnerable that node is. For example, those vulnerabilities which have a higher Threat Key value should be patched at a higher priority than others. The COTS software associated with higher ThreatKey values is more vulnerable than those that have lower values. And the attackers that have higher ThreatKey values are more dangerous than others. A Fast Algorithm to Calculate ThreatKey Although by definition, the ThreatKey of a node is the sum of the weight of all attack paths that go through that node, we found the calculation of the ThreatKey does not necessarily require emulating the Attack Paths. A fast algorithm with a time complexity of O(n) has been devised for ThreatKey calculation, where n is the sum of total number of nodes of all layers in G and the total number of edges in G. Specifically, for a given Structured Attack Graph G, the weight of each node can be calculated by multiplying the ratings of all its Attack Path relevant attributes 53 together. So, the weight of a Attack Path can be computed by multiplying the weights of all nodes on that Attack Path. We observed that in a simplified Structured Attack Graph as shown in the following Figure, for example, if ThreatKey(V 21 )= ∑ Weight(AttackPath i ) then ThreatKey(V 21 )= TopDownWeight*BottomUpWeight/W(V 21 ), where TopDownWeight = W(V 21 )*[ W(V 31 )+W(V 32 )] BottomUpWeight = W(V 21 )*[ W(V 11 )+W(V 12 )+W(V 13 )] Figure 10 ThreatKey Calculation for a Three-Layer-Attack-Graph This calculation can be expanded to a standard Structured Attack Graph. For each node, all the attack paths that go through it form two trees, one above it and one below it, as illustrated in Figure 10. The BottomUpWeight represents the ThreatKey weight contributed by the lower tree; the TopDownWeight represents the ThreatKey weight contributed by the upper tree. Both BottomUpWeight and TopDownWeight of 54 each node can be calculated recursively, as illustrated in pseudo-code in Algorithm 3 and 4. Algorithm 2 calculates the ThreatKey of each node. Furthermore, for prioritizing purpose, the nodes can be sorted in O(n*lg(n)) time with quick sort or similar algorithms. Algorithm 2 Void FastCalculateThreatKey(Structured_Attack_Graph G) Return: A updated Structured Attack Graph G wherein the ThreatKey of Each Node calculated 1: for(each v in G.Va U G.Vv U G.Vc U G.Vh U G.Vs) { //multiply the severity drivers of each vertices together as the initial severity weight 2: v.initialThreatKey = ∏ v.Attribute i ; 3: } 4: for(each v in G.Va) { 5: CalculateBottomUpWeight(G, v); 6: } 7: for(each v in G.Vs) { 8: CalculateTopDownWeight(G, v); 9: } 10: for(each v in G.Va U G.Vv U G.Vc U G.Vh U G.Vs) { //multiply the severity drivers of each vertices together as the initial severity weight 11: v.threatKey = v.bottomUpThreatKey * v.topDownThreatKey / v.initialThreatKey; 12: } 13: return; Algorithm 3 float CalculateBottomUpWeight (Structured_Attack_Graph G, Vertice v) Return: A updated Structured Attack Graph G with temp results toward getting ThreakKey 1: v.bottomUpThreatKey = 0; 2: if (v ∈ G.Vs) { //reach the top layer, the values in the graph 3: return v.initialThreatKey; 4: } else { 6: for(each v’ where v’ (is in the adjacent upper layer of v) and 55 (v is associated with v’) ) { 7: v.bottomUpThreatKey += v.initialThreatKey * CalculateBottomUpWeight(G, v’); 8: } 9: return v.bottomUpThreatKey; 10: } Algorithm 4 float CalculateTopDownWeight (Structured_Attack_Graph G, Vertice v) Return: A updated Structured Attack Graph G with temp results toward getting ThreakKey 1: v.topDownThreatKey = 0; 2: if (v ∈ G.Va) { //reach the bottom layer, the attackers in the graph 3: return v.initialThreatKey; 4: } else { 6: for(each v’ where v’ (is in the adjacent lower layer of v) and (v is associated with v’) ) { 7: v.topDownThreatKey += v.initialThreatKey * CalculateTopDownWeight(G, v’); 8: } 9: return v.topDownThreatKey; 10: } D. Effectiveness of Security Practices Under T-MAP framework, the goal of many security practices can be understood as to block a certain set of Attacking Paths from the existing Attacking Path set. For example, Firewalls are to block those Attacking Paths that pass through controlled network ports. Enforcing physical access to important computers is done to block those Attacking Paths that require local access. Patching software vulnerabilities is done to block those Attacking Paths that are caused by the vulnerabilities. 56 In this sense, the effect of a security practice can be simulated by removing the corresponding attack paths and nodes that this security practice can suppress in the graph. For example, the effect of vulnerability patching can be simulated by removing all Attacking Paths that have vulnerability patches available from the Attacking Path set that is before applying patches. Definition 6: Effectiveness of Security Practices For a given security practice SP, Effectiveness(SP)=1-TotalThreat(AfterSP)/TotalThreat(BeforeSP) It represents the percentage of the total threat weight (before the security practice) that has been suppressed because of taking that security practice. 57 Chapter 5 ITS Case Study We assessed the stakeholder/value impacts of possible compromise scenarios of server X as well as and identified/evaluated the major sources of possible attackers in section 4.3. With the T-MAP weighting system established, we continue our case study in the following steps. The results are the snapshot at the time of Sept. 2006 when the case study was conducted. Step 6 Load Vulnerability Information In this step, 17,731 vulnerabilities published by NIST from 1999 to 2006 are loaded into the Tiramisu tool. Figure 11 Screenshot of Load Vulnerability Database 58 Step 7 Determine the COTS Software Installed The COTS software that is installed on Server X is summarized in Table 10. Tiramisu takes this information as input. Table 10 COTS Software Installed on Server X Host COTS Name Version Vendor Solaris 9.0 SUN Directory Server 5.2.x SUN Oracle9i DBMS 9.2.0.4 Oracle Server X Oracle9i Client 9.2.0.2 Oracle Step 8 Calculate Attack Paths This step calculates the initial attack paths and associated severity weights of the current system without any security protection (Figure 12). The output shows that there are a total of 1314 attack paths with a total weight of 29.311. In addition, there are 106 COTS vulnerabilities reside in the current system. The top 3 vulnerabilities are CVE-2002-0858, CVE-2003-0722, and CVE-2004-1351 [NIST], associated with ThreatKey values of 1.539, 1.157 and 0.984, respectively. 59 Figure 12 Screenshot of Attack Path Calculation Results Step 9 Determine What Type of Attack Path Can Be Suppressed by Each Security Practice Inspired by the Table of Effectiveness Estimates in SAEM proposed by Butler [Butler, May 2002, Table 6], a Suppressing Matrix as shown below is used to summarize the effects of the effectiveness of each alternative security investment plan. 60 Table 11 Suppressing Matrix* Security Investment Plans Attack Path Attributes Properties Firewall Patch User Ctrl. Data Encryption Backup Server Digital Signature Official Fix 80% Temporal Fix 10% Work around 10% V.TFA None V.AN Required 100% Confidentiality 95% Integrity 100% V.TI Availability 80% AG1 - 90% 10% AG2 - 90% 90% AG3 - 100% * Attribute Definitions are in Table 1. AG1-3 are attackers, see Section 4.3 The first column represents the relevant Attack Path attributes. The second column is the possible ratings of each attribute. The percentage p in table cell at row i and column j stands for the security practice of column j can suppress p percent of the threat of those attack paths that have the attribute rating represented by row i. The suppressing percentages should be justified by experienced security managers. For example, in our case study, our client estimated that 90% of the threats from internet hackers (Attack Group AG1 and AG2) can be suppressed by firewall, thus the rows representing remote attackers are marked with 90% in column 1. The column of “Patch” represents that if the plan is to patch COTS software, in practice, only about 80% of the patches are applicable because of system compatibility and stability considerations. The column of “Acct. Ctrl.” represents that by enforcing strict user account control, it can almost avoid all the attacks that need valid user account on Server X. In addition, by periodically changing system password and 61 restricting system access, it can suppress almost all insiders, about 90% of the total threat from unskilled hackers; however, it can only suppress about 10% of the skilled hackers. The Tiramisu tool screenshot of evaluating the effectiveness of patching Server X is illustrated in Figure 13 as follows. Figure 13 Screenshot of Security Practice Simulator The Tiramisu assessment outputs for each security practice alternatives are summarized as follows. We found for Server X, the vulnerability of CVE-2005- 1495, 2002-2089, 1999-0795, and 1999-0568 do not have patch or workaround available by the time the paper is written. Assuming the top 80% of the vulnerability has applicable patches, there is about 106*80% equals 85 applicable patches. Thus the cost of applying patches increases linearly along with the number of patches to apply at a cost of $250 per patch. 62 In order to determine the optimal plan for a budget around $6,000, we plotted the relationship between investment Effectiveness and the number of patches applied, as shown in Figure 14. Assuming the vulnerabilities that have higher ThreatKey values are fixed first, clearly, the security manager should first tighten the user account on Server X, and then patch the top 22 vulnerabilities to achieve the best Effectiveness while stay in the budget. According to the Tiramisu output, the investment Effectiveness of this plan is 87.6%. Figure 14 Optimal Security Practice Plan It is important to point out that Figure 14 does not directly answer the question what will be the optimal amount to invest. As the percentage measure of the total threat weight reduced by certain security practice, Effectiveness does not reflect the absolute value of security investment benefit, thus cannot be directly used for Net Value analysis. 63 Figure 15 Sweet Spots of Investment Under the assumption that Net Value(n) = A*Effectiveness(n) – Cost(n), where A is the net value of perfect security and n is the number of patches to apply, we plotted the Relative Net Value (RVN) analysis for Alt.2 for our case study, where RVN(n)=NV(n)/NV(0). As shown in Figure 15, the sweet spots is achieved at n where 0 ) ( = dn n dRVN . The trend meets the common sense that the more valuable perfect security, the higher optimal amount to invest should be. 64 Chapter 6 Tool Support – Tiramisu We have implemented an automated T-MAP software test-bed with a project code of Tiramisu at USC. The Tiramisu takes three inputs: the general vulnerability information, an organization’s IT infrastructure information, and how an organization’s business values depend on its IT infrastructure. It calculates a complete list of associated Attacking Paths, and outputs the overall threats in terms of the total weights of Attacking Paths. Currently, the system capabilities include: • Automate vulnerability information collection from multiple authority sources including NIST NVD, CERT, Symantec BugTraq, Microsoft, FrSIRT, SANS, ISS. • Allow security professionals to specify the COTS software in their IT systems from a comprehensive COTS software inventory list. • Prioritize COTS vulnerabilities under given stakeholder value/utility context. • Estimate the effectiveness of common security practices such as vulnerability patching, firewall, data encryption, etc. • Export report data into Excel. 65 Overall, the Tiramisu component model is illustrated in Figure 17 as follows. The vulnerability information is collected from authority organization through the scrawling engine, and stored into the vulnerability DB. The Tiramisu Front End performs data analysis and interacts with system users. Figure 16 Tiramisu Component Model Internally, the Tiramisu employs layered software architecture as illustrated in Figure 16. From the bottom to the top, the “Automated Data Collecting Engine” collects the latest published vulnerability reports from CERT/CC, NIST, SANS, SecurityFocus, Symantec, and Microsoft websites automatically, formats and populates the data into the second layer “Data Storage and Management”. Currently our database contains information on 27,400 vulnerabilities that have been published on NIST since 1999 with extended information such as associated network ports, recommended solutions (by CERT, NIST, SecurityFocus respectively), if listed as top vulnerabilities by SANS, and so forth. 66 Figure 17 Tiramisu Software Architecture The “Data Storage and Management” layer includes an XML database implementing the ontology of vulnerabilities, IT asset computers, organizations’ key business values, and the relationships among these entities. Through the GUI the users can easily specify the types of operating systems and COTS software installed on their IT servers, and specify how organizations’ core business values can be affected by security breaches such as compromises of confidentiality, integrity, and/or availability on different IT servers. The “Attacking Path Calculation and Management” layer consumes the data provided in the “Data Storage and Management” layer to generate a complete list of Attacking Paths, and calculates the severity weights of Attacking Paths based on user input. The “User Definable Security Practice Simulator” layer allows user to specify what kind of attacking paths can be blocked by certain security practices. This feature of calculating the total weight of attacking paths that are suppressed by 67 certain security practices helps security managers estimate the effectiveness of different security practices. The “Report Generator” organizes the system output into Microsoft Excel friendly format and save information upon user’s request. This capability allows user leverage the data processing function provided by Excel for further analyses. 68 Chapter 7 Validation and Results 7.1 Validation Methodology Validation Strategy Clearly, the foundation of T-MAP is based on effective COTS vulnerability prioritization under stakeholder value context. However, stakeholder value is a subjective and complex concept by nature, and is very difficult to measure and quantify. Therefore, as an overall strategy, we take an empirical approach to validate T-MAP through real-world case studies by comparing the vulnerability priorities generated by T-MAP and the priorities generated by experienced security professionals. Metrics IEEE defines accuracy as a quantitative measure of the magnitude of error [Engineers, 1990]. In T-MAP, we identify prioritization errors by a vulnerability ranking system through counting the mismatches between the priorities generated by the ranking system and the ones by experienced security professionals. Furthermore, we use inaccuracy as the metric to measure the performance of vulnerability ranking systems. 69 Specifically, assume there are n software vulnerabilities exist in a COTS system S, say v 1 , v 2 … v n . Not losing generality, we assume they are sorted by the truth of priority from high to low under system stakeholder value context. Now, if ranking system R prioritizes the n vulnerabilities from high to low in a different order, say v r1 , v r2 … v rn , we take the following steps to measure the inaccuracy of ranking system R in prioritizing the n vulnerabilities. Step 1: Construct the pair-wise comparison matrix as show in Figure 18 as below, where v 1 , v 2 … v n are marked on each dimension of the matrix respectively. Figure 18 Prioritization Clash Counting 70 Step 2: Enumerate all possible pair-wise comparisons for these n vulnerabilities with the matrix. Obviously, all the enumeration can be listed by the none-gray cells in the above matrix, wherein cell(a, b) represents the pair-wise comparison between v a and v b . For each none-grey cell(a, b), if the priority order of v a and v b given by ranking system R clashes (or disagrees) with the truth priority order, then we mark cell(a, b) with a ‘X’. Thus, there are a total of 2 ) 1 ( * 2 − = n n C n possible pair-wise comparisons whose results are recorded in the none-grey cells in the comparison matrix. Step 3: Now, we count the total number of clashes in the comparison matrix and represent it with m. Thus, intuitively, the inaccuracy of ranking system R in prioritizing v 1 , v 2 … v n can be calculated by: inaccuracy = ) 1 ( * * 2 2 ) 1 ( * 2 − = − = n n m n n m C m n Obviously, inaccuracy is a percentage between 0 and 1 representing the error ratio that ranking system R makes in prioritizing v 1 , v 2 … v n . Validation Steps Currently, T-MAP is validated through three representative case studies: two were conducted inside USC and one outside USC: (1) the Server X case study conducted at the USC Information Technology Services division (ITS), (2) the Student 71 Academic Progress Web System case study conducted at the Manual Art Senior High-school (MASH); and (3) the GreenBay Server case study conducted at CSSE. Each case study was conducted through the following steps: Step 1: For each project, identify the system key stakeholders and value propositions; Step 2: Establish a set of security evaluation criteria based on stakeholder value propositions; Step 3: Use the Tiramisu tool to enumerate and analyze attack paths based on comprehensive COTS vulnerability database (Currently, the Tiramisu tool vulnerability database contains 27,400 vulnerabilities information); Step 4: Evaluate the severity of each scenario in terms of numeric ratings against the evaluation criteria established in Step 2; Step 5: the security threat of vulnerabilities will be quantified with the total severity ratings of all attack paths that are relevant to this vulnerability; Step 6 The vulnerabilities will be prioritized based on their T-MAP severity ratings; Step 7 The vulnerabilities will be prioritized based on the ratings assigned by representative of current leading approaches (such as BugTraq/Symantec, CERT, Microsoft, NIST, and SANS, if applicable) 72 Step 8 A subset of the vulnerabilities is randomly selected from the full vulnerability list; Step 9 Experienced security professionals who have well understanding on both security vulnerabilities and the system stakeholder value propositions helped us prioritize the selected vulnerabilities; Step 10 The number of clashes between the ranking-system-generated vulnerability priorities and the security-professional manually generated priorities were counted for T-MAP as well as other selected representative approaches; It is important to note, because of the subjective nature of stakeholder values/utilities, the truth of the right prioritization of vulnerabilities is very difficult to obtain. Thus, we use opinions on prioritization from experienced security managers as the best approximation to the truth. By doing so, it involves a certain degree of threats to validation. So we proactively take mitigation actions to minimize the threats to validate by carefully selecting the security professionals that we were interviewing with, as discussed in Section 7.5. Step 11 The inaccuracy of different vulnerability ranking systems was compared to prove or disprove if T-MAP is better than mainstream value neutral approaches. 73 7.2 Case Study Results and Evaluation Our evaluation focuses on testing the hypothesis “whether or not if T-MAP can outperform current value-neutral approaches in COTS security vulnerability prioritization”. In this section we present the case study results in detail and summarize results evaluations. 7.2.1 USC ITS Server X Case Study Results As the main demonstration example, we continue the discussions on the USC ITS ServerX case study in Section 5. Clearly, the correctness of the economic analyses in Figure 14 and Figure 15 in Section 5 largely relies upon a correct prioritization of COTS vulnerabilities. Thus, assuming security manager’s ranking is a good approximation to the truth, we test how well T-MAP method can captures the security manager’s opinion on vulnerability rankings in the Server X case study. Particularly, we sampled a set of Server X vulnerabilities randomly from the Tiramisu outputs. Then the security manager was asked to prioritize the sample vulnerabilities manually. His rankings were compared to the Tiramisu rankings as illustrated in Figure 18. Excluding the irrelevant outlier of CVE-2003-0064, the total number of clashes counting is 2. The total number of pair-wise comparisons made equals = 2 8 C 1+2+…+7=28. Thus, the prioritization inaccuracy equals 2/28=7.1%. The regression shows an R square value of 0.86 indicating a strong fit. The security manager ranked CVE-2003-0064 as the least because the relevant application dtterm is not enabled on Server X at all; he ranked the CVE-2005-2072 at a higher priority 74 because a program is running as setuid with root privilege on Server X, which involved more security risks. Figure 19 Vulnerability Ranking Comparison: Security Manager vs. T-MAP Furthermore, in comparisons between the security manager’s and the T-MAP’s outputs of the priorities of “organization values under most threat” and “top attackers”, the Tiramisu tool generated results well matching the security manager’s opinions as well. The tool recommended optimal plan was convincing to the security manager and matched his experience on the Server X security context. Also, recall the previous economics analysis on security patching in Figure 14, the economic curve shows that the top 30.2% vulnerabilities (32 out of 106) caused about 80% of the total threat in this case study. This result moderately matches the well known “20/80 rule” in security risk assessment. The top vulnerabilities 75 identified by the tool well met the security manager’s experience except for a few outlier vulnerabilities that are associated with disabled services. 7.2.2 Manual Art Senior High School (MASH) Case Study Results System Background Another case study was conducted on the Student Academic Progress Web System at the Manual Art Senior High School in Los Angeles. The goal of this system is to “provide a secure, user-friendly and always-available system to its students, and especially, their parents so that the website (i.e, website system) can be used at home, at local libraries or anywhere the Internet is available to help them understand where students stand vis-à-vis LAUSD graduation requirements, and allow them to realistically assess whether or not they are in the running for admission to specific sets of colleges.” Similar to the ITS ServerX, the following steps are taken: (1) identified the system stakeholder value propositions, (2) prioritized the COTS security vulnerabilities with T-MAP, and (3) compared the T-MAP priorities with the network administrator’s prioritization on a subset of randomly-selected vulnerabilities. The results are presented as follows. 76 Stakeholders and Values Propositions Taking the similar steps as the previous case study, we identified the system key stakeholder value propositions as shown in Table 12. The criteria weights are derived from AHP through interviewing with the system administrator. Table 12 MASH Server Case Study Stakeholder Value Propositions Stakeholders Weight Criteria S1 * S2 * S3 * Organizational Value Description 0.767 1. Provide the student parents easy access to their children’s academic progress information + ++ ++ Help students and their parents understand where students stand vis-à-vis LAUSD graduation requirements, and allow them to realistically assess whether or not they are in the running for admission to specific sets of colleges 0.085 2. Regulation ++ Comply with the student academic records regulations required by the city of Los Angeles 0. 148 3. Privacy + + Protect the student privacy mainly on academic progress (*) S1: MASH students; S2: student parents; S3: school academic counselors +: relevant; ++: highly relevant Through the interview, we found privacy is not the primary concern of this system because there is no sensitive information involved in the system such as SSN. However, there is a high requirement for the system to provide accurate student 77 academic information so that student parents can easily access their children’s academic progress at the high school. Based on the system stakeholder value context, we derived the severity score of possible security breach scenarios through AHP as follows: Table 13 MASH Case Study Security Breach Scenario AHP Evaluation Score MASH Server Weight Criteria Confidentiality Integrity Availability 0.767 Criteria 1 in Table 12 0.058 0.595 0.347 0.085 Criteria 2 in Table 12 0.732 0.216 0.061 0.148 Criteria 3 in Table 12 0.474 0.474 0.053 Evaluation Score 0.192 0.529 0.279 COTS Components As a MS Windows based system, the COTS components in MASH Server are listed in the following table. Table 14 COTS Software Installed on MASH Server Host COTS Name Version Vendor IIS 6.0 Microsoft SQL Server 2000 SP3 Microsoft ASP .Net 1.1 SP1 Microsoft MASH Server Windows Server 2003 SP1 Microsoft 78 Vulnerability Ranking Comparison – System Manager vs. T-MAP With the above stakeholder value proposition and COTS information as input, T-MAP identified 94 system vulnerabilities in total when the case study was conducted. Twelve vulnerabilities were selected randomly from the list and the system manager was asked to prioritize them. (The system manager has 29 years experience in computer systems, network administration, and security.) His rankings are compared with the T-MAP rankings as follows: Figure 20 Vulnerability Ranking Comparison: System Manager vs. T-MAP As shown, excluding the outliers, the total number of clash counting for T-MAP is 4. The total number of pair-wise comparisons made equals = 2 9 C 1+2+…+8=36. Thus the prioritization inaccuracy equals 4/36=11.1%. The R square value of 0.825 between T-MAP ranking and the system manager’s ranking indicates a strong fit. There are 3 out of 12 selected vulnerabilities which are outliers. Two of them are 79 associated with disabled services and not relevant. For illustration purpose, we plotted only one of the outliers, the CVE-2005-1214, as shown in red square. The system manager assigned CVE-2005-1214 low priority because he was very confident that the local system users will not make such mistakes that are necessary to enable such an exploit. 7.2.3 CSSE Greenbay Server Case Study Results Background Greenbay is a mission critical web server at CSSE. It provides web services to the CSSE hosted graduate level Software Engineering course which involves about 200 - 300 graduate students each year. As a key eServices projects collaboration point, it also serves as a file server for about 20-30 on-going CSSE eServices projects each year that involves the CSSE clients from a wide range of organizations across academic, industry, and none-profit sectors, for example, the USC Library, IBM/Rational, African Millennium Foundation, etc. Besides the supports to CSSE Software Engineering course and eServices projects, it is the file server for several research projects conducted at the center as well. Stakeholders and Values Propositions The system key stakeholder value propositions for CSSE GreenBay case study are shown in Table15. The criteria weights are derived from AHP through interviewing with the CSSE system administrator and a CISSP security expert at CSSE. 80 Through the interview, we identified the system key stakeholders are researchers, graduate students, and the CSSE eService projects clients. The system security has strong impacts on the daily research and teaching activities. In addition, it may affect the CSSE reputation because of the involvement of eService projects clients and large number of students. The regulation violation is not a concern for this system. Table 15 CSSE Greenbay Case Study Stakeholder Value Propositions Stakeholders Weight Criteria S1 * S2 * S3 * Organizational Value Description 0.723 1. Productivity ++ ++ + Support the on-going research and teaching activities at the Center 0.061 2. Privacy + The graduate students may have some grading information posted on the server, but it is not a major concern since SSN is not used at USC for course purposes any more. 0. 216 3. Reputation ++ If the Greenbay server get compromised, there may be significant impacts on the CSSE reputation because of the large number of users to the system from different sectors (*) S1: Researchers (faculties and other CSSE researchers); S2: Graduate students; S3: CSSE eServices projects clients +: relevant; ++: highly relevant Based on the stakeholder value context, we derived the severity score of possible security breach scenarios through AHP as follows: 81 Table 16 GreenBay Case Study Security Breach Scenario AHP Evaluation Score MASH Server Weight Criteria Confidentiality Integrity Availability 0.723 Criteria 1 in Table 15 0.055 0.545 0..400 0.061 Criteria 2 in Table 15 0.819 0.091 0.091 0.216 Criteria 3 in Table 15 0.058 0.596 0.347 Evaluation Score 0.103 0.528 0.370 COTS Components As a Linux based system, the COTS components in CSSE Greenbay Server are listed in Table 17 as follows. Table 17 COTS Software Installed on CSSE Greenbay Server Host COTS Name Version Vendor PHP 5.0.4 PHP Apache 2.0.54 Apache Tomcat 5.0.19 Apache Fedora Core 3 Redhat Greenbay MySQL Server 4.1.12 MySQL Vulnerability Ranking Comparison – System Manager vs. T-MAP Based on the above stakeholder value proposition and COTS input, T-MAP identified 44 system vulnerabilities for the system when the case study was conducted. A total of fifteen vulnerabilities were selected randomly from the list and 82 a CISSP certified security expert at CSSE was asked to prioritize them. Her rankings are compared with T-MAP output as plotted in Figure 21. As shown, excluding the outliers, the total number of clash counting is 7. The total number of pair-wise comparisons equals = 2 12 C 1+2+…+11=66. Thus, the prioritization inaccuracy equals 7/66=10.6%. The R square value of 0.86 between T- MAP ranking and the system manager’s ranking indicates a strong fit. There are 2 out of 14 selected vulnerabilities are not relevant to the system because one of them is associated with the disabled Certificate Revocation List services which are not relevant; the other one is only exploitable by local users through complicated steps, which is very unlikely to happen according to the security expert. Figure 21 Vulnerability Ranking Comparison: System Manager vs. T-MAP 83 7.3 Testing Hypotheses Recall the hypotheses stated in Section 1.3, our focus is to test if T-MAP, the proposed stakeholder value sensitive approach, can outperform the current State of The Art value-neutral approaches. This hypothesis is tested by measuring and comparing the prioritization inaccuracy across the vulnerability ranking systems of T-MAP, CVSS v2.0, CVSS v1.0, IBM ISS, and Microsoft in the case studies conducted. The comparison results are shown in Figure 22, 23, and 24. Figure 22 ITS Case Study Result Comparison: TMAP vs. Value Neutral Approaches 84 Figure 23 MASH Case Study Result Comparison: TMAP vs. Value Neutral Approaches 85 Figure 24 Greenbay Study Result Comparison: TMAP vs. Value Neutral Approaches Table 18 compares the prioritization performance across T-MAP and other mainstream value neutral approaches. Both CVSS v2.0 and CVSS v1.0 ranks are collected from the default score (base score) recommended in the NIST National Vulnerability Database (NVD). The case study data is categorized by columns. The ranking approaches are listed by rows. Some of the cells are marked with “N/A” because not all the vulnerabilities involved in the case study are available in the corresponding ranking system thus the ranking data is not available. For example, the COTS systems for ITS Server X and CSSE GreenBay case study are Unix/Linux based systems, thus the vulnerabilities in these two systems are not available in the Microsoft vulnerability database. 86 Table 18 Prioritization Inaccuracy Comparison across Approaches ServerX Case Study (Total Comparisons 28) MASH Case Study (Total Comparisons 36) GreenBay Case Study (Total Comparisons 66) # of Clashes Inaccuracy # of Clashes Inaccuracy # of Clashes Inaccuracy T-MAP 2 7.1% 4 11.1% 7 10.6% CVSS v2.0 5 17.9% 9 25% 27 40.9% CVSS v1.0 6 21.4% 12 33.3% 22 33.3% IBM ISS 9 32.1% 17 47.2% 46 69.7% Microsoft N/A N/A 21 58.3% N/A N/A The results demonstrate that compared to the existing mainstream value-neutral approaches, T-MAP achieved the lowest inaccuracy in all three case studies, as illustrated through box plotting as shown in Figure 25 below. Figure 25 Inaccuracy Comparison - Box Plots 87 Across all the three case studies, a total number of 28+36+66=130 pair-wise comparisons were made between the security professional’s priority and the ranking system recommended priority. The total number of clashes for T-MAP, CVSS v2, CVSS v1, and IBM ISS are 13, 41, 40, and 72 respectively. The overall inaccuracies for these approaches are 10.0%, 31.5%, 30.8, and 55.4%, respectively. T-MAP achieved the lowest overall inaccuracy of 10.0%. T-test has been conducted on the three case studies to analyses the inaccuracy differences between T-MAP and other value-neutral approaches respectively as shown in the Appendix. The initial results are strongly against the null hypotheses made in Section 1.3. However, given the limited number of case studies available, it is still premature to make any statistical conclusions at this point. So, we conclude that T-MAP achieved observable lower inaccuracy than the other mainstream approaches in all the three empirical case studies that have been conducted. Based on the 130 pair-wise comparisons made between the priorities generated by security professionals and ranking systems across all case studies, the T-MAP achieved the lowest overall inaccuracy of 10%, compared to the overall inaccuracy value of 31.5% by CVSSv2, 30.8% by CVSSv1, 55.4% by IBM ISS, and 58.3% by Microsoft. The inaccuracy measurement to Microsoft vulnerability ranking system was based on 36 pair-wise comparisons from one case study only. The initial t test results based on existing case studies are strongly against the null hypothesis with p values ranging from 0.044 to 0.118. However, this observation is based on a very limited number (three) of case studies and cannot be generalized statistically. 88 As a baseline, these results can at least serve as empirical evidences that T-MAP outperformed other mainstream value neutral approaches in COTS security vulnerability ranking. 7.4 Lessons Learned: Top Reasons for Ranking Mismatches Through out the case studies, we observed T-MAP made over-estimates and under- estimates compared to the security professional’s ranking. The top reasons of generating these mismatches are: • Disabled services/programs Often, vulnerabilities that are associated with disabled programs are considered to be less important or even irrelevant. • Importance of running services Security professionals tend to assign the vulnerabilities associated with critical operating system services higher ranks. • Security professional’s self-confidence on handling vulnerabilities The pre-conditions of exploiting a vulnerability may be different. If the security professional is very confident that certain pre-conditions will not be 89 easily met in his environment, he may assign the vulnerability lower ranks, vice verse. • Privileges of running programs Not all programs are running with the default privileges. Security professionals tend to assign the vulnerabilities associated with high privileges higher rank, vice verse. • Security manager’s confidence on insiders How much security professionals trust the insiders and the insiders’ technical skills may affect his rank to the locally-exploitable-vulnerabilities. 7.5 Threats to Validity Threat #1 The number of case studies can be accomplished is limited The feasible number of case studies that can be accomplished in my PhD time frame is limited. Thus, the conclusions will be based on a limited amount of project data set, and the results may involve certain degree of bias. Mitigations Given the limited time frame, this threat is inevitable and cannot be mitigated completely. The mitigation efforts made are: 90 • Selected diverse COTS systems from different platforms: specifically, the three case studies are conducted on Unix, Linux, and Windows based respectively. • Select representative systems and conduct case studies with experienced security professionals to assure the quality of the case study data. • Conducted case studies from real life systems inside and outside USC. • Explore more case studies in the future work. Threat #2 Using the Security Manager’s Opinion as Approximation of Truth Stakeholder value/utilities are largely subjective by nature and are difficult to measure objectively. In the case studies, we used security professional’s opinions as the approximation of truth to evaluate the performance of vulnerability ranking systems. However, unavoidably, the results may involve with security professionals’ individual bias. Mitigations In order to minimize this threat to validation, we conducted the case studies with experienced security professionals who are not only knowledgeable in security and vulnerability technologies, but also familiar with the organization value propositions, so that their opinions can serve as better approximations to the truth. The following table summarizes the profiles of security professionals who participated in our case studies. As shown, the average number of years of experience on security technology 91 and the organization of security professionals is 11.66 years and 10.33 years, respectively. Two out of three security professionals who participated our case studies are CISSP holders. Table 19 Security Professional Profiles Case study Number of Years of Security Experience Number of Years of in the organization Professional Certification ITS Server X 11 7 CISSP MASH Server 18 18 None GreenBay Server 6 6 CISSP Threat #3 The Comprehensiveness of T-MAP Vulnerability Database T-MAP requires a comprehensive and up-to-date vulnerability database to generate meaningful results. Mitigation T-MAP has grown its contents based on the current National Vulnerability Database which is the largest publicly accessible vulnerability database maintained by NIST. An automated tool has been developed to scrawl Internet resources across authority organizations such as ISS, Microsoft, SANS, CERT, NIST/CVSS, Symantec BugTraq to grow and maintain an update-to-date vulnerability database which is critical to T-MAP. By the time of the dissertation is written, the database contains more than 27,400 COTS vulnerability information. 92 Chapter 8 Using T-MAP over Software Life-cycles As presented, T-MAP is devised to (1) prioritize COTS security vulnerabilities with respect to stakeholder value context; (2) evaluate the security performance of COTS candidates; (3) estimate the effectiveness of common security practices such as Firewall, Patching, and Data Encryptions. An important question that still remains unanswered is “when and how T-MAP can be used in system and software life-cycle to maximize its value?” This section illustrates how T-MAP can be used in different software/system life-cycle processes in COTS intensive system development. Specifically, we discuss how T-MAP can be useful in four representative software/system life-cycle models: • Boehm’s spiral model [Boehm, May 1988]. • The ISO 15288 system engineering life-cycle model [ISO 15288, 2002]. • Royce’s Waterfall model [Royce, August, 1970]. • Boehm’s recent Incremental Commitment Model [Barry W. Boehm, Oct. 2007]. Table 20-23 summarizes the analyses. The T-MAP capabilities are categorized into (1) Vulnerability Prioritization, (2) COTS Evaluation and Selection, and (3) 93 Effectiveness Estimation of Security Practices, as shown in columns. The phases of each life-cycle model are listed in rows. The potential scenarios that T-MAP can help are described in the tables correspondingly. Table 20 Using T-MAP in Boehm's Spiral Model T-MAP Capabilities Software/System Life-cycle models Vulnerability Prioritization COTS Evaluation and Selection Effectiveness Estimation of Security Practices Inception (1) Determine the feasible level of security that available COTS can achieve; (2) Evaluate the security performance of COTS candidates based on cost- effectiveness analysis from security perspective (1) Determine the feasible range of the improvement that common security practices can make; (2) Determine strategy to assure system security; (3) compare and select security protection plans based on cost- effectiveness analysis Elaboration Incorporate security considerations in COTS selection; design system security protection mechanisms Construction Identify COTS vulnerabilities in a prioritized order and help to decide which ones to fix first Transition Identify high priority COTS vulnerabilities and apply necessary patches Maintenance Monitor COTS vulnerability updates and help to decide which ones to fix first (1)Re-evaluate and select the most effective security practices dynamically; (2)re-adjust security protection plans dynamically; (3) determine the high- payoff of security investment 94 Table 21 Using T-MAP in the ISO 15288 System Engineering Life-cycle Model T-MAP Capabilities Software/System Life-cycle models Vulnerability Prioritization COTS Evaluation and Selection Effectiveness Estimation of Security Practices Conceptualization (1) Determine the feasible level of security that available COTS can achieve; (2) Evaluate the security performance of COTS candidates based on cost- effectiveness analysis from security perspective (1) Determine the feasible range of the improvement that common security practices can make; (2) Determine strategy to assure system security; (3) compare and select security protection plans based on cost-effectiveness analysis Development Identify COTS vulnerabilities in a prioritized order and help to decide which ones to fix first Incorporate security considerations in COTS selection; design system security protection mechanisms Operation, Test & Evaluation Evaluate the system security risks that caused by COTS vulnerabilities Transition to Operation Identify high priority COTS vulnerabilities and apply necessary patches Operation, Maintenance Monitor COTS vulnerability updates and help to decide which ones to fix first (1)Re-evaluate and select the most effective security practices dynamically; (2)re-adjust security protection plans dynamically; (3) determine the high- payoff of security investment dynamically Replace or dismantle Evaluate the system security risks that caused by COTS vulnerabilities and make replacement/dismantle decisions 95 Table 22 Using T-MAP in Royce’s Waterfall Model T-MAP Capabilities Software/System Life-cycle models Vulnerability Prioritization COTS Evaluation and Selection Effectiveness Estimation of Security Practices Requirement (1) Determine the feasible level of security that available COTS can achieve; (2) perform relevant tradeoff/feasibility analysis; Design Incorporate security considerations in COTS selection; design system security protection mechanisms Choose security protection design that has best cost- effectiveness Construction Integration Identify COTS vulnerabilities in a prioritized order and help to decide which ones to fix first Testing Help to decide the priority order to test COTS vulnerabilities and patches Installation Help to decide the priority order to install COTS vulnerabilities patches Maintenance Monitor COTS vulnerability updates and help to decide which ones to fix first (1)Re-evaluate and select the most effective security practices dynamically; (2)re- adjust security protection plans dynamically; (3) determine the high- payoff of security investment 96 Table 23 Using T-MAP in the Incremental Commitment Model (ICM) T-MAP Capabilities Software/System Life-cycle models Vulnerability Prioritization COTS Evaluation and Selection Effectiveness Estimation of Security Practices Exploration Determine the feasible level of security that available COTS can achieve Determine the feasible range of the improvement that common security practices can make Valuation Evaluate the security performance of COTS candidates based on cost- effectiveness analysis from security perspective (1) Determine strategy to assure system security; (2) compare and select security protection plans based on cost-effectiveness analysis Architecting Incorporate security considerations in COTS selection; design system security protection mechanisms Design detailed security protection plan based on cost-effectiveness analysis Development1 Arch.2 Identify COTS vulnerabilities in a prioritized order and help to decide which ones to fix first Operation1 Dev.2 Arch. 3 Monitor COTS vulnerability updates and help to decide which ones to fix first (1)Re-evaluate and select the most effective security practices dynamically; (2)re-adjust security protection plans dynamically; (3) determine the high- payoff of security investment 97 It is important to point out that the above life-cycle processes models may not fit COTS intensive systems well because of the unique constraints and characteristics that COTS intensive systems have by nature [V. Basili, 2001; D. Reifer, 2004]. For COTS intensive systems, Boehm et al illustrated a risk driven spiral process framework for COTS Based Application (CBA) development. [Boehm et al, 2003]. A refined variance of this process with injected value based philosophy was presented in [Y. Yang, July/August 2005], as shown in Figure 7 below. As a process model that primarily helps the development of COTS Based Applications, the system maintenance is not covered in this process framework. Obviously, T-MAP has the best fit in its process phases of (1) “P1: Identify objectives, constraints, and priorities” and (2) the “P3: Assess COTS candidates”. Figure 26 The Composable CBA Assessment Process Element [Y. Yang, July/August 2005] 98 Chapter 9 Value Sensitive Firewall Rule Generation 9.1 The Problem Firewall is one of the most popular security practices that have been widely adopted for IT protections. According to the 2006 CSI/FBI survey, 98% of the organizations that participated in the survey deployed Firewalls. However, configuring Firewall policies has been very fallible and effort consuming. Ideally, a perfect set of Firewall rules opens exactly the necessary network ports and blocks all others. However, in practice, the ideal status is difficult to achieve because setting up controls for every network port may involve a large volume of rules, which may cause the following problems: (1) Significantly decrease the packet forwarding throughput on Firewalls and routers (2) Making the maintenance on Firewall rules more difficult and fallible: according to one of the security professionals whom we interviewed with, the practical number of maintainable firewall rules for single Firewall is around 50-150. 99 On the other hand, though, it is not always the less Firewall rules the better either, because reducing the number of Firewall rules will limit the granularity of control on network ports, and may significantly increase system security risks by opening unnecessary ports. The interesting tradeoffs here is between (1) security and maintainability; (2) security and usability. The key question is, how many Firewall rules will be ideal? Currently, many Firewalls are just configured based on the security administrator’s experience or best knowledge on vulnerability exploits through network ports. We are unaware of systematic approaches that explicitly inject system stakeholder values into Firewall policy generation, and guide security professionals make good decisions to balance the above tradeoffs. In this section, we present a USC CSSE approach to generated value-sensitive Firewall rules based on T-MAP. 9.2 Observation: Security Risk Distribution Over Network Ports Different network ports are associated with different risks. At USC CSSE, we collected the port information for 717 COTS vulnerabilities that were published between 1999 and 2006. The distribution of vulnerabilities over network port number is illustrated in Figure 27 as below. The X axis represents the TCP/UDP port number, and the Y axis represents the number of vulnerabilities that are associated 100 with the port. The only outlier that is not included is the Port 80: it is associated with 347 vulnerabilities which is significantly higher than other ports. The plot shows that opening the ports between 0-1000 is highly risky in general, which well matches the common sense. Figure 27 COTS Vulnerability Distribution over Network Ports (partial) Clearly, how system stakeholder values may be impacted by security vulnerabilities also drives the risk distribution over network ports. As the typical attacking scenario illustrated in Figure 7 in Chapter 3, attackers penetrate network port(s) through Firewalls; exploit the system vulnerabilities that are accessible through the port(s); damage the victim systems, and compromise system stakeholder values. So, different network port may involve different security risks depending on not only what are the vulnerabilities associated to the port, but also on how the vulnerabilities may impact stakeholder values. 101 9.3 Stakeholder Value Driven Automated Firewall Rule Generation based on T-MAP Ideally, a good Firewall rule set should suppress as much as possible risks with as less as possible number of rules. This section presents a stakeholder value driven approach based on T-MAP to automatically generate Firewall rules and maximizes risk suppression. 1. Abstracting Firewall Rules As known, Firewall rules specifies if the data packets that go through a range of adjacent network ports should be permitted or denied. (For simplicity, we limit our discussion on the Firewall rules that has the same protocols and source/dest IP addresses.) For example, the following rule in Cisco format enables the access to port 100-200 on the host of 192.168.0.2 from any source IP: access-list 110 permit tcp any host 192.168.0.2 range 100 200 The group number of 110 means this rule applies to the outbound packets; the word “tcp” specifies the network protocol; the word “any” specifies the source IP address; the words of “host 192.168.0.2” specifies the dest IP address, the words of “range 100 200” specifies the port range that is enabled. Assume a set of Firewall rules are applied to a host, the port control can be visualized as Figure 28 as follows: 102 Figure 28 Effect of Firewall Rules In our discussion, we assume all the access to any port is “deny” by default. In another word, a port is accessible from the source to dest if and only if exist one of Firewall rule enables the access to it. 2. Stakeholder Value Driven Firewall Rule Generation Based on T-MAP Our goal is to find the optimal rule set that suppress the maximum of risks while keep the number of rules within acceptable range. At current stage, for simplicity, we limit our discussion on a set of Firewall rules share the same attributes of protocol (e.g. TCP/UDP), source and dest IP, and the same network interface (e.g. network card) except for the starting and ending port. Assuming (1) the acceptable max number of Firewall rules is n, and (2) the ports for that need to be enabled for the protected system is known, the desired Firewall rule set that involves minimal ThreatKey value can be derived through the following steps: Step 1 Construct an ideal rule set that enabled the ports exactly as needed, assuming the rules controlling adjacent ports are merged; 103 Step 2 If the number of rules in step 1 is less than the acceptable max number of rules, then no more work is needed. Otherwise, continue to Step 3; Step 3 For the protected system go through the T-MAP Step 1-8 as described in Chapter 4-5, use a variance of the T-MAP Structured Attack Graph as follows to calculate the ThreatKey for each network port node (as the nodes marked grey in Figure 29). The major difference is that ethe network ports are inserted to the original graph as an explicit layer to measure the threat associated with each port. If an edge exists between a port node and a vulnerability node, then it represents that the vulnerability can be exploited through the associated port. Figure 29 Structured Attack Graph with Network Port Information Similar to the original ThreatKey definition, define: ThreatKey(Port i ) =∑(Weight of Associated Attacking Paths) 104 Step 4 Calculate the ThreatKey for each “Enabled Region” and each “Disabled Region” as illustrated in Figure 28 for the ideal rule set that was identified in Step 1. Definition 7: Region. Because each Firewall rule usually controls a range of sequential network ports, for simplicity we define a series of sequential ports as a Region, marked as Region <start, end> where the word “start” stands for the starting port number, and the word “end” stands for the ending port number. Obviously, start and end are integers between 0- 65535 and end ≥ start. So, ThreatKey(Region<start, end> ) = ∑ = end start i i Port ThreatKey ) ( Because the system stakeholder values have been taken into account in the port ThreatKey calculation, the ThreatKey value for Region is inherently sensitive to system stakeholder values as well. Assuming a set of Firewall rules share the same attributes of protocol (e.g. TCP/UDP), source and dest IP, and the same network interface (e.g. network card) except for the starting and ending port, we quantify the total threat associated with the rule set as: ThreatKey(RuleSet ) = ∑ = n j ThreatKey 1 (Enabled Region Of Rule j ) 105 where n is the number of rules in the rule set. Step 5 A Greedy algorithm can be devised to find the optimal rule set that is associated with the minimal ThreatKey value for a given limit on the max number of rules. For reader’s convenience, Figure 28 is copied as follows. Figure 28 (repeated) Definition 8: Adjacent Rules. In the ideal rule set that was identified in Step 1, assume Rule x enables Region<x start , x end >, and Rule y enables Region<y start , y end >. Not losing generality, assume y start ≥ x end. Then, Rule x and Rule y are defined as adjacent if (1) all ports in Region<x end , y start > are blocked; or, (2) y start - x end = 1, representing the case that the controlled regions of the two rules are adjacent to each other. Definition 9: Merging Rules. In the ideal rule set identified in Step 1, assume Rule x and Rule y are two adjacent rules. We assume Rule x enables port Region<x start , x end >, and Rule y enables port Region<y start , y end >. Not losing generality, assume y start ≥ x end. 106 Then, merging Rule x and Rule y generates a new rule, say Rule z , which enables the Region< x start , y end >. The Greedy Algorithm To summarize the Greedy Algorithm takes the ideal rule set identified in Step 1 as initial input. Next, it continues to merge the adjacent rules that have the minimal ThreatKey value of the interim blocked region until the number of total rules equals n. The algorithm is described as follows: Algorithm 5 RuleSet generateOptimalRuleSet(Structured_Attack_Graph G, Integer maxNumberOfRules, RuleSet originalIdealRuleSet) Return: The optimalRuleSet. (after running the algorithm, it has the minimal ThreatKey value and have less than maxNumberOfRules rules) 1: RuleSet optimalRuleSet; 2: FastCalculateThreatKey(G); //Algorithm 2 as specified in Chapter 4; 3: if (the number of rules in the originalIdealRuleSet is less than maxNumberOfRules) 4: return originalIdealRuleSet; //The number of rules meets the requirement. 5: QuickSort(The blocked regions for the originalIdealRuleSet by ThreatKey value); //Compexity: O[n*lg(n)] 6: //Greedy method 7: optimalRuleSet = originalIdealRuleSet; 8: while (number of rules in the optimalRuleSet > maxNumberOfRules) { 9: merge the two adjacent rules in the optimalRuleSet that have the least ThreatKey value of the interim blocked region; 10: } 11: return optimalRuleSet; 107 Now, we prove that at every step during the merging process, the ThreatKey value of the generated rule set, optimalRuleSet, is minimal. Proof: Assume there are n rules in the originalIdealRuleSet. Because the originalIdealRuleSet enables exactly the ports that must be opened, by definition, it has the minimal ThreatKey value among all the rule sets that have n rules. Clearly, after i times of merges, there will be n-i rules left in the optimalRuleSet. Assume after the i-th merge, the optimalRuleSet has the minimal ThreatKey among all the possible rule sets that have n-i rules. Next, we prove for the (i+1)-th merge based on Algorithm 5, the optimalRuleSet still has the minimal ThreatKey value among all the possible rule sets that have n-(i+1) rules: Obviously, the only possibility of removing one rule from the existing n-i rules while still keep all the ports enabled by this rule open is to merge this rule with one of its adjacent rules. Not losing generality, assume Rule x and Rule x+1 are the two adjacent rules that have the least value of ThreatKey of the interim Region. If there exists another two adjacent rules, say the Rule y and Rule y+1 , after merging them, the overall ThreatKey of the rule set will be less than merging Rule x and Rule x+1 . In another word, ThreatKey(optimalRuleSet after merging Rule y and Rule y+1 ) < ThreatKey(optimalRuleSet after merging Rule x and Rule x+1 ). 108 Then, based on the ThreatKey definition, after cancelling the ThreatKey values of the shared ports from each side of the above equation, we have: ThreatKey(Interim Region of Rule y and Rule y+1 ) < ThreatKey(Interim Region of Rule x and Rule x+1 ), which is contradicting with the initial assumption that the Rule x and Rule x+1 are the two adjacent rules that have the least value of ThreatKey of the interim Region. So, for the (i+1)-th merge following the Algorithm 5, the new rule set has the minimal ThreatKey value among all the rule sets that has n-(i+1) rules. End of Proof. 9.4 Preliminary Implementation Results We have implemented a concept-proof system for the above algorithm as part of the Tiramisu tool. The system inputs and outputs are illustrated in Figure 30 as follows: Figure 30 Firewall Rule Generation - System Input and Output 109 For the inputs, the Tiramisu tool contains a special database that contains the information of vulnerabilities that can be exploited through network TCP/UDP ports. By the time this dissertation is written, it has the port information of 717 vulnerabilities that were published during 1999-2007. The system stakeholder values and possible impacts from COTS security vulnerabilities can be obtained through the T-MAP steps as presented in Chapter 4 and 5. The Services and network communication ports usage information can be obtained through: • Information on default port configuration of relevant services and software; • System design and deployment documents; • The network data analyzing tools such as Tcpdump and Netflow. An example of the Netflow data that the Tiramisu tool can process is illustrated as follows: Figure 31 Example of Netflow Data Format 110 Also, the Tiramisu tool allows security professionals to define the must-enforced enabled/disabled ports, as illustrated in the user interface screenshots as Figure 32. Figure 32 Screenshot of Specifying Enforced Port Control Based on the pre-defined T-MAP system stakeholder value input, the screenshot of an example output is demonstrated as Figure 33 as follows: 111 Figure 33 Example Output of Firewall Rule Generation As shown, the max number of allowed Firewall rules is defined as 6. The IP addresses are obscured for client protection. The generated rule set is listed in the first big textbox on the top-right of the screen. As an example, for the highlighted rule #4, one of its associated vulnerabilities is the CVE-2003-0117, a Microsoft BizTalk Server vulnerability that can be exploited through sending a “certain request to the HTTP receiver”. Because the default TCP port number of the HTTP service is 80, it is within the port Region that is controlled by the rule #4 which enables the ports between 50 and 443. Furthermore, we plotted how the total number of exploitable Attack Paths that are associated with the rule set across all the rules decreases when the maximum number of rules in the rule set increases, as shown in Figure 34 as below. 112 Figure 34 Number of Firewall Rules vs. Number of Exploitable Attack Paths This result matches the common sense that the more rules are allowed, the better resolution of control can be implemented over the network ports, thus the less unused ports are opened because of the rule number limits and the less risky the system is. The preliminary result also demonstrates a clear economic curve on reducing the number of Attack Paths when the acceptable number of Firewall rules increases: the number of rules hits its high-payoff point after getting increased to a certain level. Afterwards, in the diminishing return region, the number of Attack Paths becomes to decrease slowly. Also, for the example system, we observed a significant “gap point” on the number of exploitable attack paths at the point where the number of rules increased from 10 to 12. It indicates increasing the number of rules from 10 to 12 is probably worthwhile to consider. 113 Furthermore, based on the number of data packets counting on each port by the Netflow tool, we plotted the number of data packets and the number of exploitable attack paths for each rule. The result is illustrated as follows: Figure 35 Number of Packets vs. Number of Exploitable Attack Paths The plots are divided into four regions: • Type-1. Large number of attack paths as well as large volume of data packets flows. • Type-2. Large number of attack paths but small volume of data packets flows. • Type-3. Small number of attack paths but large volume of data packets flows. • Type-4. Small number of attack paths and small volume of data packets flows. 114 In general, this information may help security professionals further analyze the Firewall rules: data through the Type 1 rules should be checked further to determine if the large volume of communications is generated by normal user applications or attacks. If it is generated by users, the system administrator may consider moving the data flow to less risky ports by re-configuring the system, because it can be difficult to detect the malicious data packets from large volume of the user data. The Type-2 rules may be re-analyzed to determine if it is really necessary to keep these ports open, because these ports are not frequently used but involves high risks to stakeholder values. The Type-3 rules are probably the most useful rules because they involves less risks than other types of rules but have frequent user packet flows. The Type-4 rules may be less important because they are less frequently used in network communications, but they are not very risky to open either. 9.5 Summary This chapter presented a stakeholder value sensitive approach to generate Firewall rules based on T-MAP. A greedy algorithm was devised and proved to generate the Firewall rule set with minimal ThreatKey value for given acceptable max number of rules. A concept-proof tool was developed as part of the Tiramisu tool. The screenshots from an example system was presented and explained. However, this work is still in its very early stage. In reality, the Firewall rule definition can be much more complex if take other factors into consideration, for example, the trust to 115 source/desk IP addresses, technical characteristics of network protocols, etc. Considering the method potentials demonstrated in our preliminary exploration, maturing this work to real life use can be a very interesting future work. 116 Chapter 10 Conclusions and Future Work 10.1 Conclusions T-MAP is a stakeholder value centric security threat modeling approach devised in light of a large body of previous works across Value Based Software Engineering, Security Economics, COTS Vulnerability Studies, and Attack Graph. It defines a formal framework that prioritizes COTS security vulnerabilities under system stakeholder value context, distilling the technical details of thousands of published software vulnerabilities into executive-friendly language at a high-level. Furthermore, it establishes a quantitative framework to identify the sweet-spot in security investment as well as estimate the effectiveness of common security practices such as Firewall, data encryption, patching, and creating redundant systems. An O(n) algorithm was devised to calculate the associated threat weight (ThreatKey) in Structured Attack Graph, which considerably increased the T-MAP scalability to systems that involve large volume of hosts, software, and vulnerabilities. This scalability is achieved at the tradeoff on limiting the analyses scope to single-step-exploits. A stakeholder value sensitive Firewall rule generation method based on T-MAP was introduced. The output rule set was proven to have the minimal ThreatKey value among all the possible rule sets that have the same number 117 of rules. A software tool, the Tiramisu, was developed to automate T-MAP, which significantly reduces the human effort involved in security threat evaluation. Compared to current value-neutral approaches, T-MAP systematically establishes the traceability and consistency from organizational-level value propositions to technical-level security threats and corresponding mitigation strategies. Compared to traditional risk management approaches, T-MAP does not require accurate values of probabilities, frequencies, and size of loss which are very difficult to estimate in practice. In all the three case studies conducted on real-life systems, T-MAP well captured the stakeholder value priorities through AHP pair-wise comparisons and injected the value priorities in attack path evaluation. The vulnerability rankings generated by T- MAP demonstrated strong correlations with the rankings generated by security professionals manually, with the R square values vary from 0.82 to 0.86. In the total number of 130 pair-wise vulnerability-rank-comparisons across all case studies, the T-MAP achieved an overall inaccuracy of 10%, which is observably lower than other value-neutral approaches: the inaccuracy value of 31.5% by CVSSv2, 30.8% by CVSSv1, 55.4% by IBM ISS, and 58.3% by Microsoft Vulnerability ranks (the Microsoft result is based on one case study that involves 36 pair-wise comparisons). Although the preliminary t test results based on the three case studies cannot be generalized in a statistical sense, the initial results are strongly against the null hypothesis that the inaccuracy of T-MAP will not differ from existing value neutral approaches. The case study analyses also found that T-MAP generated the over- 118 estimates and under-estimates mainly because of disabled services, the system privilege that the program were running, and the professionals’ confidence on themselves, insiders, and the environment. Based on the reasonably sound COTS vulnerability prioritization, we used T-MAP to help our client to compare security practice alternative plans based on economic analyses in the ITS Server X case study, wherein T-MAP demonstrated significant strength in estimating the effectiveness of security practices: the recommendation generated by T-MAP well met with the security manager’s experience. So, we can conclude that at least in these case studies T-MAP well captured the human perceptions on stakeholder values through its method steps, and reflected these value propositions in the automated security threat evaluation. However, since the method and the tool still generate some vulnerability over-estimates and under- estimates, it is still important for security professionals to balance T-MAP recommendations with those based on expert judgment. These conclusions are specific to the case studies we have conducted based on single-host COTS systems. However, though, the method demonstrated some perceivable potential even for bigger systems. For example, it can help: y Executives using cost-effectiveness analyses for their security practices and investments. y Security administrators identifying key vulnerability based on organizational value preferences. 119 y IT system architects evaluating the security performance of COTS systems. 10.2 Limitations At current stage, T-MAP still has several none-trivial limitations. First, constrained by the fact that the existing vulnerability databases do not have enough information to support multi-step-exploit attack graph modeling, our discussion on T-MAP is limited on single-step-exploit only in order to keep the method practical. But, as a bottom-line, a successful multi-step-exploit can be treated as a chain of several successful single-step-exploits. Thus, the failure of any of the single-step- exploits may make the multi-step-exploit unsuccessful. In this sense, T-MAP still can capture a large portion of the overall threat even after making the “one-step- exploit” simplifications. Moreover, perceivably, T-MAP has some potential to be extended to integrate more complex multi-step-exploit attack graph by chaining the steps between the layers of “IT hosts”, “COTS Software”, and “Vulnerability” in the Structured Attack Graph, if the connect-ability of pre-condition and post-conditions between exploits were known. Second, as a practical approach, the T-MAP method quantifies security threats all based on reported vulnerabilities of software, thus is not sensitive to unpublished vulnerabilities. As known, the impact from published vulnerabilities is much less significant than from published ones. An empirical study conducted by Arora shows that the average attacks per host per day dramatically jumped from 0.31 to 5.45 like a 120 step function after vulnerability status changed from “secret” to “published” [A. Arora, June 2004]. In this sense, the T-MAP method can still capture the major part of the security threats for COTS software systems. Third, many organizations have their own in-house developed software. For the security vulnerabilities reside in in-house software, if the vulnerability is known, it can be handled by adding its information to the T-MAP vulnerability database as a COTS vulnerability entry. If the vulnerability is unknown, it falls into the limitations of “unknown vulnerabilities” as discussed above. Finally, the T-MAP method requires comprehensive, accurate and up-to-date vulnerability information. Currently we are experimenting using web search and AI text reading technologies in growing our vulnerability database based on the NIST National Vulnerability Database. For example, our automated data-collecting engine can classify the “vulnerability solution” on securityfocus.com into four categories of “patching”, “change configurations”, “block access”, and “no solutions found” with accuracy of 87.3% according to our latest testing results. We believe further updates and improvements in the vulnerability database are critical and valuable. 10.3 Feedbacks The initial feedbacks on T-MAP from other researchers and our clients are quite positive. 121 • It was enlisted in the Software Security Assurance, State of The Art Report (Page 140) prepared by the Department of Defense(DoD) Information Assurance Technology Analysis Center, July 2007 [IATAC & DACS, 2007]. • It was commented as one of the most promising emerging methods for assessing software security risks by the Department of Homeland Security publication, Security in Software Lifecycle, Draft 1.2 (Page 46), August 2006 [K. M. Goertzel, T. Winograd, H. L. McKinley, P. Holley, B. A. Hamilton (2006)]. Our clients commented that T-MAP was: • “A valuable way of quantifying the very difficult tradeoffs that we have to make everyday.” • “T-MAP results well fit my past experience.” Also, a US patent was applied for the T-MAP methodology. The patent application Serial Number is 60/894,431, assigned by the United States Patent and Trademark Office. 10.4 Future Work The current research has focused on establishing the T-MAP conceptual framework, the methodology validation, and developing the Tiramisu tool. The 122 following directions are identified as most valuable future work to evolve T-MAP into its next stage: • Extend the T-MAP to support capturing the consensus of more than one security professionals. Limited by the availability of resources, each of the current case studies was conducted with one security expert only. Perceivably, using the consensus of a group of security experts’ opinions may further reduce the human bias in method validation. • Integrate T-MAP with System Engineering processes and refine it from a system engineering perspective. • Improve the vulnerability data collection algorithm to achieve higher data collecting accuracy. • Grow and improve the vulnerability database in the Tiramisu tool. • Conduct more empirical case studies on larger systems to further validate the T-MAP framework. • Mature the Automated Firewall Rule Generation method based on T-MAP. 123 References P. Ammann, D. Wijesekera, S. Kaushik (Nov. 2002). Scalable, graph-based network vulnerability analysis. Proceedings of the 9th ACM conference on Computer and Communications Security. R. Anderson (2001). "Why information security is hard – an economic perspective." 17th Annual Computer Security Applications Conference: 358–365 A. Arora, R. Krishnan, A. Nandkumar, R. Telang, Y. Yang (June 2004). Impact of Vulnerability Disclosure and Patch Availability - An Empirical Analysis. Workshop on Economics of Information Security. AS/NZS 4360 (1999). Australian Standard: Risk Management, Standards Australia T. August, T. Tunca (Nov. 2006). "Network Software Security and User Incentives." Management Science 52(11): 1703-1720. Baldwin, R. (1988). Rule based analysis of computer security, MIT Technical Report TR-401, MIT LCS Lab. V. Basili, B. Boehm (2001). "COTS Based System Top 10 List." Computer 34(5): 91-93. H. Berghel, D. Village (April 2005). "The two sides of ROI: return on investment vs. risk of incarceration." Communications of the ACM. L. D. Bodin, L. A. Gordon, M. P. Loeb (Feb. 2005). "Evaluating Information Security Investment Using the Analytic Hierarchy Process." Communications of The ACM. Boehm, B. W. (May 1988). "A Spiral Model of Software Development and Enhancement." IEEE Software. B. Boehm, R. Ross (1989). "Theory-W Software Project Management: Principles and Examples." IEEE Transactions Software Engineering: 902-916. B. Boehm, K. Sullivan (2000). "Economics: A Roadmap, The Future of Software Engineering." ACM Transactions: 319-343. 124 B. Boehm, D.Port, Y. Yang, J. Bhuta (2003). Not All CBS Are Created Equally. Proceedings of Second International Conference on COTS Based Software Systems (ICCBSS). B. Boehm, L. Huang, A. Jain, R. Madachy (2004). "Nature of Information System Dependability – A Stakeholder/Value Approach." CSSE Technical Report. B. Boehm, A. Jain (2005). "An Initial Theory of Value-Based Software Engineering in the book of S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, P. Gruenbacher (eds.) Value-Based Software Engineering." Springer Verlag. Boehm, B. (1981). Software Engineering Economics, Prentice Hall PTR. Barry W. Boehm, Jo Ann Lane (Oct. 2007). "Using the Incremental Commitment Model to Integrate System Acquisition, Systems Engineering, and Software Engineering." Crosstalk. S. Boyer, O. Dain, R. Cunningham (Mar. 2005). Stellar: A Fusion System for Scenario Construction and Security Risk Assessment. Proceedings, Third IEEE International Workshop on Information Assurance. Butler, S. A. (May 2002). "Software evaluation: Security attribute evaluation method: a cost-benefit approach." Proceedings of the 24th International Conference on Software Engineering. H. Cavusoglu, H. Cavusoglu, J. Zhang (June 2006). "Economics of Security Patch Management." The Fifth Workshop on the Economics of Information Security (WEIS 2006). H. Cavusoglu, B. Mishra, S. Raghunathan (July 2004). "A model for evaluating IT security investments." Communications of the ACM. Y. Chen, B. Boehm, L. Sheppard (June 2007). "Measuring Security Investment Benefit for Off The Shelf Systems." The 2007 Workshop on the Economics of Information Security (WEIS 2007), CMU, PA, June 2007. Y. Chen, B. Boehm, L. Sheppard (Sept. 2006). "Measuring Security Investment Benefit for COTS Based Systems." CSSE Technical Reports #2006-609. Y. Chen, B. Boehm, L. Sheppard (May 2007). "Stakeholder Value Driven Threat Modeling for Off The Shelf Based Systems." 29th International Conference on Software Engineering (ICSE), Doctoral Symposium, . 125 Y. Chen, B. Boehm, L. Sheppard (Jan. 2007). "Value Driven Security Threat Modeling Based on Attack Path Analysis." 40th Hawaii International Conference on System Sciences. CMMS (2002). "CMS Information Security Risk Assessment Methodology, v1.1." Centers for Medicare and Medicaid Services. CORAS (2003). http://heim.ifi.uio.no/~ketils/coras/. Corporation, M. "Common Vulnerability and Exposures." http://cve.mitre.org/. G. C. Dalton II, et al (2006). "Analyzing Attack Trees using Generalized Stochastic Petri Nets." Proceddings of IEEE Workshop on Information Assurance. D. Farmer, E. H. Spafford (Sept. 1991). "The cops security checker system." Technical Report CSDTR-993, Purdue University. FiSIRT "FiSIRT Security Advisories, http://www.frsirt.org/english." K. M. Goertzel, T. Winograd, H. L. McKinley, P. Holley, B. A. Hamilton (2006). “Security in Software Life-cycle, Making Software Development Processes and Software Produced by Them - More Secure, Draft 1.2.” Department of Homeland Security, Pp 46. L. Gordon, M. P. Loeb (Jan 2006). "Budgeting process for information security expenditures." Communications of The ACM. L. Gordon, M. P. Loeb, W. Lucyshyn, R. Richardson (2006). "2006 CSI/FBI COMPUTER CRIME AND SECURITY SURVEY." L. Gordon, M. Loeb (2002). "The economics of information security investment." ACM Trans, Inf. Syst. Sec. 5, 4: 438–457. Hoo, K. J. S. (2000). "How much is enough? A risk management approach to computer security." Ph.D. Dissertation, Stanford University. M. Howard, D. LeBlanc Writing Secure Code, Chapter 4, Pp 69-124, Microsoft Press. M. Howard, J. P., J. M. Wing (Dec. 2003). "Measuring Relative Attack Surfaces." Proceedings of Workshop on Advanced Developments in Software and Systems Security. Engineers, I. o. E. a. E. (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. 126 IATAC & DACS (2007). Software Security Assurance State-of-the-Art Report (SOAR). ISO 15288 (2002). Systems Engineering–System Life Cycle Processes. ISO 15408 (1999). "The Common Criteria for Information Technology Security Evaluation (CC) version 2.1." ISS. http://xforce.iss.net/. Kontio, J. (1996). "A Case Study in Applying a Systematic Method for COTS Selection." Eighteenth International Conference on Software Engineering. R. Lippmann, K. Ingols (March 2005). "An annotated review of past papers on attack graphs." Technical report, MIT Lincoln Laboratory. P. K. Manadhata, J.M. Wing (Aug. 2006). "An Attack Surface Metric." USENIX Security Workshop on Security Metrics (MetriCon). Martin, R. A. (2002). "Managing Vulnerabilities in Your Commercial-Off-The-Shelf (COTS) Systems Using An Industry Standards Effort." IEEE. Microsoft. "Microsoft Security Alert Severity Matrix, http://www.microsoft.com/technet/security/alerts/matrix.mspx." Microsoft. "Microsoft Security Bulletin, http://www.microsoft.com/technet/security/bulletin/." MITRE. "Open Vulnerability Assess Language, http://oval.mitre.org/oval/." Nessus. "http://www.nessus.org/." NIST. "National Vulnerability Database, http://nvd.nist.gov/." S. Noel, S. Jajodia, B. O. Berry, M. Jacobs (Dec. 2003). "Efficient minimum-cost network hardening via exploit dependency graphs." 19th Annual Computer Security Applications Conference (ACSAC). OMG. "http://www.uml.org/." X. Ou, W. F. Boyer, M. A. McQueen (Oct. 2006). "A scalable approach to attack graph generation." 13th ACM Conference on Computer and Communications Security (CCS 2006). 127 D. Port, Z. H. Chen, (2004). "Assessing COTS Assessment: How much is Enough?" Proc. 3rd Int’l Conf. COTS Based Software Systems (ICCBSS): 183-198 PTA Technologies. http://www.ptatechnologies.com/. D. Reifer (2002). "Making the Software Business Case." Addison Wesley. D. Reifer, et al (2004). "COTS Based Systems: Twelve Lessons Learned." Proc. 4rd Int’l Conf. COTS-Based Software Systems (ICCBSS 04): 137-145. N. H. Roberts, WE Vesely, D. F. Haasl, F. F. Goldberg, (Jan. 1981). "Fault Tree Handbook." Systems Reliability Research Office of U.S. Nuclear Regulatory Commission, Washington DC, 20555. Royce, Winston (August, 1970). "Managing the Development of Large Software Systems." Proceedings of IEEE WESCON 26. T. L. Saaty (1980). "The Analytic Hierarchy Process." McGraw-Hill . P. Saitta, B. Larcom, M. Eddington (2005). Trike v.1 Methodology Document (Draft). http://www.octotrike.org/. SANS. "Top 20 Most Critical Vulnerabilities, http://www.sans.org/top20 ". SATAN. "SATAN: http://www.ibiblio.org/pub/packages/security/Satan-for-Linux/." S. E. Schechter (Jan. 2005). "Toward econometric models of the security risk from remote attacks." Security & Privacy Magazine, IEEE 3(1): 40-44. M. Schiffman "Common Vulnerability Scoring System (CVSS), http://www.first.org/cvss/." B. Schneier (Dec. 1999). "Attack trees: Modeling security threats." Dr. Dobb's Journal. Sci-tech-today. "Man Charged with Hacking USC Database, http://www.sci-tech- today.com/story.xhtml?story_id=11200CI6KD4W." SECURIS. http://heim.ifi.uio.no/~ketils/securis/. Securityfocus. "http://www.securityfocus.com/." O. Sheyner, J. Haines, S. Jha, R. Lippmann, J.M. Wing, (May 2002). "Automated Generation and Analysis of Attack Graphs " Proceedings of the IEEE Symposium on Security and Privacy. 128 O. Sheyner (April 2004). "Scenario Graphs and Attack Graphs " Carnegie Mellon University, PhD Thesis. Skybox, http://www.skybox.com/. StandishGroup (2001). "Extreme CHAOS." tech. report. G. Stoneburner, A. Goguen, A. Feringa (2002). "Risk Management Guide for IT Systems, NIST Special Publication 800-30." F. Swiderski, W. Snyder, (2004). "Threat Modeling." Microsoft Press. Symantec. "Symantec Threat Severity Assessment, http://www.symantec.com/avcenter/threat.severity.html." S. Templeton, K. Levitt (2000). "A requires/provides model for computer attacks." Proceedings of the 2000 New Security Paradigmes Workshop P. Torr (Sept. 2005). "Demystifying the threat modeling process." Security & Privacy Magazine, IEEE 3(5): 66-77. US-CERT. "http://www.cert.org/octave." US-CERT (DB). "US-CERT Vulnerability Database, http://www.kb.cert.org/vuls/." US-CERT (Metrics). "US-CERT Vulnerability Metrics, http://www.kb.cert.org/vuls/html/fieldhelp#metric." US-GAO (1999). "US. General Accounting Office, Information Security Risk Assessment: Practices of Leading Organizations." D. Verdon, G McGraw (July 2004). "Risk analysis in software design." Security & Privacy Magazine, IEEE 2(4): 79-84. J. M. Wing (Mar. 2005). "Scenario Graphs Applied to Security . Summary Paper." Proceedings of Workshop on Verification of Infinite State Systems with Applications to Security. Y. Yang, J. Bhuta, B. Boehm, D. Port (July/August 2005). "Value-Based Processes for COTS-Based Applications." IEEE Software 22(4). 129 Appendix Initial t test Results T test on inaccuracy values between T-MAP and other approaches respectively has been conducted to evaluate the difference. However, given the limited number of case studies that have been conducted so far, it is premature to make any statistical conclusions. We list the results as follows. Table 24 T Test Analysis between T-MAP and Mainstream Approaches Ranking System Inaccuracy Mean Inaccuracy Var t value p value (alpha = 0.05, 2 tail) CVSS v2.0 0.280 0.014 -2.643 0.118 CVSS v1.0 0.290 0.005 -4.585 0.044 IBM ISS 0.497 0.037 -3.594 0.069 Microsoft 0.583 n/a n/a n/a The t test results shows t value varies in a -2.643 to -4.585 range, suggesting the mean value of T-MAP inaccuracy is considerably smaller than other approaches; the p value varies in a 0.044 to a 0.118 range, suggesting the case study results is against the null hypotheses as stated in Section 1.3. However, because of the limited number of data points available (three), it is premature to generalize any of these conclusions statistically.

Abstract (if available)

Abstract The thesis presents the Threat Modeling Method Based on Attack Path Analysis (T-MAP) which quantifies security threats by calculating the total severity weights of relevant attacking paths for Commercial Off The Shelf (COTS) based systems. Further security economic analysis enabled by T-MAP is demonstrated. Compared to existing approaches, T-MAP is sensitive to system stakeholder value priorities and organizational IT environment. It distills the technical details of thousands of relevant software vulnerabilities into management-friendly numbers at a high-level

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Software quality analysis: a value-based approach

PDF

A value-based theory of software engineering

PDF

The effects of required security on software development effort

PDF

Attacks and defense on privacy of hardware intellectual property and machine learning

Asset Metadata

Creator Chen, Yue (author)

Core Title Software security economics and threat modeling based on attack path analysis; a stakeholder value driven approach

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 12/07/2007

Defense Date 10/10/2007

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag COTS,OAI-PMH Harvest,security economics,software engineering,threat modeling,value based,vulnerability ranking

Language English

Advisor Boehm, Barry W. (committee chair), Huang, Ming-Deh (committee member), Neuman, Clifford (committee member), Steece, Bert M. (committee member)

Creator Email yuec@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m967

Unique identifier UC1358387

Identifier etd-Chen-20071207 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-595157 (legacy record id),usctheses-m967 (legacy record id)

Legacy Identifier etd-Chen-20071207.pdf

Dmrecord 595157

Document Type Dissertation

Rights Chen, Yue

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu