Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
COSYSMO 3.0: an extended, unified cost estimating model for systems engineering
(USC Thesis Other)
COSYSMO 3.0: an extended, unified cost estimating model for systems engineering
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
v5
Copyright 2018 James P Alstad
COSYSMO 3.0: An Extended, Unified Cost
Estimating Model for Systems Engineering
By James Patrick Alstad
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(ASTRONAUTICAL ENGINEERING)
December 2018
ii
Dedication
This dissertation is dedicated to Roberta,
for her unwavering support and understanding.
iii
A. Acknowledgements
It is common in an acknowledgement to start by quoting Newton, “If I have been
able to see further than others, it is because I have stood on the shoulders of
giants.”
For the last 30 years, I have been trying to stand on the shoulders of Dr Barry W
Boehm, and I think I have now succeeded. His intellectual leadership of the field
of, first, software engineering, and, now, systems engineering has been an
inspiration to me and to dozens of others, especially to his students; this includes
his teaching, his administration of the Center for Systems and Software
Engineering, and his research. I know no one else with as deep an understanding
of software and systems engineering. Recently, I am also grateful to him for
taking me as a PhD student and supporting me in my efforts.
As explained more fully below, the support of the COSYSMO 3.0 Working Group
has been essential to this work, and some of them deserve special mention. Gan
Wang is second only to Dr Boehm in his support of COSYSMO 3.0; he has
published frequent foundational papers in the field, and he has provided key help
and advice to me personally. Garry Roedler also has supported COSYSMO since
its beginning, and he provided important and novel advice on COSYSMO 3.0
specifix. Jo Ann Lane was a leader in getting COSYSMO to apply to system-of-
systems, and I’m glad I could further that goal. Marilee Wheaton gave good all-
around advice from her broad systems engineering background, and was very
regular at attending Working Group meetings. Last but not least, Dan Ligett read a
draft of this dissertation and survived, which was encouraging; he also offered
helpful criticism.
My Committee has been helpful, providing direction and comments: Dr Boehm,
Dr Azad Madni, Dr Dan Erwin, Dr Neil Siegel, Dr Aiichiro Nakano (Qualifying),
and Dr Jim Moore (Defense).
I am grateful for the support of the Center for Systems and Software Engineering,
which has been helping Dr Boehm to lead us for the last couple decades. I am also
iv
grateful for the support of the Systems Engineering Research Center at Stevens
University for supporting Dr Boehm and me.
B. Abstract
The discipline of systems engineering continues to increase in importance. There
are more projects, projects are larger, and projects are more critical, and all of these
mean that more and better systems engineering is required. It follows that the cost
of systems engineering continues to increase in importance. In turn, it follows that
accurate estimation of systems engineering costs continues to increase in
importance, as systems engineering results suffer if a project either underestimates
or overestimates its cost.
There are models for estimating systems engineering cost, notably COSYSMO 1.0,
but all these models leave out some aspect of modern practices, and therefore tend
to estimate a modern systems engineering cost inaccurately, or not at all. These
modern practices include reuse of engineering artifacts, requirements volatility,
and engineering in a system-of-systems context. While all of these practices have
been addressed to some extent in research papers, there is no all-encompassing
model that integrates all of them.
My research has resulted in an integrated model that includes the features of
COSYSMO 1.0 and covers those modern practices. It is open and therefore widely
available. I have completed a comprehensive model based, in part, on actual
project data.
v
C. Table of Contents
Dedication .............................................................................................................................................. ii
A. Acknowledgements .................................................................................................................... iii
B. Abstract .......................................................................................................................................... iv
C. Table of Contents ........................................................................................................................... v
D. List of Tables ............................................................................................................................... vii
E. List of Figures ............................................................................................................................. viii
1 Introduction .................................................................................................................................... 1
2 Statement of the Problem ........................................................................................................... 3
2.1 The Problem ........................................................................................................................................... 3
2.2 Research Objectives ............................................................................................................................. 4
2.3 Contributions to Society ..................................................................................................................... 4
3 Literature Review ......................................................................................................................... 5
3.1 COSYSMO Literature ............................................................................................................................ 5
3.2 Space Systems Engineering Literature .......................................................................................... 7
4 Background of Proposed Research ......................................................................................... 9
4.1 General Background ............................................................................................................................ 9
4.2 General Structure of a COCOMO-Family Model ....................................................................... 11
4.3 Methodology for Creating the New Model ................................................................................. 13
5 Research Results and Conclusions ........................................................................................ 14
5.1 Research Hypothesis ........................................................................................................................ 14
5.2 Preliminary Results .......................................................................................................................... 14
5.2.1 How the Expert-Based Model Was Determined .............................................................................. 14
5.2.2 Summary of Driver Changes ..................................................................................................................... 17
5.2.3 Expert-Based COSYSMO 3.0 ...................................................................................................................... 19
5.2.3.1 Foundational Elements ........................................................................................................................................ 20
5.2.3.1.1 Model Scope .................................................................................................................................................... 20
5.2.3.1.2 Top-Level Equation ...................................................................................................................................... 21
5.2.3.1.3 Size Formula ................................................................................................................................................... 22
5.2.3.1.4 Activity Levels ................................................................................................................................................ 22
5.2.3.1.5 Exponent Formula ........................................................................................................................................ 28
5.2.3.1.6 Allowed Numeric Levels for Cost Driver and Scale Factor Ratings ........................................ 28
5.2.3.1.7 Multi-Subproject Model ............................................................................................................................. 28
5.2.3.2 Size Drivers ............................................................................................................................................................... 30
5.2.3.2.1 System Requirements ................................................................................................................................. 30
5.2.3.2.2 System Interfaces .......................................................................................................................................... 31
vi
5.2.3.2.3 Algorithms ....................................................................................................................................................... 32
5.2.3.2.4 Operational Scenarios ................................................................................................................................. 33
5.2.3.3 Cost Drivers .............................................................................................................................................................. 35
5.2.3.3.1 CONOPS and Requirements Understanding ..................................................................................... 35
5.2.3.3.2 Architecture Understanding .................................................................................................................... 36
5.2.3.3.3 Stakeholder Team Cohesion .................................................................................................................... 37
5.2.3.3.4 Level of Service Requirements ............................................................................................................... 38
5.2.3.3.5 Technology Risk ............................................................................................................................................ 39
5.2.3.3.6 # of Recursive Levels in the Design ...................................................................................................... 39
5.2.3.3.7 Development For Reuse ............................................................................................................................. 40
5.2.3.3.8 # and Diversity of Installations/Platforms ........................................................................................ 40
5.2.3.3.9 Migration Complexity ................................................................................................................................. 41
5.2.3.3.10 Interoperability ........................................................................................................................................... 42
5.2.3.3.11 Personnel/Team Capability ................................................................................................................... 44
5.2.3.3.12 Process Capability ...................................................................................................................................... 45
5.2.3.3.13 Personnel Experience/Continuity ...................................................................................................... 46
5.2.3.3.14 Multisite Coordination ............................................................................................................................. 47
5.2.3.3.15 Tool Support ................................................................................................................................................. 48
5.2.3.4 Scale Factors ............................................................................................................................................................. 50
5.2.3.4.1 Risk/Opportunity Resolution .................................................................................................................. 50
5.2.3.4.2 Process Capability ........................................................................................................................................ 51
5.2.3.4.3 Requirements Volatility ............................................................................................................................. 51
5.2.3.5 Numeric Values of Ratings ................................................................................................................................. 53
5.2.4 Continuity and The Rosetta Stone ......................................................................................................... 54
5.3 Final Results ........................................................................................................................................ 56
5.3.1 Determining the Final Model ................................................................................................................... 56
5.3.1.1 What Is Undetermined about COSYSMO 3.0 at this Point? .................................................................. 56
5.3.1.2 The Dataset ............................................................................................................................................................... 57
5.3.1.3 Model Adjustments ................................................................................................................................................ 58
5.3.1.4 The CEM Prior Information ............................................................................................................................... 59
5.3.1.5 Determining the Placement of Process Capability ................................................................................... 60
5.3.1.6 One Last Issue before Doing Fits ..................................................................................................................... 62
5.3.1.7 Determining the Values of the Final Model Parameters ....................................................................... 63
5.3.1.7.1 What “Doing a Linear Regression Fit” Means in this Context .................................................... 63
5.3.1.7.2 Bayes .................................................................................................................................................................. 65
5.3.1.7.3 Determining the Final Model of COSYSMO 3.0 ................................................................................. 65
5.3.2 Calibrating to a Data Set ............................................................................................................................. 67
5.3.2.1.1 Removing Outliers: the PL4O Dataset ................................................................................................ 67
5.3.2.1.2 PL4O Model and Fit ...................................................................................................................................... 68
5.3.2.1.3 Some Observations ...................................................................................................................................... 69
5.3.2.1.4 Hill-Climbing and the nlm function ....................................................................................................... 70
5.3.2.1.5 Properties of the PL4O.nlm Calibration .............................................................................................. 71
5.3.3 Attributes of the Final Model ................................................................................................................... 73
5.4 Research Contributions ................................................................................................................... 77
5.4.1 Summary of Research Contributions .................................................................................................... 77
5.4.2 The Benefits of the COSYSMO 3.0 Model ............................................................................................. 78
5.4.3 Improvements to Bayesian Linear Regression Practice .............................................................. 78
6 Evaluation and Validation of the Final Model ................................................................... 81
6.1 Appropriate Use of the Model ....................................................................................................... 81
6.2 Threats to Validity ............................................................................................................................. 82
6.3 Evaluation ............................................................................................................................................ 84
vii
7 Schedule of Remaining Research Tasks .............................................................................. 85
7.1 Submission to Journal ...................................................................................................................... 85
8 Future Research .......................................................................................................................... 86
9 References ..................................................................................................................................... 87
Appendix A: Correlation Coefficients ......................................................................................... 90
Appendix B: Counts of Ratings ...................................................................................................... 92
Appendix C: Regression Results ................................................................................................... 93
D. List of Tables
Table 3-1. Which Concepts are Used in Which Estimating Models? ............................................ 7
Table 3-2. USCM8 Wrap Factors for Systems Engineering. ......................................................... 8
Table 5-1. Core Members of the COSYSMO 3.0 Working Group. ............................................ 15
Table 5-2. The Cost Drivers for the Coalesced Effort Multipliers version of COSYSMO 3.0. .. 16
Table 5-3. Summary of Driver Changes. ..................................................................................... 17
Table 5-4. Correspondence between COSYSMO 3.0 Stages and TR 24728 Stages. .................. 21
Table 5-5. Overview of the Activity Levels. ............................................................................... 24
Table 5-6. Numerical Values for ALFraction for DWR and DFR Activity Levels. ................... 24
Table 5-7. Definitions of the DWR Activity Levels. ................................................................... 25
Table 5-8. Definitions of the DFR Activity Levels. .................................................................... 27
Table 5-9. Table for Assigning eReqs to a Size Driver. .............................................................. 30
Table 5-10. Numeric Values of Ratings for Cost Drivers. .......................................................... 53
Table 5-11. Numeric Values of Ratings for Exponents. .............................................................. 54
Table 5-12. The Rosetta Stone. .................................................................................................... 55
Table 5-13. The Parameters of COSYSMO 3.0. ......................................................................... 56
Table 5-14. Disposition of Parameters without Ratings in the Dataset. ...................................... 58
Table 5-15. EMRs and Step Sizes for the 12 CEM Priors. .......................................................... 60
Table 5-16. Comparison of Cost Driver and Scale Factor Fits for Process Capability as Cost
Driver and as Scale Factor. ................................................................................................... 61
Table 5-17. The Final Model Parameters. ................................................................................... 63
Table 5-18. Rated Parameters for the Final Model. ..................................................................... 66
Table 5-19. Constant Parameters for the Final Model. ................................................................ 67
Table 5-20. Analysis of Productivities for Outliers. .................................................................... 68
Table 5-21. Comparison of Linear Regressions with and without Priors on the PL4O Dataset. 68
Table 5-22. Two Types of Tension between Different Fit Approaches. ..................................... 69
Table 5-23. Parameters of the PL4O.nlm Calibration. ................................................................ 72
Table 5-24. Summary of Sources of the Model. .......................................................................... 74
Table 5-25. COSYSMO 3.0 Features versus Previous COSYSMO Models. .............................. 75
Table 5-26. Numerical Properties of COSYSMO 3.0 versus Previous Models. ......................... 77
Table 0-1. Counts of the Number of Times Each Rating Was Used for Each Driver. ................ 92
Table 0-2. Printed Results of the Regression for the Final Model Cost Driver Fit. .................... 93
viii
Table 0-3. Printed Results of the Regression for the Final Model Exponent Fit. ........................ 94
E. List of Figures
Figure 1-1. A Satellite Project Org Chart, Focusing on Systems Engineering. ............................. 1
Figure 3-1. NICM Estimate for satellite system cost of systems engineering. .............................. 8
Figure 4-1. The COCOMO Family of Estimating Models. ........................................................... 9
Figure 4-2. The Family of COSYSMO Models. ......................................................................... 11
Figure 4-3. This equation applies to all COCOMO-family estimating models. .......................... 12
Figure 4-4. The USC CSSE methodology for generating a cost estimating model. .................... 13
Figure 5-1. This Top-Level Equation Computes the Estimated Effort for a Project. .................. 21
Figure 5-2. The equations for the adjusted size and for the partial development factor. ............. 22
Figure 5-3. The Systems Engineering V. ..................................................................................... 23
Figure 5-4. Diagrammatic View of the DWR Activity Levels. ................................................... 25
Figure 5-5. Diagrammatic View of the DFR Activity Levels. .................................................... 26
Figure 5-6. The Equation for the Exponent E. ............................................................................. 28
Figure 5-7. The Formula for Total Project Effort When Subprojects are Used. ......................... 29
Figure 5-8. Structural View of Bayesian Data Analysis. ............................................................. 65
Figure 5-9. The Result of the PL4O.nlm Fit on the PL4O Dataset. ............................................ 73
Figure 5-10. COSYSMO 3.0 Cost Drivers, in order of Increasing EMR. ................................... 76
1
1 Introduction
Systems engineering is that engineering discipline that works at the system-as-a-
whole level. Simplifying somewhat, it starts from customer needs and
requirements and produces (among other artifacts) a system architecture and
specifications for all the subsystems in the system; furthermore, systems
engineering conducts the validation testing on the complete, constructed system.
By definition, a systems engineer has to be a generalist, one who is familiar with
several specific engineering disciplines.
Figure 1-1. A Satellite Project Org Chart, Focusing on Systems Engineering.
This is a partial, simplified diagram of the role of systems engineering on a satellite project.
Systems engineering receives needs and requirements from the customer, and produces a system
architecture and subsystem specifications for use by specialized engineers.
Systems engineering is particularly important on large programs to ensure
technical coordination between subsystems and between subcontractors. To give
an idea of the magnitude of the projects where systems engineering is important,
consider just the US Department of Defense; in fiscal year 2014 Stephen Welby,
the Deputy Assistant of Defense for Systems Engineering, “performed systems
engineering oversight of 182 programs with acquisition costs of $1.8T” (Baldwin,
2015).
2
If systems engineering is important on large programs, then accurate cost
estimation for systems engineering is important also.
There are three recognized types of engineering cost estimating models: similarity,
bottom-up, and parametric; the present research is concerned with a parametric
model, where attributes of a project are input to a model and the model provides a
cost estimate.
Specifically, the starting point for my research is COSYSMO 1.0, Ricardo
Valerdi’s parametric systems engineering cost estimating model from 2005
(Valerdi R. , 2005). (COSYSMO 1.0 itself was strongly influenced by the
software engineering cost estimating model COCOMO II.2000 (Boehm B. W.,
2000), which in turn was based on the earlier COCOMO 81 (Boehm B. W., 1981).)
COSYSMO 1.0 did a respectable job of covering the specifics of systems
engineering projects in 2005, but since then additional project phenomena have
become common, and are expected to become more widespread in the future.
Specifically, a contemporary systems engineering project may:
• Reuse systems engineering artifacts produced on a previous project, or
produce artifacts to be reused on later projects, or both;
• Encounter significant volatility in system requirements; and/or
• Be part of a system-of-systems project. (A system-of-systems is one where
each constituent system is independently managed and has a distinct
mission.)
The goal of my research is to create COSYSMO 3.0, a modern parametric systems
engineering cost estimating model, that builds upon the features of COSYSMO 1.0
and includes the three project features just listed (reuse, requirements volatility,
and a system-of-systems context).
3
2 Statement of the Problem
2.1 The Problem
Systems engineering is becoming more widespread; for example, INCOSE now
has “nearly 10,000 members” (INCOSE, 2017). Likewise, it’s becoming a more
important part of large projects; the US Department of Defense’s Better Buying
Power Initiative 3.0 specifies that one of its key goals is to “achieve dominant
capabilities while controlling lifecycle costs” (Kendall, 2015), and achieving
capabilities and controlling lifecycle costs are within the domain of systems
engineering.
On large projects, it is certainly necessary for the developer to estimate engineering
costs in general and systems engineering costs in particular; obviously, accuracy of
such estimates is important. Accurate estimates provide a sound foundation for
bidding, and for tracking the progress of ongoing projects. However, some
organizations do not do very accurate estimates. A closely related issue is that
procuring organizations also may not do accurate estimates, so that they also are
vulnerable to the possible negative consequences.
One particular problem with an inaccurate estimate is that it might be too high, so
that the excess budget is wasted due to Parkinson’s Law (Parkinson, 1957). A
subtler problem is that a systems engineering estimate might be too low; this could,
for example, disallow sufficient investigation of a new technology that could
reduce lifecycle costs.
There are system engineering cost models in existence; they can help with the
above problems. A prime example is COSYSMO 1.0 (Valerdi R. , 2005). It
covers simple projects well, but current projects can be more complex, involving
reuse of engineering artifacts, requirements volatility, and/or a system-of-systems
context. There are partial cost models, each based on COSYSMO 1.0, for some of
these situations, but they are not integrated into a single model (this is explained
more fully below).
To sum up the problem: systems engineering is increasing in importance, so an
accurate cost estimating model is needed for it; projects are getting more
complicated, so a new model addressing those complexities in an integrated
fashion is needed.
4
2.2 Research Objectives
The goal of this research is to develop a systems engineering cost estimating
model, called “COSYSMO 3.0”, with these properties:
§ Is applicable to a wide range of systems engineering projects;
§ Includes all the features of COSYSMO 1.0, plus these features: reuse,
requirements volatility, system-of-systems context (in part);
§ Provides continuity to users of previous COSYSMO-family models;
§ When calibrated to data from a particular organization, estimates actual
systems engineering costs with a PRED(.30) accuracy of 50%.
2.3 Contributions to Society
One contribution is that organizations that use COSYSMO 3.0 will tend to have
more accurate cost estimates for systems engineering. This implies that
engineering projects that include systems engineering will tend to overrun budgets
less, but will be more effective because the budget for systems engineering is
adequate.
Because COSYSMO 3.0 is more comprehensive than earlier models (covering
reuse, requirements volatility, and use in systems-of-systems), it will be applicable
to more projects than earlier models. Furthermore, it is a non-proprietary model
and freely available. Therefore it will be used on more projects, so the benefits of
using a parametric estimating model will be more widespread; this is the second
contribution.
Because COSYSMO 3.0 is an open and non-proprietary model, the system
engineering profession, including professional organizations like INCOSE, can use
its existence to upgrade their standards by, for example, specifying that creation of
a formal cost estimate is a part of good systems engineering practice. This is a
third contribution.
5
3 Literature Review
3.1 COSYSMO Literature
Before there was a parametric cost estimating model for systems engineering, two
versions of COCOMO providing parametric cost estimating models were extant:
COCOMO 81 (Boehm B. W., 1981) and COCOMO II (Boehm B. W., 2000).
COCOMO 81 used these concepts in its model: use of a specific size measure
(source lines of code); the cost driver concept, which is to rate several attributes of
the project, and for each to apply a multiplier to increase or decrease the estimated
cost based on its rating; to apply an exponent to the size to account for
diseconomies of scale; to adjust the estimated cost by adjusting the size when
artifacts (code) are reused; a subproject model, which showed how subprojects
with different cost driver ratings could be estimated under a single project; and to
encourage an organization to tailor the model to its own data by adjusting the
calibration parameter. In 2000, the COCOMO II model used those concepts and
added these: a more sophisticated model of the effect of reuse; the use of rated
values (“scale factors”) in the exponent; an acknowledgement that developing
software for later reuse had higher cost.
COSYSMO 1.0 was the first parametric cost estimating model for systems
engineering; Ricardo Valerdi published it in 2005 (Valerdi R. , 2005). It used
some, but not all, of the concepts from COCOMO II; notably, there was no
provision for estimating with reuse, and the exponent was constant (similar to
COCOMO 81). Also, COSYSMO 1.0 added use of multiple artifacts to measure
size.
Wang and his co-authors introduced a model incorporating reuse of previously
developed artifacts (Wang, Ankrum, Valerdi, & Millar, 2008), and that submodel
was incorporated into COSYSMO 2.0 (Fortune, 2009). The underlying concept,
called Development With Reuse, was that reused artifacts reduced the cost because
some phases in the life cycle could be skipped. (For example, if artifacts being
reused had already been carried through the architecture activity, then the current
project could skip activities through architecture.)
COSYSMO with Requirement Volatility (“COSYSMO RV”) introduced a model
adjusting cost depending on the degree of volatility of system requirements. Such
6
volatility was accounted for by putting its driver into the exponent as a scale factor.
This captured the phenomenon that the penalty for a given amount of volatility
went up as the size of the project increased, and went up more than linearly.
Wang enhanced his Development With Reuse concept (Wang G. , 2016) by
extending the idea of skipping development activities to Development For Reuse,
where a project, such as an IR&D project, might skip activities at the end of
development, thereby reducing the cost on the present project. He continued to use
the term Generalized Reuse Framework (GRF) to encompass Development For
Reuse. However, the GRF did not account for the intrinsically greater cost of
developing for reuse.
Lane made two contributions toward applying COSYSMO in a system-of-systems
context. One was applying COCOMO II’s subproject model to show how to
estimate systems engineering at both the system-of-systems level and at the
constituent system level in an integrated way (Lane, Cost Model Extensions to
Support Systems Engineering Cost Estimation for Complex Systems and Systems
of Systems, 2009). The other was to notice that development in a system-of-
systems context seemed to be more costly, and to propose that the degree of
interoperability be used to account for this in COSYSMO (Lane & Valerdi,
Systems Interoperabililty Influence on System of Systems Engineering Effort,
2011).
7
Table 3-1. Which Concepts are Used in Which Estimating Models?
This summarizes the discussion in the text above.
Concept
COCO-
MO 81
COCO-
MO II
COSYS-
MO 1.0
COSYS-
MO 2.0
COSYS-
MO RV
GRF
Sys-of-
Sys
Use of a specific size measure X X X X X X X
Cost drivers X X X X X X X
Use of exponent on size X X X X X X X
Address reuse X X X X
Tailoring via calibration
parameter
X X X X X X X
Sophisticated reuse model X X
Use of rated exponents X X
Account for increased cost when
developing for reuse.
X
Subproject model X X X
Size with multiple artifacts X X X X X
Address interoperability X
3.2 Space Systems Engineering Literature
For space systems development, Space Mission Engineering provides several
parametric cost estimating models specific to this application (Apgar, 2011); I
cover two of them here, to show how they estimate systems engineering cost and to
contrast them with the COCOMO-family models. One of these space systems
models is the NASA Instrument Cost Model (NICM) (Habib-agahi, 2010). It
functions by first using the satellite type and satellite parameters to estimate a basic
system cost from that specific model, and then calculating systems engineering
cost from the basic system cost per a parametric equation (see Figure 3-1).
8
Y = 0.4931×S
0.8645
where S= basic system cost
Y = Systems engineering cost
Figure 3-1. NICM Estimate for satellite system cost of systems engineering.
This is taken from Table 11-15 of (Apgar, 2011).
The other space systems model for systems engineering cost is USCM8 (Tecolete
Research, 2002). Similarly to NICM, it first estimates a basic system cost. From
that it computes systems engineering cost as a “wrap factor”, a percentage, of the
basic system cost, based on a judgment of the difficulty of the systems engineering
(see Table 3-2).
Table 3-2. USCM8 Wrap Factors for Systems Engineering.
The selected wrap factor is multiplied by the estimated basic system cost
to yield the estimated systems engineering cost.
This is extracted from Table 11-24 of (Apgar, 2011).
Cost Element
Mini-
mum
Factor
Average
Factor
Maxi-
mum
Factor
Reasons for Using the
Maximum Factor
Systems Engineering 15% 20% 35%
Frequently changing require-
ments, multiple interfaces.
It can be seen that these satellite system cost models are very simple, in that they
base their systems engineering cost estimates on the estimated cost of the entire
system, without any consideration of the specific attributes of the systems
engineering effort on the project. (I note in passing that some of the system
models do include some consideration of system-level attributes like reuse and
degree of challenge.) Because this class of models is too simplistic, I do not
consider it further.
9
4 Background of Proposed Research
4.1 General Background
The background of this research is the COCOMO family of cost estimating models
(Figure 4-1). The first models were software engineering cost models, namely
COCOMO 81 (Boehm B. W., 1981) and COCOMO II.2000 (Boehm B. W., 2000).
(Further software engineering cost models have been added to the family since.)
Figure 4-1. The COCOMO Family of Estimating Models.
Most of the models are software estimating models, but those in the upper right are other types,
including COSYSMO (1.0). A date indicates the year a model was first published.
Part of the COCOMO II effort was the creation of a general methodology for
developing a parameterized cost estimation model (chapter 4 of (Boehm B. W.,
2000)). This methodology was used in the present research.
10
COSYSMO 1.0 was created in 2005 by Ricardo Valerdi (Valerdi R. , 2005)
(following the model development methodology just discussed). Since then, a
number of models extending COSYSMO 1.0 have been created (as discussed more
fully in section 3). Each of these is based directly on COSYSMO 1.0; otherwise,
they are basically independent of each other (compare Table 3-1). I call each of
these an “extension model”. Here is the set of extension models:
• Systems engineering with reuse ( (Fortune, 2009), (Wang, Ankrum, Valerdi,
& Millar, 2008))
• Systems engineering for reuse ( (Wang, Roedler, Pena, & Valerdi, 2014),
(Wang G. , 2016))
• Systems engineering in the presence of requirements volatility (Pena, 2012)
• Systems engineering in the system-of-systems context ( (Lane, Cost Model
Extensions to Support Systems Engineering Cost Estimation for Complex
Systems and Systems of Systems, 2009), (Lane & Valerdi, 2011))
Figure 4-2 summarizes the status of COSYSMO 1.0 and its extensions.
11
Figure 4-2. The Family of COSYSMO Models.
Each of the four “extension models” is based directly on COSYSMO 1.0, but is otherwise
independent of the others.
4.2 General Structure of a COCOMO-Family Model
At a high level, creating an estimate for a project via a COCOMO-family model
means:
• Projecting (i.e., estimating) values for a set of attributes of the project; then
• Plugging those values into the model to arrive at an estimated number of
labor hours for the project.
For the COCOMO family, each attribute used in the estimate is called a “driver”.
There are four different kinds of drivers; namely:
• Size drivers,
• Reuse factors (which adjust the size),
12
• Scale factors, and
• Cost drivers.
For an estimator to apply a particular COCOMO-family model to arrive at a cost
estimate for a specific project involves going through these steps:
1. The estimator projects the numeric value of each size driver.
2. The estimator determines a textual rating for each of the other drivers. A
textual rating is a qualitative projection of the value of that driver on a
model-provided scale; a typical textual rating scale would be Very
Low/Low/Nominal/High/Very High. The model provides a definition of
each of the textual ratings for each of these drivers, so that the estimator is
choosing a textual rating from the model-provided definitions.
3. For each of the other drivers, the model converts the textual rating into a
numeric rating.
4. The model provides an equation that describes how to mathematically
combine all the numeric ratings into the estimate.
Remarkably, a single equation form is applicable to all the COCOMO-family
models (Figure 4-3). (The details for this equation for the present model will be
presented in section 5.2.3 below.)
Figure 4-3. This equation applies to all COCOMO-family estimating models.
A = a calibration factor, accounting for productivity.
PH = the estimated amount of labor for the project, in person-hours.
AdjSize = the adjusted size of the project, encompassing size drivers and reuse factors.
E = an exponent, based on scale factor numeric values, accounting for diseconomies of scale.
EM
j
= the numeric value of a cost driver.
13
4.3 Methodology for Creating the New Model
A general methodology for developing a parameterized cost estimation model was
created as part of the COCOMO II effort. The methodology is detailed in chapter
4 of (Boehm B. W., 2000); it is summarized in Figure 4-4. The present research is
using this methodology.
Figure 4-4. The USC CSSE methodology for generating a cost estimating model.
(Figure 4.1 from (Boehm B. W., 2000).)
Steps 1-5 (Determine model needs; Analyze existing literature; Perform behavioral
analyses; Define relative significance, data, and ratings; and Perform expert-
judgment Delphi assessment, formulate a priori model) were completed in May of
2017. I call the resulting a priori model “Expert-Based COSYSMO 3.0”; this is
presented in section 5.2. As will be brought out below, the model has now been
finalized.
14
5 Research Results and Conclusions
5.1 Research Hypothesis
In brief, the goal of the present research is to create COSYSMO 3.0, a model based
on COSYSMO 1.0 and integrating the features from the extension models.
More fully, the goal of the present research is to develop a systems engineering
cost estimating model with these properties:
§ Is applicable to a wide range of systems engineering projects;
§ Includes all the major features of COSYSMO 1.0 and its extension models;
§ Provides continuity to users of previous COSYSMO-family models;
§ When calibrated to data from a particular organization, estimates actual
systems engineering costs with a PRED(.30) accuracy of 50%.
5.2 Preliminary Results
This section gives important results on the “Expert-Based Model”, which is one of
the major pieces of information that feeds into the Final Model.
5.2.1 How the Expert-Based Model Was Determined
Early in my model development effort (in May 2014), I formed the advisory
COSYSMO 3.0 Working Group, under the aegis of the Center for Systems and
Software Engineering (CSSE). It consisted of many people interested in the
development of COSYSMO 3.0. Now there are about 25 total people in the
Working Group, though not all of them are active. However, the core of the
Working Group includes eleven people, all experienced, senior systems engineers;
they are from industry, government, and academia; and they all have been advising
COSYSMO model developers going back to COSYSMO 1.0; most of them are
members of a CSSE Affiliate (a company that supports USC’s Center for Systems
and Software Engineering). The Working Group has a regular telecon, usually
15
weekly, where issues in the COSYSMO 3.0 model are discussed and resolved.
The Working Group’s advice has been invaluable in my model development.
Table 5-1. Core Members of the COSYSMO 3.0 Working Group.
These are members that contributed regularly over the years. Members are listed in alphabetical
order.
Member Present Affiliation
Barry Boehm USC
Brad Clark Software Metrics
Bob Epps Retired
Bill Golaz Lockheed Martin
Gary Hafen Retired
Jo Ann Lane San Diego State University
Dan Ligett Softstar Systems
Garry Roedler Lockheed Martin
John Sautter Northrup-Grumman
Gan Wang BAE Systems
Marilee Wheaton The Aerospace Corporation
Perhaps the biggest issue discussed was the overall structure of the cost drivers for
COSYSMO 3.0. Dr Boehm and I proposed a model called the Coalesced Effort
Multipliers (CEM) model. This consisted of only six cost drivers (Table 5-2);
COSYSMO 1.0’s 14 cost drivers were grouped as attributes under these six, as
well as some additional cost driver attributes. There was sentiment in the Working
Group that this was a conceptually simpler cost driver submodel, and it had the
advantage that fewer data points would have been required to validate the model.
However, a stronger sentiment on the Working Group was that this was too
inconsistent with COSYSMO 1.0 and the other extension models; in particular,
organizations that had a large set of existing, rated projects would not be able to
carry those project ratings forward into COSYSMO 3.0 without significant effort.
(I.e., the CEM submodel would violate the “Provides continuity to users of
previous COSYSMO-family models” property from my Research Hypothesis
(section 5.1).) Therefore the CEM submodel for cost drivers was dropped, in favor
of using COSYSMO 1.0’s as a baseline. (Note that this CEM submodel should not
16
be confused with the Composite Effort Multiplier prior data used in generating the
Final Model.)
Table 5-2. The Cost Drivers for the Coalesced Effort Multipliers version of COSYSMO
3.0.
Problem Understanding
Solution Understanding and
Difficulty
Personnel/Team Capability
Productivity Aids
Performance Constraints
Step 5 of the methodology specifies using an expert-judgment “Delphi”
assessment. Delphi is a long-standing technique for determining group consensus
(Helmer, 1966), having originated at Rand in 1948. The original Delphi technique
was enhanced by Boehm and Farquhar to allow discussion between rounds of
voting, so as to enhance the amount of information exchanged among experts; they
called this enhanced technique Wideband Delphi (section 22.2 of (Boehm B. W.,
1981)). I used Wideband Delphi to determine numeric driver values in Expert-
Based COSYSMO 3.0. Here is a brief summary of the steps by which Wideband
Delphi determines a group consensus:
1. The Coordinator presents the experts with the problem (for COSYSMO 3.0,
the numeric value of a driver).
2. Each expert turns in his vote anonymously.
3. The Coordinator summarizes the votes (mean value, spread, etc).
4. The Coordinator presents the summary, and the group discusses the results
of this round of voting. During the discussion, each expert can choose to
discuss his/her vote openly, or can keep it anonymous.
17
5. If the result of the discussion is that the group accepts the mean value as the
consensus, the process terminates; otherwise, the process returns to step 2
for another round of voting.
An alternate method for gaining group consensus is the Analytic Hierarchy Process
(AHP) (Analytic Hierarchy Process, 2015). The Working Group did a small
exercise in assessing the value of a parameter using AHP. One hoped-for benefit
of using AHP is that votes could be gathered through online ballots, with group
meetings not being necessary; however, only a few Working Group members
submitted ballots. Also, we needed an ad hoc method for coming to group
consensus, as AHP just delivers strength of preference among several alternatives,
given a (single) set of pairwise preferences. And, some members of the Working
Group felt that real-time discussions of parameter values can be quite valuable.
For these reasons, the upshot of our AHP trial was that we did not use it further.
5.2.2 Summary of Driver Changes
During the course of Working Group meetings, each driver of COSYSMO 1.0 and
the extension models was examined. Some of these elements were modified, some
were accepted unchanged, and some were deleted; the results are summarized in
Table 5-3.
Table 5-3. Summary of Driver Changes.
Element and Source (COSYSMO 1.0,
if not otherwise noted)
Definition Changes from Source
Size Drivers
System Requirements
Text definition; row added to rating scale with
guidance.
System Interfaces
Text definition.
Algorithms
Text definition.
Operational Scenarios
Text definition.
18
Element and Source (COSYSMO 1.0,
if not otherwise noted)
Definition Changes from Source
Reuse Factors
Development with Reuse factors
(COSYSMO 2.0)
Updated to (Wang G. , 2016).
Development for Reuse factors (Wang
G. , 2016)
None
Scale Factors
Base Exponent
None
Risk/Opportunity Resolution
New
Process Capability
New
Requirements Volatility
(COSYSMO RV)
None
Cost Drivers
CONOPS and Requirements
Understanding
Enhanced from “Requirements
Understanding”.
Architecture Understanding
Viewpoint dropped; definition changed.
Stakeholder Team Cohesion
Minor wording change in one viewpoint.
Level of Service Requirements
Minor change in text definition.
Technology Risk
Minor structural change in viewpoints.
# of Recursive Levels in the Design
A viewpoint dropped; minor definition change.
Development for Reuse
New.
# and Diversity of
Installations/Platforms
Moderate wording change in a viewpoint.
19
Element and Source (COSYSMO 1.0,
if not otherwise noted)
Definition Changes from Source
Migration Complexity
None
Interoperability (Lane & Valerdi, 2011)
Moderate wording changes to definition.
Personnel/Team Capability
Moderate wording changes to text definition.
Process Capability
New.
Personnel Experience/Continuity
Ratings on one viewpoint shifted down a level.
Multisite Coordination
None.
Tool Support
Ratings shifted up one level.
Documentation Match to Life Cycle
Needs
Dropped.
5.2.3 Expert-Based COSYSMO 3.0
This section defines Expert-Based COSYSMO 3.0.
An initial subsection describes Foundational Elements; subsections on Size
Drivers, Cost Drivers, and Scale Factors follow.
There were two unresolved issues in Expert-Based COSYSMO 3.0. In each case,
a new feature of COSYSMO 3.0 has two choices for how to address it. With the
Working Group, I decided to resolve each by deciding to gather project data and
see which choice fit the data better; therefore both choices appear in the present
model. These are the unresolved issues and the two choices for each:
• For interoperability, the suggestion from (Lane & Valerdi, 2011) was to
consider the two choices of adding a new cost driver (found in section
5.2.3.3.10) and of adding a new row to the System Interfaces rating table
(row is highlighted in section 5.2.3.2.2).
20
• For Process Capability, the Working Group could not decide whether
treating it as a cost driver (section 5.2.3.3.12) or treating it as a scale factor
(section 5.2.3.4.2) was more appropriate.
When the Final Model was generated, the Process Capability issue was resolved
(per section 5.3.1.5), and the interoperability cost driver was dropped.
5.2.3.1 Foundational Elements
5.2.3.1.1 Model Scope
The issue is, what system engineering labor cost elements of a project or program
are to be estimated by the model?
The cost elements which might be estimated are all the stages in ISO/IEC15288
(ISO/IEC, 2002). The stages used in the present model are adjusted for
consistency with those in COSYSMO 1.0. (The nominal COSYSMO 1.0 phases of
“Operate, Maintain, or Enhance” and “Replace or Dismantle” are covered in
COSYSMO 3.0 under the latter 15288 stages.)
Section 5.2.2 of (Valerdi R. , 2005) says that the in-scope stages for COSYSMO
1.0, adapted from 15288, are Conceptualize, Develop, Operational Test and
Evaluation, and Transition to Operation (there is also a paragraph on this topic in
section 1.2 of (Valerdi R. , 2005) immediately following its Figure 1.) COSYSMO
3.0 follows this definition of what effort is in its scope.
The costs covered by the COSYSMO model are systems engineering labor costs
for a particular project for a system-of-interest; costs for subsystems on other
projects are not included.
Table 5-4 gives the correspondence between COSYSMO stages and stages from
15288, as specified in chapter 4 of TR 24748-1 (ISO/IEC, 2010).
21
Table 5-4. Correspondence between
COSYSMO 3.0 Stages and TR 24728 Stages.
COSYSMO Stage Corresponding 24748 Stage
Conceptualize Concept
Develop Development, except as covered
in the stage below
Operational Test and Evaluation These Development activities:
verifying and validating the system,
performing appropriate inspections
Transition to Operation Utilization activities directed at
the transition from development
Production Production
Utilization Utilization, except as covered in the
“Transition to Operation” stage
Support Support
Retirement Retirement
5.2.3.1.2 Top-Level Equation
Figure 5-1. This Top-Level Equation Computes the Estimated Effort for a Project.
A = a calibration factor, accounting for productivity.
PH = the estimated amount of labor for the project, in person-hours.
AdjSize = the adjusted size of the project, encompassing size drivers and reuse factors
(covered further in section 5.2.3.1.3).
E = an exponent, based on scale factor numeric values, accounting for diseconomies of scale.
EM
j
= the numeric value of a cost driver (covered further in section 5.2.3.2).
(This expands upon Figure 4-3.)
There is a circumstance when a different formula should be used;
this is described in section 5.2.3.1.7 Multi-Subproject Model.
22
5.2.3.1.3 Size Formula
where:
SD = the particular size driver (requirement, interface, algorithm, or
scenario);
eReq = the raw size value looked up per section 5.2.3.2;
AL
x
= an Activity Level per Table 5-5;
RType = the reuse type (DWR (Developed With Reuse) or DFR (Developed
For Reuse)); and
ALFraction = a value from Table 5-6 as described below.
Figure 5-2. The equations for
the adjusted size and for the partial development factor.
5.2.3.1.4 Activity Levels
The concepts of activity and activity levels are used to handle the partial
development aspect of reuse. That is, Development With Reuse (DWR) and
Development For Reuse (DFR) can imply that some normal activities are skipped
AdjSize
C3
=
SizeDrivers
∑
eReq(Type(SD),Difficulty(SD))×
PartialDevFactor(AL
Start
(SD),AL
End
(SD),RType(SD))
PartialDevFactor(AL
Start
,AL
End
,RType)=
ALFraction(a,RType)
a=AL
Start
AL
End
∑
23
on the project, leading to a reduction in effort; that phenomenon is accounted for in
this section. (Material in this section is adapted from (Wang G. , 2016).)
Figure 5-3. The Systems Engineering V.
This V diagram shows how systems engineers evolve a system, staring with Concept of
Operations and going through Operation and Maintenance.
The activities for Development With Reuse and Development For Reuse
are defined based on this V.
The activity levels are outlined in Table 5-5, and fully defined below. The activity
levels are what is summed over in the equation for Partial Development Factor in
Figure 5-2. There are different activity levels for DWR and DFR.
24
Table 5-5. Overview of the Activity Levels.
DWR AL DFR AL AL #
New Conceptualized for Reuse 1
Design Modified (not used) 2
Design Implemented Designed for Reuse 3
Adapted for Integration Constructed for Reuse 4
Adopted for Integration (not used) 5
Managed Validated for Reuse 6
Before giving the full definitions of DWR and DFR activity levels, the numerical
results are given in Table 5-6; this table gives numeric values for ALFraction for
use in the PartialDevFactor equation in Figure 5-2.
Table 5-6. Numerical Values for
ALFraction for DWR and DFR Activity Levels.
The blue rows give the values for ALFraction
for use in the PartialDevFactor equation in Figure 5-2.
The pink rows give running totals for convenience.
25
Table 5-7 gives full definitions of the DWR activity levels, while Figure 5-4 gives
a diagrammatic view based on the systems engineering V.
Figure 5-4. Diagrammatic View of the DWR Activity Levels.
Table 5-7. Definitions of the DWR Activity Levels.
DWR AL Definition AL
#
New
System attribute that is new, which requires developing from scratch;
or from previously defined system design or constructed product
components but requiring near-complete changes in system architecture
as a result of modified or extended system functionalities.
1
Design Modified
System attribute that is designed and developed by leveraging
previously defined system concept, functional and logical reference
architecture; or from previously designed physical architecture or
constructed product components which requires significant design and
implementation changes or refactoring but without major changes in
system functionalities
2
Design
Implemented
System attribute that is implemented from an inherited, completed
system design or a previously constructed product component that may
require only limited design changes in the physical architecture to an
extent that it will not impact or change the basic design but that may
require reimplementation of the component.
3
26
DWR AL Definition AL
#
Adapted for
Integration
System attribute that is integrated from adaptation or tailoring (by
limited modification of interfaces) of previously constructed or
deployed product components without changes in the system
architecture and design or the physical implementation except for those
related to interface changes so that the adapted element can be
effectively integrated or form fit into the new system. The effort
required is relatively lower than that of the Design Implemented
category. This category includes removal of system element from
previously developed or deployed system baseline.
4
Adopted for
Integration
System attribute that is incorporated or integrated from previously
developed or deployed product components without modification,
which requires complete integration, assembly, test and checkout
activities as well as V&V testing. This is also known as “black-box”
reuse or simple integration.
5
Managed
System attribute that is inherited from previously developed and
validated product components without modification and the integration
of such an element, if required, is through significantly reduced V&V
testing effort by means of inspection or provided test services,
procedures and equipment. Most of the systems engineering effort
incurred is a result of technical management.
6
Table 5-8 gives full definitions of the DWR activity levels, while Figure 5-5 gives
a diagrammatic view based on the systems engineering V.
Figure 5-5. Diagrammatic View of the DFR Activity Levels.
27
Table 5-8. Definitions of the DFR Activity Levels.
DFR AL Definition AL
#
Conceptualized for
Reuse
This Activity Level encapsulates a set of front-end
systems engineering activities from which reusable
resource produced is a logical or functional architecture
that must be further developed through a series of
detailed design, implementation, verification and
validation testing activities to realize the final deployable
product.
1
Designed for Reuse
This Activity Level encapsulates a set of front-end of
system design activities from which reusable resource
produced is a complete system design or physical
architecture that must be further developed through a
series of implementation, integration, verification and
validation testing activities to realize the final deployable
product.
3
Constructed for Reuse
This Activity Level encapsulates a set of system
development activities from which reusable resource
produced is a physical product or component that has
been implemented and independently verified through
verification testing but has not been deployed or used in
an end system. This requires all levels of system
development activities short of final system-level
integration, transition, verification and validation testing.
4
Validated for Reuse
This Activity Level encapsulates the entire set of system
development activities from which reusable resource
produced is a physical product or component that has
been developed, deployed, and operational validated
through its use in an end system.
6
28
5.2.3.1.5 Exponent Formula
Figure 5-6. The Equation for the Exponent E.
All exponents (scale factors) are described in section 5.2.3.3.1.
5.2.3.1.6 Allowed Numeric Levels for Cost Driver and Scale Factor Ratings
Rating levels for a cost driver or a scale factor are on a scale of
Very Low/Low/Nominal/High/Very High/Extra High, or some contiguous subset
of that. A simple correspondence of these to integers is -2/-1/0/1/2/3.
Non-integer rating levels are also allowed; specifically, a number that is 25%,
50%, or 75% of the way from one allowed integer to another is allowed.
5.2.3.1.7 Multi-Subproject Model
A project will have one or more subprojects. A subproject is a part of the project’s
work where effort multiplier ratings are all the same. If there are different parts of
the work that have different effort multiplier ratings, then each such part of the
work is a different subproject. On the other hand, if all parts of the work have the
same effort multiplier ratings, then the project has a single subproject.
A common cause of separate subprojects is when part of a project is subcontracted
or otherwise performed by a separate team. Another possible multi-subproject
situation is one where a system has multiple subsystems, the project is at the
system level, and work on each subsystem is a separate subproject.
E= E
Base
+SF
ROR
+SF
PC
+SF
RV
29
Many projects are performed by multiple organizations within a company; a
typical case is where different engineering specialties belong to separate
organizations. By itself, this does not mean there are multiple subprojects. Rather,
separate subprojects may occur when distinct teams perform the same
15288/24748 (ISO/IEC, 2010) stages of work.
The formula that applies to the multi-subproject project is given in Figure 5-7; this
supersedes the formula in section 5.2.3.1.2.
Figure 5-7. The Formula for Total Project Effort
When Subprojects are Used.
The subscript “C3” just means COSYSMO 3.0, and can be ignored;
PM
C3M
is the total COSYSMO 3.0 effort when subprojects are used.
This formula is taken from (Lane, 2009).
If a project has only one subproject, the multi-subproject model should be ignored.
(If it is used anyway, the results will be the same.)
PM
C3M
= A
C3
⋅(TotalSize
C3
)
E
C3
⋅ (
Subproject
s
Size
C3
TotalSize
C3
⋅ EM
C3:s,j
j=1
15
∏
)
s∈Subprojects
∑
30
5.2.3.2 Size Drivers
There are four types of size driver: system requirements, system interfaces,
algorithms, and operational scenarios. Each size driver needs to be given an
easy/nominal/difficult ranking to determine its size in eReqs (equivalent nominal
requirements) according to Table 5-9.
Table 5-9. Table for Assigning eReqs to a Size Driver.
Each size driver includes a rating scale. A person doing a rating is to examine the
table, and choose an overall Easy/Nominal/Difficult rating based on his/her
judgment.
5.2.3.2.1 System Requirements
Text Definition: This driver represents the number of requirements for the system-
of-interest at the system level or the level of “sell-off” to the customer, which may
include derived requirements at the same Level. The quantity of requirements
includes those related to the effort involved in engineering the system interfaces,
system specific algorithms, and operational scenarios. Requirements may be
functional, performance, feature, or service-oriented in nature depending on the
methodology used for specification. They may also be defined by the customer or
contractor. Each requirement must have systems engineering effort associated
with it such as verification and validation, functional decomposition, functional
allocation, etc. System requirements can typically be quantified by counting the
number of applicable “shalls” in the system or marketing specification. Note on
“shall”: that word is used by the US Department of Defense to flag requirements.
In other contexts other words may be used for this purpose, such as “will”, “must”,
“should”, “may”, or “provides”; use a consistent word or combination of words
appropriate to your context.
31
Rating Scale:
Easy Nominal Difficult
Simple to implement Moderately difficult to implement Complex to implement or
engineer
Traceable to source Can be traced to source with some
effort
Hard to trace to source
Little requirements overlap Some overlap High degree of requirements
overlap
Verification of the requirement
is easy; this is typically the case
for verification techniques of
Inspection, Analysis, Analogy,
Demonstration, and Sampling.
Verification of the requirement is
nominal; this is typically the case
for the verification technique of
Testing in normal cases.
Verification of the requirement
is difficult; this is typically the
case for the verification
technique of Testing in difficult
cases, such as where
additionally verifying via a
second technique, or where a
new test station is needed.
Guidance: When evaluating the verification method (the fourth viewpoint in the
rating table), the cases given are merely “typical”; judgment should be used in
determining the rating for this viewpoint. For example, a requirement with a
difficult Analysis may be rated Nominal or even Difficult on this viewpoint,
depending on the difficulty of the analysis.
5.2.3.2.2 System Interfaces
Text Definition: This driver represents the number of shared physical and logical
boundaries between system components or functions (internal interfaces) and those
external to the system (external interfaces). These interfaces typically can be
quantified by counting the number of external and internal system interfaces
among ISO/IEC 15288-defined system elements.
32
Rating Scale:
Easy Nominal Difficult
Simple & straightforward Moderate complexity Complex protocol(s)
Uncoupled Loosely coupled Highly coupled
Strong consensus Moderate consensus Low consensus
Well behaved Predictable behavior Poorly behaved
Domain or enterprise
standards employed
Functional standards employed
Isolated or connected systems
with few or no standards
Per section 5.2.3, the last viewpoint is under evaluation as one way of capturing
interoperability; it may or may not end up in the final model.
5.2.3.2.3 Algorithms
Text Definition: This driver represents the number of mathematical algorithms to
be derived in order to achieve the system functional and performance
requirements. The number can be quantified by counting the number of unique
algorithms needed to realize key system requirements specified in the system
specification or architecture description document. As an example, this could
include a complex aircraft tracking algorithm like a Kalman Filter being derived
using existing experience as the basis for the all aspect search function. Another
example could be a discrimination algorithm being derived to identify friend or foe
function in space-based applications.
33
Rating Scale:
Easy Nominal Difficult
- Algebraic - Straight forward calculus
- Complex constrained
optimization; pattern recognition
- Straightforward structure
- Nested structure with decision
logic
- Recursive in structure with
distributed control
- Simple data - Relational data - Noisy, ill-conditioned data
- Timing not an issue - Timing a constraint
- Dynamic, with timing and
uncertainty issues
- Adaptation of library-based
solution
- Some modeling involved
- Simulation and modeling
involved
5.2.3.2.4 Operational Scenarios
Text Definition: This driver represents the number of operational scenarios that a
system must satisfy in order to accomplish its intended mission or mission
objectives. An operational scenario must be end-to-end and triggered by an
operational event. Such scenarios include both the nominal stimulus-response
thread plus all of the off-nominal threads resulting from bad or missing data,
unavailable processes, or other exceptional conditions. The number of scenarios
can typically be quantified by counting the number of system-level use cases
developed as part of the operational architecture or by counting operational modes
captured in the user manual.
34
Rating Scale:
Easy Nominal Difficult
- Well defined - Loosely defined - Ill defined
- Loosely coupled - Moderately coupled
- Tightly coupled or many
dependencies/conflicting
requirements
- Timelines not an issue - Timelines a constraint
- Tight timelines through scenario
network
- Few, simple off-nominal threads
- Moderate number or complexity of
off-nominal threads
- Many or very complex off-
nominal threads
35
5.2.3.3 Cost Drivers
5.2.3.3.1 CONOPS and Requirements Understanding
”CONOPS” means “concept of operations”. Usually a project is guided by a
Concept of Operations Document (“the CONOPS”), which provides a customer’s
concepts on how a system needs to operate in key scenarios.
Text Definition: The extent to which the Stakeholders and Team understand the
system's concept of operations. In addition, this cost driver rates the level of
understanding of the system requirements by all stakeholders including systems,
software, hardware, customers, team members, users, etc. Primary sources of
added systems engineering effort are unprecedented systems, unfamiliar domains,
or systems whose requirements are emergent with use.
Rating Scale:
Viewpoints
:
Very low Low Nominal High Very High
Degree of
Understand
-ing of
CONOPS
The
Stakeholders
and Team
have a poor
understanding
of the
CONOPS.
The
Stakeholders
and Team
have a
mediocre
understanding
of the
CONOPS.
The
Stakeholders
and Team
have an
average
understanding
of the
CONOPS.
The
Stakeholders
and Team
have a good
understanding
of the
CONOPS.
The
Stakeholders
and Team
have a strong
understanding
of the
CONOPS.
Unresolved
Issues in
the
CONOPS
Multiple
critical issues
are
unresolved in
Stakeholders’
understanding
of the
CONOPS
One critical
issue is
unresolved in
Stakeholders’
understanding
of the
CONOPS
No critical
issues are
unresolved in
Stakeholders’
understanding
of the
CONOPS, but
several
significant
issues are
unresolved
No critical
issues are
unresolved in
Stakeholders’
understanding
of the
CONOPS, but
a few
significant
issues are
unresolved
No critical or
significant
issues are
unresolved in
Stakeholders’
understanding
of the
CONOPS
36
Viewpoints
:
Very low Low Nominal High Very High
Requireme
nts Under-
standing
Poor:
emergent
requirements
or
unprecedente
d system
Minimal:
many
undefined
areas
Reasonable:
some
undefined
areas
Strong: few
undefined
areas
Full
understanding
of
requirements,
familiar
system
User
Training
The user
community
has received
little or no
training on
the
new/modified
system,
including on
any new
technology in
use.
The user
community
has received a
less than
average
amount of
training on
the
new/modified
system,
including on
any new
technology in
use.
The user
community
has received
an average
amount of
training on
the
new/modified
system,
including on
any new
technology in
use.
The user
community
has received a
greater than
average
amount of
training on
the
new/modified
system,
including on
any new
technology in
use.
The user
community
has received a
superior
amount of
training on
the
new/modified
system,
including on
any new
technology in
use.
Guidance: There is a potential problem in the interaction between the
Requirements Understanding Viewpoint rating and the Requirements Volatility
scale factor rating (section 5.2.3.4.3); it is possible to interpret high Requirements
Volatility as low Requirements Understanding, which would lead to counting one
phenomenon in both ratings. If you feel that you have a sound understanding of
the distinction between requirements understanding and requirements volatility
influences on your project, then no special action is needed. On the other hand, if
you are uncertain in this area, then it is recommended that whenever the
Requirements Volatility scale factor is rated above Very Low, the Requirements
Understanding viewpoint should be ignored in rating CONOPS and Requirements
Understanding.
5.2.3.3.2 Architecture Understanding
Text Definition: This cost driver rates the degree of understanding of determining
and managing the system architecture in terms of platforms, standards, new and
NDI (COTS/GOTS) components, connectors (protocols), and constraints. This
37
includes tasks like systems analysis, tradeoff analysis, modeling, simulation, case
studies, etc.
Rating Scale:
Very low Low Nominal High Very High
Poor
understan
ding of
architectu
re and
NDI,
unpreced
ented
system
Minimal
understanding
of architecture
and NDI, many
unfamiliar
areas
Reasonable
understanding of
architecture and
NDI, some
unfamiliar areas
Strong
understanding of
architecture and
NDI, few
unfamiliar areas
Full understanding of
architecture, familiar
system and NDI
5.2.3.3.3 Stakeholder Team Cohesion
Text Definition: Represents a multi-attribute parameter, which includes
leadership, shared vision, diversity of stakeholders, approval cycles, group
dynamics, Integrated Product Team framework, team dynamics, trust, and amount
of change in responsibilities. It further represents the heterogeneity in stakeholder
community of the end users, customers, implementers, and development team.
Rating Scale:
Very Low Low Nominal High Very High
Culture
Stakeholders
with diverse
expertise,
task nature,
language,
culture,
infrastructure
. Highly
heterogeneou
Heterogeneo
us
stakeholder
community.
Some
similarities in
language and
culture
Shared
project
culture
Strong team
cohesion and
project
culture.
Multiple
similarities in
language and
expertise
Virtually
homogeneous
stakeholder
communities.
Institutionaliz
ed project
culture
38
s stakeholder
communities
Compatibility
Highly
conflicting
organizationa
l objectives
Converging
organizationa
l objectives
Compatible
organizationa
l objectives
Clear roles &
responsibiliti
es
Strong
mutual
advantage to
collaboration
Familiarity
and Trust
Complete
lack of
familiarity
Willing to
collaborate,
little
familiarity
Some
familiarity
and trust
Extensive
successful
collaboration
Very high
level of
familiarity
and trust
5.2.3.3.4 Level of Service Requirements
Text Definition: This cost driver rates the difficulty and criticality of satisfying the
ensemble of level of service requirements, such as security, safety, response time,
maintainability, Key Performance Parameters (KPPs), system qualities (formerly
known as the “ilities”), etc.
Rating Scale:
Very low Low Nominal High Very High
Difficult
y
Simple; single
dominant KPP
Low, some
coupling
among KPPs
Moderately
complex,
coupled KPPs
Difficult,
coupled KPPs
Very
complex,
tightly
coupled KPPs
Criticalit
y
Slight
inconvenience
Easily
recoverable
losses
Some loss
High financial
loss
Risk to
human life
39
5.2.3.3.5 Technology Risk
Text Definition: The maturity, readiness, and obsolescence of the technology
being implemented. Immature or obsolescent technology will require more
Systems Engineering effort.
Rating Scale:
Very Low Low Nominal High Very High
Lack of
Maturity/
Readiness
Technology
proven and
widely used
throughout
industry.
Mission proven.
(TRL 9)
Proven through
actual use and
ready for
widespread
adoption.
Concept
qualified. (TRL
8)
Proven on pilot
projects and
ready to roll- out
for production
jobs. Concept
has been
demonstrated.
(TRL 7)
Ready for pilot
use. Proof of
concept
validated. (TRL
5 & 6)
Still in the
laboratory.
Concept
defined. (TRL 3
& 4)
Obsolesce
nce
- Technology is
the state-of-the-
practice -
Emerging
technology could
compete in
future
- Technology is
stale - New and
better
technology is
ready for pilot
use
- Technology is
outdated and use
should be
avoided in new
systems - Spare
parts supply is
scarce
5.2.3.3.6 # of Recursive Levels in the Design
Text Definition: The number of levels of design related to the system-of-interest
(as defined by ISO/IEC 15288).
40
Rating Scale:
Very
Low
Low Nominal High Very High
Number
of levels
1 2 4 6 >7
5.2.3.3.7 Development For Reuse
Text Definition: Is the project (or subproject) developing artifacts to be reused on
later project(s)? (“Development for Reuse”, or “DFR”.) If so, what is the extent
of the planned reuse?
Rating Scale:
Low Nominal High Very High Extra High
No reuse at all. Artifacts will be
reused only on
the current
project.
Artifacts will
be reused
across the
program.
Artifacts will
be reused
across a
product line.
Artifacts will
be reused
across
multiple
product lines.
5.2.3.3.8 # and Diversity of Installations/Platforms
Text Definition: The number of different platforms that the system will be hosted
and installed on. The complexity in the operating environment (space, sea, land,
fixed, mobile, portable, information assurance/security, constraints on size weight,
and power). For example, in a wireless network it could be the number of unique
installation sites and the number of and types of fixed clients, mobile clients, and
servers. Number of platforms being implemented should be added to the number
being phased out (dual count).
41
Rating Scale:
Nominal High Very High Extra High
Sites/
installations
Single installation
site or configuration
2-3 sites or diverse
installation
configurations
4-5 sites or diverse
installation
configurations
>6 sites or diverse
installation
configurations
Operating
environment
Existing facility
meets all known
environmental
operating
requirements
Moderate
environmental
constraints;
controlled
environment (i.e.,
A/C, electrical)
Ruggedized mobile
land-based
requirements; some
information security
requirements.
Coordination between
1 or 2 regulatory or
cross functional
agencies required.
Harsh environment
(space, sea airborne)
sensitive information
security requirements.
Coordination between
3 or more regulatory
or cross functional
agencies required.
Platforms
<3 types of platforms
being installed and/or
being phased
out/replaced
4-7 types of
platforms being
installed and/or being
phased out/replaced
8-10 types of
platforms being
installed and/or being
phased out/replaced
>10 types of
platforms being
installed and/or being
phased out/replaced
Homogeneous
platforms
Compatible platforms
Heterogeneous, but
compatible platforms
Heterogeneous,
incompatible
platforms
Networked using a
single protocol
Networked using a
single protocol and
multiple operating
systems
Networked using a
mix of protocols;
single operating
system
Networked using a
mix of protocols;
multiple operating
systems
5.2.3.3.9 Migration Complexity
Text Definition: This cost driver rates the extent to which the legacy system
affects the migration complexity, if any. Legacy system components, databases,
workflows, environments, etc., may affect the new system implementation due to
new technology introductions, planned upgrades, increased performance, business
process reengineering, etc.
42
Rating Scale:
Nominal High Very High Extra High
Legacy
contractor
Self; legacy system is
well documented.
Original team largely
available
Self; original
development team not
available; most
documentation
available
Different contractor;
limited
documentation
Original contractor
out of business; no
documentation
available
Effect of
legacy system
on new
system
Everything is new;
legacy system is
completely replaced
or non-existent
Migration is restricted
to integration only
Migration is related to
integration and
development
Migration is related
to integration,
development,
architecture and
design
5.2.3.3.10 Interoperability
Note: While Interoperability is part of Expert-Based COSYSMO 3.0, it is not part
of the Final Model, per section 5.3.1.3.
Text Definition: How extensive are the interoperability requirements?
Interoperability is defined as “The ability of a system to work with another system
or group of systems”. External interoperability (interoperability with other
systems) is always considered. When the system of interest is a system-of-
systems, internal interoperability (interoperability between constituent systems)
also applies.
Rating Scales:
There are two different rating scales; the appropriate one should be selected,
depending on whether the project is for an existing system or for a new system.
43
Existing System Rating Scale:
Viewpoint Very Low Low Nominal High Very High
External In-
teroperability (What is
the system’s status with
regard to
interoperability (before
being integrated into
the system-of-
systems)?)
Isolated. Connected.
No interopera-
bility require-
ments; or
functional
standards em-
ployed.
Domain stand-
ards employed.
Enterprise
standards em-
ployed.
Internal In-
teroperability
There are a
very large
number of
significant
inconsistency/
incompatibility
issues in
standards, da-
tabases, and
interfaces
among con-
stituent sys-
tems.
There are a
large number
of significant
inconsistency/
incompatibility
issues in
standards,
databases, and
interfaces
among con-
stituent sys-
tems.
This is not a
system-of-
systems; or
there are a
moderate
number of
significant
inconsistency/
incompatibility
issues in
standards, da-
tabases, and
interfaces
among con-
stituent sys-
tems.
There are a
few significant
inconsistency/
incompatibility
issues in
standards, da-
tabases, and
interfaces
among con-
stituent sys-
tems.
There are no
significant
inconsistency/
incompatibility
issues in
standards,
databases, and
interfaces
among
constituent
systems.
44
New System Rating Scale:
Viewpoint Very Low Low Nominal High Very High
External
Interoperability
System-
specific data.
Documented
data.
No
interoperability
requirements;
or aligned static
data.
Aligned
dynamic data.
Harmonized
data.
Internal
Interoperability
Existing
constituent
systems do
not
interoperate,
and they are
large and
complex.
Existing
constituent
systems do not
interoperate, but
they are simple
and/or small.
This is not a
system-of-
systems; or all
constituent
systems are
new; or all
existing
constituent
systems
presently
interoperate.
5.2.3.3.11 Personnel/Team Capability
Text Definition: Composite systems engineering capability of a team of Systems
Engineers (compared to the national pool of SEs) on the attributes of analyzing
complex problems and synthesizing solutions, being efficient and thorough, and
having the ability to communicate and cooperate.
Rating Scale:
Very Low Low Nominal High Very High
15
th
percentile 35
th
percentile 55
th
percentile 75
th
percentile 90
th
percentile
45
5.2.3.3.12 Process Capability
Note: Process Capability is considered as a possible cost driver for the Expert-
Based Model (per section 5.2.3), but was decided to be (only) a scale factor in the
Final Model (per section 5.3.1.5). In any case, this section provides its definition.
Text Definition: The consistency and effectiveness of the project team at
performing SE processes. This may be based on assessment ratings from a
published process model (e.g., CMMI, EIA-731, SE-CMM, ISO/IEC15504). It can
alternatively be based on project team behavioral characteristics, if no assessment
has been made.
46
Rating Scale (“SEMP” = Systems Engineering Management Plan):
Very
Low
Low Nominal High Very High Extra High
Assessment
Rating
Level 0
(if
contin-
uous
model)
Level 1 Level 2 Level 3 Level 4 Level 5
Project Team Behavioral
Characteristics
Ad Hoc
approac
h to
process
perform
ance.
Performed SE
process;
activities
driven only by
immediate
contractual or
customer
requirements;
SE focus
limited.
Managed SE
process;
activities
driven by
customer and
stakeholder
needs in a
suitable
manner; SE
focus is
requirements
through design;
project- centric
approach – not
driven by
organizational
processes.
Defined SE
process;
activities
driven by
benefit to
project; SE
focus is
through
operation;
process
approach
driven by
organizational
processes
tailored for the
project.
Quantitatively
Managed SE
process;
activities
driven by SE
benefit; SE
focus on all
phases of the
life cycle.
Optimizing SE
process;
continuous
improvement;
activities
driven by
system
engineering
and
organizational
benefit; SE
focus is product
life cycle &
strategic
applications.
SEMP Sophistication
Manage
ment
judgmen
t is used
SEMP is used
in an ad-hoc
manner only
on portions of
the project that
require it
Project uses a
SEMP with
some
customization
Highly
customized
SEMP exists
and is used
throughout the
organization
The SEMP is
thorough and
consistently
used;
organizational
rewards are in
place for those
that improve it
Organization
develop best
practices for
SEMP; all
aspects of the
project are
included in the
SEMP;
organizational
rewards exist
for those that
improve it
5.2.3.3.13 Personnel Experience/Continuity
Text Definition: The applicability and consistency of the staff at the initial stage of
the project with respect to the domain, customer, user, technology, tools, etc.
47
Rating Scale:
Very low Low Nominal High Very High
Experience Up to 1 year
experience
3 years of
continuous
experience
5 years of
continuous
experience
10 years of
continuous
experience
20 years of
continuous
experience
Annual
Turnover
48% 24% 12% 6%
3%
5.2.3.3.14 Multisite Coordination
Text Definition: Location of stakeholders, team members, resources, corporate
collaboration barriers.
48
Rating Scale:
Very Low Low Nominal High Very High Extra High
Collocation
International,
severe time
zone impact
Multi-city and
multi-
national,
considerable
time zone
impact
Multi-city or
multi-
company,
some time
zone effects
Same city or
metro area
Same building or
complex, some
co-located
stakeholders or
onsite
representation
Fully co-
located
stakeholders
Commun-
ications
Some phone,
mail
Individual
phone, FAX
Narrowband
e-mail
Wideband
electronic
communication
Wideband
electronic
communication,
occasional video
conference
Interactive
multimedia
Corporate Collaboration
Barriers
Severe export
and security
restrictions
Mild export
and security
restrictions
Some
contractual &
Intellectual
property
constraints
Some
collaborative
tools &
processes in
place to
facilitate or
overcome,
mitigate barriers
Widely used and
accepted
collaborative
tools &
processes in
place to facilitate
or overcome,
mitigate barriers
Virtual team
environment
fully
supported by
interactive,
collaborative
tools
environment
5.2.3.3.15 Tool Support
Text Definition: Coverage, integration, and maturity of the tools in the Systems
Engineering environment.
49
Rating Scale:
Very low Low Nominal High Very High
No SE
tools, or
simple
SE tools
with little
integratio
n.
Basic SE tools
moderately
integrated
throughout the
systems
engineering
process
Strong, mature SE
tools, moderately
integrated with
other disciplines.
Cover many parts
of the life cycle.
Strong, mature
domain model-
based life cycle
tools. Cover all
important parts
of the life cycle.
Strong model
and consistency
checking,
integration with
management
tools.
Very strong, mature,
domain model-based,
knowledge-based life
cycle tools. Cover the
complete life cycle.
Thorough integration
across life cycle and
management tools.
Advanced knowledge-
based diagnosis of
leading risk indicators
50
5.2.3.4 Scale Factors
5.2.3.4.1 Risk/Opportunity Resolution
Text Definition: This driver captures the project’s use of a comprehensive,
effective risk/opportunity management process, and the amount of risk on the
current project.
Rating Scale:
Viewpoint:
Very
Low
Low Nominal High Very High Extra High
A life cycle-
long, funded
process for
identifying,
tracking, and
resolving risks
and
opportunities
is carried out.
No such
process,
or the
process
is very
weak.
The
process
is weak.
The process is
moderate.
The process is
fairly strong.
The process is
strong.
The process is
very strong.
A culture of
risk and
opportunity
identification,
tracking, and
resolution is
part of the
organization.
Very
weak
culture.
Weak
culture.
Moderate
culture,
including
experience in
risk/opportunity
management.
Fairly strong
culture,
including fairly
successful
experience in
risk/
opportunity
management.
Strong culture,
including
mostly
successful
experience in
risk/
opportunity
management.
Very strong
culture,
including very
successful
experience in
risk/
opportunity
management.
Number and
criticality of
risk items.
> 10
Critical
5-10
Critical
2-4 Critical 1 Critical
> 5 Non-
Critical
< 5 Non-
Critical
51
5.2.3.4.2 Process Capability
Note: Process Capability is considered as a possible scale factor in the Expert-
Based Model. It was decided to be (only) a scale factor in the Final Model (per
section 5.3.1.5).
The definition of Process Capability is given in section 5.2.3.3.12.
5.2.3.4.3 Requirements Volatility
Text Definition: Requirements volatility is defined as unplanned changes in
requirements over a given set of process stages during the system’s life cycle.
These changes may include additions, modifications or deletions.
52
Rating Scale:
Characteristic:
Very
Low
Low
Moderat
e
High
Very
High
Weight
< 1.5 >1.5-2.5 >2.5-3.5 >3.5-4.5 > 4.5
System requirements baselined and
agreed to by key stakeholders
Fully 1
Mostly 2
Generally
3
Some-
what 4
No
Agree-
ment 5
26%
Level of uncertainty in key customer
requirements, mission objectives, and
stakeholder needs
Very Low
1
Low 2
Moderate
3
High 4
Very
High 5
22%
Number of co-dependent systems with
influence on system requirements
Very Low
1
Low 2
Moderate
3
High 4
Very
High 5
16%
Strength of your organization’s re-
quirements development process and
level of change control rigor
Very
High 1
High 2
Moderate
3
Low 4
Very Low
5
8%
Precedentedness of the system, use of
mature technology
Very
High 1
High 2
Moderate
3
Low 4
Very Low
5
9%
Stability of stakeholders' organizations
(developer, customer)
Very
High 1
High 2
Moderate
3
Low 4
Very Low
5
14%
Experience level of the systems engi-
neering team in requirements analysis
and development
Very
High 1
High 2
Moderate
3
Low 4
Very Low
5
6%
53
5.2.3.5 Numeric Values of Ratings
This section gives numeric values for the parameters of Expert-Based COSYSMO
3.0.
The numeric values for the reuse factors are found above in Table 5-6. The
numeric values for cost drivers are given in Table 5-10; those for scale factors are
given in Table 5-11.
Table 5-10. Numeric Values of Ratings for Cost Drivers.
This table allows a rating for a cost driver
to be translated into the corresponding numerical value.
The EMR is the ratio between a cost driver’s (numerically) highest and lowest values.
The ratings for a cost driver form a geometric sequence.
54
Table 5-11. Numeric Values of Ratings for Exponents.
This table allows a rating for a scale factor
to be translated into the corresponding numerical value.
The ratings for a scale factor form an arithmetic sequence.
5.2.4 Continuity and The Rosetta Stone
As detailed in Table 5-3, some of the driver definitions have changed since
COSYSMO 1.0. Users of older versions of COSYSMO need to be able to
meaningfully apply COSYSMO 3.0 to their past projects (“continuity”). A Rosetta
Stone document is provided for this purpose, telling COSYSMO 3.0 users how to
re-rate drivers from previous versions of COSYSMO. Instructions for re-rating are
either “Same rating” or “Adjust old rating by x steps”.
55
Table 5-12. The Rosetta Stone.
This table gives instructions to users of previous versions of COSYSMO as to how to re-rate
their drivers for COSYMO 3.0.
56
5.3 Final Results
5.3.1 Determining the Final Model
Given Expert-Based COSYSMO 3.0 (described in section 5.2) these are the steps
in determining the Final Model:
1. Obtaining a dataset;
2. Seeing what parameters the data set covers, and adjusting the model if
necessary;
3. Generating any additional prior data (beyond the Expert-Based Model);
4. Determining the resolution on the placement of Process Capability in the
model; and
5. Determining the model parameters based on the above results
As the above implies, the Final Model is determined by the dataset, the Delphi
results in the Expert-Based Model, and the additional prior information.
The following subsections go through the steps in that list. There are a couple
additional subsections: the initial subsection is a summary of what remains
undetermined at this point; a subsection after step 4 deals with the remaining
issues.
5.3.1.1 What Is Undetermined about COSYSMO 3.0 at this Point?
The structure of COSYSMO 3.0 is known from section 5.2 results, notably the top-
level equation in section 5.2.3.1.2, except for the exact placement of Process
Capability. What is left to determine is the values of the parameters given in this
table:
Table 5-13. The Parameters of COSYSMO 3.0.
Parameter Group Parameters
Size Multipliers
11 of them: Easy/Nominal/Difficult versus System requirements/
System interfaces/Algorithms/Operational Scenarios, except
Nominal System requirements = 1.0.
57
Parameter Group Parameters
Reuse Level Factors
5 DWR levels plus 4 DFR levels plus “Deleted”, but not “None”,
which = 100%.
Cost Drivers
CONOPS and Requirements Understanding
Architecture Understanding
Stakeholder Team Cohesion
Level of Service Requirements
Technology Risk
# of Recursive Levels in the Design
Development for Reuse
# and Diversity of Installations/Platforms
Migration Complexity
Interoperability
Personnel/Team Capability
Personnel Experience/Continuity
Multisite Coordination
Tool Support
Scale Factors
Risk & Opportunity Management
Requirements Volatility
Cost Driver or Scale
Factor
Process Capability
Constants
A (“Productivity” factor)
Ebase (Constant part of exponent)
5.3.1.2 The Dataset
An anonymous COSYSMO industry expert contributed a dataset of actual
projects—sort of.
What she actually contributed was detailed statistical parameters about a dataset of
actual aerospace projects. From the statistical parameters, I created a synthetic
dataset with the same statistical parameters. Each (synthetic) project includes
ratings on most COSYSMO drivers, which I adjusted to COSYSMO 3.0
parameters using the Rosetta Stone (per section 5.2.4), and actual project effort.
The COSYSMO expert rigorously applied COSYSMO 1.0 definitions to her
ratings, so the resulting ratings should be valid COSYSMO 3.0 ratings.
The result was a dataset of 44 projects, known as the PL4 dataset.
58
The dataset did not have ratings on these drivers: Requirements Volatility,
Development for Reuse, and Interoperability; furthermore, the ratings on sizes and
reuse levels did not turn out to be helpful in fitting corresponding model
parameters.
5.3.1.3 Model Adjustments
In deciding what to do about those uncovered drivers (from the section above), I
considered the Delphi value for the parameter, and whether the Delphi-determined
value had a good grounding.
Table 5-14. Disposition of Parameters without Ratings in the Dataset.
This table gives the disposition of drivers that don’t have ratings in the dataset.
Driver Have
Parameter
Value from
Delphi?
Is Delphi Value Well-
Grounded?
Disposition
Size
Multipliers
Yes Yes: initial values in Delphi
were from COSYSMO 1.0
Use Delphi values.
Reuse Level
Factors
Yes Yes: initial values in Delphi
were from Generalized Reuse
Framework paper (Wang G. ,
2016)
Use Delphi values.
Requirements
Volatility
Yes Yes: initial value in Delphi
was from COSYSMO RV
(Pena, 2012)
Use Delphi values.
Development
for Reuse
Yes Yes: initial value in Delphi
was from COCOMO II
Use Delphi values.
Interoperability Yes No; no initial value was used Drop from model
As can be seen, a consequence of this analysis is that the Interoperability cost
driver is dropped from the model.
59
5.3.1.4 The CEM Prior Information
In 2008, Wang, Valerdi, Boehm, and Shernoff published a paper (Wang G. V.,
2008) suggesting that the step sizes in COSYSMO 1.0 were generally too large,
causing deviations from Nominal ratings to tend to have an unrealistically large
impact on an estimate. That paper proposed as a solution a modification to the
COSYSMO 1.0 estimating formula, called the Composite Effort Multiplier
formula (called “CEM” here; however, not the same as the Coalesced Effort
Multiplier model discussed in section 5.2.1.)
Generalizing a little, that CEM formula for the product of effort multipliers could
be formulated as:
GeoMean(Group1of EffortMultipliers)×GeoMean(Group2of EffortMultipliers)×…
where “GeoMean” means “geometric mean of”. (The paper also includes some
“Early Impact Factors” and some “Late Impact Factors”, but I left that aspect out,
as they’re not part of COSYSMO 3.0.) Now the geometric mean is defined as
GeoMean(EM
1
,…,EM
n
)= EM
i
i=1
n
∏
⎛
⎝
⎜
⎞
⎠
⎟
1/n
.
But, mathematically,
EM
i
i=1
n
∏
⎛
⎝
⎜
⎞
⎠
⎟
1/n
= EM
i
( )
1/n
i=1
n
∏
,
and the effect of that 1/n in the exponent of EM
i
is to raise the EMR of that cost
driver to the 1/n power, thereby reducing its impact as the authors desired.
In discussions with the Working Group, I learned that several Working Group
members were in agreement with this CEM sentiment (and few were opposed). So
the COSYSMO 3.0 Final Model includes a CEM influence, which generally
reduces EMRs.
60
To formulate this CEM influence, I tried to adopt the CEM principle of raising
EMRs to a fractional power. I examined the actual fractions proposed in the CEM
paper (either 1/4 or 1/3), and applied those fractions to get my CEM EMRs. For
COSYSMO 3.0 cost drivers not assigned an exponent in the paper, I tried to guess
the appropriate exponent. I excluded from my CEM priors any COSYSMO 3.0
cost driver not included in COSYSMO 1.0. Here are the results:
Table 5-15. EMRs and Step Sizes for the 12 CEM Priors.
For the choice of Exponents, see the text above.
Driver
COSYSMO
1.0 Step Size
Chosen CEM
Exponent
Resulting CEM
Step Size
Log
1
of CEM
Step Size
CONOPS & Requirements
Understanding 0.754 0.250 0.9319
-0.030648745
Architecture Understanding 0.802 0.250 0.9462 -0.024022088
Stakeholder Team Cohesion 0.798 0.333 0.9275 -0.032680762
Level of Service Requirements 1.280 0.250 1.0636 0.026778673
Technology Risk 1.267 0.250 1.0611 0.02573675
# of Recursive Levels in the
Design 1.179 0.250 1.0420
0.017875445
# and Diversity of Installations/
Platforms 1.244 0.250 1.0561
0.023692376
Migration Complexity 1.259 0.250 1.0593 0.025013378
Personnel/Team Capability 0.781 0.333 0.9208 -0.03581269
Personnel Experience/
Continuity 0.811 0.333 0.9324
-0.03037925
Multisite Coordination 0.890 0.250 0.9714 -0.012606228
Tool Support 0.858 0.250 0.9625 -0.016595333
For the variances for these cost driver priors, I supplied a figure based on a typical
variance for the Delphi priors; in log space, that variance is 0.0029608.
5.3.1.5 Determining the Placement of Process Capability
Process Capability was a cost driver in all earlier COSYSMO models, but it was
suggested by some members of the Working Group that it should be investigated
1
All logarithms in this document (and in the computations underlying COSYSMO 3.0) are to the
base 10, even though some people think that’s common and unnatural.
61
whether Process Capability “fits” better as a cost driver or as a scale factor
(compare section 5.2.3).
So with a preliminary version of the model (actually, with two preliminary
versions: one had Process Capability as a cost driver but not as a scale factor, the
other with it as a scale factor but not a cost driver, but they were otherwise the
same) I determined the respective fits by checking the values of three fit statistics
of each version’s cost driver fit and scale factor fit, whence Table 5-16:
Table 5-16. Comparison of Cost Driver and Scale Factor Fits for Process
Capability as Cost Driver and as Scale Factor.
In the case of Standard Error of the Residuals, a smaller number is better; in the case of R
squared, a number closer to 1.0 is better; in the case of the F-statistic, a larger number is better.
The Working Group and I decided to take Process Capability as a scale factor
based on three arguments:
• Its cost driver fit that way is only a little worse, but its scale factor fit that
way is much better (i.e., the “PROC as CD” column entries of the above
table are, considered together, worse than the “PROC as SF” column.)
• Generally speaking, members of the Working Group tended to have the
intuition that this is likely better, because to them it seemed intuitively likely
that, for example, a poor process would have a proportionally greater impact
on a larger project; and
62
• This way, its placement agrees with the placement of the corresponding
driver in COCOMO II (PMAT).
5.3.1.6 One Last Issue before Doing Fits
Ebase, the constant element of the exponent equation (per section 5.2.3.1.5),
remains to be determined.
Consider the situation of Requirements Volatility. From section 5.3.1.2 we know
that the data does not cover the Requirements Volatility parameter. When fitting a
model to the data, then, I will assume that each project rated Requirements
Volatility as Nominal.
Compare this to the situation of Development for Reuse. When fitting a model to
the data, I will also assume that each project rated that cost driver as Nominal.
This assumption works very nicely, as rating a cost driver as Nominal is the same
as multiplying its estimated effort by 1.0, so there’s no difference between the
driver not being there and its having a rating of Nominal.
Going back to Requirements Volatility, we see that this situation is not the same: a
Requirements Volatility rating of Nominal means that a value of 0.01894 has to be
added to the exponent (per the Requirements Volatility Delphi result, extracted
from section 5.2.3.5). Therefore the existence of the Requirements Volatility scale
factor cannot be ignored when fitting a model to data.
But it can be adjusted for. A fit would normally provide a value for Ebase, the
exponent constant. But suppose we relabel the fit constant “BEXP_RVOL”; the fit
won’t know the difference, and we can treat BEXP_RVOL as = Ebase + 0.01894.
All we have to do is perform our fit, then do a manual calculation Ebase =
BEXP_RVOL – 0.01894.
With that in mind, and referencing Table 5-13, our present state is now given in
Table 5-17:
63
Table 5-17. The Final Model Parameters.
These are the parameters to be determined by performing fits.
Parameter Group Parameters
Cost Drivers
CONOPS and Requirements Understanding
Architecture Understanding
Stakeholder Team Cohesion
Level of Service Requirements
Technology Risk
# of Recursive Levels in the Design
# and Diversity of Installations/Platforms
Migration Complexity
Personnel/Team Capability
Personnel Experience/Continuity
Multisite Coordination
Tool Support
Scale Factors
Risk & Opportunity Management
Process Capability
Constants
A (“Productivity” factor)
BEXP_RVOL (Constant part of exponent + Nominal RV)
5.3.1.7 Determining the Values of the Final Model Parameters
Subsections of this section cover a couple preliminaries: what linear regression
means in the present context, and a quick tutorial on the applicable parts of
Bayesian data analysis. Then the last subsection covers how the Final Model
parameters were determined.
5.3.1.7.1 What “Doing a Linear Regression Fit” Means in this Context
(This section is based on section 4.6.1 of the COCOMO II book (Boehm B. W.,
2000).)
“Doing a linear regression fit” means finding values for all the parameters in Table
5-17; that means performing more than one linear regression. Here is the
procedure:
64
1. Use the logarithmic form of the estimating equation
and perform a linear regression.
2. That yields fitted values for log(A), E, and log(CD
i
StepSize), E is discarded.
(Note that the CD
i
StepSize can be used to populate a row of a cost driver
value lookup table, because an effort multiplier = the step size to the power
of the numeric rating. The EMR can also be calculated.)
3. This is the “cost driver fit”.
4. Use this form of the estimating equation:
to calculate, with the cost driver fit values, the indicated response variable
(left-hand side).
5. Use that (and the step size ratings) to perform the second linear regression.
This yields fitted values for EBase (or BEXP_RVOL) and SF
i
StepSize.
6. This is the “exponent fit”.
log(ActualEffort)= log(A)+E*log(Size)
+ log(CD
i
StepSize)*CD
i
Rating
i
∑
log(ActualEffort)−log(A)− log(CD
i
StepSize)*CD
i
Rating
i
∑
log(Size)
=EBase+ SF
i
StepSize*SF
i
Rating
i
∑
65
5.3.1.7.2 Bayes
For purposes of this dissertation, Bayesian data analysis lets us determine a model
based on both a dataset and prior information about what a model should look like.
The structure of the process can be diagramed as
Figure 5-8. Structural View of Bayesian Data Analysis.
On both the input and output side of the Bayesian Combination,
“parameter values” includes parameter variance information.
(A wide-ranging reference on Bayesian data analysis is “BDA3”: (Gelman, 2013);
only a few specific BDA3 results on linear regression are used here.)
Linear regressions for this dissertation are done sometimes without prior
information, and sometimes with.
5.3.1.7.3 Determining the Final Model of COSYSMO 3.0
With those preliminaries taken care of, the Final Model was generated by doing a
linear regression against the PL4 dataset using both the Delphi (as represented by
the Expert-Based Model in section 5.2.3) and the CEM information (section
5.3.1.4) as prior information.
The next two tables give the parameters of the Final Model, determined per various
parts of section 5.3.1:
66
Table 5-18. Rated Parameters for the Final Model.
Text Rating: Very Low Low Nominal High
Very
High
Extra
High
Numeric Rating: -2.0 -1.0 0.0 1.0 2.0 3.0
Cost Driver
Step
Size Effort Multipliers
CONOPS &
Requirements
Understanding 0.765 1.71 1.31 1.00 0.76 0.59
(Invalid)
Architecture
Understanding
0.805 1.54 1.24 1.00 0.81 0.65
(Invalid)
Stakeholder Team
Cohesion
0.802 1.55 1.25 1.00 0.80 0.64
(Invalid)
Level of Service
Requirements 1.277 0.61 0.78 1.00 1.28 1.63
(Invalid)
Technology Risk
1.262 0.63 0.79 1.00 1.26 1.59
(Invalid)
# of Recursive
Levels in the
Design 1.179 0.72 0.85 1.00 1.18 1.39
(Invalid)
# and Diversity of
Installations/
Platforms 1.238 (Invalid)
(Invalid)
1.00 1.24 1.53
1.90
Migration
Complexity
1.252 (Invalid)
(Invalid)
1.00 1.25 1.57
1.96
Personnel/Team
Capability
0.831 1.45 1.20 1.00 0.83 0.69
(Invalid)
Personnel
Experience/
Continuity 0.858 1.36 1.17 1.00 0.86 0.74
(Invalid)
Multisite
Coordination 0.812 1.52 1.23 1.00 0.81 0.66
0.54
Tool Support 0.892 1.26 1.12 1.00 0.89 0.80
(Invalid)
Scale Factor
Step
Size Scale Factor Values
Risk &
Opportunity
Management
-0.0120 0.0602 0.0482 0.0361 0.0241 0.0120 0.0000
Process Capability -0.0107 0.0536 0.0429 0.0322 0.0214 0.0107 0.0000
Requirements
Volatility 0.0095 0.0000 0.0095 0.0189 0.0284 0.0379 (Invalid)
67
Table 5-19. Constant Parameters for the Final Model.
A Productivity Factor 26.33
EBase Exponent Base 1.0332
5.3.2 Calibrating to a Data Set
This section determines a model that fulfills the final unmet clause of the research
hypothesis (section 5.1): estimating actual systems engineering costs to an
accuracy of PRED(.30), when calibrated to data from a particular organization. It
turns out that multiple steps are needed.
5.3.2.1.1 Removing Outliers: the PL4O Dataset
The Final Model does not do particularly well on the PL4 database, in the sense
that its PRED(.30) = 29.4%.
The next step is to try to improve accuracy by removing outliers. I used the
“Productivity” measure
2
of a project introduced by COSYSMO 1.0 for this
purpose, the ratio of Actual Hours over Size.
I looked for unusually high Productivity, defined by:
• Is this Productivity at or near the extreme of high Productivities; and
• Is this Productivity separated from the bulk of Productivities by a
significant gap with no projects?
I also looked for unusually low Productivities, with a corresponding definition.
I started by sorting Productivities into order, then looking at the ratio of each
Productivity to the previous one. The resulting list is excerpted in Table 5-20:
2
“Productivity” is a misnomer, in that the ratio yields the inverse of productivity, which is unit
cost.
68
Table 5-20. Analysis of Productivities for Outliers.
The two segments below are from the list of sorted Productivities; the lowest Productivities are
in the left segment, and the highest are in the right segment.
Looking at the low Productivities, the ratio at AR is high, separating the previous
projects from the bulk of Productivities; therefore AM, BF, and AP are considered
outliers. Looking at the high Productivities, the ratio at BT is high, separating BT
and following projects from the bulk of Productivities; therefore BT and BK are
considered outliers.
These five projects were removed from the dataset, resulting in the PL4O dataset.
5.3.2.1.2 PL4O Model and Fit
Two linear regression fits were done on the PL4O dataset; results are as given in
Table 5-21.
Table 5-21. Comparison of Linear Regressions with and without Priors
on the PL4O Dataset.
Model (fit) name: PL4O PL4O.Bayes
Priors used in the regression: None Delphi and CEM Values
F statistic value (on CD fit): 1668 103.1
Fit statistix, in general: Good Moderate
69
PRED(.30): 66.7% 28.2%
How many coefficients fit
significantly?
A few All but 3
Credibility of coefficient
values
Some are non-credible All are credible
The PL4O fit has to be rejected because of non-credible coefficient values; the
PL4O.Bayes fit has to be rejected because of a low PRED(.30).
5.3.2.1.3 Some Observations
Table 5-21, along with other several other fits I did (but are not detailed here), lead
me to suggest that there’s an intrinsic tension between types of fits, per the upper
part of Table 5-22:
Table 5-22. Two Types of Tension between Different Fit Approaches.
“MMRE” means Mean Magnitude of Relative Errors:
A Tension between Types of Fits
Data-Only Fits Bayesian Fits
Data tends to be close to fit Data may be dispersed from
fit
May have non-credible
coefficient values
Coefficients tend to have
credible values
70
Another Tension between Types of Fits
Least-squares Fits Absolute-Deviation Fits
Example fitting techniques:
linear regression, F statistic
Example fitting techniques:
MMRE, PRED
Data far from the middle have
more influence on the fit
All data have equal influence
on the fit
I also note the tension in the second part of the table, based on a theoretical
concern: a least-squares fit tries to minimize the sum of squared errors.
5.3.2.1.4 Hill-Climbing and the nlm function
The practical and theoretical difficulties led me to look for an approach to this fit
not based on linear regression. I found a general hill-climbing function called
“nlm” built into the R statistix package and tried that.
One argument for dropping least-squares not given above is this: A least-squares
approach tries to minimize the sum of squared errors; but that’s not what I’m
looking for. I’m looking to maximize PRED(.30), the percentage of projects
whose estimate is within .30 of the actual. In short, least-squares and I had a
different definition of success, but with nlm hill-climbing I could specify my own
definition of success.
Here is my sketch of how the hill-climbing algorithm works in the context of
model-fitting:
• The user provides the algorithm with:
+ An initial value for the model parameters; and
+ An error function that, given a set of model parameters, yields a
number saying how far those model parameters are from
optimal.
71
• The algorithm starts by trying small variations in the current
parameter values, seeing if any lead to a reduction in the error
function
+ And, if so, which variation leads to the biggest improvement;
+ If no improvement, stop with this local maximum.
• The algorithm modifies the parameter values by taking a step in the
direction of biggest improvement, and repeat previous step.
I applied nlm to the PL4O data set. I gave nlm these arguments:
• The PL4O.Bayes parameter values as an initial value; and
• An error function of PRED(.30). This was computed on the PL4O data set,
as Actuals, with the Estimates generated by applying the current parameter
values to the data set. (In fact, this description of the error function is a
simplification; see section 5.3.2.1.5 for the details.)
The result was a model “PL4O.nlm” with these properties:
• All coefficient values are credible; and
• PRED(.30) on the PL4O dataset was 51.3%.
5.3.2.1.5 Properties of the PL4O.nlm Calibration
In addition to the PRED(.30) = 51.3%, the other property of interest is that all
coefficient values are credible. Table 5-23 gives the fitted parameters, while Figure
5-9 shows the results of the fit.
The exposition in the previous section suggests that the error function used in this
fit is simply PRED(.30); but that wouldn’t work. PRED(.30) is a step function,
constant until it jumps discontinuously, and that doesn’t work with the hill-
climbing algorithm.
I defined and used a function PRED_near which is continuous; it goes up as the
value gets closer to the “hit” range. In other words, it gives partial credit for a
near-miss.
72
I also used a function PRED_count_wrong that counts the number of parameter
values that are in the wrong direction and gives a penalty accordingly.
I also adjusted PRED and PRED_count_wrong so that each gives a bonus if the
parameters are perfect; i.e., in case the PRED value was >= 50% and in case no
parameters are in the wrong direction, respectively.
My error function was PRED * PRED_near * PRED_count_wrong.
Table 5-23. Parameters of the PL4O.nlm Calibration.
Requirements Volatility is not part of this model.
Cost Drivers Step Size
CONOPS & Requirements Understanding 0.905
Architecture Understanding 0.793
Stakeholder Team Cohesion 0.888
Level of Service Requirements 1.324
Technology Risk 1.125
# of Recursive Levels in the Design 1.301
# and Diversity of Installations/ Platforms 1.182
Migration Complexity 1.245
Personnel/Team Capability 0.870
Personnel Experience/ Continuity 0.988
Multisite Coordination 0.793
Tool Support 0.922
Scale Factors Step Size
Risk & Opportunity Management -0.0121
Process Capability -0.0108
Constants
A (Productivity constant) 29.09
Ebase (Exponent constant) 1.0742
73
Figure 5-9. The Result of the PL4O.nlm Fit on the PL4O Dataset.
5.3.3 Attributes of the Final Model
This section summarizes some attributes of the COSYSMO 3.0 Final Model.
Below are: Table 5-24. Summary of Sources of the Model., Table 5-25.
COSYSMO 3.0 Features versus Previous COSYSMO Models., Figure 5-10.
COSYSMO 3.0 Cost Drivers, in order of Increasing EMR., and Table 5-26.
Numerical Properties of COSYSMO 3.0 versus Previous Models.
74
Table 5-24. Summary of Sources of the Model.
C1 = COSYSMO 1.0, PE = published enhancements to COSYSMO 1.0,
WG = COSYSMO 3.0 Working Group.
Element of the Model Source(s)
Basic structure (size, exponent,
cost drivers, equation)
C1, RV
thesis
Reuse submodel PE, Delphi
New drivers (requirements
volatility, risk/opportunity
management, development for
reuse)
PE, WG
Parameters values for size
multipliers, reuse level factors,
requirements volatility,
development for reuse
Delphi
Interoperability (Dropped)
Other parameter values Fits
Placement of process capability Calibration
75
Table 5-25. COSYSMO 3.0 Features versus Previous COSYSMO Models.
Feature
COSYSMO
1.0
COSYSMO
2.0
COSYSMO
RV
GRF
Sys-
of-
Sys
COSYSMO
3.0
Use of a
specific size
measure
X X X X X X
Cost drivers X X X X X X
Use of
exponent on
size
X X X X X X
Addresses
reuse
X X X
Tailoring via
calibration
parameter
X X X X X X
Sophisticated
reuse model
X X
Use of rated
exponents
X X
Account for
increased cost
when
developing for
reuse
X
Subproject
model
X X
Size includes
multiple
artifacts
X X X X X X
Address
interoperability
X
76
Figure 5-10. COSYSMO 3.0 Cost Drivers, in order of Increasing EMR.
The EMR (Effort Multiplier Ratio) of a cost driver is its maximum possible value divided by its
minimum possible value; this is the impact of the cost driver. The product of all EMRs (here,
2198) is an indicator of how sensitive the model is to non-Nominal ratings; one goal of
COSYSMO 3.0 is to reduce this sensitivity from its levels in COSYSMO 1.0.
2.908
2.454
2.381
2.331
2.231
1.888
1.638
1.601
1.530
1.450
1.439
1.349
1.223
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50
Level of Service Requirements
# of Recursive Levels in the Design
Multisite Coordination
Architecture Understanding
Stakeholder/Team Cohesion
Personnel Experience/ Continuity
Development for Reuse
Technology Risk
CONOPS & Requirements
# and Diversity of Installations/
Personnel/Team Capability
Migration Complexity
Tool Support
Cost Driver Impacts (EMRs) in Final COSYSMO 3.0
77
Table 5-26. Numerical Properties of COSYSMO 3.0 versus Previous Models.
Parameter
COSYSMO
1.0
Previous
Proposals
(Together)
Expert-
Based C3
Final
COSYSMO
3.0 PL4O.nlm
# Cost Drivers 14 - 14 13 13
# Scale Factors 0 2 3 3 3
Total EMR 44095 - 145745 2063 2198
A (Productivity) 38.55 38.55 26.33 29.09
Nominal Exponent 1.060 1.060 1.116 1.122 1.162
Multiplier, on
25,000 eReq
project 1.84 1.84 3.24 3.45 5.14
Max - Min
Exponent 0.000 0.038 0.153 0.151 0.152
Multiplier, on
25,000 eReq
project 1.00 1.47 4.73 4.63 4.66
5.4 Research Contributions
5.4.1 Summary of Research Contributions
The contributions of the present research are:
• Creation of the COSYSMO 3.0 Model
• Enhancing the practice of fitting cost-estimating models:
+ Using hill-climbing to fit
+ Improvements in the practice of Bayesian linear regression fits
Using hill-climbing is covered above in sections 5.3.2.1.3 through 5.3.2.1.5. The
benefits of the COSYSMO 3.0 Model are covered in section 5.4.2, while section
5.4.3 covers linear regression fit improvement.
78
5.4.2 The Benefits of the COSYSMO 3.0 Model
There are four benefits from the COSYSMO 3.0 Model:
1. Organizations that use COSYSMO 3.0 will tend to have more accurate
systems engineering cost estimates. This will lead to less budget
overrunning and more effective systems engineering.
2. COSYSMO 3.0 will be applicable to more projects than previous models,
due to its being comprehensive. This will lead to the Model being used on
more projects.
3. COSYSMO 3.0 will tend to be accepted by industry and government, due to
the close collaboration of the Working Group in its development.
4. COSYSMO 3.0 will be non-proprietary and freely available. Therefore
professional organizations can use its existence to upgrade their standard
processes by specifying that creation of a formal cost estimate is a part of
good systems engineering practice.
5.4.3 Improvements to Bayesian Linear Regression Practice
This section is based on material in section 14.2 of BDA3 (Gelman, 2013).
Current Practice. The current practice is to compute a linear regression with
prior data in two steps:
1. Perform an ordinary linear regression, yielding coefficients (parameter
values), both means and variances.
2. Adjust the coefficients’ means and variances one at a time using the
corresponding prior mean and variance. (This is based on a statistical model
called the “single variable model”.) Weight the data and prior information
by the respective precisions (where precision = 1/variance) and compute
weighted sums for mean and precision.
This has a deficiency of not taking into account covariance in prior data (my
Delphi data was based on ballots, and so included covariance information) and a
deficiency of computing results in two steps.
79
Terminology and No-Bayes Solution. To establish terminology for solutions for
linear regression, here is the solution for the no-Bayes case:
X = the matrix of project ratings
y = the vector of actual efforts
Beta-hat = the result vector of coefficients
V-beta = the result covariance matrix for the coefficients.
Then the solution of a linear regression with all variables having equal variance
and no covariance, and with no priors, is
Beta-hat = (X
T
X)
-1
X
T
y
V-beta = (X
T
X)
-1
Including a Covariance Matrix. Suppose X has a covariance matrix Sigma-y
(still with no prior).
One could imagine computing a “Cholesky” factor Sigma-y
-1/2
from Sigma-y.
This Cholesky factor would have the property that Sigma-y
-1/2
times
Sigma-y
-1/2
= Sigma-y
-1
.
Can further imagine multiplying both y and X by Sigma-y
-1/2
and then
plugging those into the linear regression solution
In the process, the covariance matrix becomes
Sigma-y
-1
times Sigma-y = the identity matrix,
so all variances are equal and the previous solution applies.
The solution with a covariance matrix therefore becomes
Beta-hat = (X
T
Sigma-y
-1
X)
-1
X
T
Sigma-y
-1
y
V-beta = (X
T
Sigma-y
-1
X)
-1
.
Note that the result requires computing Sigma-y
-1
but not Sigma-y
-1/2
.
Including Prior Information.
80
Key insight:
Suppose one had prior information (mean and covariance) about
one of the variables Beta
j
.
One could treat this piece of prior information as an additional data point:
An added element of y and an added row of X.
The added element of y would be the prior mean; the added row of X
would include the prior covariance of Beta
j
, plus
“ratings” having a 1 under Beta
j
and zeroes elsewhere.
Intuition: This is an additional “vote” on the value of Beta
j
and
its covariance.
The solution:
Add additional rows of y and X where prior information is available.
(Allows flexibility:
Can include prior information on only some of the variables.
Can include prior information from 2 sources (just add 2 rows).
Can handle variance-only information (single-variable case):
covariance row has zeroes elsewhere.)
Just plug the enhanced y and X into the previous formulas:
Gives Bayesian results in a single computation.
An Implementation Note. I wrote R functions to compute the linear regression
with and without prior information, and used these for a while to get fits.
However, R’s built-in “lm” function computes lots of supplementary information
(residuals, F statistic) that I wanted to get. Recalling the computation (above) that
included a covariance matrix, I wrote another function that computed Sigma-y
1/2
* y and Sigma-y
-1/2
* X, then I feed those results into lm, so that I get the
supplementary information.
81
6 Evaluation and Validation of the Final Model
6.1 Appropriate Use of the Model
This section provides guidance on correct use of COSYSMO 3.0.
The COSYSMO 3.0 Final Model is based, in part, on a dataset; the Model should
not be used to estimate projects that are significantly outside the range covered by
the dataset. The acceptable range for projects is approximately 3,000 to 300,000
hours.
The COSYSMO 3.0 Final Model is a point estimation model; that is, a user puts in
a set of ratings, and the Model produces one number, the estimated effort.
However, simply making a point estimate and then basing one’s project planning
on that estimate is bad practice. Good estimation practice requires making
multiple estimates on a range of ratings. Here are some examples, of increasing
sophistication, of how to structure this process to better guide project planning:
• Pick a few key parameters, and give each one three ratings: Pessimistic,
Neutral, and Optimistic. Then produce three estimates, one with all
Pessimistic ratings, one with all Neutral, and one with all Optimistic. Use
(and maintain) this set of estimates in project planning.
• If a project has a list of anticipated risks and opportunities (compare section
5.2.3.4.1), use those risks to guide Pessimistic, Neutral, and Optimistic
ratings and make estimates as above.
• Use Monte Carlo simulation to get a distribution of possible project
estimates, and use that distribution to guide project planning. Inputs to the
simulation can be intuitive Low/Medium/High ratings, or, preferably, can be
based on a Risk and Opportunity Plan.
One final guidance on good estimating practice: in the real world, it seems more
common to have multiple things go wrong on a project than one would expect by
chance, so try to take multiple pessimistic ratings into account in project planning.
82
6.2 Threats to Validity
Many possible threats to validity were identified in (Valerdi R. L., 2010),
(Griffiths, 1993), and (Valerdi R. , 2005); the ones discussed below seem to be the
most applicable to the COSYSMO 3.0 Final Model. There are three groups of
threats: those applicable to the model as a whole, those applicable to the use of the
Delphi process, and those applicable to the use of the dataset.
Threats applicable to the model as a whole:
Construct validity refers to the correspondence between the elements of the model
and the underlying systems engineering phenomena. Most of the model elements
should correspond well, because they are used in previous models; in the cases of
Development for Reuse and Risk and Opportunity Management, the previous
model is COCOMO II. Changes to model elements were reviewed by the Working
Group, whose members applied their extensive systems engineering experience.
Internal validity refers to the fact that the elements of the model actually contribute
to increased or decreased systems engineering effort. Again, the use of model
elements from previous models, plus the input of the Working Group to element
definition, tends to provide internal validity.
External validity refers to having an appropriately bounded realm of applicability.
COSYSMO 3.0 users are always advised to calibrate the model to their data, and
are advised that the Final Model’s coefficients were based on data from the
aerospace industry. I also note that the dataset used in the Final Model spans a
wide range of project sizes (over two orders of magnitude). Furthermore, section
Error! Reference source not found. is intended to steer users toward valid uses
of the Model.
Threats applicable to the use of the Delphi process: Threats applicable to the use of the Delphi process:
(Before considering this group of threats, let me remind of the Delphi process used
on COSYSMO 3.0, as explained in section 5.2.1. The procedure for each Delphi
workshop was that each expert turned in his or her opinions via ballot, and those
ballots were anonymous.)
Sample self-selection refers to experts who have non-representative opinions
voting more than other experts. This can have occurred during the Delphi, but any
effect might be reduced because of the anonymity of the ballots.
83
Group conformity refers to experts feeling pressured to vote similarly to others in
the group. Again, this could have happened during a Delphi, but the effect would
be mitigated by the anonymity of the ballots.
Measurement reliability refers to having reported numbers actually reflect the
underlying quantity. In the context of the COSYSMO 3.0 Delphi process, this
means that an expert’s vote is desired to be reliable. This is subjective, but the fact
that the ballots are anonymous should help measurement reliability.
Threats applicable to the use of the dataset:
Noisy data refers to using data that has a high degree of variability in it. In fact,
the PL4 data does seem to have noticeable variability. However, the Final Model
fit had highly significant coefficients, and PRED(.30) > 50% was achieved on the
PL4O dataset.
Large amount of data available needs no explanation. There was a moderate
amount of data available for the Final Model, 44 projects. As just noted, this was
sufficient to achieve significance in all coefficients. (Note: this item and the three
below are assumptions involved in using the Ordinary Least Squares model, which
underlies the linear regression fits produced by the lm function in R (Griffiths,
1993).)
No outliers refers to not using data points that are actually outside the assumptions
of the model. Indeed, the 5 outliers that were identified in producing the PL4O
dataset were at the edges of acceptable Productivity (section 5.3.2.1.1).
Nevertheless, the Final Model achieved significance, even with the outliers
included.
Predictor variables are not correlated refers to the coefficients not having high
correlations. In fact, there are only three instances of correlations higher than 0.60
in magnitude: CONOPS & Requirements Understanding versus Architecture
Understanding = 0.602, Stakeholder Team Cohesion versus Personnel/Team
Capability = 0.607, and Personnel/Team Capability versus Personnel Experience
and Continuity = 0.618. These are all below the traditional limit of 0.66.
Predictors are all continuous needs no explanation. While COSYSMO 3.0 ratings
are suggested to be in quarter-steps (section 5.2.3.1.6), in fact the underlying
mathematics is continuous in all ratings.
84
6.3 Evaluation
The research hypothesis (section 5.1) is satisfied when there is a systems
engineering cost estimating model with four specific properties. The present
section argues that the COSYSMO 3.0 Final Model essentially meets those
properties. In some cases, similarity to COSYSMO 1.0 is part of the argument.
Here are the four properties and their respective arguments:
1. Is applicable to a wide range of systems engineering projects
Like COSYSMO 1.0, the model is in principle applicable to a wide range of
projects; all systems engineering projects have the attributes used in forming
a COSYSMO 3.0 estimate. As the Final Model’s coefficients are based on
data (PL4) from the aerospace industry, a user in a different type of industry
should definitely calibrate the model to their own projects.
The PL4 data does cover more than two orders of magnitude in project size,
which makes it wide-ranging in that regard.
2. Includes all the major features of COSYSMO 1.0 and its extension models
All the features of COSYSMO 1.0 are included (the model definition in
section 5.2.3 is essentially a superset of COSYSMO 1.0’s).
The extension models covering reuse and requirements volatility are
included.
The extension model covering applicability to systems-of-systems is partly
included (the multi-subproject model), but interoperability is not included
due to the absence of project data (as explained in section 5.3.1.3).
3. Provides continuity to users of previous COSYSMO-family models
Continuity is provided via the Rosetta Stone (see section 5.2.4).
4. When calibrated to data from a particular organization, estimates actual
systems engineering costs with a PRED(.30) accuracy of 50%
This was demonstrated via the calibration to the PL4O dataset (section
5.3.2), which resulted in PRED(.30) = 51.3%.
I claim the research hypothesis is satisfied to an acceptable degree.
85
7 Schedule of Remaining Research Tasks
7.1 Submission to Journal
I plan to submit a paper on my results to the INCOSE journal Systems Engineering.
Should there be any difficulty there, I would submit to the Journal of Cost Analysis
and Parametrics and/or IEEE’s Journal of Systems Engineering and Electronics.
86
8 Future Research
An obvious research topic is to produce a validated model for interoperability. Our
work on interoperability (compare section 5.2.3.3.10) should provide a solid
foundation.
One possible line of future research is to provide tailored models for different types
of project; for example, for defense projects, or for software-intensive systems.
Such a tailored model would consist of a number of pre-set values for some driver
ratings (for example, for a defense project, Development for Reuse might have the
pre-set rating of High). These pre-set values would be determined from an
analysis of actual data from projects of each type.
Another possible line for future research is developing an estimating model for
total development cost based primarily on COSYSMO 3.0 drivers. (A starting
point for this line might be the work of Reggie Cole of Lockheed-Martin (Cole &
Roedler, 2013).)
A small improvement to the activity level definitions in section 5.2.3.1.4 is
desirable: it would be better if the DWR and DFR activity levels integrated more
simply.
87
9 References
Apgar, H. (2011). Cost Estimating. In J. R. Wertz, D. F. Evertt, & J. J. Puschell,
Space Mission Engineering: The New SMAD (pp. 289-324). Hawthorne,
CA: Microcosm Press.
Baldwin, K. (2015). DoD Systems Engineering Update. US Department of
Defense, Office of the Deputy Assistant Secretary of Defense for Systems
Engineering. Washington: DoD.
Boehm, B. W. (2000). Software Cost Estimation with COCOMO II. Upper Saddle
River, NJ: Prentice-Hall, Inc.
Boehm, B. W. (1981). Software Engineering Economics. Upper Saddle River, NJ:
Prentice-Hall, Inc.
Cole, R., & Roedler, G. (2013). COSYSMO Extension as a Proxy Systems Cost
Estimation. Lockheed-Martin. Los Angeles: USC CSSE.
Fortune, J. (2009). Estimating Systems Engineering Reuse with the Constructive
Systems Engineering Cost Model (COSYSMO 2.0). Los Angeles, CA: USC.
Gelman, A. e. (2013). Bayesian Data Analysis (Third Edition ed.). Boca Raton,
FL, USA: Chapman & Hall/CRC Texts in Statistical Science.
Griffiths, W. E. (1993). Learning and Practicing Econometrics. John Wiley &
Sons, Inc.
Habib-agahi, H. (2010). NASA Instrument Cost Model (NICM) Verion IV.
Pasadena CA: NASA JPL.
Helmer, O. (1966). Social Technology. New York, NY: Basic Books.
INCOSE. (2017, 4 25). INCOSE Membership. Retrieved 4 25, 2017, from
INCOSE: http://www.incose.org/about/Membership
ISO/IEC. (2010). Systems and Software Engineering—Life Cycle Management—
Part 1: Guide for Life Cycle Management (ISO/IEC TR 24748-1:2010(E)).
Geneva, Switzerland: ISO/IEC.
88
ISO/IEC. (2002). Systems engineering--system life cycle processes (ISO/IEC
15288). I. Geneva, Switzerland: nternational Standardization
Organization/International Electrotechnical Commission.
Kendall, F. (2015). Implementation Directive for Better Buying Power 3.0. US
Department of Defense. Washington DC: US DOD.
Lane, J. A. (2009). Cost Model Extensions to Support Systems Engineering Cost
Estimation for Complex Systems and Systems of Systems. 7th Annual
Conference on Systems Engineering Research. CSER.
Lane, J. A., & Valerdi, R. (2011). Systems Interoperabililty Influence on System of
Systems Engineering Effort. CSER. CSER.
Parkinson, G. N. (1957). Parkinson's Law and Other Studies in Administration.
Boston, MA: Houghton-Mifflin.
Pena, M. (2012). Quantifying the Impact of Requirements Volatility on Systems
Engineering Effort. Los Angeles, CA: USC.
Tecolete Research. (2002). Unmanned Space Vehicle Cost Model (USMC). El
Segunco, CA: US Air Force, Space and Missle Systems Center.
Unknown. (2015, October 18). Analytic Hierarchy Process. Retrieved October 18,
2015, from Wikipedia:
https://en.wikipedia.org/wiki/Analytic_hierarchy_process
Valerdi, R. L. (2010). Lessons Learned about Mixed Methods Research Strategies
in Systems Engineering: Evidence from PhD Dissertations. 8th Conference
on Systems Engineering Research. Hoboken NJ.
Valerdi, R. (2005). The Constructive Systems Engineering Cost Model. Los
Angeles, CA: University of Southern California.
Wang, G. (2016). The Generalized Reuse Framework--Strategies and the Decision
Process for Planned Reuse. INCOSE 2016. INCOSE.
Wang, G. V. (2008). Proposed Modification to COSYSMO Estimating
Relationship. INCOSE International Symposium , 18 (1), 249-262.
Wang, G., Ankrum, A., Valerdi, R., & Millar, C. (2008). COSYSMO Reuse
Extension. 18th INCOSE Symposium. Utrecht, The Netherlands: INCOSE.
89
Wang, G., Roedler, G. J., Pena, M., & Valerdi, R. (2014). A Generalized Systems
Engineering Reuse Framework and its Cost Estimating Relationship.
INCOSE 2014. INCOSE.
90
Appendix A: Correlation Coefficients
This section gives the correlation coefficients for the cost driver fit in the COSYSMO 3.0 Final
Model. These are the correlations that are > 0.60:
corr[CRQU,ARCH] = 0.602017
corr[TEAM,PCAP] = 0.6071545
corr[PCAP,PEXC] = 0.6181853.
const LogSize CRQU ARCH TEAM LSVC
const 1.000000000 -0.5998231449 0.028283098 0.012779697 0.016136035 -0.001178475
LogSize -0.599823145 1.0000000000 0.007096049 0.006166456 0.001956481 -0.003420924
CRQU 0.028283098 0.0070960492 1.000000000 0.602017034 0.519473680 -0.274145928
ARCH 0.012779697 0.0061664557 0.602017034 1.000000000 0.338382689 0.010872616
TEAM 0.016136035 0.0019564809 0.519473680 0.338382689 1.000000000 -0.219176001
LSVC -0.001178475 -0.0034209239 -0.274145928 0.010872616 -0.219176001 1.000000000
TRSK 0.003834842 -0.0045011718 -0.499194950 -0.482846407 -0.262690839 -0.119071575
RECU 0.049955629 -0.0023024341 0.075472520 -0.082063870 0.368426884 -0.099447451
INST -0.038602364 -0.0049652056 -0.514641777 -0.409264630 0.028482133 0.172166704
MIGR -0.047751718 -0.0029313156 -0.439214100 -0.583355006 -0.464968126 -0.283945399
PCAP -0.009957140 0.0022581908 0.384056957 0.209313076 0.607154535 -0.310888235
PEXC 0.008290243 0.0014791799 0.268126002 0.046202672 0.145218560 -0.086429377
SITE 0.024494322 0.0008610044 0.165294491 0.129537678 0.432929070 -0.109095069
TLSP
0.067099144 -0.0002516986 0.261646975 -0.077979916 0.280175296 -0.192169588
TRSK RECU INST MIGR PCAP PEXC
const 0.003834842 0.049955629 -0.038602364 -0.047751718 -0.009957140 0.008290243
LogSize -0.004501172 -0.002302434 -0.004965206 -0.002931316 0.002258191 0.001479180
CRQU -0.499194950 0.075472520 -0.514641777 -0.439214100 0.384056957 0.268126002
ARCH -0.482846407 -0.082063870 -0.409264630 -0.583355006 0.209313076 0.046202672
TEAM -0.262690839 0.368426884 0.028482133 -0.464968126 0.607154535 0.145218560
LSVC -0.119071575 -0.099447451 0.172166704 -0.283945399 -0.310888235 -0.086429377
TRSK 1.000000000 0.123042737 0.269694774 0.252668553 -0.334693320 -0.352848360
RECU 0.123042737 1.000000000 -0.238551779 -0.353636082 0.226030590 0.218065515
INST 0.269694774 -0.238551779 1.000000000 0.323736517 0.198782882 0.039011587
MIGR 0.252668553 -0.353636082 0.323736517 1.000000000 -0.193379281 -0.088284870
PCAP -0.334693320 0.226030590 0.198782882 -0.193379281 1.000000000 0.618185251
PEXC -0.352848360 0.218065515 0.039011587 -0.088284870 0.618185251 1.000000000
SITE -0.062344358 0.050760533 -0.257180010 -0.257207838 -0.014032219 -0.239658313
TLSP 0.074741753 0.348700442 -0.024504981 -0.101880152 0.102404728 0.163895620
91
SITE TLSP
const 0.024494322 0.0670991442
LogSize 0.000861004 -0.0002516986
CRQU 0.165294491 0.2616469750
ARCH 0.129537677 -0.0779799158
TEAM 0.432929070 0.2801752964
LSVC -0.109095068 -0.1921695876
TRSK -0.062344357 0.0747417532
RECU 0.050760533 0.3487004422
INST -0.257180010 -0.0245049811
MIGR -0.257207838 -0.1018801520
PCAP -0.014032219 0.1024047278
PEXC -0.239658312 0.1638956196
SITE 1.000000000 0.3053722673
TLSP 0.305372267 1.0000000000
92
Appendix B: Counts of Ratings
This table shows, for the PL4 dataset, how many times each rating was used with
each driver.
Table 0-1. Counts of the Number of Times Each Rating Was Used
for Each Driver.
Text
Rating: VL VL-L L L-N N N-H H
H-
VH VH
VH-
EH EH
Numeric
Rating: -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
CRQU 0 0 6 3 25 2 7 0 1 0 0
ARCH 0 0 6 3 22 5 7 1 0 0 0
TEAM 0 2 3 4 22 3 8 1 1 0 0
LSVC 0 2 2 5 19 3 9 2 2 0 0
TRSK 0 1 4 7 22 3 6 0 1 0 0
RECU 0 0 7 9 20 2 6 0 0 0 0
INST 0 0 0 0 34 2 5 1 1 0 1
MIGR 0 0 0 0 28 2 6 3 5 0 0
PCAP 0 0 0 1 24 7 10 1 1 0 0
PROC 0 0 4 0 20 2 10 1 3 1 3
PEXC 0 3 3 23 1 12 1 1 0 0 0
SITE 1 0 3 3 25 1 7 0 3 0 1
TLSP 7 2 30 1 3 1 0 0 0 0 0
ROPM 0 0 2 0 34 0 6 0 2 0 0
Totals: 8 10 70 59 299 45 88 11 20 1 5
%ages: 1 2 11 10 49 7 14 2 3 0 1
93
Appendix C: Regression Results
This section gives the results of the regression runs for the Final Model, one for the
cost driver fit and one for the exponent fit.
Table 0-2. Printed Results of the Regression
for the Final Model Cost Driver Fit.
[1] #
[1] # Details on fit (log space)
[1] #
Call:
lm(formula = C3D_CDnP_model2_const, data =
C3DPL4_CDnP_B6$yX_adj)
Residuals:
Min 1Q Median 3Q Max
-0.78611 -0.12216 -0.00345 0.12835 0.92235
Coefficients:
Estimate Std. Error t value Pr(>|t|)
const 1.420404 0.071217 19.95 < 2e-16 ***
LogSize 1.173908 0.016970 69.17 < 2e-16 ***
CRQU -0.116400 0.008916 -13.06 < 2e-16 ***
ARCH -0.093946 0.004041 -23.25 < 2e-16 ***
TEAM -0.095678 0.005490 -17.43 < 2e-16 ***
LSVC 0.106198 0.003755 28.28 < 2e-16 ***
TRSK 0.101158 0.004012 25.21 < 2e-16 ***
RECU 0.071477 0.006395 11.18 2.64e-14 ***
INST 0.092595 0.005565 16.64 < 2e-16 ***
MIGR 0.097515 0.007499 13.00 < 2e-16 ***
PCAP -0.080263 0.004183 -19.19 < 2e-16 ***
PEXC -0.066446 0.002062 -32.22 < 2e-16 ***
SITE -0.090461 0.002560 -35.33 < 2e-16 ***
TLSP -0.049602 0.004219 -11.76 5.10e-15 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.3757 on 43 degrees of freedom
Multiple R-squared: 0.9975, Adjusted R-squared: 0.9967
F-statistic: 1215 on 14 and 43 DF, p-value: < 2.2e-16
94
Table 0-3. Printed Results of the Regression
for the Final Model Exponent Fit.
[1] #
[1] # Details on fit (log space)
[1] #
Call:
lm(formula = PL4nP_Expb_model, data =
PL4nP_Expb_C3D_B6$yX_adj)
Residuals:
Min 1Q Median 3Q Max
-0.15831 -0.00136 0.11124 0.19213 0.57277
Coefficients:
Estimate Std. Error t value Pr(>|t|)
Exp_RVOL_Base 1.0521550 0.0053196 197.79 <2e-16 ***
PROC -0.0107210 0.0008478 -12.64 3e-16 ***
ROPM -0.0120484 0.0003521 -34.22 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.2121 on 44 degrees of freedom
Multiple R-squared: 0.999, Adjusted R-squared: 0.9989
F-statistic: 1.441e+04 on 3 and 44 DF, p-value: < 2.2e-16
Abstract (if available)
Abstract
The discipline of systems engineering continues to increase in importance. There are more projects, projects are larger, and projects are more critical, and all of these mean that more and better systems engineering is required. It follows that the cost of systems engineering continues to increase in importance. In turn, it follows that accurate estimation of systems engineering costs continues to increase in importance, as systems engineering results suffer if a project either underestimates or overestimates its cost. ❧ There are models for estimating systems engineering cost, notably COSYSMO 1.0, but all these models leave out some aspect of modern practices, and therefore tend to estimate a modern systems engineering cost inaccurately, or not at all. These modern practices include reuse of engineering artifacts, requirements volatility, and engineering in a system-of-systems context. While all of these practices have been addressed to some extent in research papers, there is no all-encompassing model that integrates all of them. ❧ My research has resulted in an integrated model that includes the features of COSYSMO 1.0 and covers those modern practices. It is open and therefore widely available. I have completed a comprehensive model based, in part, on actual project data.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0)
PDF
Extending systems architecting for human considerations through model-based systems engineering
PDF
Incorporation of mission scenarios in deep space spacecraft design trades
PDF
Organizing complex projects around critical skills, and the mitigation of risks arising from system dynamic behavior
PDF
Integration of digital twin and generative models in model-based systems upgrade methodology
PDF
Modeling and simulation testbed for unmanned systems
PDF
Context-adaptive expandable-compact POMDPs for engineering complex systems
PDF
Quantifying the impact of requirements volatility on systems engineering effort
PDF
Quantifying the effect of orbit altitude on mission cost for Earth observation satellites
PDF
Impacts of system of system management strategies on system of system capability engineering effort
PDF
Domain-based effort distribution model for software cost estimation
PDF
A declarative design approach to modeling traditional and non-traditional space systems
PDF
Designing an optimal software intensive system acquisition: a game theoretic approach
PDF
A model for estimating schedule acceleration in agile software development projects
PDF
Quantitative and qualitative analyses of requirements elaboration for early software size estimation
PDF
The development of an autonomous subsystem reconfiguration algorithm for the guidance, navigation, and control of aggregated multi-satellite systems
PDF
Techniques for analysis and design of temporary capture and resonant motion in astrodynamics
PDF
A model for estimating cross-project multitasking overhead in software development projects
PDF
Improved size and effort estimation models for software maintenance
PDF
The effects of required security on software development effort
Asset Metadata
Creator
Alstad, James Patrick
(author)
Core Title
COSYSMO 3.0: an extended, unified cost estimating model for systems engineering
School
Viterbi School of Engineering
Degree
Doctor of Philosophy
Degree Program
Astronautical Engineering
Publication Date
10/15/2018
Defense Date
08/10/2018
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
cost estimation,COSYSMO,OAI-PMH Harvest,systems engineering
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Boehm, Barry William (
committee chair
), Erwin, Dan (
committee member
), Madni, Azad M. (
committee member
), Moore, James E. (
committee member
), Siegel, Neil Gilbert (
committee member
)
Creator Email
alstad@acm.org,jalstad@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-76678
Unique identifier
UC11671823
Identifier
etd-AlstadJame-6836.pdf (filename),usctheses-c89-76678 (legacy record id)
Legacy Identifier
etd-AlstadJame-6836.pdf
Dmrecord
76678
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Alstad, James Patrick
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
cost estimation
COSYSMO
systems engineering