Economic Value of Systems Engineering
Trends Toward Increasing the Value of Systems Engineering
With traditional projects such as railroads, reservoirs, and refrigerators, the systems engineer faced a self-contained system, with relatively stable requirements, a sound scientific base, and numerous previous precedents. As more systems are intended to become parts within one or more evolving systems of systems (SoS) featuring rapidly increasing scale, dynamism, interdependence, human-intensiveness, sources of vulnerability, and novelty, the performance of effective SE takes on an ever-higher economic value.
This is corroborated by the Case Studies and Vignettes in Part 7. Shortfalls in SE in the United States Federal Aviation Administration (FAA) Advanced Automation System (AAS), United States Federal Bureau of Investigation (FBI) Virtual Case File System, the Hubble Space Telescope, the Boeing 787 Dreamliner, and the Therac-25 medical linear accelerator led to either extremely expensive cancelled systems or much more expensive systems in terms of total cost of ownership or loss of human lives. On the other hand, the Global Positioning System (GPS), Miniature Seeker Technology Integration Project (MSTI), and Next Generation Medical Infusion Pump Project all demonstrate that investment in thorough SE led to highly cost-effective systems. Figure 1 summarizes analyses data by Werner Gruhl relating investment levels in SE to cost overruns of United States National Aeronautics and Space Administration (NASA) projects (Stutzke 2005). The results demonstrate a general correlation between the amount invested in SE within a program and cost overruns, suggesting the critical role of properly allocating SE resources.
Further Quantitative Evidence of the Value of Systems Engineering
Analysis of the effects of shortfalls in systems architecture and risk resolution (the results of inadequate SE) for software-intensive systems in the 161-project Constructive Cost Model II (COCOMO™ II) database, show a statistically significant increase in rework costs as a function of project size measured in source lines of code (SLOC): averages of 18% rework for ten-thousand-SLOC projects, and 91% rework for ten-million-SLOC projects. This result has helped major system projects to rectify initial underinvestments in SE (e.g., Boehm et al. 2004), and to determine “how much SE is enough” in general by balancing the risks of underinvesting in SE against those of over-investing (often called “analysis paralysis”), as shown in Figure 2 (Boehm-Valerdi-Honour 2008).
In general, small projects can quickly compensate for neglected SE interface definition and risk resolution, but as projects get larger, with more independently-developed components, late-rework costs rise much more rapidly than the savings in reduced SE effort. Medium-sized projects have relatively flat operating regions, but very large projects pay extremely large penalties for neglecting thorough SE. Extensive surveys and case study analyses corroborate these results.
Survey data on software cost and schedule overruns in My Life Is Failure: 100 Things You Should Know to Be a Better Project Leader (Johnson 2006) indicate that the primary sources of the roughly 50% of the commercial projects with serious “software overruns” were actually due to shortfalls in SE (lack of user input, incomplete requirements, unrealistic expectations, unclear objectives, and unrealistic schedules). The extensive Software Engineering Institute (SEI)/National Defense Industrial Association (NDIA) survey of 46 government-contracted industry projects showed a strong correlation between higher project SE capability and higher project performance (Elm et al. 2007). Ongoing research combining project data and survey data reported in “Toward An Understanding of The Value of SE” (Honour 2003) and “Effective Characterization Parameters for Measuring SE” (Honour 2010) provide additional evidence of the economic value of SE, and further insights on critical SE success factors.
A calibrated model for determining “how much SE is enough” has been developed in (Valerdi 2008). It estimates the number of person-months that a project needs for SE as a function of system size modified by 14 factors that affect SE effort needed. System size is defined in terms of numbers and complexity of requirements, interfaces, operational scenarios, and key algorithms. The factors that affect SE effort include architecture understanding, technology maturity, legacy-system migration, personnel capabilities, process maturity, and tool support.
References
Works Cited
Boehm, B., Brown, A.W., Basili, V., and Turner, R. 2004. "Spiral Acquisition of Software-Intensive Systems of Systems." CrossTalk. May, pp. 4-9.
Boehm, B., R. Valerdi, and E.C. Honour. 2008. "The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems." Systems Engineering. 11(3): 221-234.
Elm, J. P., D.R. Goldenson, K. El Emam, N. Donatelli, and A. Neisa. 2008. A Survey of Systems Engineering Effectiveness-Initial Results (with Detailed Survey Response Data). Pittsburgh, PA, USA: Software Engineering Institute, CMU/SEI-2008-SR-034. December 2008.
Honour, E.C. 2003. "Toward An Understanding of The Value of Systems Engineering." Proceedings of the First Annual Conference on Systems Integration, March 2003, Hoboken, NJ, USA.
Honour, E.C. 2010. "Effective Characterization Parameters for Measuring Systems Engineering." Proceedings of the 8th Annual Conference on Systems Engineering Research (CSER). March 17-19, 2010. Hoboken, NJ, USA.
Johnson, J. 2006. My Life Is Failure: 100 Things You Should Know to Be a Better Project Leader. Boston, MA, USA: Standish Group International.
Stutzke, R. 2005. Estimating Software-Intensive Systems. Boston, MA, USA: Addison Wesley.
Valerdi, R. 2008. The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems. Saarbrücken, Germany: VDM Verlag.
Primary References
Boehm, B., R. Valerdi, and E.C. Honour. 2008. "The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems." Systems Engineering, 11(3): 221-234.
Honour, E.C. 2010. "Effective Characterization Parameters for Measuring Systems Engineering." Proceedings of the 8th Annual Conference on Systems Engineering Research (CSER). March 17-19, 2010. Hoboken, NJ, USA.
Valerdi, R. 2008. The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems. Saarbrücken, Germany: VDM Verlag.
Additional References
Elm, J.P., D.R. Goldenson, K. El Emam, N. Donatelli, and A. Neisa. 2008. A Survey of Systems Engineering Effectiveness-Initial Results (with Detailed Survey Response Data). Pittsburgh, PA, USA: Software Engineering Institute, CMU/SEI-2008-SR-034. December 2008.
Johnson, J. 2006. My Life Is Failure: 100 Things You Should Know to Be a Better Project Leader. Boston, MA, USA: Standish Group International.
Vanek, F., R. Grzybowski, P. Jackson, and M. Whiting. 2010. "Effectiveness of Systems Engineering Techniques on New Product Development: Results from Interview Research at Corning Incorporated." Proceedings of the 20th Annual INCOSE International Symposium. 12-15 July 2010. Chicago, IL.
SEBoK Discussion
Please provide your comments and feedback on the SEBoK below. You will need to log in to DISQUS using an existing account (e.g. Yahoo, Google, Facebook, Twitter, etc.) or create a DISQUS account. Simply type your comment in the text field below and DISQUS will guide you through the login or registration steps. Feedback will be archived and used for future updates to the SEBoK. If you provided a comment that is no longer listed, that comment has been adjudicated. You can view adjudication for comments submitted prior to SEBoK v. 1.0 at SEBoK Review and Adjudication. Later comments are addressed and changes are summarized in the Letter from the Editor and Acknowledgements and Release History.
If you would like to provide edits on this article, recommend new content, or make comments on the SEBoK as a whole, please see the SEBoK Sandbox.
blog comments powered by Disqus