Economic Value of Systems Engineering
The Increasing Value of Systems Engineering
With traditional projects, such as railroads, reservoirs, and refrigerators, a systems engineer faced a self-contained system that typically had relatively stable requirements, a sound scientific base, and numerous previous precedents. As most modern systems become parts within one or more evolving systems of systems (SoS), the performance of effective SE now takes on an ever-higher economic value, as the systems feature a rapidly increasing scale, dynamism, interdependence, human-intensiveness, number of sources of vulnerability, and novelty.
This is corroborated by the implementation examples in Part 7. Shortfalls in SE lead to either cancellation of already expensive systems or even more expensive systems in terms of total cost of ownership or loss of human life. Part 7 presents the problems in the United States Federal Aviation Administration (FAA) Advanced Automation System (AAS), United States Federal Bureau of Investigation (FBI) Virtual Case File System, the Hubble Space Telescope Case Study, and the Therac-25 medical linear accelerator.
On the other hand, the Global Positioning System (GPS), Miniature Seeker Technology Integration Project (MSTI), and Next Generation Medical Infusion Pump Project all demonstrate that investment in thorough SE results in highly cost-effective systems. Figure 1 summarizes the analyses data by Werner Gruhl, which relates investment levels in SE to cost overruns of the United States National Aeronautics and Space Administration (NASA) projects (Stutzke 2005). The results indicate that there is a general correlation between the amount invested in SE within a program and cost overruns, demonstrating the critical role of properly allocating SE resources.
Further Quantitative Evidence of the Value of Systems Engineering
Analysis of the effects of shortfalls in systems architecture and risk resolution (the results of insufficient SE) for software-intensive systems in the 161-project Constructive Cost Model II (COCOMO™ II) database shows a statistically significant increase in rework costs as a function of project size measured in source lines of code (SLOC): averages of 18% rework for ten-thousand-SLOC projects and 91% rework for ten-million-SLOC projects. This data has influenced many major system projects to reconsider initial underinvestment in SE (e.g., Boehm et al. 2004), as well as to address “how much SE is enough” by balancing the risks of under-investing in SE against those of over-investing (often called “analysis paralysis”), as shown in Figure 2 (Boehm, Valerdi, and Honour 2008).
Typically, small projects can quickly compensate for neglected SE interface definition and risk resolution; however, as projects grow larger and have more independently-developed components, the cost of late rework negates any savings in reduced SE effort. Additionally, medium-sized projects have relatively flat operating regions, while very large projects pay extremely large penalties for neglecting thorough SE. Extensive surveys and case study analyses corroborate these results.
Survey data on software cost and schedule overruns in My Life Is Failure: 100 Things You Should Know to Be a Better Project Leader (Johnson 2006) indicates that the primary sources of the roughly 50% of commercial projects with serious “software overruns” are the result of shortfalls in SE (lack of user input, incomplete requirements, unrealistic expectations, unclear objectives, and unrealistic schedules). The extensive survey of 46 government-contracted industry projects conducted by the Software Engineering Institute (SEI)/National Defense Industrial Association (NDIA) illustrated a strong correlation between higher project SE capability and higher project performance (Elm et al. 2007). Ongoing research that combined project data and survey data reported in “Toward an Understanding of The Value of SE” (Honour 2003) and “Effective Characterization Parameters for Measuring SE” (Honour 2010) has provided additional evidence as to the economic value of SE and further insights on critical factors that affect SE success.
The Constructive Systems Engineering Cost Model (COSYSMO), a calibrated model for determining “how much SE is enough,” has been developed and is discussed in (Valerdi 2008). It estimates the number of person-months that a project needs for SE as a function of system size (i.e., requirements, interfaces, algorithms, and operational scenarios), modified by 14 factors (i.e., requirements understanding, technology risk, personnel experience, etc.), which dictates the amount of SE effort needed. Other economic considerations of SE include the costs and benefits of reuse (Wang, Valerdi and Fortune 2010), the management of SE assets across product lines (Fortune and Valerdi 2013), the impact of SE on project risk (Madachy and Valerdi 2010), and the role of requirements volatility on SE effort (Pena and Valerdi 2010).
References
Works Cited
Boehm, B., Brown, A.W., Basili, V., and Turner, R. 2004. "Spiral acquisition of software-intensive systems of systems," CrossTalk. May, pp. 4-9.
Boehm, B., R. Valerdi, and E.C. Honour. 2008. "The ROI of systems engineering: Some quantitative results for software-intensive systems," Systems Engineering, vol. 11, no. 3, pp. 221-234.
Elm, J. P., D.R. Goldenson, K. El Emam, N. Donatelli, and A. Neisa. 2008. A Survey of Systems Engineering Effectiveness-Initial Results (with Detailed Survey Response Data). Pittsburgh, PA, USA: Software Engineering Institute, CMU/SEI-2008-SR-034. December 2008.
Fortune, J., and R. Valerdi. 2013. "A framework for systems engineering reuse," Systems Engineering, vol. 16, no. 2.
Honour, E.C. 2003. "Toward an understanding of the value of systems engineering," Proceedings of the First Annual Conference on Systems Integration, Hoboken, NJ, USA, March 2003.
Honour, E.C. 2010. "Effective characterization parameters for measuring systems engineering," Proceedings of the 8th Annual Conference on Systems Engineering Research (CSER), Hoboken, NJ, USA, March 17-19, 2010.
Johnson, J. 2006. My Life Is Failure: 100 Things You Should Know to Be a Better Project Leader. Boston, MA, USA: Standish Group International.
Madachy, R., and R. Valerdi. 2010. Automating systems engineering risk assessment, 8th Conference on Systems Engineering Research, Hoboken, NJ.
Pena, M., and R. Valerdi. 2010. "Characterizing the impact of requirements volatility on systems engineering effort," 25th Forum on COCOMO and Systems/Software Cost Modeling, Los Angeles, CA.
Stutzke, R. 2005. Estimating Software-Intensive Systems. Boston, MA, USA: Addison Wesley.
Valerdi, R. 2008. The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems. Saarbrücken, Germany: VDM Verlag.
Wang, G., R. Valerdi, and J. Fortune. 2010. “Reuse in systems engineering,” IEEE Systems Journal, vol. 4, no. 3, pp. 376-384.
Primary References
Boehm, B., R. Valerdi, and E.C. Honour. 2008. "The ROI of systems engineering: Some quantitative results for software-intensive systems," Systems Engineering, vol. 11, no. 3, pp. 221-234.
Honour, E.C. 2010. "Effective characterization parameters for measuring systems engineering," Proceedings of the 8th Annual Conference on Systems Engineering Research (CSER), Hoboken, NJ, USA, March 17-19, 2010
Valerdi, R. 2008. The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems. Saarbrücken, Germany: VDM Verlag.
Additional References
Hughes, T.P. 2000. Rescuing Prometheus: Four Monumental Projects that Changed the Modern World. New York, NY, USA: Vintage Books.
Vanek, F., R. Grzybowski, P. Jackson, and M. Whiting. 2010. "Effectiveness of systems engineering techniques on new product development: Results from interview research at corning Incorporated." Proceedings of the 20th Annual INCOSE International Symposium, Chicago, IL, USA, July 12-15, 2010.