Product Systems Engineering Special Activities

Readiness Level Assessments

As a new complex product or service is developed it is essential to verify and validate that the developed system is sufficiently mature to release as an operational product or service. Technology Readiness Assessments (TRAs) are established tools used to qualify technology development and help make investment decisions within complex development programs in order to deploy systems or elements of technology to an end user in a timely fashion.

This question of maturity was formalized by NASA (Mankins 1995) and later modified for use by the DOD [DOD 2006], AFRL [AFRL], and DOE [DOE 2008] as well as a growing number of non-governmental organizations. Technology Readiness Levels is a metric developed to summarize the degree of maturity of a technology. The original NASA TRL scale have 9 different levels from the basic principles observed (TRL 1) to actual systems flight proven through successful mission implementation. The TRL utilized by the DOD is portrayed in Table 1.

Table 1: Technology Readiness Levels for Assessing Critical Technologies (Mankins 1995)

The utilization of Readiness Levels has an impact upon the structure and operation of life cycles as described in Part 3; they allow better management and control of technology risk and program’s development cost and schedules. However, TRL do not provide an assessment of programmatic influence on TRL, technology criticality and priority, software aging and readiness context as pointed out by (Smith 2005). While TRLs have proved to be useful in evaluating a technology’s performance, as demonstrated in the laboratory or in a test environment, TRLs do not measure whether the technology product can actually be produced in an affordable manner. The concept of manufacturing readiness levels (MRL) has been incorporated to expand the TRL idea so that it can incorporate producibility concerns. The MRL approach addresses questions such as technology reproducibility, the cost of production, technology manufacturing production environment, etc. early in the development phase (GAO 2003).

Figure 1. Technology Readiness Levels and Their Relationship to System Acquisition Milestones

Readiness levels are an active research area within academia and government agencies in regards to integration of technology components into complex systems (Integration Readiness Levels-IRL) to address interface maturity among existing and maturing technology developments. TRLs apply to the critical enabling technologies, which are usually embodied at subsystem or assembly level or system component. Systems Readiness Levels (SRL) is used when going from individual technologies to the “whole” system. The SRL Model is a function of the individual Technology Readiness Levels (TRL) in a system and their subsequent integration points with other technologies, the Integration Readiness Level (IRL). (Sauser 2006)

Another related maturity aspect is related to the provisioning of products that are generally available (offered) and referred to as COTS (Commercial-Off-The-Shelf). Such products, be they hardware, software or a mixture of both hopefully have achieved the degree of maturity so that those acquiring them can rely upon their operational properties and that the documentation of the COTS is sufficient to provide the proper guidance in their utilization.

The PSE should realize that the TRL assessment for COTS changes dramatically if the operational environment or other requirements are imposed that exceed the design limits of the COTS product. For example, operations at very high or very cold temperature, high shock levels or vibration.

Product Certification

Product Certifications are domain and product specific and typically relate to human safety & health, the need to meet a specific government regulation, or are required by underwriters for insurance purposes. Certifications are performed by a third party (independent of the developer) who provides a guarantee of quality, safety and reliability of the product to the customer or user.

According to the INCOSE Handbook, Section 8.7.2 Quality Assurance: Product certification is the process of certifying that a certain product has passed performance or quality assurance tests or qualification requirements stipulated in regulations such as a building code or nationally accredited test standards, or that it complies with a set of regulations governing quality or minimum performance requirements.

The INCOSE Handbook, Section 8.10 Verification, defines four methods for verification, inspection, analysis, demonstration and test. In addition it defines certification as a fifth verification method, which is defined as verification against legal or industrial standards by an outside authority without direction to that authority as to how the requirements are to be verified. For example, this method is used for electronic devices; CE certification in Europe, and UL certification in the US and Canada.

The best known certification is airworthiness certification, related to safety of flight for aircraft. In the U.S. this is performed by the Federal Aviation Administration (FAA) (reference FAA web site). Government certifications are also common in the medical systems field where the Federal Drug Administration (FDA) is the primary certification agency. Some certifications are based on standards defined by technical societies, such as the ASME (American Society of Mechanical Engineers). The combination of the technical standards and a certification allows product developer to perform certifications that meet government standards without having the government directly involved in the process.

There are equivalent government organizations in other countries and for other regulated areas such as communications, building safety, nuclear systems, transportation systems to include ships, trains and automobiles, environmental impact and energy use. Systems engineers must learn the certifications that are required for the domain and product they are developing and work with the certification agents early in the development effort to ensure the necessary certifications are included in the system requirements, the system development plan and the funding provided to accomplish the development. When system changes and upgrades are necessary the systems engineers must determine if product re-certification is necessary and include it in the plans and funding for the system upgrade. (FAA.gov)

Enabling Product Certifications

The section above on Product Certification dealt with certification of various attributes of the product (e.g., safety, security, environmental impact, electromagnetic interference, etc.). There may be other certifications of enabling products that must be considered and appreciated by Product SE activity, such as operator certification of airplane pilots to ensure safety in flight and of nuclear plant operators to ensure prevention or mitigation of damaging nuclear radiation effects. An example of this is shown in the certification program by the North American Electric Reliability Corporation (NERC):

In support of NERC’s mission, the System Operator Certification Program’s mission is to ensure that employers have a workforce of system operators that meet minimum qualifications. These minimum qualifications are set through internationally recognized processes and procedures for agencies that certify persons. The Certification Program promotes excellence in the area of system operator performance and encourages system operators to be inquisitive and informed. (nerc.com)

Production qualification testing (PQT) is another type of certification:

A technical test completed prior to the Full-Rate Production (FRP) decision to ensure the effectiveness of the manufacturing process, equipment, and procedures. This testing also serves the purpose of providing data for the independent evaluation required for materiel release so that the evaluator can address the adequacy of the materiel with respect to the stated requirements. These tests are conducted on a number of samples taken at random from the first production lot, and are repeated if the process or design is changed significantly and when a second or alternative source is brought online. (https://dap.dau.mil/glossary/pages/2451.aspx)

Security certification and accreditation (C&A) is often required for deployment of computing and networking equipment in a classified environment. Facility certification may be required to ensure a building housing your equipment can provide the proper environment for safe and efficient operation of the equipment. HEMP certification may be required to ensure a building and its equipment can withstand the effects of high-altitude electromagnetic pulse (HEMP) from nuclear weapon. A similar type of certification to HEMP is for TEMPEST testing to ensure that sensitive electronic emissions are not allowed to leave a high security facility. (TEMPEST is a code name referring to investigations and studies of compromising emission, not an acronym.)

Technology Planning & Insertion

Technology planning can be an enterprise function or a program function. Technology planning, as an enterprise function, typically occurs on an annual basis to determine the funding for independent research and development in the coming year. Technology planning, as a program function, occurs early in the program and often continues through the life of the system. The design of the product system is highly dependent on the availability of technologies that have acceptable risk and that meet the customer's cost, schedule and performance requirements. These critical technologies will only be available when necessary if the systems engineers perform concept design and technology assessments and trade studies that define the critical technologies and the capabilities necessary in advance of the system development activities that will use the critical technologies.

The Mitre Systems Engineering Guide provides the following definition for technology planning: Technology Planning is the process of planning the technical evolution of a program or system to achieve its future vision or end-state. Technology planning may include desired customer outcomes, technology forecasting and schedule projections, technology maturation requirements and planning, and technology insertion points. The goal is a defined technical end-state enabled by technology insertion over time.

Systems engineers who participate in technical planning must understand the future vision and system requirements and relate these to the current and expected future technologies that can be applied to the system design during the current development and for future upgrades. To do this, systems engineers must acquire and maintain knowledge of the existing and developing technology in their design domain. The systems engineer will also provide the essential connection between the system user and research communities to provide alignment between the technology developers and the system designers.

Technology planning and insertion usually requires that the systems engineer perform technology readiness assessment that rate the maturity levels and the risk associated with the planned technologies. Immature, risky technologies require risk reduction activities that include prototyping and product development and test activities that provide quantification of the capabilities and risks. The risk reduction activities provide the data necessary to assess and update the design to reduce its risk.

Product Road Mapping & Release Planning

Product roadmaps provide an outline that shows when products are scheduled for release and include an overview of the product's primary and secondary features. Both internal and external product maps should be created. The form of the roadmap will depend on the development methodology being used to develop the system. Waterfall, iterative and spiral development models result in different roadmaps and release plans. The systems engineers must be an integral member of the team creating the roadmaps. Requirements should be mapped to each of the planned releases. Test plans must be adapted to the development model and the release plans.

Product roadmaps should be aligned with the technology roadmaps that are applicable to the product. Technology maturity should be accomplished before the technologies are included in the product development plans and the roadmap for the product release that includes those technologies.

Product roadmaps are essential for software intensive systems that have many releases of software and capability upgrades. The identification of the requirements for each release, the test plans and the features provided in each release is an essential driver of the product development process. Clear definition of these items can make the difference between delivering the capabilities the customer in looking for and will support or a product that fails to meet the needs of the customer and is abandoned.

Intellectual Property Management

Systems Engineers create and must manage intellectual property as part of their job. Existing systems engineering literature rarely mentions or covers this topic. However, there are many text books and management related literature that provide additional information, such as (Irish 2005). There are many types of intellectual property. Listed below are some of the more important types with brief explanations. Intellectual property – Intangible output of the rational thought process that has some intellectual or informational value and is normally protected via using copyrights, patents, and/or trade secrets. (Vivien 2005)

1. Proprietary Information - Any information which gives a company (or enterprise) an advantage over its competitors is usually proprietary.

2. Patents – A patent is the principle mechanism for protecting rights in an invention or discovery. In exchange for a full disclosure of how to practice it, the issuing government will grant the right to exclude others from practicing the invention for a limited prior of time, usually 15 to 20 years (17 years from date of issue in the United States).

3. Design Patents--In some countries these are referred to by the more appropriate term "design registrations" or some other name. Basically, they protect rights in ornamental designs provided the designs are new and inventive - i.e., "non-obvious" at the time they are made. In the United States the maximum term of a design patent is 14 years.

4. Trademarks--A trademark identifies the source of origin of goods in commerce, and is not stronger than the actual use to which it has been put and the diligence with which it has been protected from infringement, encroachment, dilution. Under some circumstances, a trademark may be registered with Governmental agencies, which services as notice to others and also confers certain procedural advantages. Among a company's most valuable assets is the corporate name, which also is the company's primary trademark.

5. Copyrights A claim of copyright protects such matters as writings, musical compositions, or works of art, from copying by others, i.e., from plagiarism. A notice of claim of copyright must be made in the manner prescribed by law at the time of first publication.

Parts, Material & Process Management

The consequences of mission failure or inability to deploy the system on time due to parts, materials, and process (PM&P) issues need to be clearly understood by the SE team, as these elements are fundamental to the overall mission reliability and program success. PM&P management is especially important in harsh environments (like outer space and underwater) and in situations where system failure can have catastrophic impacts on public safety (like nuclear power, bridges and tunnels, and chemical processing plants).

Generally, original equipment manufacturers (OEM’s) engaged in design and fabrication of electronic systems have a documented policy that deals with PM&P, sometimes in the form of a PM&P Management Manual. The elements of a PM&P control program include things such as:

• PM&P requirements that apply to a system

• Generation of a program-approved parts list (PAPL)

• Appointment of a PM&P Control Board (PMPCB)

• Development of part stress derating policy and part parameter derating policy for end of life

• Definition of minimum qualification, quality control, and screening requirements for parts

PM&P management guidance is provided by MIL-HDBK-512 and ANSI/AIAA R-100, which identify the overall management process elements of a PM&P program. Additional issues to be addressed by PM&P include: hazardous materials, rare earth elements, conflict materials, and counterfeit materials.

References

Citations

ANSI 2001, Recommended Practice for Parts Management (ANSI/AIAA R-100A-2001)

DoD 5000.1. (October 23, 2000). The Defense Acquisition System.

FAA http://www.faa.gov/aircraft/air_cert/airworthiness_certification/aw_overview/

Irish, V. 2005. Intellectual Property Rights for Engineers, Second Edition. IET.

Mankins, J. 1995. Technology Readiness Levels—A White Paper. Advanced Concepts Office, Office of Space Access and Technology, NASA.

MITRE Systems Engineering Guide on line.

MIL-HDBK-512A, Parts Management DTD 31 Oct 01

Sauser, B., Verma, D., Ramirez-Marquez, J. and Gove, R. 2006. From TRL to SRL: The Concept of System Readiness Levels. Conference on Systems Engineering Research, Los Angeles, CA.

Smith, J. 2005. An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. Proceedings of the 38th Hawaii International Conference on Systems Sciences

https://dap.dau.mil/glossary/pages/2451.aspx

http://www.nerc.com/page.php?cid=6%7C84

Primary References

FAA http://www.faa.gov/aircraft/air_cert/airworthiness_certification/aw_overview/

Irish, V. 2005. Intellectual Property Rights for Engineers, Second Edition. IET.

Mankins, J. 1995. Technology Readiness Levels—A White Paper. Advanced Concepts Office, Office of Space Access and Technology, NASA.

MITRE Systems Engineering Guide on line.

Sauser, B., Verma, D., Ramirez-Marquez, J. and Gove, R. 2006. From TRL to SRL: The Concept of System Readiness Levels. Conference on Systems Engineering Research, Los Angeles, CA.

Smith, J. 2005. An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. Proceedings of the 38th Hawaii International Conference on Systems Sciences

https://dap.dau.mil/glossary/pages/2451.aspx

http://www.nerc.com/page.php?cid=6%7C84

Additional References

ANSI 2001, Recommended Practice for Parts Management (ANSI/AIAA R-100A-2001)

DoD 5000.1. (October 23, 2000). The Defense Acquisition System.

MIL-HDBK-512A, Parts Management DTD 31 Oct 01

Comments from 0.5 Wiki

This article is new to the SEBoK for version 0.75. As such, there are no associated 0.5 comments.

SEBoK v. 1.9.1 released 30 September 2018

SEBoK Discussion

Please provide your comments and feedback on the SEBoK below. You will need to log in to DISQUS using an existing account (e.g. Yahoo, Google, Facebook, Twitter, etc.) or create a DISQUS account. Simply type your comment in the text field below and DISQUS will guide you through the login or registration steps. Feedback will be archived and used for future updates to the SEBoK. If you provided a comment that is no longer listed, that comment has been adjudicated. You can view adjudication for comments submitted prior to SEBoK v. 1.0 at SEBoK Review and Adjudication. Later comments are addressed and changes are summarized in the Letter from the Editor and Acknowledgements and Release History.

If you would like to provide edits on this article, recommend new content, or make comments on the SEBoK as a whole, please see the SEBoK Sandbox.

blog comments powered by Disqus