System Verification

Verification is a set of actions used to check the correctness of any element such as a System Element, a system, a document, a service, a task, a requirement, etc. Validation is a set of actions used to check the compliance of these elements to their purpose.

Overview

The Verification Process is performed on an iterative basis on every produced engineering element throughout the life cycle.

The Validation Process generally occurs at the end of a set of life cycle tasks or activities, and at least at the end of every milestone of a development project. The Validation Process applied onto the system when completely integrated is often called Final Validation – see System Integration Figure 1.

The purpose of Verification is to show that the element meets the requirements.

The purpose of Validation is to show that the requirements were correct.

Validation is used to ensure that “one is working the right problem” whereas Verification is used to ensure that “one has solved the problem right”. (Martin. 1997)

Remarks about definitions:

There are several books and standards that provide different definitions of Verification and of Validation. The most and general accepted definitions can be found in [ISO-IEC 12207:2008, ISO-IEC 15288:2008, ISO25000:2005, ISO 9000:2005]:

Verification: confirmation, through the provision of objective evidence, that specified requirements have been fulfilled. With a note added in ISO-IEC 15288: Verification is a set of activities that compares a system or system element against the required characteristics. This may include, but is not limited to, specified requirements, design description and the system.

Validation: confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled. With a note added in ISO 9000:2005: Validation is the set of activities ensuring and gaining confidence that a system is able to accomplish its intended use, goals and objectives (i.e., meet stakeholder requirements) in the intended operational environment.

Principles

Concept of Verification and Validation Action

The performance of the Verification and Validation Action includes:

• to obtain a result from the performance of the Verification and Validation Action onto the submitted element,
• to compare the obtained result with the expected result,
• to deduce a degree of correctness and of conformance/compliance of the submitted element,
• to decide about the acceptability of this conformance/compliance, because sometimes the result of the comparison may require a judgment of value regarding the relevance in the context of use to accept or not the obtained result (generally analyzing it against a threshold, a limit).

Note: If there is uncertainty about the conformance/compliance, the cause could come from ambiguity in the requirements; the typical example is the case of a measure of effectiveness expressed without a "limit of acceptance" (above or below threshold the measure is declared unfulfilled).

Figure 1 – Definition and performance of a Verification and Validation Action (Faisandier, 2011)

What to verify and validate? – Any engineering element can be verified and validated using a specific reference for comparison: Stakeholder Requirement, System Requirement, Function, System Element, Document, etc. Examples are provided in Table 1 below.

Table 1 – Examples of verified and validated items

Integration, Verification, Validation, Final Validation, Operational Validation

Verification activities begin during development (definition and realization) and continue into deployment and use.

Once the System Elements have been realized, they begin integration into the complete system, and this integration includes Verification and Validation activities as stated in System Integration. These activities are done in parallel, reducing the tasks at Final Validation.

System Validation concerns the global system (Product, Service, or Enterprise) seen as a whole and is based on the totality of the requirements (System Requirements, Stakeholder Requirements). Three methods are used:

• First, the V&V results are aggregated from the application of the Verification Process and the Validation Process to every definition element and to every integration element;
• Second, final Validation is performed for the complete integrated system in an industrial environment (as close as possible to the operational environment);
• Third, operational Validation is performed on the complete system in its operational environment (context of use - higher level of system).

Operational Validation relates to the operational mission of the system and shows if the system is ready for use or for production.

Integration, Verification and Validation

Verificationand validation occur at each stage of the development of a system, as shown in Figure 2..

Figure 2 – Verification and Validation level per level (Faisandier & Roussel, 2011)

Verification Actions and Validation Actions inside and across levels

Inside each level of system decomposition, Verification and Validation Actions are performed during System Definition and System Realization as represented in Figure 3 for the upper levels and in Figure 4 for lower levels. The Stakeholder Requirements Definition and the Operational Validation make the link between two levels of the system decomposition.

Figure 3 – Verification and Validation Actions in upper levels of system decomposition (Faisandier, 2011)

The System Elements Requirements and the End products Operational Validation make the link between the two lower levels of the decomposition – see Figure 4.

Figure 4 - Verification and Validation Actions in lower levels of system decomposition (Faisandier, 2011)

Process Approach

Introduction

Several approaches exist for defining the Verification and Validation Processes. INCOSE defines two main steps: plan and perform the Verification Actions. (INCOSE. 2010)

NASA has a slightly more detailed approach that includes five main steps: prepare verification, perform verification, analyze outcomes, produce report, and capture work products. (NASA. 2007) page 102

Any approach may be used, provided that it is appropriate to the scope of the system, the constraints of the project, includes the activities listed above in some way, and is appropriately coordinated with other activities (including System Definition, System Realization, and extension to the rest of the life cycle).

Verification Process

Major activities and tasks performed during this process include:

1. Establish a verification strategy drafted in a Verification and Validation Plan (glossary) (this activity is carried out concurrently to System Definition activities) obtained by the following tasks:
   1. Identify the verification scope in listing as exhaustive as possible the characteristics or properties that should be checked; the number of Verification and Validation Actions can be very high;
   2. Identify the constraints according to their origin (technical feasibility, management constraints as cost, time, availability of verification and validation means or qualified personnel, contractual constraints as criticality of the mission) that limit potentially the Verification and Validation Actions;
   3. Define the appropriate verification and validation techniques to be applied such as inspection, analysis, simulation, peer-review, testing, etc., depending of the best step of the project to perform every Verification and Validation Action according to constraints;
   4. Trade off of what should be verified (scope) taking into account all the constraints or limits and deduce what can be verified; the selection of Verification and Validation Actions would be made according to the type of system, objectives of the project, acceptable risks and constraints;
   5. Optimize the Verification and Validation strategy defining the most appropriate verification technique for every Verification and Validation Action, defining the necessary verification and validation means (tools, test-benches, personnel, location, facilities) according to the selected verification technique, scheduling the Verification and Validation Actions execution in the project steps or milestones, defining the configuration of the elements submitted to Verification and Validation Actions (mainly about testing on physical elements).
2. Perform the Verification and Validation Actions includes the following tasks:
   1. Detail each Verification and Validation Action, in particular the expected results, the verification technique to be applied and corresponding means (equipments, resources and qualified personnel);
   2. Acquire the verification and validation means used during the system definition steps (qualified personnel, modeling tools, mocks-up, simulators, facilities); then those during the integration step (qualified personnel, Verification and Validation Tools, measuring equipments, facilities, Verification and Validation Procedures, etc.);
   3. Carry out the Verification and Validation Procedures at the right time, in the expected environment, with the expected means, tools and techniques;
   4. Capture and record the results obtained when performing the Verification and Validation Actions using Verification and Validation Procedures and means.
3. Analyze obtained results and compare them to the expected results; record the status compliant or not; generate verification reports and potential issue/trouble reports and change requests on the design as necessary.
4. Control the process includes the following tasks:
   1. Update the Verification and Validation Plan according to the progress of the project; in particular the planned Verification and Validation Actions can be redefined because of unexpected events (addition, deletion or modification of actions);
   2. Coordinate the verification activities with the project manager for schedule, acquisition of means, personnel and resources, with the designers for issue/trouble/non conformance reports, with configuration manager for versions of physical elements, design baselines, etc.

Validation Process

Purpose of the Validation Process

The purpose of the [System] Validation Process is to provide objective evidence that the services provided by a system when in use comply with stakeholder requirements, achieving its intended use in its intended operational environment. (ISO/IEC. 2008)

This process performs a comparative assessment and confirms that the stakeholders' requirements are correctly defined. Where variances are identified, these are recorded and guide corrective actions. System validation is ratified by stakeholders. (ISO/IEC. 2008)

The validation process demonstrates that the realized end product satisfies its stakeholders' (customers and other interested parties) expectations within the intended operational environments, with validation performed by anticipated operators and/or users. (NASA. 2007)

It is possible to generalize the process using an extended purpose as follows: the purpose of the Validation Process applied to any element is to demonstrate or prove that this element complies with its applicable requirements achieving its intended use in its intended operational environment.

Each System Element, system, and the complete System of Interest are compared against their own applicable requirements (System Requirements, Stakeholder Requirements). This means that the Validation Process is instantiated as many times as necessary during the global development of the system. The Validation Process occurs at every different level of the system decomposition and as necessary all along the system development. Because of the generic nature of a process, the Validation Process can be applied to any engineering element that has conducted to the definition and realization of the system elements, the systems and the System of Interest.

In order to ensure that validation is feasible, the implementation of requirements must be verifiable onto the submitted element. Ensuring that requirements are properly written, i.e. quantifiable, measurable, unambiguous, etc., is essential. In addition, verification/validation requirements are often written in conjunction with Stakeholder and System Requirements and provide the method for demonstrating the implementation of each System Requirement or Stakeholder Requirement.

The generic inputs are the baseline references of requirements applicable to the submitted element. If the element is a system, the inputs are the System Requirements and Stakeholder Requirements.

The generic outputs are elements of the Verification and Validation Plan.

Activities of the ValidationProcess

Major activities and tasks performed during this process include:

1. Establish a validation strategy drafted in a Verification and Validation Plan (this activity is carried out concurrently to System Definition activities) obtained by the following tasks:
   1. Identify the validation scope that is represented by the [System and or Stakeholder] Requirements; normally, every requirement should be checked; the number of Verification and Validation Actions can be high;
   2. Identify the constraints according to their origin : same as for Verification Process;
   3. Define the appropriate verification/validation techniques to be applied : same as for Verification Process;
   4. Trade off of what should be validated (scope) : same as for Verification Process;
   5. Optimize the Verification and Validation strategy : same as for Verification process
2. Perform the Verification and Validation Actions includes the following tasks: same as for Verification Process
3. Analyze obtained results and compare them to the expected results; decide about the acceptability of the conformance/compliance; record the decision and the status compliant or not; generate validation reports and potential issue/trouble reports and change requests on the [System or Stakeholder] Requirements as necessary.
4. Control the process includes the following tasks: same as for Verification Process

Artifacts and Ontology Elements

These processes may create several artifacts such as:

1. Verification and Validation Plan (contains in particular the Verification and Validation strategy with objectives, constraints, the list of the selected Verification and Validation Actions, etc.)
2. Verification and Validation Matrix (contains for each Verification and Validation Action, the submitted element, the applied technique / method, the step of execution, the system block concerned, the expected result, the obtained result, etc.)
3. Verification and Validation Procedures (describe the Verification and Validation Actions to be performed, the Verification and Validation Tools needed, the Verification and Validation Configuration, resources, personnel, schedule, etc.)
4. Verification and Validation Reports
5. Verification and Validation Tools
6. Verified and validated element (system, system element, etc.)
7. Issue / Non Conformance / Trouble Reports
8. Change Requests on requirement, product, service, enterprise

These processes handles the ontology elements of Table 3.

Table 3 - Main ontology elements as handled within Verification and Validation

The main relationships between ontology elements are presented in Figure 5.

Figure 5 – Verification and Validation elements relationships with other engineering elements (Faisandier, 2011)

Note: "Design Reference" is a generic term; instances depend of the type of submitted engineering elements, for example: specified requirements, description of design characteristics or properties, drafting rules, standards, regulations, etc.

Checking and Correctness of Verification and Validation

The main items to be checked during the Verification and Validation Processes concern the items produced by the Verification and Validation processes:

• The Verification and Validation Plan, the Verification and Validation Actions, the Verification and Validation Procedures, Verification and Validation reports respect their corresponding template.
• Every verification and validation activity has been planned, performed, recorded and has generated outcomes as defined in the processes description above.

Methods and Techniques

Verification and Validation techniques

There are several verification/validation techniques / method to check that an element or a system complies to its [System, Stakeholders] Requirements. These techniques are common to verification and validation. The purposes are different; verification is used to detect faults/defects, whereas validation is to prove satisfaction of [System and/or Stakeholders] Requirements. Table 4 below provides synthetic descriptions of some techniques.

Table 4 – Verification and Validation techniques

Note: Demonstration and testing can be functional or structural. Functional demonstration and testing are designed to ensure that correct outputs are produced given specific inputs. For structural demonstration and testing, there are performance, recovery, interface, and stress considerations. These considerations will determine the system’s ability to perform and survive given expected conditions.

Validation/ Traceability Matrix

The importance of traceability is introduced in every topic of the System Definition KA using a Traceability Matrix. It may also be extended and used to record data such as the Verification and Validation Actions list, the selected Verification / validation Technique to verify / validate the implementation of every engineering element (in particular Stakeholder and System Requirement), the expected results, the obtained results when the Verification and Validation Action has been performed. The use of such a matrix enables the development team to ensure that the selected Stakeholder and System Requirements have been verified, or to evaluate the percentage of Verification and Validation Actions completed. In addition, the matrix helps to check the performed Verification and Validation activities against the planned activities as outlined in the Verification and Validation Plan, and finally to ensure that System Validation has been appropriately conducted.

Application to Product systems, Service systems, Enterprise systems

Because of the generic aspect of the process, this one is applied as defined above. The main difference resides on the detailed implementation of the verification techniques described above in Table 5.

Table 5 – Verification techniques and types of system

Practical Considerations

Major pitfalls encountered with System Verification and Validation are presented in Table 6.

Table 6 – Major pitfalls with System Verification and Validation

Major proven practices encountered with System Verification and Validation are presented in Table 7.

Table 7 – Proven practices with System Verification and Validation

References

Citations

Buede, D. M. 2009. The engineering design of systems: Models and methods. 2nd ed. Hoboken, NJ: John Wiley & Sons Inc.

INCOSE. 2010. INCOSE systems engineering handbook, version 3.2. San Diego, CA, USA: International Council on Systems Engineering (INCOSE), INCOSE-TP-2003-002-03.2.

ISO/IEC. 2008. Systems and software engineering - system life cycle processes. Geneva, Switzerland: International Organization for Standardization (ISO)/International Electronical Commission (IEC), ISO/IEC 15288:2008 (E).

NASA. 2007. Systems Engineering Handbook. Washington, D.C.: National Aeronautics and Space Administration (NASA), NASA/SP-2007-6105, December 2007.

Primary References

INCOSE. 2010. Systems Engineering Handbook: A Guide to Life Cycle Processes and Activities, version 3.2. San Diego, CA, USA: International Council on Systems Engineering (INCOSE), INCOSE-TP-2003-002-03.2.

ISO/IEC. 2008. Systems and Software Engineering - System Life Cycle Processes. Geneva, Switzerland: International Organization for Standardization (ISO)/International Electronical Commission (IEC), ISO/IEC/IEEE 15288:2008 (E).

NASA. 2007. Systems Engineering Handbook. Washington, D.C.: National Aeronautics and Space Administration (NASA), NASA/SP-2007-6105, December 2007.

Additional References

Buede, D. M. 2009. The engineering design of systems: Models and methods. 2nd ed. Hoboken, NJ: John Wiley & Sons Inc.

DAU. February 19, 2010. Defense acquisition guidebook (DAG). Ft. Belvoir, VA, USA: Defense Acquisition University (DAU)/U.S. Department of Defense.

ECSS. 6 March 2009. Systems engineering general requirements. Noordwijk, Netherlands: Requirements and Standards Division, European Cooperation for Space Standardization (ECSS), ECSS-E-ST-10C.

SAE International. 1996. Certification considerations for highly-integrated or complex aircraft systems. Warrendale, PA, USA: SAE International, ARP475

SEI. 2007. Capability maturity model integrated (CMMI) for development, version 1.2, measurement and analysis process area. Pittsburg, PA, USA: Software Engineering Institute (SEI)/Carnegie Mellon University (CMU).

---

Article Discussion

[Go to discussion page]

<- Previous Article | Parent Article | Next Article ->