System Verification

Introductory Paragraph(s)

Introduction, Definition and Purpose

Introduction - Verification is a set of actions used to check the correctness of any element such as a System Element, a system, a document, a service, a task, a requirement, etc. Validation is a set of actions used to check the compliance of these elements to their purpose.

These verification and validation actions are planned and carried out throughout the life cycle of the system. Verification and validation are generic terms that need to be instantiated within the context they occur. Verification and Validation understood as processes are transverse activities to every life cycle stage of the system. In particular during the development cycle of the system, the Verification Process and the Validation Process are performed in parallel of the System Definition and System Realization processes, and apply onto any activity and product resulting from the activity. The activities of every life cycle process and those of the Verification Process and of the Validation Process fit into each others.

The Integration Process uses intensively the Verification Process. The Verification Process is performed on an iterative basis on every produced engineering element.

The Validation Process generally occurs at the end of a set of life cycle tasks or activities, and at least at the end of every milestone of a development project. The Validation Process is not limited to a phase at the end of the development of the system. It might be performed on an iterative basis on every produced engineering element during the development and might begin with the validation of the expressed Stakeholders' Requirements as engineering elements. The Validation Process applied onto the system when completely integrated is often called Final Validation – see System Integration Figure 1.

Definitions and Purposes

The purpose of Verification, as a generic action, is to identify the faults/defects introduced at the time of any transformation of inputs into outputs. Verification is used to prove that the transformation was made according to the selected and appropriate methods, techniques, standards or rules. If the verification cannot be performed only on the transformation itself, one uses the outcomes of the transformation to establish the evidence these outcomes own the expected characteristics. Verification is used to prove that the transformation was done in the “right” way.

The purpose of Validation, as a generic action, is to establish the compliance of any activity output, compared to the inputs of this activity. Validation is used to prove that the transformation of inputs produced the expected, the "right" result.

The term Verification is often associated with the term Validation and understood as a single concept Verification and Validation (V & V). Validation is used to ensure that “one is working the right problem” whereas Verification is used to ensure that “one has solved the problem right”. (Martin. 1997)

Verification and/or Validation are based on tangible evidences; this means based on information whose veracity can be demonstrated, based on factual results obtained by techniques such as inspection, measurement, test, analysis, calculation, etc.

Verify a system (Product, Service, or Enterprise) consists in comparing the realized characteristics or properties of the Product, Service or Enterprise against its expected Design Properties. These Design Properties are either independent of the System Requirements (state of the art), or specific i.e. derived from the System Requirements.

Validate a system (Product, Service, or Enterprise) consists in demonstrating that the Product, Service or Enterprise satisfies its System Requirements. System Validation is related first to System Requirements, and eventually to Stakeholders Requirements depending of the contractual practices of the concerned industrial sector. From a purpose and a global point of view, validate a system consists in acquiring confidence in its ability to achieve its intended mission or use under specific operational conditions.

Remarks about definitions:

There are several books and standards that provide different definitions of Verification and of Validation. The most and general accepted definitions can be found in [ISO-IEC 12207:2008, ISO-IEC 15288:2008, ISO25000:2005, ISO 9000:2005]:

Verification: confirmation, through the provision of objective evidence, that specified requirements have been fulfilled. With a note added in ISO-IEC 15288: Verification is a set of activities that compares a system or system element against the required characteristics. This may include, but is not limited to, specified requirements, design description and the system.

Validation: confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled. With a note added in ISO 9000:2005: Validation is the set of activities ensuring and gaining confidence that a system is able to accomplish its intended use, goals and objectives (i.e., meet stakeholder requirements) in the intended operational environment.

Principles

Concept of Verification and Validation Action

Verification and Validation Action – The terms "verification" and "validation" are generic and are used in association with other terms to define engineering elements. So, one uses hereafter the terms verification and validation action to mention an action of verification or of validation.

A Verification and Validation Action is defined then performed.

The definition of a Verification and Validation Action applied to an engineering element includes – see Figure 1:

• to identify the element on which the Verification and Validation Action will be performed,
• to identify the reference/baseline in order to define the expected result of the Verification and Validation Action.

The performance of the Verification and Validation Action includes:

• to obtain a result from the performance of the Verification and Validation Action onto the submitted element,
• to compare the obtained result with the expected result,
• to deduce a degree of correctness and of conformance/compliance of the submitted element,
• to decide about the acceptability of this conformance/compliance, because sometimes the result of the comparison may require a judgment of value regarding the relevance in the context of use to accept or not the obtained result (generally analyzing it against a threshold, a limit).

Note: If there is uncertainty about the conformance/compliance, the cause could come from ambiguity in the requirements; the typical example is the case of a measure of effectiveness expressed without a "limit of acceptance" (above or below threshold the measure is declared unfulfilled).

Figure 1 – Definition and performance of a Verification and Validation Action (Faisandier, 2011)

What to verify and validate? – Any engineering element can be verified and validated using a specific reference for comparison: Stakeholder Requirement, System Requirement, Function, System Element, Document, etc. Examples are provided in Table 1 below.

Table 1 – Examples of verified and validated items

Validation versus Verification

The section discusses the fundamental differences between the two concepts and associated processes.

Source of the terms – From an actual and etymological meaning, the term verification comes from the Latin "verus" – that means truth – and "facere" – that means make/perform. So verification means to prove that something is “true” or correct (a property, a characteristic, etc.). The term validation comes from the Latin "valere" – that means become strong – and has the same root as value. So validation means to prove that something has the right features to produce the expected effects. Reference: Verification and Validation in plain English – (Jerome, Lake, INCOSE 1999).

Process similarities and differences - There are similarities between the Verification Process and the Validation Process in term of activities. The techniques used to define and perform the Verification Actions and those for Validation Actions are identical. The main differences concern the reference used to check the correctness of an element, and the acceptability of the effective correctness. Within verification, the comparison between the expected result and the obtained result is generally binary; it is true or not. Within validation the result of the comparison may require a judgment of value to accept or not the obtained result compared to a threshold, a limit.

The following Table 2 provides supplementary and synthetic information to help understanding the differences depending of the point of view.

Table 2 – Synthetic differences between Verification and Validation TO BE CHANGED

Verification vs. Validation (Citation) : According to the NASA Systems Engineering Handbook from a process perspective, the product verification and product validation processes may be similar in nature, but the objectives are fundamentally different. : The distinction between Verification and Validation is significant. Verification consists of proof of compliance with specifications and may be determined by tests, analysis, demonstration, inspection, etc…. Validation consists of proof that the system accomplishes its purpose. It is usually much more difficult (and much more important) to validate a system than to verify it. Strictly speaking, validation can be accomplished only at System level, while verification must be accomplished throughout the entire system architectural hierarchy. (NASA. 2007).

Integration, Verification, Validation, Final Validation, Operational Validation

There is sometimes a misconception that Verification occurs after Integration and before Validation. In most of the cases, it is more appropriate to begin verification activities during development (definition and realization) and to continue them into deployment and use.

Once the System Elements have been realized, their integration to form the complete system is performed. Integration consists to assemble and to perform Verification and Validation Action as stated in System Integration. A Final Validation activity generally occurs when the system is integrated, but a certain number of Verification and Validation Actions are also performed in parallel of System Integration in order to reduce as much as possible the number of Verification and Validation Actions while controlling the risks that could be generated if some checks are dropped out. Integration, Verification and Validation are intimately processed together due to the necessity of optimizing the Verification and Validation strategy and the Integration strategy.

System Validation concerns the global system (Product, Service, or Enterprise) seen as a whole and is based on the totality of the requirements (System Requirements, Stakeholder Requirements). But it is obtained gradually throughout the development stage of the system by pursuing three non exclusive ways:

• First by cumulating the Verification and Validation Actions results provided by the application of the Verification Process and the Validation Process to every definition element and to every integration element;
• Second by performing final Validation onto the complete integrated system in an industrial environment (as close as possible from the operational environment);
• Third by performing operational Validation Actions onto the complete system in its operational environment (context of use - higher level of system).

Operational Validation relates to the operational mission of the system and relate to the acceptance of the system ready for use or for production. For example, operational Validation will force to show in the operational environment that a vehicle has the expected autonomy (is able to cover a defined distance), can cross obstacles, performs safety scenarios as required, etc.

Integration, Verification and Validation level per level

It is impossible to carry out only a single global Validation on a complete integrated complex system. The sources of faults/defects could be important and it would be impossible to determine the causes of a non conformance raised during this global check. As generally the System of Interest has been decomposed during design in a set of blocks and layers of systems and System Elements, thus every system, System Element is verified, validated and possibly corrected before being integrated into the parent system block of the higher level, as shown on Figure 2.

Figure 2 – Verification and Validation level per level (Faisandier & Roussel, 2011) TO BE CHANGED

As necessary, the systems, System Elements are partially integrated in sub-sets in order to limit the number of properties/characteristics to be verified within a single step - see System Integration. For each level, it is necessary to make sure by a set Verification and Validation Actions that the features stated at the preceding level are not damaged. Moreover, a compliant result obtained in a given environment (for example: final validation environment) can turn into non compliant if the environment changes (for example: operational validation environment). So, as long as the sub-system is not completely integrated and/or does not operate in the real operational environment, no result must be regarded as definitive.

During modifications made to a system, the temptation is to focus on the new adapted configuration forgetting the environment and the other configurations. However, a modification can have significant consequences on other configurations. Thus, any modification requires regression Verification and Validation Actions (often called Regression Testing).

Verification Actions and Validation Actions inside and transverse to levels

Inside each level of system decomposition, Verification and Validation Actions are performed during System Definition and System Realization as represented in Figure 3 for the upper levels and in Figure 4 for lower levels. The Stakeholder Requirements Definition and the Operational Validation make the link between two levels of the system decomposition.

Figure 3 – Verification and Validation Actions in upper levels of system decomposition (Faisandier, 2011)

The System Elements Requirements and the End products Operational Validation make the link between the two lower levels of the decomposition – see Figure 4.

Figure 4 - Verification and Validation Actions in lower levels of system decomposition (Faisandier, 2011)

Note 1: The two figures above show a perfect dispatching of verification and validation activities on the right part, using the corresponding references provided by the System Definition processes on the left part. Some times in the real practices, the outputs of the Stakeholder Requirements Definition Process are not sufficiently formalized or do not contain sufficient operational scenarios and cannot serve as a reference to define Verification and Validation Actions to be performed in the operational environment. In this case, the System Requirements Definition Process outputs may be used in place of.

Note 2: The last level of the system decomposition is dedicated to the Realization of the System Elements, and the vocabulary and the number of activities used on the Figure 4 may be different – see System Implementation.

Verification and Validation strategy

The difference between verification and validation is especially useful for elaborating the Integration strategy, the Verification strategy, and the Validation strategy. In fact the efficiency of the System Realization is gained optimizing the three strategies together to form what is often called Verification & Validation strategy. The optimization consists to define and to perform the minimum of Verification and Validation Actions but detecting the maximum of errors/faults/defects and getting the maximum of confidence in the use of the Product, Service, or Enterprise. Of course the optimization takes into account the potential risks potentially generated if Verification and Validation Actions are dropped out.

Process Approach

Introduction

As explained above, there are two processes: the Verification Process and the Validation Process.

Because of the generic nature of these processes, they can be applied to any engineering element that has conducted to the definition and realization of the System Elements, the systems and the System of Interest.

But facing the huge number of potential Verification and validation Actions that may be generated by this normal approach, it is necessary to optimize the verification and validation strategy. This strategy is based on the balance between what should be verified as a must, the constraints such as time, cost, and feasibility of testing that limit naturally the number of Verification and Validation Actions, and the risks one accepts dropping out some Verification and Validation Actions.

Several approaches exist that may be used for defining the Verification and Validation Processes. INCOSE defines two main steps: plan and perform the Verification Actions. (INCOSE. 2010)

NASA has a slightly more detailed approach that includes five main steps: prepare verification, perform verification, analyze outcomes, produce report, and capture work products. (NASA. 2007) page 102

Any approach may be used, provided that it is appropriate to the scope of the system, the constraints of the project, includes the activities listed above in some way, and is appropriately coordinated with other activities (including System Definition, System Realization, and extension to the rest of the life cycle).

Verification Process

Purpose of the Verification Process

The purpose of the [System] Verification Process is to confirm that the specified design requirements are fulfilled by the system. This process provides the information required to effect the remedial actions that correct non-conformances in the realized system or the processes that act on it. (ISO/IEC. 2008)

It is possible to generalize the process using an extended purpose as follows: the purpose of the Verification Process applied to any element is to confirm that the applicable design reference is fulfilled by this element.

Each System Element, system, and the complete System of Interest should be compared against its own design references. As stated by Dennis Buede, “verification is the matching of [configuration items], components, sub-systems, and the system to corresponding requirements to ensure that each has been built right.” (Buede 2009) This means that the Verification Process is instantiated as many times as necessary during the global development of the system. The Verification Process occurs at every different level of the system decomposition and as necessary all along the system development.

The generic inputs are the baseline references of the submitted element. If the element is a system, the inputs are the functional and physical architectures elements as described in a System Design Document, the design description of the internal interfaces (Input/Output Flows, Physical Interfaces) to the system and the Interfaces Requirements external to the system.

The generic outputs are elements of the Verification and Validation Plan that includes the verification and validation strategy, the selected Verification and Validations Actions, the Verification and Validation Procedures, the Verification and Validation Tools, the verified element or system, the verification reports, the issue/trouble reports and change requests on the design.

Activities of the Verification Process

Major activities and tasks performed during this process include:

1. Establish a verification strategy drafted in a Verification and Validation Plan (glossary) (this activity is carried out concurrently to System Definition activities) obtained by the following tasks:
   1. Identify the verification scope in listing as exhaustive as possible the characteristics or properties that should be checked; the number of Verification and Validation Actions can be very high;
   2. Identify the constraints according to their origin (technical feasibility, management constraints as cost, time, availability of verification and validation means or qualified personnel, contractual constraints as criticality of the mission) that limit potentially the Verification and Validation Actions;
   3. Define the appropriate verification and validation techniques to be applied such as inspection, analysis, simulation, peer-review, testing, etc., depending of the best step of the project to perform every Verification and Validation Action according to constraints;
   4. Trade off of what should be verified (scope) taking into account all the constraints or limits and deduce what can be verified; the selection of Verification and Validation Actions would be made according to the type of system, objectives of the project, acceptable risks and constraints;
   5. Optimize the Verification and Validation strategy defining the most appropriate verification technique for every Verification and Validation Action, defining the necessary verification and validation means (tools, test-benches, personnel, location, facilities) according to the selected verification technique, scheduling the Verification and Validation Actions execution in the project steps or milestones, defining the configuration of the elements submitted to Verification and Validation Actions (mainly about testing on physical elements).
2. Perform the Verification and Validation Actions includes the following tasks:
   1. Detail each Verification and Validation Action, in particular the expected results, the verification technique to be applied and corresponding means (equipments, resources and qualified personnel);
   2. Acquire the verification and validation means used during the system definition steps (qualified personnel, modeling tools, mocks-up, simulators, facilities); then those during the integration step (qualified personnel, Verification and Validation Tools, measuring equipments, facilities, Verification and Validation Procedures, etc.);
   3. Carry out the Verification and Validation Procedures at the right time, in the expected environment, with the expected means, tools and techniques;
   4. Capture and record the results obtained when performing the Verification and Validation Actions using Verification and Validation Procedures and means.
3. Analyze obtained results and compare them to the expected results; record the status compliant or not; generate verification reports and potential issue/trouble reports and change requests on the design as necessary.
4. Control the process includes the following tasks:
   1. Update the Verification and Validation Plan according to the progress of the project; in particular the planned Verification and Validation Actions can be redefined because of unexpected events (addition, deletion or modification of actions);
   2. Coordinate the verification activities with the project manager for schedule, acquisition of means, personnel and resources, with the designers for issue/trouble/non conformance reports, with configuration manager for versions of physical elements, design baselines, etc.

Validation Process

Purpose of the Validation Process

The purpose of the [System] Validation Process is to provide objective evidence that the services provided by a system when in use comply with stakeholder requirements, achieving its intended use in its intended operational environment. (ISO/IEC. 2008)

This process performs a comparative assessment and confirms that the stakeholders' requirements are correctly defined. Where variances are identified, these are recorded and guide corrective actions. System validation is ratified by stakeholders. (ISO/IEC. 2008)

The validation process demonstrates that the realized end product satisfies its stakeholders' (customers and other interested parties) expectations within the intended operational environments, with validation performed by anticipated operators and/or users. (NASA. 2007)

It is possible to generalize the process using an extended purpose as follows: the purpose of the Validation Process applied to any element is to demonstrate or prove that this element complies with its applicable requirements achieving its intended use in its intended operational environment.

Each System Element, system, and the complete System of Interest are compared against their own applicable requirements (System Requirements, Stakeholder Requirements). This means that the Validation Process is instantiated as many times as necessary during the global development of the system. The Validation Process occurs at every different level of the system decomposition and as necessary all along the system development. Because of the generic nature of a process, the Validation Process can be applied to any engineering element that has conducted to the definition and realization of the system elements, the systems and the System of Interest.

In order to ensure that validation is feasible, the implementation of requirements must be verifiable onto the submitted element. Ensuring that requirements are properly written, i.e. quantifiable, measurable, unambiguous, etc., is essential. In addition, verification/validation requirements are often written in conjunction with Stakeholder and System Requirements and provide the method for demonstrating the implementation of each System Requirement or Stakeholder Requirement.

The generic inputs are the baseline references of requirements applicable to the submitted element. If the element is a system, the inputs are the System Requirements and Stakeholder Requirements.

The generic outputs are elements of the Verification and Validation Plan.

Activities of the ValidationProcess

Major activities and tasks performed during this process include:

1. Establish a validation strategy drafted in a Verification and Validation Plan (this activity is carried out concurrently to System Definition activities) obtained by the following tasks:
   1. Identify the validation scope that is represented by the [System and or Stakeholder] Requirements; normally, every requirement should be checked; the number of Verification and Validation Actions can be high;
   2. Identify the constraints according to their origin : same as for Verification Process;
   3. Define the appropriate verification/validation techniques to be applied : same as for Verification Process;
   4. Trade off of what should be validated (scope) : same as for Verification Process;
   5. Optimize the Verification and Validation strategy : same as for Verification process
2. Perform the Verification and Validation Actions includes the following tasks: same as for Verification Process
3. Analyze obtained results and compare them to the expected results; decide about the acceptability of the conformance/compliance; record the decision and the status compliant or not; generate validation reports and potential issue/trouble reports and change requests on the [System or Stakeholder] Requirements as necessary.
4. Control the process includes the following tasks: same as for Verification Process

Artifacts and Ontology Elements

These processes may create several artifacts such as:

1. Verification and Validation Plan (contains in particular the Verification and Validation strategy with objectives, constraints, the list of the selected Verification and Validation Actions, etc.)
2. Verification and Validation Matrix (contains for each Verification and Validation Action, the submitted element, the applied technique / method, the step of execution, the system block concerned, the expected result, the obtained result, etc.)
3. Verification and Validation Procedures (describe the Verification and Validation Actions to be performed, the Verification and Validation Tools needed, the Verification and Validation Configuration, resources, personnel, schedule, etc.)
4. Verification and Validation Reports
5. Verification and Validation Tools
6. Verified and validated element (system, system element, etc.)
7. Issue / Non Conformance / Trouble Reports
8. Change Requests on requirement, product, service, enterprise

These processes handles the ontology elements of Table 3.

Table 3 - Main ontology elements as handled within Verification and Validation

The main relationships between ontology elements are presented in Figure 5.

Figure 5 – Verification and Validation elements relationships with other engineering elements (Faisandier, 2011)

Note: "Design Reference" is a generic term; instances depend of the type of submitted engineering elements, for example: specified requirements, description of design characteristics or properties, drafting rules, standards, regulations, etc.

Checking and Correctness of Verification and Validation

The main items to be checked during the Verification and Validation Processes concern the items produced by the Verification and Validation processes:

• The Verification and Validation Plan, the Verification and Validation Actions, the Verification and Validation Procedures, Verification and Validation reports respect their corresponding template.
• Every verification and validation activity has been planned, performed, recorded and has generated outcomes as defined in the processes description above.

Methods and Techniques

Verification and Validation techniques

There are several verification/validation techniques / method to check that an element or a system complies to its [System, Stakeholders] Requirements. These techniques are common to verification and validation. The purposes are different; verification is used to detect faults/defects, whereas validation is to prove satisfaction of [System and/or Stakeholders] Requirements. Table 4 below provides synthetic descriptions of some techniques.

Table 4 – Verification and Validation techniques NEW TABLE

Note: Demonstration and testing can be functional or structural. Functional demonstration and testing are designed to ensure that correct outputs are produced given specific inputs. For structural demonstration and testing, there are performance, recovery, interface, and stress considerations. These considerations will determine the system’s ability to perform and survive given expected conditions.

Validation/ Traceability Matrix

The importance of traceability is introduced in every topic of the System Definition KA using a Traceability Matrix. It may also be extended and used to record data such as the Verification and Validation Actions list, the selected Verification / validation Technique to verify / validate the implementation of every engineering element (in particular Stakeholder and System Requirement), the expected results, the obtained results when the Verification and Validation Action has been performed. The use of such a matrix enables the development team to ensure that the selected Stakeholder and System Requirements have been verified, or to evaluate the percentage of Verification and Validation Actions completed. In addition, the matrix helps to check the performed Verification and Validation activities against the planned activities as outlined in the Verification and Validation Plan, and finally to ensure that System Validation has been appropriately conducted.

Application to Product systems, Service systems, Enterprise systems

Because of the generic aspect of the process, this one is applied as defined above. The main difference resides on the detailed implementation of the verification techniques described above in Table 5.

Table 5 – Verification techniques and types of system

Practical Considerations

Major pitfalls encountered with System Verification and Validation are presented in Table 6.

Table 6 – Major pitfalls with System Verification and Validation

Major proven practices encountered with System Verification and Validation are presented in Table 7.

Table 7 – Proven practices with System Verification and Validation

References

Please make sure all references are listed alphabetically and are formatted according to the Chicago Manual of Style (15th ed). See the BKCASE Reference Guidance for additional information.

Citations

List all references cited in the article. Note: SEBoK 0.5 uses Chicago Manual of Style (15th ed). See the BKCASE Reference Guidance for additional information.

Buede, D. M. 2009. The engineering design of systems: Models and methods. 2nd ed. Hoboken, NJ: John Wiley & Sons Inc.

INCOSE. 2010. INCOSE systems engineering handbook, version 3.2. San Diego, CA, USA: International Council on Systems Engineering (INCOSE), INCOSE-TP-2003-002-03.2.

ISO/IEC. 2008. Systems and software engineering - system life cycle processes. Geneva, Switzerland: International Organization for Standardization (ISO)/International Electronical Commission (IEC), ISO/IEC 15288:2008 (E).

NASA. 2007. Systems Engineering Handbook. Washington, D.C.: National Aeronautics and Space Administration (NASA), NASA/SP-2007-6105, December 2007.

Primary References

INCOSE. 2010. Systems Engineering Handbook: A Guide to Life Cycle Processes and Activities, version 3.2. San Diego, CA, USA: International Council on Systems Engineering (INCOSE), INCOSE-TP-2003-002-03.2.

ISO/IEC. 2008. Systems and Software Engineering - System Life Cycle Processes. Geneva, Switzerland: International Organization for Standardization (ISO)/International Electronical Commission (IEC), ISO/IEC/IEEE 15288:2008 (E).

NASA. 2007. Systems Engineering Handbook. Washington, D.C.: National Aeronautics and Space Administration (NASA), NASA/SP-2007-6105, December 2007.

Additional References

Buede, D. M. 2009. The engineering design of systems: Models and methods. 2nd ed. Hoboken, NJ: John Wiley & Sons Inc.

DAU. February 19, 2010. Defense acquisition guidebook (DAG). Ft. Belvoir, VA, USA: Defense Acquisition University (DAU)/U.S. Department of Defense.

ECSS. 6 March 2009. Systems engineering general requirements. Noordwijk, Netherlands: Requirements and Standards Division, European Cooperation for Space Standardization (ECSS), ECSS-E-ST-10C.

SAE International. 1996. Certification considerations for highly-integrated or complex aircraft systems. Warrendale, PA, USA: SAE International, ARP475

SEI. 2007. Capability maturity model integrated (CMMI) for development, version 1.2, measurement and analysis process area. Pittsburg, PA, USA: Software Engineering Institute (SEI)/Carnegie Mellon University (CMU).

---

Article Discussion

[Go to discussion page]

<- Previous Article | Parent Article | Next Article ->