Periodic Reporting for period 3 - VALU3S (Verification and Validation of Automated Systems' Safety and Security)
Período documentado: 2022-05-01 hasta 2023-07-31
Manufacturers of automated systems have been allocating an enormous amount of time and effort in the past years developing and conducting research on automated systems. Manufacturers of these systems need to make sure that the systems function in the intended way and according to specifications which is not a trivial task as system complexity rises dramatically the more integrated and interconnected these systems become with the addition of automated functionality and features to them.
With rising complexity, it is necessary to conduct thorough verification and validation (V&V) of these systems. Through the V&V of automated systems, the manufacturers of these systems are able to ensure safe, secure and reliable systems for society to use since failures in highly automated systems can be catastrophic.
The high complexity of automated systems incurs an overhead on the V&V process making it time-consuming and costly. VALU3S aimed to design, implement and evaluate state-of-the-art V&V methods and tools in order to reduce the time and cost needed to verify and validate automated systems with respect to safety, cybersecurity and privacy (SCP) requirements.
In VALU3S, 13 use cases with specific SCP requirements have been studied in detail. Several state-of-the-art V&V methods have been investigated and further enhanced in addition to implementing new methods aiming for reducing the time and cost needed to conduct V&V of automated systems. The V&V methods investigated were then used to design improved process workflows for V&V of automated systems. Several tools and use cases have been enhanced supporting the improved processes which were evaluated through quantitative and qualitative evaluation of a large set of demonstrators, 21 of which were presented publicly at the Final Demonstration session in June 2023.
The high-level objectives of the project were as follows:
Objective 1. To develop a Multi-layered framework enabling more effective verification and validation
Objective 2. To overcome the SCP gaps and limitations of cyber-physical systems
Objective 3. To present a novel, standards compliant V&V workflow that is generic to reference methods in selected cyber-physical domains
Objective 4. To demonstrate, verify and validate the usefulness and wider acceptance of the proposed framework by realistic pilots
Objective 5. To suggest and validate new as well as state-of-the-art evaluation scenarios for safety, cybersecurity and privacy evaluation
Objectives 6. To develop and improve V&V tools and evaluation criteria
Objective 7. To revisit and identify the weaknesses of relevant safety and security standards and develop a concrete strategy to influence the development of new standards
Objective 8. To present guidelines for end users and practitioners as well as to disseminate the project results aiming to increase the awareness on the importance of conducting SCP V&V
With rising complexity, it is necessary to conduct thorough verification and validation (V&V) of these systems. Through the V&V of automated systems, the manufacturers of these systems are able to ensure safe, secure and reliable systems for society to use since failures in highly automated systems can be catastrophic.
The high complexity of automated systems incurs an overhead on the V&V process making it time-consuming and costly. VALU3S aimed to design, implement and evaluate state-of-the-art V&V methods and tools in order to reduce the time and cost needed to verify and validate automated systems with respect to safety, cybersecurity and privacy (SCP) requirements.
In VALU3S, 13 use cases with specific SCP requirements have been studied in detail. Several state-of-the-art V&V methods have been investigated and further enhanced in addition to implementing new methods aiming for reducing the time and cost needed to conduct V&V of automated systems. The V&V methods investigated were then used to design improved process workflows for V&V of automated systems. Several tools and use cases have been enhanced supporting the improved processes which were evaluated through quantitative and qualitative evaluation of a large set of demonstrators, 21 of which were presented publicly at the Final Demonstration session in June 2023.
The high-level objectives of the project were as follows:
Objective 1. To develop a Multi-layered framework enabling more effective verification and validation
Objective 2. To overcome the SCP gaps and limitations of cyber-physical systems
Objective 3. To present a novel, standards compliant V&V workflow that is generic to reference methods in selected cyber-physical domains
Objective 4. To demonstrate, verify and validate the usefulness and wider acceptance of the proposed framework by realistic pilots
Objective 5. To suggest and validate new as well as state-of-the-art evaluation scenarios for safety, cybersecurity and privacy evaluation
Objectives 6. To develop and improve V&V tools and evaluation criteria
Objective 7. To revisit and identify the weaknesses of relevant safety and security standards and develop a concrete strategy to influence the development of new standards
Objective 8. To present guidelines for end users and practitioners as well as to disseminate the project results aiming to increase the awareness on the importance of conducting SCP V&V
The following results have been obtained up to now in the project:
* Delivery of 80 deliverables.
* Detailing of all 13 project use cases, expressing the stakeholders’ concerns.
* Creation of an 8-dimensional framework for V&V of automated systems and implementing a web-based repository for storing the elements of the framework.
* Population of the web-based repository with 115 V&V methods, 45 improved/developed V&V tools, 61 supporting tools, more than 57 evaluation scenarios, 192 test cases and 239 requirements, and results of 27 demonstrators associated to the use cases.
* Identification and detailing of 400 gaps on V&V methods with potential for improvements.
* Design of a dedicated description language for V&V workflows, the V&V modelling language (VVML).
* Designing and modelling several workflows representing partners V&V activities, tools, and use cases.
* Designing, implementing, and improving of 44 V&V tools and 48 V&V methods.
* Qualitative and quantitative evaluation of a large set of demonstrators, 21 of which have been presented at the Final Demo session.
* Collection, presentation, and evaluation of project use cases using 15 evaluation criteria measuring SCP attributes and 13 criteria for measuring V&V processes.
* Creation and maintaining internal processes and organizational setup, analyses of risks and their impacts on the project and its objectives, analyses of project’s upcoming milestones, validation of past milestones as well as introduction of members of external advisory board to the project.
* Organization of several training sessions on: V&V methods, standards and standardization activities, and exploitation activities.
* Identification of the set of standards which are relevant for the foreseen technical work in the project and defining actions that will enable the results of VALU3S to contribute to the corresponding standardization efforts.
* Preparation and presentation of various types of communication materials (leaflets, brochures, videos, presentations, posters, etc.).
* Organisation of two press conferences.
* Publication of over 90 scientific articles (journal, conference, workshop articles) as well as giving multiple public speeches and different venues.
* Publication of 11 newsletters and one patent in addition to planning for submission of another patent as well as creation of a spin-off.
* Delivery of 80 deliverables.
* Detailing of all 13 project use cases, expressing the stakeholders’ concerns.
* Creation of an 8-dimensional framework for V&V of automated systems and implementing a web-based repository for storing the elements of the framework.
* Population of the web-based repository with 115 V&V methods, 45 improved/developed V&V tools, 61 supporting tools, more than 57 evaluation scenarios, 192 test cases and 239 requirements, and results of 27 demonstrators associated to the use cases.
* Identification and detailing of 400 gaps on V&V methods with potential for improvements.
* Design of a dedicated description language for V&V workflows, the V&V modelling language (VVML).
* Designing and modelling several workflows representing partners V&V activities, tools, and use cases.
* Designing, implementing, and improving of 44 V&V tools and 48 V&V methods.
* Qualitative and quantitative evaluation of a large set of demonstrators, 21 of which have been presented at the Final Demo session.
* Collection, presentation, and evaluation of project use cases using 15 evaluation criteria measuring SCP attributes and 13 criteria for measuring V&V processes.
* Creation and maintaining internal processes and organizational setup, analyses of risks and their impacts on the project and its objectives, analyses of project’s upcoming milestones, validation of past milestones as well as introduction of members of external advisory board to the project.
* Organization of several training sessions on: V&V methods, standards and standardization activities, and exploitation activities.
* Identification of the set of standards which are relevant for the foreseen technical work in the project and defining actions that will enable the results of VALU3S to contribute to the corresponding standardization efforts.
* Preparation and presentation of various types of communication materials (leaflets, brochures, videos, presentations, posters, etc.).
* Organisation of two press conferences.
* Publication of over 90 scientific articles (journal, conference, workshop articles) as well as giving multiple public speeches and different venues.
* Publication of 11 newsletters and one patent in addition to planning for submission of another patent as well as creation of a spin-off.
* Definition of a multi-dimensional framework and creation of a web-based repository for storing elements of V&V activities. The repository has been made public to offer the community, practitioners and researchers of autonomous systems a web tool that allows to learn about and identify suitable V&V processes.
* The web-based repository allows storing V&V methods and tools as well as artifacts of the application of those methods and tools in use cases such as requirements, test cases, evaluation criteria and demonstrators results. To do so, 12 web content types have been created: 1) Use cases, 2) Evaluation Scenarios, 3) Requirements, 4) Test Cases, 5) Methods, 6) Improved/Developed tools, 7) Supporting tools, 8) Organizations, 9) Standards, 10) Demonstrator results, 11) Measurement of V&V process improvements and 12) Measurement of SCP V&V improvements.
* Enabling tool-support for modelling V&V workflows of project partners using and extending the modelling framework Enterprise Architect (EA VVML Profile).
* Development of innovative V&V tools and tool chains.
* Evaluation of a development process mostly focuses on workforce needed for engineering phases. We propose to consider several different aspects of developed automated systems at the same time to provide more objective evaluation. We provided the list of 15 evaluation criteria aimed at safety, cybersecurity, and privacy attributes and 13 criteria for evaluating our V&V process, out of which we proposed 7 new criteria for SCP evaluation and 4 new criteria for V&V process evaluation.
* 21 demonstrators have been proposed to showcase the results consisting of the use of state-of-the-art technologies, technologies new or improved, as well as evaluation of their impact and wider acceptance.
* Publication of over 90 scientific articles and presentation of 15 open-source software resulting from technical work developed along the project. These scientific/technical outputs contributed with new results, pushing therefore the state-of-the-art of V&V.
* Development of standardization landscapes for all the application domains considered in the project.
* The web-based repository allows storing V&V methods and tools as well as artifacts of the application of those methods and tools in use cases such as requirements, test cases, evaluation criteria and demonstrators results. To do so, 12 web content types have been created: 1) Use cases, 2) Evaluation Scenarios, 3) Requirements, 4) Test Cases, 5) Methods, 6) Improved/Developed tools, 7) Supporting tools, 8) Organizations, 9) Standards, 10) Demonstrator results, 11) Measurement of V&V process improvements and 12) Measurement of SCP V&V improvements.
* Enabling tool-support for modelling V&V workflows of project partners using and extending the modelling framework Enterprise Architect (EA VVML Profile).
* Development of innovative V&V tools and tool chains.
* Evaluation of a development process mostly focuses on workforce needed for engineering phases. We propose to consider several different aspects of developed automated systems at the same time to provide more objective evaluation. We provided the list of 15 evaluation criteria aimed at safety, cybersecurity, and privacy attributes and 13 criteria for evaluating our V&V process, out of which we proposed 7 new criteria for SCP evaluation and 4 new criteria for V&V process evaluation.
* 21 demonstrators have been proposed to showcase the results consisting of the use of state-of-the-art technologies, technologies new or improved, as well as evaluation of their impact and wider acceptance.
* Publication of over 90 scientific articles and presentation of 15 open-source software resulting from technical work developed along the project. These scientific/technical outputs contributed with new results, pushing therefore the state-of-the-art of V&V.
* Development of standardization landscapes for all the application domains considered in the project.