Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies

Project Information

U-Test

Grant agreement ID: 645463

DOI

10.3030/645463

Project closed

EC signature date 1 December 2014

Start date 1 January 2015

End date 31 December 2017

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 3 713 233,75

EU contribution

€ 3 713 233,75

Coordinated by

NORWAY HEALTH TECH
Norway

Periodic Reporting for period 3 - U-Test (Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies)

Reporting period: 2017-01-01 to 2017-12-31

Summary of the context and overall objectives of the project

Cyber-Physical Systems (CPS) work in the uncertain real-world, and thus must be able to work correctly in the unpredictable world. The project was directed towards enabling such systems to work correctly in real-world uncertainty by providing new ways to test them. Given their countless applications (e.g. in robotics, transportation), it is compulsory to build a unified framework that enables testing various types of such systems in different domains with various types of testing methods. The project provided such framework that allows adding new testing methods into the developed framework.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

There are three outputs of the project: 1) Uncertainty Taxonomy (UT), 2) Uncertainty Modeling Framework (UMF), and 3) Uncertainty Testing Framework (UTF).
First, we developed an Uncertainty Taxonomy (UT). At the current state-of-the-art and practice, Uncertainty in CPS during their development and testing is not studied much. Due to this reason, the UT is designed based on the existing work on uncertainty from other fields such as philosophy, physics, statistics, and healthcare. The UT was validated with the use cases provided by the two pilots. We have also initiated a new modelling standard based on the UT at the Object Management Group (OMG).
We developed the Uncertainty Modeling Framework (UMF). The UMF has implemented several key items: 1) Implementation of Uncertainty Taxonomy as a UML profile called the UML Uncertainty Profile, 2) A set of model libraries to enable modeling advanced uncertainty-related measurements, 3) Implementation of the UML Testing Profile—one of the model-based testing standard, 4) A set of modelling methodologies to create test ready models with the developed profiles and model libraries. We demonstrated the completeness of the UMF by creating test ready models of the use cases provided by the two pilots.
We have developed Uncertainty Testing Framework (UTF). The UTF have implemented two key items: 1) the definition of test strategies for testing based on test ready models developed with the Uncertainty Modelling Framework, 2) strategies to improve the quality of test ready models. These strategies have been implemented and prototyped in several research tools developed by researchers. These strategies were also evaluated with the use cases provided by the two pilots.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The central value proposition of U-Test has been defined centered on modeling and testing toolchain and framework, complemented by support and awareness services. To check the exploitation hypotheses, a series of product-market fit activities have been carried engaging potential users and customers. The two project pilots’ technology transfer activities, two sectoral workshops, and 14 market exploration activities have allowed engaging CPS value chain key actors such as large CPS integrators and specialised SMEs.
Pilots’ insights gathering activities have collected the feedback evidence from the pilots regarding the following aspects: technical perspective regarding solution maturity level, added value regarding the product performance and provided benefits, adoption requirements regarding complementary infrastructure and learning curve, and support services requirements for implementation and after implementation.
Standardization plays an important role in a modern society, which is being increasingly globe. To be competitive internationally, many industries are encouraged or forced to conform to various standards. In other words, standards make an important contribution to national and international competitiveness, which is now a well-recognized fact, as one can see from the list of membership of OMG (http://www.omg.org/cgi-bin/apps/membersearch.pl) where organizations such as NASA, THALES, Airbus Group play roles in OMG standardization activities. To be more specific, in the context of the project, our aim of conforming our solutions to existing and widely adopted standards is an important strategy to improve the competitiveness of the consortium.
We put an effort to initiate a new Uncertainty Modelling standard at the Object Management Group (OMG) called “Precise Semantics for Uncertainty Modelling”. Based on the Request for Information (RFI) of the standard, we established an international working group for the development of the Request for Proposal (RFP) for the standard. The working group together developed the RFP, which was initially presented to the OMG in June 2017. Based on the feedback, the RFP was revised and prepared to be presented for voting to issue in December 2017’s OMG meeting. The proposed RFP was presented during the OMG meeting in December 2017. The latest draft focuses on core concepts and uncertainty measurements, with additional RFPs on uncertainty modelling planned to be issued in the future. The RFP was issued at the OMG meeting that took place in Burlingame, San Francisco, USA, in December 2017. The RFP is published here: http://www.omg.org/cgi-bin/doc?ad%2F2017-12-01.
We have been participating in the standardization work of the UTP 2 revised submission (Figure 7). As the leader (Fraunhofer Fokus (FF)) of this initiative, FF attended the OMG meeting in June 2017 and communicated the current status of UTP 2 to the ADTF. 5-9 June 2017, representatives from the U-Test consortium participated in the Object Management Group (OMG) meeting held in Brussels. On June 7, FF presented the revised version of the UML Testing Profile (UTP) to the ADTF. OMG voted to adopt the revised UTP. The next phase of in the standardization activity is the Finalization Task Force (FTF). FTF aims at transforming the adopted specification into a formal specification.
The project team focused on consolidating not only the results of research work but also extensively promoting the U-Test’s solution to the CPS market. In particular, our dissemination and communication activities have purposefully reached the core groups of the U-Test’s community and beyond. The core groups of the U-Test’s community are composed of end-user community, academia, and industry, which develop and deploy CPS in real life applications. First and foremost, the U-Test’s research results have been significantly disseminated to the academic world regarding research articles written by U-Test project team. The acceptance of our research work for publication in top publication venues with rigorous peer review processes has shown the recognition of the academic world to the U-Test’s research results.
The positive results of our research work have provided a strong foundation for promoting the U-Test’s values further to the other core groups of the U-Test’s community and beyond that. Another ultimate goal of U-Test is to promote U-Test’s values for adoption by industry, e.g. via public awareness events. U-Test had made its presence noted in different public awareness events, such as IoT conference in Oslo, Road2CPS in Vienna. To enable adoption by industry, the U-CertifyIt tool is developed that can be used by practitioners for uncertainty testing of CPS in real life applications. These tool chains and the technical guides for using them are being transferred first to the internal industry partners of U-Test. Our exploitation plan has been in place for making U-Test’s solution to be available for industry in the European market and worldwide in the future.

Project Logo