Periodic Reporting for period 2 - SAFEXPLAIN (SAFE AND EXPLAINABLE CRITICAL EMBEDDED SYSTEMS BASED ON AI)
Berichtszeitraum: 2024-04-01 bis 2025-09-30
SAFEXPLAIN aims at devising ways to use Artificial Intelligence (AI) software, and more specifically, Deep Learning (DL) software in safety critical systems so that such DL software inherits safety requirements and hence, it must be developed following the same principles as any other software in those systems. This poses a number of challenges because functional safety standards are often incompatible with the way DL software is developed, and such software is not amenable “as it is” to be included in safety-critical systems. In SAFEXPLAIN, we contend that such a huge challenge can only be addressed holistically, adapting coordinately the safety development processes, the DL software architecture, and the way high-performance platforms are used to run such software. For that purpose, SAFEXPLAIN devises concepts and principles that allow addressing the challenge, and specific realizations of software architectures and tools, on industrially-relevant platforms, applied to automotive, space, and railway case studies.
SAFEXPLAIN partners have devised the development processes to follow for DL software, as well as how software architectures must include it, have devised DL solutions capable of complementing DL software realizing the system functionality (e.g. detecting and classifying objects) with information about uncertainties and anomalies in the data and inference process, have set a specific middleware capable of providing all services needed for the AI-based application, have found ways to smartly use high-performance platforms in a predictable and traceable manner, and have realized those solutions in the context of three domain-specific case studies (automotive, space and railway) and a generic fully open source case study (aka “core demo) intended to ease adoption.
In more detail, SAFEXPLAIN has achieved its goals in the different fronts of the project: (A) safety lifecycle and safety architecture design, (B) solutions to provide meta-information along with DL software outcomes that allows quantifying uncertainty and detecting anomalies, (C) platform support and system services where to practically integrate the AI-based software architectures, and (D) case study integration and evaluation to assess SAFEXPLAIN technologies and concepts.
(A) A safety lifecycle describing how to set training and validation data for safety-relevant DL software, training processes, and inference processes during operation has been defined. Such safety lifecycle has already been assessed positively by relevant certification entities and domain experts. Also, safety architectures amenable for different degrees of integrity levels for DL software have been devised, and some of them explicitly applied to the project case studies. Such integrity levels relate to the cases where DL software provides complementary information related to the safety of the system, whether it intervenes in the safety management of the system, or whether it implements the safety functionality.
(B) Processes and solutions to assess the trustability of DL software have been identified, and applied to the challenges exposed in the case studies. Such solutions allow telling whether the system is being fed with data different to that used for training (e.g. system trained to identify people only, but a dog appears in the scene), whether the DL model used for raising predictions is capable of raising trustable predictions even if input data is similar to the one used for training, whether input data offers insufficient information to raise trustable outcomes (e.g. detecting and classifying overlapped objects), and whether any unforeseen anomaly occurred during the process.
(C) A suitable middleware has been deployed allowing to integrate DL-based safety-relevant applications onto the platform with appropriate levels of abstraction and providing the services needed by the applications. All services required by the case studies have been successfully implemented, as well as those features needed to properly control the application above. Related to the latter, the target platform of the project (NVIDIA Orin) has been carefully analyzed identifying how to master it, setting convenient configurations, and providing real-time guarantees to the DL-based applications to be run on top.
(D) Case studies have been integrated and assessed on top of the target platform, which includes the high-performance hardware platform, the middleware interfacing services for the application and control tasks, and the SAFEXPLAIN technologies needed for the execution of DL-based applications adhering to their safety requirements. Moreover, a generic fully open case study (aka core demo) has been also integrated and released openly to ease technology adoption across applications and domains.
Apart from all the technical advances achieved by the project, huge effort has been done to disseminate and communicate the achievements of the project in a wide variety of communities and audiences, spanning from technical specialists to industrial stakeholders and general audiences. Hand in hand with the dissemination efforts, exploitation efforts have allowed identifying the exploitable items of the project, defining exploitation paths for each one of them, and creating a dialogue between the relevant standardization bodies and the project. In particular, standardization bodies are working toward defining ways to enable the incorporation of DL software in safety-critical systems, and explicit communication with the project has provided those bodies with practical processes and examples that some of them have already incorporated to their syllabus. As part of the dialogue, SAFEXPLAIN partners have been exposed to the directions that those standards are taking so that project solutions already match those standards whenever they become final.
(A) A safety lifecycle describing how to set training and validation data for safety-relevant DL software, training processes, and inference processes during operation has been defined. Such safety lifecycle has already been assessed positively by relevant certification entities and domain experts. Also, safety architectures amenable for different degrees of integrity levels for DL software have been devised, and some of them explicitly applied to the project case studies. Such integrity levels relate to the cases where DL software provides complementary information related to the safety of the system, whether it intervenes in the safety management of the system, or whether it implements the safety functionality.
(B) Processes and solutions to assess the trustability of DL software have been identified, and applied to the challenges exposed in the case studies. Such solutions allow telling whether the system is being fed with data different to that used for training (e.g. system trained to identify people only, but a dog appears in the scene), whether the DL model used for raising predictions is capable of raising trustable predictions even if input data is similar to the one used for training, whether input data offers insufficient information to raise trustable outcomes (e.g. detecting and classifying overlapped objects), and whether any unforeseen anomaly occurred during the process.
(C) A suitable middleware has been deployed allowing to integrate DL-based safety-relevant applications onto the platform with appropriate levels of abstraction and providing the services needed by the applications. All services required by the case studies have been successfully implemented, as well as those features needed to properly control the application above. Related to the latter, the target platform of the project (NVIDIA Orin) has been carefully analyzed identifying how to master it, setting convenient configurations, and providing real-time guarantees to the DL-based applications to be run on top.
(D) Case studies have been integrated and assessed on top of the target platform, which includes the high-performance hardware platform, the middleware interfacing services for the application and control tasks, and the SAFEXPLAIN technologies needed for the execution of DL-based applications adhering to their safety requirements. Moreover, a generic fully open case study (aka core demo) has been also integrated and released openly to ease technology adoption across applications and domains.
Apart from all the technical advances achieved by the project, huge effort has been done to disseminate and communicate the achievements of the project in a wide variety of communities and audiences, spanning from technical specialists to industrial stakeholders and general audiences. Hand in hand with the dissemination efforts, exploitation efforts have allowed identifying the exploitable items of the project, defining exploitation paths for each one of them, and creating a dialogue between the relevant standardization bodies and the project. In particular, standardization bodies are working toward defining ways to enable the incorporation of DL software in safety-critical systems, and explicit communication with the project has provided those bodies with practical processes and examples that some of them have already incorporated to their syllabus. As part of the dialogue, SAFEXPLAIN partners have been exposed to the directions that those standards are taking so that project solutions already match those standards whenever they become final.