Immersive Visual Technologies for Safety-critical Applications

Project Information

ImmerSAFE

Grant agreement ID: 764951

DOI

10.3030/764951

Project closed

EC signature date 2 August 2017

Start date 1 January 2018

End date 30 November 2022

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 3 957 576,84

EU contribution

€ 3 957 576,84

Coordinated by

TAMPEREEN KORKEAKOULUSAATIO SR
Finland

Periodic Reporting for period 2 - ImmerSAFE (Immersive Visual Technologies for Safety-critical Applications)

Reporting period: 2020-01-01 to 2022-11-30

Summary of the context and overall objectives of the project

In safety-critical environments, Immersive Visual Technologies (IVT) are the key for providing full situational awareness to human operators to allow them making decisions with far-reaching consequences on productivity, environment, and even human lives. IVT are developed and deployed by multi-disciplinary experts, who understand the core imaging technologies and the requirements set by the safety-critical applications, and who can account for the human user in the design of related systems. ImmerSAFE trained fifteen such experts by involving them into a diverse research programme, and challenging them with the following overall objectives:
• O1: Develop new robust and reliable approaches to sensing, visualisation and reliable data transmission
• O2: Determine optimal implementation of IVT under challenging conditions and integration into existing frameworks in two example use cases: mobile work machines and control centres, with specific focus on interface design, embedded systems implementation, and system dependability analysis
• O3: Increase knowledge on the human factors affecting the adoption and use of IVT through user experience testing, task performance metrics, and organisational level studies
Throughout the project, the employed ESRs developed new methods, procedures, algorithms, and materials, as detailed in Section 2, which in their integrity fully achieved the research objectives.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The research in ImmerSAFE has been structured in three scientific and technological work packages (WP) as illustrated in Fig. 1.

The ESRs of WP1 delivered the following main outcomes, which together led to achieving O1.
• Laura Ribeiro developed a method for automatic joint calibration of multiple cameras utilizing an industrial robotic arm, published in Int. Symposium on Image and Signal Processing and Analysis
• Laura also developed a retro-reflective marker-based approach for remote operation and alignment of tools in harsh industrial environments, published in Fusion Engineering and Design, Applied Sciences and EUVIP
• Yuta Miyanishi developed a rigorous modelling framework for simulating observed retinal images in reaction to a light field, applicable for the next-generation near-eye displays, published in Optics Express
• Anirudh Puligandla developed a multi-resolution approach for optimizing camera placement on a heavy machine, which delivers a complete surround view with a minimal number of cameras, published in IEEE Access
• Shahriar Hasan developed a PlatoonSAFE tool that simulates realistic vehicle dynamics, road traffic, wireless communications, fail-operational and fail-safe states, published in IEEE Vehicular Technology Conference and IEEE Open Journal of Intelligent Transportation Systems
• Shahriar also developed machine learning-based methods for predicting communication delays in vehicular networks
• Nayee Muddin Khan Dousai developed a method for detection of humans in UAV images based on ensemble learning, published in IEEE Access
• Emanuele Palma developed a segmentation-based point cloud compression method and an LF sensor image lossless compression method exploiting pixel predictions, published in EUSIPCO, EUVIP and IEEE Access

ESRs within WP2 delivered outcomes achieving O2
• Soheila Sheikh Bahaei extended human and organization modelling elements in existing metamodels used for modelling socio-technical systems to model AR-equipped socio-technical systems, published in Int. Conf. on Computer Safety, Reliability and Security
• She also developed a framework for risk assessment of AR-equipped socio-technical systems, published in Journal of Systems Architecture
• Taufik Akbar Sitompul designed visual representations to be displayed on the windshield of excavators and mobile cranes, published in VRCAI and Multimodal Technologies and Interaction and together with José Rosa prototyped a transparent display with extended visual functionalities, published in ISARC
• Jose also developed new transparent materials that emit red and blue colors, published in Materials
• Saif Hadj Sassi developed a firefighter training simulator for earthquake situations based on virtual reality, published in Sensors
• Dinar Khowaja developed a simulation system for comparing VR and LCD displays in various driving conditions

The ESRs organized in WP3 delivered outcomes toward O3:
• Simone Grassini evaluated stressful and non-stressful conditions utilizing the Stroop and arithmetic test and developed a driving simulator equipped with non-invasive medical devices for monitoring physiological parameters when using AR, published in Frontiers in Psychology and Brain and Behavior
• Syed Muhammad Umair Arif created a multi-platform AR web-based application for enhancing the user experience in control rooms, published in Frontiers in Neuroscience
• Moses Mariajoseph analyzed the operators’ reaction time when using AR, published in IEEE Vehicular Technology Conference
• Mina Saghafian developed cognitive tasks for evaluating human factors (e.g. presence, motion sickness, workload) during the use of IVTs and proposed a model covering the key issues at each stage of organizational technological changes, published in Frontiers in Psychology.

The network training programme, illustrated in Fig. 2, included 3 Training Schools, 3 Tech Days, 2 Webinar Series, and the final ImmerSAFE cluster day.

The project results have been published in 62 scientific publications and were disseminated to the public through presentation at 28 events. The project also contributed to the work of the JPEG and MPEG standardization bodies.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

ImmerSAFE delivered advances in three core areas: Technology Components, Systems, and Human Factors. In the first area, the ESRs delivered novel LF compression and vision enhancement algorithms, novel technologies for multi-coloured transparent display designs, ultra-reliable wireless communication, hardware-friendly imaging algorithm implementations, and multimodal rendering approaches for high quality of experience. In the third area, the advances resulted in novel AR attention models, methodologies for evaluating task-based performance when using IVT and identifying factors within organisations which have an effect to the adoption of new IVT. Finally, the novel IVT technologies were evaluated for usage on system level in work machines and control centres. The project advances will considerably increase safety of construction sites, cargo terminals, work machine operators, and operation in hazardous conditions. They will enable more efficient use of resources, and faster and more efficient reactions in the case of emergencies.

On an individual level, the ESRs gathered expert knowledge on core IVT topics, such as sensing, data communications, display technologies, user experience and safety culture. They also acquired multidisciplinary knowledge and skills needed in academia and industry. Their expertise will be a driving force for the next generation of innovative solutions related to IVT for safety critical applications.

On European level, the ImmerSAFE research results and the successfully trained experts will strengthen the European innovation capacity by enhancing and expanding the research base of the academic partners in important fields of research, deploying the developed technologies in all industries located along the IVT chain, and promoting the changes to organisations employing IVT for commercial or public safety applications.

Overview of the research topics addressed in the three scientific work packages of ImmerSAFE

Overview of the ImmerSAFE training structure