Skip to main content

Immersive Visual Technologies for Safety-critical Applications

Periodic Reporting for period 1 - ImmerSAFE (Immersive Visual Technologies for Safety-critical Applications)

Reporting period: 2018-01-01 to 2019-12-31

In many contexts – heavy work machines, emergency response, control centres – human operators face complex and demanding situations where their decisions can have far-reaching consequences on productivity, environment, and even human lives. In order to deal with such challenges, the operator needs full situational awareness, which can be achieved by sensing relevant visual data about the operating environment, refining it into mission-critical information, and presenting it in an immersive, intuitively comprehensible manner. This can be achieved by Immersive Visual Technologies (IVT) delivering ultra-realistic and interactive visual experience.

The aim of ImmerSAFE is to train cross-disciplinary experts who understand imaging and display technology, reliable data transmission, embedded hardware and software system design, user-centred design and quality of experience, and cognitive and organisational psychology, and are able to apply that integral knowledge to safety-critical applications. The network advances the state of the art by working towards three specific objectives:
• Develop new robust and reliable IVT; in particular novel approaches to sensing, visualisation and reliable data transmission.
• Determine optimal implementation of IVT under challenging conditions and integration into existing frameworks in two example use cases: mobile work machines and control centres. These aspects are considered through interface design, embedded systems implementation and system dependability analysis.
• Increase knowledge on the human factors affecting the adoption and use of IVT through user experience testing, task performance metrics and organisational level studies.
The research in ImmerSAFE has been structured in three scientific and technological work packages (WP) as illustrated in Fig. 1, aligned to work towards the three project objectives.

WP1 (immersive visual technologies) deals with the problems related to sensing, processing and transmission of rich visual data. Current state and results include:
• Method for automatic joint calibration of multiple cameras utilizing an industrial robotic arm that can be used even in configurations in which neither of the cameras’ field of views intersect.
• State of the art overview of see-through display that could simultaneously provide the accommodation and convergence cues.
• Approach for optimizing camera placement on a heavy machine with the aim of obtaining complete surround view with a minimal number of cameras.
• Simulation and performance evaluation of fail-operational and fail-safe vehicle platooning in the presence of transient communication errors.
• Evaluation of deep learning approaches for object detection in UAV images.
• Point cloud compression method that first segments the point clouds based on the color attributes at each node and then separately compresses the segments.

WP2 (Embedded systems for safety-critical applications) addresses issues related with system aspects of technologies developed in WP1 and refined by user studies in WP3. Current state and results include:
• Extension of human and organization modelling elements in existing metamodel used for modelling socio-technical systems, to be able to model AR-equipped socio-technical systems
• Design of visual representations, on the windshield of excavators and mobile cranes, that can be used to assist operators to perform safe operations.
• Firefighter training simulator for earthquake situations based on virtual reality.
• Transparent material that emits red color and its novel configuration found to emit blue color.

WP3 (User experience, performance and organisational change) addresses needs and requirements of the user of immersive visual solutions developed in WP1 and WP2. The issue is analysed from the productivity and safety perspective. Current state and results include:
• Evaluation of stressful and non-stressful conditions utilizing the Stroop and arithmetic test.
• Conceptual layer-based augmented reality / mixed reality model of the key aspects of the quality of experience.
• Development of cognitive tasks for evaluating human factors during the use of immersive visual technologies.
• Model covering the key issues at each stage of organizational technological changes.

Organization of training within ImmerSAFE is presented in Fig. 2.

The network organized two Training Schools, first one focused on image processing algorithms and methods for sensing, processing, and visualization of 3D content and second one focused on basics of distributed embedded systems, communication within and between embedded systems, and concepts of safety assurance and certification. The network also held two Tech Days that were more practically oriented, that is, the ESRs were exposed to use cases of IVT and met in person with professionals from relevant industries. It also organized first series of webinars that included state-of-the art presentations by ESRs and moderated discussions around the research topics.
ImmerSAFE will deliver advances beyond the state of the art in three core areas: Technology Components, System (Application) Level, and Human Factors. In the area of Technology Components, the research is expected to result in novel light field vision enhancement algorithms, novel head up and multi-coloured transparent display designs, algorithms for multi-modal 3D reconstruction, ultra-reliable wireless communication, hardware-friendly imaging algorithm implementations, and multimodal rendering approaches for high quality of experience. In the area of Human Factors, the advances will result in novel AR attention models, methodologies for evaluating task-based performance when using IVT and Identifying factors within organisations which have an effect to the adoption of new IVT. Finally, in the area of System Level, the novel IVT technologies will be employed to cabins of work machines and in control centres – the two use cases the project is focusing on. When implemented in the corresponding work pipelines, the aforementioned advances will considerably increase safety of construction sites and cargo terminals, work machine operators, remote machine operation in hazardous conditions, will enable more efficient use of resources, and faster and more efficient reactions in the case of emergencies.

On an individual level, the 15 ESRs who are working on the project will gather expert knowledge on various IVT topics, such as sensing, data communications, representations, display technologies, user experience and safety culture. This will be achieved through multidisciplinary projects and courses, and inter-sectorial secondments. They will also acquire a multidisciplinary knowledge and the skills that are needed in academia and industry. This will make them very competitive in the job markets. It is expected that their expertise will be a driving force for the next generation of innovative solutions related to IVT for safety critical applications.
Overview of the research topics addressed in the three scientific work packages of ImmerSAFE
Overview of the ImmerSAFE training structure