Intersubjective AI-driven multimodal interaction for advanced user-centric human robot collaborative applications

Project Information

JARVIS

Grant agreement ID: 101135708

DOI

10.3030/101135708

EC signature date 20 October 2023

Start date 1 January 2024

End date 31 December 2027

Funded under

• Digital, Industry and Space

Total cost

€ 10 338 223,75

EU contribution

€ 9 319 266,26

Coordinated by

PANEPISTIMIO PATRON
Greece

Periodic Reporting for period 1 - JARVIS (Intersubjective AI-driven multimodal interaction for advanced user-centric human robot collaborative applications)

Reporting period: 2024-01-01 to 2024-12-31

Summary of the context and overall objectives of the project

The JARVIS project, as an Innovation Action, aspires to bring together a team of “Best in Class” experts that can combine all the aforementioned technological enablers to:
“develop a reusable set of tools that enable AI driven multimodal means of interaction: a) involving interfaces for physical and remote information exchange, robot control and programming, b) providing social skills to a variety of robots to achieve seamless user-centric interaction that extends human ability for complex tasks and c) demonstrating scalability of application and ability to achieve economies at scale.”

The proposed technologies will be tested for flexibility, scalability, trustworthiness and user acceptance in four use cases, aiming to demonstrate production line and cross sector applicability: Pillar 1 - Agile Manufacturing: a) Aeronautics production (COLLINS-BE), b) Automotive production (TOFAS) ii) Pillar 2 - Inspection and Maintenance (I&M): c) Energy – Nuclear plants decommissioning (EDF), d) Energy – offshore energy production (EQUINOR).

The workplan of JARVIS has as a target to comply with the following 6 objectives:
Objective O1: Development of AI enhanced means of interaction for seamless human-robot communication, control and programming
Objective O2: Development of socially interactive robots
Objective O3: Security, privacy and safety towards trustworthy AI
Objective O4: Development of cognitive and intelligent mechatronics for advanced HRI, dynamic scene modeling and understanding
Objective O5: Application of HRI in large scale pilots
Objective O6: Boosting companies, in particular SMEs and start-ups, to adopt industrial HRI solutions

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the first reporting period (M1-M12) the effort was focused on:
i) Definition of Industrial pilot cases scenarios & validation metrics
ii) Requirement analysis and the specifications for the technologies to be developed
iii) Definition of architecture for interoperable interaction modules, which complies with the requirements
iv) Establishment of the exploitation and dissemination strategy to be followed
v) Initialization of the development of the first prototypes of the JARVIS enabling technologies (to be delivered on M18)

Thus, consortium managed to achieve three Milestones (MS1, MS2), by the submission of the relevant deliverables.

The definition of the architecture enabled us to define 13 software and hardware modules which are currently on the developing phase towards the first prototypes, namely:
- Robot Control Module (RCM)
- Human-Robot Interaction Module (HRIM)
- VR Teleoperation (VRT)
- Generation Of Robot Programs (GORP)
- Teaching By Demonstration Module (TDM)
- Robot Behavior Adaptation (ROBA)
- User-Centric Interfaces (UCI)
- OpenFlow Orchestrator (OFO)
- Multi-Sensory Perception (MSP)
- Human Intention Perception And Prediction (HIPAP)
- Task Planner Module (TPM)
- Intelligent DTs (IDT)
- Smart Mechatronics Control (SMC)

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The JARVIS solutions have the potential to provide wider scientific, economic and societal effects, contributing to the expected impacts as follows:
- Reduction of human exposure to harmful environments through teleoperation: reduction of up to 50%, to be demonstrated in two use cases
- Operator acceptance level: acceptance score of >60% comparing to the baseline
- Reduction of reconfiguration and programming time: decrease of 30% of robot programming time compared to conventional techniques
- Portfolio of re-usable HRI tools that can be applied in various use-cases/sectors: demonstration of application of the JARVIS tools in 4 internal and 15 external use cases
- Productivity increase: decrease of cycle time by 40% in average
- Ergonomic conditions for HRC vs manual: ergonomics scores (RULA/ OCRA/ NIOSH) and acceptance for human involvement: reduction 15%
- Impact creation in the academic community: peer review papers submitted to well-known scientific journals and conferences at the end of the project: 20
- Impact creation through pilots’ dissemination: videos of each case study published inside wide range social media and industrial/ academic networks: 10
- Impact creation through involvement of SMEs: involvement/funding of >24 SMEs through open calls