Periodic Reporting for period 1 - JARVIS (Intersubjective AI-driven multimodal interaction for advanced user-centric human robot collaborative applications)
Reporting period: 2024-01-01 to 2024-12-31
The JARVIS project, as an Innovation Action, aspires to bring together a team of “Best in Class” experts that can combine all the aforementioned technological enablers to:
“develop a reusable set of tools that enable AI driven multimodal means of interaction: a) involving interfaces for physical and remote information exchange, robot control and programming, b) providing social skills to a variety of robots to achieve seamless user-centric interaction that extends human ability for complex tasks and c) demonstrating scalability of application and ability to achieve economies at scale.”
The proposed technologies will be tested for flexibility, scalability, trustworthiness and user acceptance in four use cases, aiming to demonstrate production line and cross sector applicability: Pillar 1 - Agile Manufacturing: a) Aeronautics production (COLLINS-BE), b) Automotive production (TOFAS) ii) Pillar 2 - Inspection and Maintenance (I&M): c) Energy – Nuclear plants decommissioning (EDF), d) Energy – offshore energy production (EQUINOR).
The workplan of JARVIS has as a target to comply with the following 6 objectives:
Objective O1: Development of AI enhanced means of interaction for seamless human-robot communication, control and programming
Objective O2: Development of socially interactive robots
Objective O3: Security, privacy and safety towards trustworthy AI
Objective O4: Development of cognitive and intelligent mechatronics for advanced HRI, dynamic scene modeling and understanding
Objective O5: Application of HRI in large scale pilots
Objective O6: Boosting companies, in particular SMEs and start-ups, to adopt industrial HRI solutions
“develop a reusable set of tools that enable AI driven multimodal means of interaction: a) involving interfaces for physical and remote information exchange, robot control and programming, b) providing social skills to a variety of robots to achieve seamless user-centric interaction that extends human ability for complex tasks and c) demonstrating scalability of application and ability to achieve economies at scale.”
The proposed technologies will be tested for flexibility, scalability, trustworthiness and user acceptance in four use cases, aiming to demonstrate production line and cross sector applicability: Pillar 1 - Agile Manufacturing: a) Aeronautics production (COLLINS-BE), b) Automotive production (TOFAS) ii) Pillar 2 - Inspection and Maintenance (I&M): c) Energy – Nuclear plants decommissioning (EDF), d) Energy – offshore energy production (EQUINOR).
The workplan of JARVIS has as a target to comply with the following 6 objectives:
Objective O1: Development of AI enhanced means of interaction for seamless human-robot communication, control and programming
Objective O2: Development of socially interactive robots
Objective O3: Security, privacy and safety towards trustworthy AI
Objective O4: Development of cognitive and intelligent mechatronics for advanced HRI, dynamic scene modeling and understanding
Objective O5: Application of HRI in large scale pilots
Objective O6: Boosting companies, in particular SMEs and start-ups, to adopt industrial HRI solutions
During the first reporting period (M1-M12) the effort was focused on:
i) Definition of Industrial pilot cases scenarios & validation metrics
ii) Requirement analysis and the specifications for the technologies to be developed
iii) Definition of architecture for interoperable interaction modules, which complies with the requirements
iv) Establishment of the exploitation and dissemination strategy to be followed
v) Initialization of the development of the first prototypes of the JARVIS enabling technologies (to be delivered on M18)
Thus, consortium managed to achieve three Milestones (MS1, MS2), by the submission of the relevant deliverables.
The definition of the architecture enabled us to define 13 software and hardware modules which are currently on the developing phase towards the first prototypes, namely:
- Robot Control Module (RCM)
- Human-Robot Interaction Module (HRIM)
- VR Teleoperation (VRT)
- Generation Of Robot Programs (GORP)
- Teaching By Demonstration Module (TDM)
- Robot Behavior Adaptation (ROBA)
- User-Centric Interfaces (UCI)
- OpenFlow Orchestrator (OFO)
- Multi-Sensory Perception (MSP)
- Human Intention Perception And Prediction (HIPAP)
- Task Planner Module (TPM)
- Intelligent DTs (IDT)
- Smart Mechatronics Control (SMC)
i) Definition of Industrial pilot cases scenarios & validation metrics
ii) Requirement analysis and the specifications for the technologies to be developed
iii) Definition of architecture for interoperable interaction modules, which complies with the requirements
iv) Establishment of the exploitation and dissemination strategy to be followed
v) Initialization of the development of the first prototypes of the JARVIS enabling technologies (to be delivered on M18)
Thus, consortium managed to achieve three Milestones (MS1, MS2), by the submission of the relevant deliverables.
The definition of the architecture enabled us to define 13 software and hardware modules which are currently on the developing phase towards the first prototypes, namely:
- Robot Control Module (RCM)
- Human-Robot Interaction Module (HRIM)
- VR Teleoperation (VRT)
- Generation Of Robot Programs (GORP)
- Teaching By Demonstration Module (TDM)
- Robot Behavior Adaptation (ROBA)
- User-Centric Interfaces (UCI)
- OpenFlow Orchestrator (OFO)
- Multi-Sensory Perception (MSP)
- Human Intention Perception And Prediction (HIPAP)
- Task Planner Module (TPM)
- Intelligent DTs (IDT)
- Smart Mechatronics Control (SMC)
The JARVIS solutions have the potential to provide wider scientific, economic and societal effects, contributing to the expected impacts as follows:
- Reduction of human exposure to harmful environments through teleoperation: reduction of up to 50%, to be demonstrated in two use cases
- Operator acceptance level: acceptance score of >60% comparing to the baseline
- Reduction of reconfiguration and programming time: decrease of 30% of robot programming time compared to conventional techniques
- Portfolio of re-usable HRI tools that can be applied in various use-cases/sectors: demonstration of application of the JARVIS tools in 4 internal and 15 external use cases
- Productivity increase: decrease of cycle time by 40% in average
- Ergonomic conditions for HRC vs manual: ergonomics scores (RULA/ OCRA/ NIOSH) and acceptance for human involvement: reduction 15%
- Impact creation in the academic community: peer review papers submitted to well-known scientific journals and conferences at the end of the project: 20
- Impact creation through pilots’ dissemination: videos of each case study published inside wide range social media and industrial/ academic networks: 10
- Impact creation through involvement of SMEs: involvement/funding of >24 SMEs through open calls
- Reduction of human exposure to harmful environments through teleoperation: reduction of up to 50%, to be demonstrated in two use cases
- Operator acceptance level: acceptance score of >60% comparing to the baseline
- Reduction of reconfiguration and programming time: decrease of 30% of robot programming time compared to conventional techniques
- Portfolio of re-usable HRI tools that can be applied in various use-cases/sectors: demonstration of application of the JARVIS tools in 4 internal and 15 external use cases
- Productivity increase: decrease of cycle time by 40% in average
- Ergonomic conditions for HRC vs manual: ergonomics scores (RULA/ OCRA/ NIOSH) and acceptance for human involvement: reduction 15%
- Impact creation in the academic community: peer review papers submitted to well-known scientific journals and conferences at the end of the project: 20
- Impact creation through pilots’ dissemination: videos of each case study published inside wide range social media and industrial/ academic networks: 10
- Impact creation through involvement of SMEs: involvement/funding of >24 SMEs through open calls