Periodic Reporting for period 1 - ARISE (Advanced AI and RobotIcS for autonomous task pErformance)
Período documentado: 2024-01-01 hasta 2025-06-30
The European Green Deal aims to make Europe the first climate neutral continent by 2050, setting extremely ambitious targets such as >55% less net greenhouse gas emissions by 2030 & 25% of agricultural land under organic farming.
Achieving such targets requires massive scale-up of renewable energy sources and eco-friendly cultivation systems. Among the largest obstacles for achieving this scale-up however is the fact that both energy and agriculture involve physically demanding and often dangerous jobs while they are both being hit by a massive manual labour shortage.
ARISE aims to introduce a combination of perception and control modules around a reconfigurable robotic manipulator, enabling a step change in the level of automation of complex manipulation tasks in the key sectors of Solar Energy & Hydroponic cultivation.
Our objectives can be summarised as follows:
(i) Develop a novel, pneumatic-based reconfigurable manipulator with advanced soft end-effectors capable of operating in environments with high risk of dust or water ingression while carrying out tasks involving both high forces/torques & delicate, complex manipulation,
(ii) Develop a Hierarchical Imitation Learning module, grounded on acquired knowledge alongside task planning algorithms with reactive planning capabilities including human-robot interaction.
(iii) Develop an Ontological framework for Knowledge Representation to enable robust & fault-tolerant collaboration & autonomous task completion through reasoning based on domain- specific fact understanding,
(iv) Develop a combination of Active Perception, Semantic Mapping & Localisation capabilities to fuse and orchestrate perception modalities in a dynamic context-aware that will enable identification changes in the environment and autonomous operation for longer periods while maintaining trustworthiness & dependability,
(v) Implement an edge-native, resource-optimised & automated computing infrastructure able to support dynamic computing at the Cloud-Edge continuum realising an ecosystem where ML-models for new tasks and applications are efficiently updated deployed on-board the robot or in a distributed manner.
Achieving such targets requires massive scale-up of renewable energy sources and eco-friendly cultivation systems. Among the largest obstacles for achieving this scale-up however is the fact that both energy and agriculture involve physically demanding and often dangerous jobs while they are both being hit by a massive manual labour shortage.
ARISE aims to introduce a combination of perception and control modules around a reconfigurable robotic manipulator, enabling a step change in the level of automation of complex manipulation tasks in the key sectors of Solar Energy & Hydroponic cultivation.
Our objectives can be summarised as follows:
(i) Develop a novel, pneumatic-based reconfigurable manipulator with advanced soft end-effectors capable of operating in environments with high risk of dust or water ingression while carrying out tasks involving both high forces/torques & delicate, complex manipulation,
(ii) Develop a Hierarchical Imitation Learning module, grounded on acquired knowledge alongside task planning algorithms with reactive planning capabilities including human-robot interaction.
(iii) Develop an Ontological framework for Knowledge Representation to enable robust & fault-tolerant collaboration & autonomous task completion through reasoning based on domain- specific fact understanding,
(iv) Develop a combination of Active Perception, Semantic Mapping & Localisation capabilities to fuse and orchestrate perception modalities in a dynamic context-aware that will enable identification changes in the environment and autonomous operation for longer periods while maintaining trustworthiness & dependability,
(v) Implement an edge-native, resource-optimised & automated computing infrastructure able to support dynamic computing at the Cloud-Edge continuum realising an ecosystem where ML-models for new tasks and applications are efficiently updated deployed on-board the robot or in a distributed manner.
The ARISE consortium has so far (M18) accomplished the following:
1) A detailed analysis of key long-horizon tasks that the robotic platform must be able to perform autonomously under the high-level guidance of a human supervisor or in physical human interaction with a worker.
These tasks include solar panel backsheet repair, rack bolt fastening and carrying solar panels for the solar farm environment & lettuce transplanting & harvesting for the hydroponic cultivation environment.
2) An extensive mapping of the functional requirements & the technical specifications derived from the analysis above, pertaining to the robotic manipulator, including its novel pneumatic driven actuators and grippers as well as the sensorial capabilities of the robot.
3) A comprehensive design of the hardware & software architecture ensuring that the low-level control and the high-level cognitive functions of the robot can smoothly interface with each other.
4) A novel delta-compression algorithm that can significantly reduce the latency of deploying containerised AI modules under bandwidth constraints. This has led to a conference publication.
5) Adapted versions of the well-known Universal Manipulation Interface have been fabricated to enable RGB-D and force stream recording for data collection.
6) Demonstration sessions for all Use case Scenarios have been conducted resulting in comprehensive datasets encompassing multiple modalities.
7) An initial version of the simulation framework has been implemented bridging the XDE physics engine, with accuracy in modelling of contact interactions using the well-trusted NVIDIA Omniverse platform.
8) AI pipelines for scene graph models extraction & multi-modal latent motion policies learning have been established.
9) A Semantic rule generalisation framework has been implemented alongside a behaviour tree model for planning & execution.
10) Knowledge Distillation pipelines for computational complexity reduction have been established.
1) A detailed analysis of key long-horizon tasks that the robotic platform must be able to perform autonomously under the high-level guidance of a human supervisor or in physical human interaction with a worker.
These tasks include solar panel backsheet repair, rack bolt fastening and carrying solar panels for the solar farm environment & lettuce transplanting & harvesting for the hydroponic cultivation environment.
2) An extensive mapping of the functional requirements & the technical specifications derived from the analysis above, pertaining to the robotic manipulator, including its novel pneumatic driven actuators and grippers as well as the sensorial capabilities of the robot.
3) A comprehensive design of the hardware & software architecture ensuring that the low-level control and the high-level cognitive functions of the robot can smoothly interface with each other.
4) A novel delta-compression algorithm that can significantly reduce the latency of deploying containerised AI modules under bandwidth constraints. This has led to a conference publication.
5) Adapted versions of the well-known Universal Manipulation Interface have been fabricated to enable RGB-D and force stream recording for data collection.
6) Demonstration sessions for all Use case Scenarios have been conducted resulting in comprehensive datasets encompassing multiple modalities.
7) An initial version of the simulation framework has been implemented bridging the XDE physics engine, with accuracy in modelling of contact interactions using the well-trusted NVIDIA Omniverse platform.
8) AI pipelines for scene graph models extraction & multi-modal latent motion policies learning have been established.
9) A Semantic rule generalisation framework has been implemented alongside a behaviour tree model for planning & execution.
10) Knowledge Distillation pipelines for computational complexity reduction have been established.
The project is expected to create the following:
Two reconfigurable pneumatic-based robotic manipulators mounted on a mobile robotic platform,
Novel soft end-effectors with variable stiffness that will allow for a diverse set of manipulation tasks,
A Robotic perception module comprising 3D vision algorithms,
A Semantic Mapping module powered by scene understanding algorithms,
A Knowledge Representation framework capturing important information regarding objects’ properties & relationships,
A Hierarchical Imitation Learning framework for acquiring robotic skills to accomplish complex tasks directly from human demonstrator,
A Human-Interaction conditioned Path &Task planning module enabling reactive robot control,
An edge-AI framework for deploying ML models & computer vision algorithms at the edge in a streamlined fashion,
Two demonstrators in solar & hydroponic farms and Open datasets for Learning-by-Demonstration of complex tasks in challenging environments.
Leading to the following impacts:
Scientific: step-change towards the automation of manual tasks, paving the way for further applications across a vast range of industries; combination of several disciplines and automation advances to contribute to the great scale workforce transition
Economic: enabling the reduction of labour costs and the increase of productivity, improved yields and higher-quality produce, lower production costs and higher profitability, creating new promising market opportunities
Social: transition to automation of work, unleashing workers from challenging, repetitive, heavy, complex, long-duration, demanding, dangerous tasks and giving the opportunity and time for skill upgrade; future job opportunities will possibly require creativity and analytical abilities to complement robotics and automation capabilities during tasks; integrating automation advances helps enhance performance in all areas, prevent accidents or hazards, and reduce human errors, ensuring a safe working environment. Increased automation in agriculture and solar installation/maintenance will safeguard against food security threats and energy poverty
Environmental: minimisation of resource waste and environmental pollution, reducing energy use while increasing energy production and resources efficiency, decreasing emissions and man-made pollution
Two reconfigurable pneumatic-based robotic manipulators mounted on a mobile robotic platform,
Novel soft end-effectors with variable stiffness that will allow for a diverse set of manipulation tasks,
A Robotic perception module comprising 3D vision algorithms,
A Semantic Mapping module powered by scene understanding algorithms,
A Knowledge Representation framework capturing important information regarding objects’ properties & relationships,
A Hierarchical Imitation Learning framework for acquiring robotic skills to accomplish complex tasks directly from human demonstrator,
A Human-Interaction conditioned Path &Task planning module enabling reactive robot control,
An edge-AI framework for deploying ML models & computer vision algorithms at the edge in a streamlined fashion,
Two demonstrators in solar & hydroponic farms and Open datasets for Learning-by-Demonstration of complex tasks in challenging environments.
Leading to the following impacts:
Scientific: step-change towards the automation of manual tasks, paving the way for further applications across a vast range of industries; combination of several disciplines and automation advances to contribute to the great scale workforce transition
Economic: enabling the reduction of labour costs and the increase of productivity, improved yields and higher-quality produce, lower production costs and higher profitability, creating new promising market opportunities
Social: transition to automation of work, unleashing workers from challenging, repetitive, heavy, complex, long-duration, demanding, dangerous tasks and giving the opportunity and time for skill upgrade; future job opportunities will possibly require creativity and analytical abilities to complement robotics and automation capabilities during tasks; integrating automation advances helps enhance performance in all areas, prevent accidents or hazards, and reduce human errors, ensuring a safe working environment. Increased automation in agriculture and solar installation/maintenance will safeguard against food security threats and energy poverty
Environmental: minimisation of resource waste and environmental pollution, reducing energy use while increasing energy production and resources efficiency, decreasing emissions and man-made pollution