Skip to main content
Go to the home page of the European Commission (opens in new window)
English en
CORDIS - EU research results
CORDIS

Scalable Manipulation Learning through AR-enhanced Teleoperation enabling Intuitive Interactive Instructions

Project description

Scaling up robot skill-learning capabilities

Significant advancements in AI, robotics, and digital technologies have led to breakthroughs in several sectors, especially automation. Unfortunately, both AI and robotics still struggle to engage with diverse objects, complex scenarios, and environments. There are also limitations in their capacity to manipulate objects of varying properties and geometries. Following intricate instruction manuals is also a challenge. With this in mind, the ERC-funded SMARTI3 project will streamline the generation of human and synthetic data for robot learning, greatly boosting and scaling up robot skill-learning capabilities. This approach combines data generation, leveraging human feedback for continuous self-improvement, fine-tuning foundational knowledge, and incorporating offline instructions and geometric information to develop key models for perception, prediction, and skill.

Objective

SMARTI3 addresses three pivotal visions in robotics: enabling robots to handle diverse objects in complex scenes, manipulate objects with varying geometries and physical properties, and interpret and follow intricate instruction manuals. While AI is predestined to realize these visions, its promises to fulfill these visions have not been fully realized yet. Despite significant breakthroughs of AI in many domains, robotics has not seen a parallel surge, primarily due to challenges in task-specific data collection, the high demand for training data, and the extensive expertise required for applying AI in robotics.

Our innovative approach integrates intuitive data generation, fine-tuning of foundational knowledge, leveraging human feedback for continual self-improvement, and incorporating offline instructions with semantic and geometric information into skill, perception, and prediction models. Our objectives are to streamline the generation of human and synthetic data for robot learning, develop few-shot adaptable foundational representations, scale-up robot skill-learning capabilities, and enhance learning from interactive human feedback and offline instructions.

The project showcases its advancements through three use cases that mirror future industrial applications: sorting and disassembling a variety of Lego pieces, origami folding from visual instructions, and assembling complex Lego structures from unsorted pieces using instruction manuals. These cases highlight our approach's adaptability and potential for industrial application.

SMARTI3 aims to revolutionize robotic skill, perception, and prediction models, significantly broadening the scope of robotic technology in various industries. This groundbreaking project lays the foundation for a new era of intelligent, adaptable, and user-friendly robotic solutions, set to rival human manipulation capabilities.

Fields of science (EuroSciVoc)

Programme(s)

Topic(s)

Funding Scheme

HORIZON-ERC - HORIZON ERC Grants

See all projects funded under this funding scheme

Call for proposal

(opens in new window) ERC-2024-COG

See all projects funded under this call

Host institution

KARLSRUHER INSTITUT FUER TECHNOLOGIE
Net EU contribution
€ 2 445 808,00
Address
KAISERSTRASSE 12
76131 Karlsruhe
Germany
Region
Baden-Württemberg Karlsruhe Karlsruhe, Stadtkreis
Activity type
Higher or Secondary Education Establishments
Links
Total cost
€ 2 445 808,00

Beneficiaries (1)