Robots learning about objects from externalized knowledge sources

Informations projet

RoboExNovo

N° de convention de subvention: 637076

DOI

10.3030/637076

Projet clôturé

Date de signature de la CE 11 Mai 2015

Date de début 1 Juin 2015

Date de fin 31 Mai 2021

Financé au titre de

• EXCELLENT SCIENCE - European Research Council (ERC)

Coût total

€ 1 496 277,00

Contribution de l’UE

€ 1 496 277,00

Coordonné par

FONDAZIONE ISTITUTO ITALIANO DI TECNOLOGIA
Italy

Periodic Reporting for period 4 - RoboExNovo (Robots learning about objects from externalized knowledge sources)

Période du rapport: 2019-04-01 au 2021-05-31

Résumé du contexte et des objectifs généraux du projet

Todays robots are able to perform sophisticated tasks: they can flip a pancake, load and unload a dishwasher, vacuum clean an apartment and pick up fruits from trees, and so forth. Still, robots can only do these things and more when all the knowledge about the objects involved in the tasks has been manually encoded into them. For instance, to load a dishwasher a robot must be programmed to recognize dishes and pans by looking at them; to manipulate glasses without shattering them, or let them fall; to put each of these objects in a pre-definite set of locations within the dish washer, and so on. Hence, robot can only do the things they know how to do, acting on objects they already know, in situations they already understand. This is a severe limitation: any robot will inevitably face novel situations in unconstrained settings, and thus will always have knowledge gaps. For instance, a robot might have to put into the dish washer a new coffee cup, or a pan might have lost one handle.

Travail effectué depuis le début du projet jusqu’à la fin de la période considérée dans le rapport et principaux résultats atteints jusqu’à présent

We as humans are very good at generalizing over our own previous experiences to deal with new, unexpected situations, and learn continuously over time about new things. To do so, we need to make new experiences, sometimes make mistakes, and learn from them. All of this is long, expensive and possibly dangerous for robots. The project RoboExNovo (Robots learning about objects from externalized knowledge sources) approaches this problem bringing a key insight: that robots can learn about what they don't know by searching on the Web, exactly as we do. This would have deep effects on the degree of autonomy that a robot can have: if it is sees a cereal box that it has never seen before in the grocery delivery box, it can look it up on the web, learn autonomously what it is, and store it in the right cupboard without human supervision; and so forth.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.

Progrès au-delà de l’état des connaissances et impact potentiel prévu (y compris l’impact socio-économique et les conséquences sociétales plus larges du projet jusqu’à présent)

Thanks to these conceptual and algorithmic advances, RoboExNovo has made it possible for robots to use knowledge resources on the Web that were not explicitly designed to be accessed for this purpose: in this demo video (https://www.youtube.com/watch?v=eIb9GjIOYXo) we showed how an R1 humanoid platform could learn on the fly to recognize flowers by mining the Web for images of flowers. In 2018, the main results of the project has been showcased in a TED talk in Milan, Italy (https://www.ted.com/talks/barbara_caputo_is_for_a_robot_the_essential_invisible_to_the_eye?language=it). In the same year, the project has been showcased in the event 'Frontier Research and Artificial Intelligence', organized by the European Research Council, in the opening session 'Foundations of Artificial Intelligence' (https://erc.europa.eu/event/frontier-research-and-artificial-intelligence https://www.youtube.com/watch?v=ZjHYi2Kb4Jk).

start-end-min.png