Robots learning about objects from externalized knowledge sources

Información del proyecto

RoboExNovo

Identificador del acuerdo de subvención: 637076

DOI

10.3030/637076

Proyecto cerrado

Fecha de la firma de la CE 11 Mayo 2015

Fecha de inicio 1 Junio 2015

Fecha de finalización 31 Mayo 2021

Financiado con arreglo a

• EXCELLENT SCIENCE - European Research Council (ERC)

Coste total

€ 1 496 277,00

Aportación de la UE

€ 1 496 277,00

Coordinado por

FONDAZIONE ISTITUTO ITALIANO DI TECNOLOGIA
Italy

Periodic Reporting for period 4 - RoboExNovo (Robots learning about objects from externalized knowledge sources)

Período documentado: 2019-04-01 hasta 2021-05-31

Resumen del contexto y de los objetivos generales del proyecto

Todays robots are able to perform sophisticated tasks: they can flip a pancake, load and unload a dishwasher, vacuum clean an apartment and pick up fruits from trees, and so forth. Still, robots can only do these things and more when all the knowledge about the objects involved in the tasks has been manually encoded into them. For instance, to load a dishwasher a robot must be programmed to recognize dishes and pans by looking at them; to manipulate glasses without shattering them, or let them fall; to put each of these objects in a pre-definite set of locations within the dish washer, and so on. Hence, robot can only do the things they know how to do, acting on objects they already know, in situations they already understand. This is a severe limitation: any robot will inevitably face novel situations in unconstrained settings, and thus will always have knowledge gaps. For instance, a robot might have to put into the dish washer a new coffee cup, or a pan might have lost one handle.

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

We as humans are very good at generalizing over our own previous experiences to deal with new, unexpected situations, and learn continuously over time about new things. To do so, we need to make new experiences, sometimes make mistakes, and learn from them. All of this is long, expensive and possibly dangerous for robots. The project RoboExNovo (Robots learning about objects from externalized knowledge sources) approaches this problem bringing a key insight: that robots can learn about what they don't know by searching on the Web, exactly as we do. This would have deep effects on the degree of autonomy that a robot can have: if it is sees a cereal box that it has never seen before in the grocery delivery box, it can look it up on the web, learn autonomously what it is, and store it in the right cupboard without human supervision; and so forth.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

Thanks to these conceptual and algorithmic advances, RoboExNovo has made it possible for robots to use knowledge resources on the Web that were not explicitly designed to be accessed for this purpose: in this demo video (https://www.youtube.com/watch?v=eIb9GjIOYXo) we showed how an R1 humanoid platform could learn on the fly to recognize flowers by mining the Web for images of flowers. In 2018, the main results of the project has been showcased in a TED talk in Milan, Italy (https://www.ted.com/talks/barbara_caputo_is_for_a_robot_the_essential_invisible_to_the_eye?language=it). In the same year, the project has been showcased in the event 'Frontier Research and Artificial Intelligence', organized by the European Research Council, in the opening session 'Foundations of Artificial Intelligence' (https://erc.europa.eu/event/frontier-research-and-artificial-intelligence https://www.youtube.com/watch?v=ZjHYi2Kb4Jk).

start-end-min.png