Robots learning about objects from externalized knowledge sources

Projektinformationen

RoboExNovo

ID Finanzhilfevereinbarung: 637076

DOI

10.3030/637076

Projekt abgeschlossen

EK-Unterschriftsdatum 11 Mai 2015

Startdatum 1 Juni 2015

Enddatum 31 Mai 2021

Finanziert unter

• EXCELLENT SCIENCE - European Research Council (ERC)

Gesamtkosten

€ 1 496 277,00

EU-Beitrag

€ 1 496 277,00

Koordiniert durch

FONDAZIONE ISTITUTO ITALIANO DI TECNOLOGIA
Italy

Periodic Reporting for period 4 - RoboExNovo (Robots learning about objects from externalized knowledge sources)

Berichtszeitraum: 2019-04-01 bis 2021-05-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Todays robots are able to perform sophisticated tasks: they can flip a pancake, load and unload a dishwasher, vacuum clean an apartment and pick up fruits from trees, and so forth. Still, robots can only do these things and more when all the knowledge about the objects involved in the tasks has been manually encoded into them. For instance, to load a dishwasher a robot must be programmed to recognize dishes and pans by looking at them; to manipulate glasses without shattering them, or let them fall; to put each of these objects in a pre-definite set of locations within the dish washer, and so on. Hence, robot can only do the things they know how to do, acting on objects they already know, in situations they already understand. This is a severe limitation: any robot will inevitably face novel situations in unconstrained settings, and thus will always have knowledge gaps. For instance, a robot might have to put into the dish washer a new coffee cup, or a pan might have lost one handle.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

We as humans are very good at generalizing over our own previous experiences to deal with new, unexpected situations, and learn continuously over time about new things. To do so, we need to make new experiences, sometimes make mistakes, and learn from them. All of this is long, expensive and possibly dangerous for robots. The project RoboExNovo (Robots learning about objects from externalized knowledge sources) approaches this problem bringing a key insight: that robots can learn about what they don't know by searching on the Web, exactly as we do. This would have deep effects on the degree of autonomy that a robot can have: if it is sees a cereal box that it has never seen before in the grocery delivery box, it can look it up on the web, learn autonomously what it is, and store it in the right cupboard without human supervision; and so forth.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

Thanks to these conceptual and algorithmic advances, RoboExNovo has made it possible for robots to use knowledge resources on the Web that were not explicitly designed to be accessed for this purpose: in this demo video (https://www.youtube.com/watch?v=eIb9GjIOYXo) we showed how an R1 humanoid platform could learn on the fly to recognize flowers by mining the Web for images of flowers. In 2018, the main results of the project has been showcased in a TED talk in Milan, Italy (https://www.ted.com/talks/barbara_caputo_is_for_a_robot_the_essential_invisible_to_the_eye?language=it). In the same year, the project has been showcased in the event 'Frontier Research and Artificial Intelligence', organized by the European Research Council, in the opening session 'Foundations of Artificial Intelligence' (https://erc.europa.eu/event/frontier-research-and-artificial-intelligence https://www.youtube.com/watch?v=ZjHYi2Kb4Jk).

start-end-min.png