Robots learning about objects from externalized knowledge sources

Project Information

RoboExNovo

Grant agreement ID: 637076

DOI

10.3030/637076

Project closed

EC signature date 11 May 2015

Start date 1 June 2015

End date 31 May 2021

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 496 277,00

EU contribution

€ 1 496 277,00

Coordinated by

FONDAZIONE ISTITUTO ITALIANO DI TECNOLOGIA
Italy

Periodic Reporting for period 4 - RoboExNovo (Robots learning about objects from externalized knowledge sources)

Reporting period: 2019-04-01 to 2021-05-31

Summary of the context and overall objectives of the project

Todays robots are able to perform sophisticated tasks: they can flip a pancake, load and unload a dishwasher, vacuum clean an apartment and pick up fruits from trees, and so forth. Still, robots can only do these things and more when all the knowledge about the objects involved in the tasks has been manually encoded into them. For instance, to load a dishwasher a robot must be programmed to recognize dishes and pans by looking at them; to manipulate glasses without shattering them, or let them fall; to put each of these objects in a pre-definite set of locations within the dish washer, and so on. Hence, robot can only do the things they know how to do, acting on objects they already know, in situations they already understand. This is a severe limitation: any robot will inevitably face novel situations in unconstrained settings, and thus will always have knowledge gaps. For instance, a robot might have to put into the dish washer a new coffee cup, or a pan might have lost one handle.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We as humans are very good at generalizing over our own previous experiences to deal with new, unexpected situations, and learn continuously over time about new things. To do so, we need to make new experiences, sometimes make mistakes, and learn from them. All of this is long, expensive and possibly dangerous for robots. The project RoboExNovo (Robots learning about objects from externalized knowledge sources) approaches this problem bringing a key insight: that robots can learn about what they don't know by searching on the Web, exactly as we do. This would have deep effects on the degree of autonomy that a robot can have: if it is sees a cereal box that it has never seen before in the grocery delivery box, it can look it up on the web, learn autonomously what it is, and store it in the right cupboard without human supervision; and so forth.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Thanks to these conceptual and algorithmic advances, RoboExNovo has made it possible for robots to use knowledge resources on the Web that were not explicitly designed to be accessed for this purpose: in this demo video (https://www.youtube.com/watch?v=eIb9GjIOYXo) we showed how an R1 humanoid platform could learn on the fly to recognize flowers by mining the Web for images of flowers. In 2018, the main results of the project has been showcased in a TED talk in Milan, Italy (https://www.ted.com/talks/barbara_caputo_is_for_a_robot_the_essential_invisible_to_the_eye?language=it). In the same year, the project has been showcased in the event 'Frontier Research and Artificial Intelligence', organized by the European Research Council, in the opening session 'Foundations of Artificial Intelligence' (https://erc.europa.eu/event/frontier-research-and-artificial-intelligence https://www.youtube.com/watch?v=ZjHYi2Kb4Jk).

start-end-min.png