Periodic Reporting for period 4 - RoboExNovo (Robots learning about objects from externalized knowledge sources)
Berichtszeitraum: 2019-04-01 bis 2021-05-31
Todays robots are able to perform sophisticated tasks: they can flip a pancake, load and unload a dishwasher, vacuum clean an apartment and pick up fruits from trees, and so forth. Still, robots can only do these things and more when all the knowledge about the objects involved in the tasks has been manually encoded into them. For instance, to load a dishwasher a robot must be programmed to recognize dishes and pans by looking at them; to manipulate glasses without shattering them, or let them fall; to put each of these objects in a pre-definite set of locations within the dish washer, and so on. Hence, robot can only do the things they know how to do, acting on objects they already know, in situations they already understand. This is a severe limitation: any robot will inevitably face novel situations in unconstrained settings, and thus will always have knowledge gaps. For instance, a robot might have to put into the dish washer a new coffee cup, or a pan might have lost one handle.
We as humans are very good at generalizing over our own previous experiences to deal with new, unexpected situations, and learn continuously over time about new things. To do so, we need to make new experiences, sometimes make mistakes, and learn from them. All of this is long, expensive and possibly dangerous for robots. The project RoboExNovo (Robots learning about objects from externalized knowledge sources) approaches this problem bringing a key insight: that robots can learn about what they don't know by searching on the Web, exactly as we do. This would have deep effects on the degree of autonomy that a robot can have: if it is sees a cereal box that it has never seen before in the grocery delivery box, it can look it up on the web, learn autonomously what it is, and store it in the right cupboard without human supervision; and so forth.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.
To make this possible, RoboExNovo has achieved several breakthroughs, from developing the first algorithm for bridging between web stored perceptual knowledge, like RGB images, and robot knowledge, like depth maps to introducing the first approach for creating new robot memories about unknown objects by automatic web mining; from introducing the theory and algorithms for online learning of new things and experiences in deep robot perception, to the first deep architecture able to teach robots how to learn from Web data by solving self-created riddles and jigsaw puzzles.
Thanks to these conceptual and algorithmic advances, RoboExNovo has made it possible for robots to use knowledge resources on the Web that were not explicitly designed to be accessed for this purpose: in this demo video (https://www.youtube.com/watch?v=eIb9GjIOYXo) we showed how an R1 humanoid platform could learn on the fly to recognize flowers by mining the Web for images of flowers. In 2018, the main results of the project has been showcased in a TED talk in Milan, Italy (https://www.ted.com/talks/barbara_caputo_is_for_a_robot_the_essential_invisible_to_the_eye?language=it). In the same year, the project has been showcased in the event 'Frontier Research and Artificial Intelligence', organized by the European Research Council, in the opening session 'Foundations of Artificial Intelligence' (https://erc.europa.eu/event/frontier-research-and-artificial-intelligence https://www.youtube.com/watch?v=ZjHYi2Kb4Jk).