Final Report Summary - L3VISU (Life Long Learning for Visual Scene Understanding (L3ViSU))
Classical machine learning systems are able to solve many different tasks, but they do so in an isolated manner. Each tasks requires a new set of examples to learn from and a new training (optimization) step. Contrary to what we are used from in natural systems, new tasks do not benefit from what the system has learned before. Even worse, the learning system tends to forget any old task when learning a new one. The objective of the project was to change these fact, giving machine learning systems the ability of lifelong learning: with every tasks learned, the learning of the next task should get simpler or more efficient. Do demonstrate the practical benefits, we chose the application area of visual scene understanding in computer vision.
The outcomes of the project advanced the state of the art on several directions. In particular, we obtained many new theoretical insights how information transfer between tasks can occur. It allowed formulating new theoretical guarantees that, in particular, showed the importance of regularizing not only the individual learning steps but also the process of information transfer itself. Additionally, we were able to show for the first time the practical possibility of learning by conditional risk minimization, i.e. continual learning under time-varying conditions where the systems searches the best prediction function specifically for at each time step. Previously, systems could only be trained to perform well in a long-term –potentially infinite-time– average.
Besides theoretical contributions, we also obtained practical algorithms for lifelong, multi-task and time-varying learning in a computer vision context. One noteworthy example is SEC, a method that allows the learning of semantic segmentation models for natural images with much less annotation effort, because it transfers useful prior knowledge from a pre-trained image classification model. A second example is iCaRL, a technique for class-incremental learning that makes it possible to train a object classification system first from a small number of classes, but then adding new classes later without having the system forget about the originally learned ones.
The insights and techniques developed in L3ViSU have been picked up and further developed by several research groups as well as practitioners worldwide, and companies have expressed their interest as well. We expect that within the next few years, lifelong learning techniques will become part of many widely used machine learning (artificial intelligence) platforms, e.g. reducing the necessary training effort for cloud services or smart phone applications.
The outcomes of the project advanced the state of the art on several directions. In particular, we obtained many new theoretical insights how information transfer between tasks can occur. It allowed formulating new theoretical guarantees that, in particular, showed the importance of regularizing not only the individual learning steps but also the process of information transfer itself. Additionally, we were able to show for the first time the practical possibility of learning by conditional risk minimization, i.e. continual learning under time-varying conditions where the systems searches the best prediction function specifically for at each time step. Previously, systems could only be trained to perform well in a long-term –potentially infinite-time– average.
Besides theoretical contributions, we also obtained practical algorithms for lifelong, multi-task and time-varying learning in a computer vision context. One noteworthy example is SEC, a method that allows the learning of semantic segmentation models for natural images with much less annotation effort, because it transfers useful prior knowledge from a pre-trained image classification model. A second example is iCaRL, a technique for class-incremental learning that makes it possible to train a object classification system first from a small number of classes, but then adding new classes later without having the system forget about the originally learned ones.
The insights and techniques developed in L3ViSU have been picked up and further developed by several research groups as well as practitioners worldwide, and companies have expressed their interest as well. We expect that within the next few years, lifelong learning techniques will become part of many widely used machine learning (artificial intelligence) platforms, e.g. reducing the necessary training effort for cloud services or smart phone applications.