Exploration of Unknown Environments for Digital Twins

Objective

"In the 'explorer' project, we will develop methods for automatically capturing and labelling video data in ""open worlds"". The ultimate goal is the great facilitation of the creation and maintenance of Digital Twins: Digital Twins are virtual 3D copies of complex scenes such as cities, factories, or construction sites. Not just a 3D reconstruction, they should capture the scene's semantics, i.e. the identity of each object and the scene's dynamics, i.e. how objects move. Because Digital Twins have the potential to be extremely useful for monitoring large complex sites and planning the development of these sites, their forecast market is huge, they remain mostly a concept because of important limitations of the current technology. Our methods will guide autonomous systems such as robotic platforms and UAVs through complex and unknown environments to capture visual data for creating and maintaining Digital Twins. This is extremely challenging as these systems will encounter objects without any prior knowledge about them and will have to collect sufficient data about them. To the best of our knowledge, this active and automatic capture in complex real environments is a new problem. It is however very important to solve it as this will relax the need for human expertise and time: Currently, capturing such data is done manually only by researchers and requires strong understanding of what the learning algorithms require. To tackle the complexity of this problem, our approach is inspired by techniques from Artificial Intelligence applied to the exploration of extremely large trees. This approach will allow us to bring the perception part and the planning part of the problem together under the same optimization framework, to formalize it and solve it efficiently. To evaluate our developments, we will create a dataset of annotated video sequences from working sites, which we will share with the community."