We have tackled the above challenge from multiple angles:
1. We developed novel algorithms for reconstructing the 3D world from multiple images. In the paper [Demmel et al. CVPR 2021], we developed a novel numerical solution for the classical problem of Bundle Adjustment which aims to reconstruct the 3D world and the camera locations from a multitude of images. Compared to the state of the art in this field, the proposed algorithm is numerically more stable and significantly faster (often up to 40% faster) than the fastest competing method.
2. We developed a neural network approach called MonoRec [Wimbauer et al. CVPR 2021] that allows us to generate a dense reconstruction of a large-scale world from a single drive-through with a single camera. The resulting model is an almost photorealistic copy of the world, something often called a digital twin. It can serve as the basis for augmented reality applications or for testing self-driving cars in a simulated world that is a fairly exact copy of the real world.
3. We developed a neural network approach called Neurmorph [Eisenberger et al. CVPR 2021] that allows us to compute the exact pointwise correspondence between two given 3D shapes and a family of interpolating 3D shapes. We demonstrate that it can be used for digital pupeteering - i.e. transfering the dynamics of an observed 3D shape onto another 3D shape.
4. We developed a method called i3DMM [Yenamandra et al. CVPR 2021]. It makes use of deep networks to synthesize 3D head models including aspects like hair styles and others. This extends the classical deformable shape approaches to the age of deep learning.
5. In [Eisenberger et al. CVPR 2022], we derived a unified mathematical framework for computing correspondence in a deep network architecture. This approach will likely be of value to any deep network that aims to compute correspondence - correspondence between points on two 3D shapes, correspondence between pixels in an image or other.
6. In [Hofherr et al. WACV 2023], we will present a method that allows us to compute a physical simulation directly from video. In contrast to our earlier work in [Weiss et al. CVPR 2020], the current approach makes use of a neural network and learning.
