Skip to main content

Acquiring and Responding to the 3D World, Smartly

Final Report Summary - AR3WS (Acquiring and Responding to the 3D World, Smartly)

Recent advances in sensors and imaging systems have greatly simplified scanning and acquisition both in terms of setup cost and acquisition times. As scanners and sensors become ubiquitous, efficient registration of multi-modal sensor inputs (e.g. images, 3D scans, depth scans), suitably representing them, and most importantly the ability to automatically reason and efficiently interact with the acquired data becomes crucial for enabling a smart world. This project advances all the stages of such a digital shape acquisition and analysis pipeline by developing tools necessary for appropriate use the stream of 3D content that is to follow in next years. More specifically, we developed: (i) algorithms to efficiently combine photographs and 3D LiDAR scans for large scale acquisitions ranging from facades to city blocks, (ii) analyzed acquired scenes with man-made objects (e.g. mechanical parts and assemblies) to reveal underlying data decomposition and relations such as coplanarity, orthogonality, symmetry, regularity, etc., and (iii) extracted relations to support interactive navigation of the scenes for detecting damaged scene parts, plan building modifications and predicting possible functions of the individual segmented parts. We now highlight the main findings resulting from the project.

Structured 3D geometry acquisition:
We developed algorithms to exploit multi-modal acquisition (e.g. time of flight scans, photographs, GPS readings, keywords) and dominant global relations present in indoor and outdoor scenes. Note that in such a data-driven model, the relations are learned from the data rather than specified a priori. As a result, with increasing amount of data the system stabilizes and incoming data is used to validate the current model of the world and adapt accordingly. This allows large scale structured reconstructions of building interiors at a scale previously not possible. For details, please visit:

Analysis of acquired data:
Data acquisition in large-scale scenes regularly involves accumulating information across multiple scans. We developed an optimal algorithm for global registration algorithm allowing scans to be in arbitrary initial poses. Specifically, we introduced Super4PCS for global pointcloud registration that is optimal, i.e. runs in linear time (in the number of data points) and is also output sensitive in the complexity of the alignment problem based on the (unknown) overlap across scan pairs. Technically, we map the algorithm as an ‘instance problem’ and solve it efficiently using a smart indexing data organization. The algorithm is simple, memory-efficient, and fast. We demonstrate that Super4PCS results in significant speedup over alternative approaches and allows unstructured efficient acquisition of scenes at scales previously not possible. For more details and code, please visit:

Interaction metaphors for the analysed 3D content
Preservation of model relations among various parts of the object is critical to enable smart (i.e. intuitive) manipulations. Designers often create physical works-like prototypes early in the product development cycle to explore possible mechanical architectures for a design. Yet, creating functional prototypes requires time and expertise, which discourages rapid design iterations. Designers must carefully specify part and joint parameters to ensure that parts move and fit and together in the intended manner. We developed an interactive system that streamlines the process by allowing users to annotate rough 3D models with high-level functional relationships (e.g. part A fits inside part B). Based on these relationships, our system optimizes the model geometry to produce a working design. The software prototype was used design a variety of works-like prototypes as can be seen at: