These five years of research allowed us to better understand how designers draw, and to propose the first methods capable of automatically reconstructing 3D shapes from design drawings. Overall, we are happy to say that we achieved our initial goals to a large extent, as we proved that despite their inherent complexity and ambiguity, design drawings can be reconstructed because they follow specific drawing principles.
To better understand design drawing, we have collected a dataset of more than 400 professional sketches [Gryaditskaya et al. 2019]. We manually labeled the techniques used in each drawing, and we registered all drawings to reference 3D models. Analyzing this data revealed systematic strategies employed by designers to convey 3D shapes, which then inspired the development of novel algorithms for drawing interpretation. In addition, our annotated drawings and associated 3D models form a challenging benchmark to test these algorithms.
We proposed several methods to recover 3D information from drawings. A first family of method employs deep learning to recognize what 3D shape is represented in a drawing. We applied this strategy in the context of architectural design, where we reconstruct 3D building by recognizing their constituent components (building mass, façade, window) [Nishida et al. 2018]. We also presented an interactive system that allows users to create 3D objects by drawing from multiple viewpoints [Delanoy et al. 2018, 2019]. Finally, we leveraged recent developments in natural language processing to propose a neural network architecture capable of parsing line drawings into sequences of Computer-Aided-Design commands [Li et al. 2020, 2022].
The second family of methods leverages geometric properties of the lines to optimize 3D reconstruction. In particular, we exploit properties of developable surfaces to reconstruct sketches of fashion items [Fondevilla et al. 2017, 2021], and properties of construction lines to reconstruct human-made objects [Gryaditskaya et al. 2020]. More recently we leveraged symmetry to further improve the quality of these reconstructions [Hähnlein et al. 2022]. While our original focus was on 2D drawings, we extended our methodology to also consider 3D drawings in Virtual Reality [Yu et al. 2021], and to reconstruct a 3D surface from such drawings [Yu et al. 2022].
A long-term goal of our research is to evaluate the physical validity of a concept directly from a drawing. We obtained promising results towards this goal for the particular case of mechanical objects. We proposed an interactive system where users design the shape and motion of an articulated object, and our method automatically synthesize a mechanism that animates the object while avoiding collisions [Nishida et al. 2019]. The geometry synthesized by our method is ready to be fabricated for rapid prototyping. We also studied innovative fabrication techniques, in particular printing-on-fabric that allows to rapidly prototype freeform surfaces [Jourdan et al. 2020, 2022].
