Final Report Summary - RECOVER (Photorealistic 3D Reconstruction of Perspective Paintings and Pictures)
The 'Photorealistic 3D Reconstruction of Perspective Paintings and Pictures' (RECOVER) project aimed at the development of a system for the semi-automatic extraction of three-dimensional (3D) models of scenes depicted in perspective paintings. 3D models of paintings constitute a new and exciting way for the general public to experience and appreciate fine art. The viewer can experience a feeling of immersion; paintings are no longer perceived as static artefacts from a long-gone past but as living, vibrant entities. With the aid of appropriate software, the viewer can literally dive into the painting, interacting with it and observing it from various viewpoints in impressive walk-throughs and inspiring fly-bys. This enables non-specialists to step into history and experience the scene in the space and time frame perceived by the artist. Ultimately, the viewing of paintings becomes a more appealing, exploratory endeavour, arousing the public's interest in fine art and cultural heritage in general. In addition to them being used in interactive multimedia applications, textured 3D models reconstructed with the RECOVER system can be employed in applications such as virtual reality, video games, 3D photography, digital visualisation, visual metrology, art history study, etc.
Widespread deployment of information and telecommunications technologies has triggered an ever increasing demand for digital content to support a variety of applications. In particular, advances in three-dimensional (3D) model rendering and visualisation have emphasised the need for 3D digital content to be employed in computer graphics, mixed reality and communication. This in turn, has created a tremendous potential for techniques capable of producing digital 3D models corresponding to scenes and objects, transforming the relevant research into a hot topic for several years. Equally important to the production of digital content is the capture of the content's semantics with appropriate metadata, aiming to improve content reuse, personalisation, searchability, interchange and management.
Being non-intrusive and cheap in terms of the required equipment, approaches that are based on the processing of images provide a particularly attractive paradigm for achieving 3D reconstruction and at the same time adding geometric semantics to images. According to the current state-of-practice, fully manual reconstruction techniques based on the use of CAD and 3D modelling tools for reconstructing paintings are quite tedious and labour-intensive, therefore time-consuming and costly. Laser scanning techniques cannot be applied due to the fact that the canvas used for painting is 2D. Conventional photogrammetric approaches and multi-view geometry vision techniques are also inapplicable due to their need for several images acquired from different viewpoints. Recover's approach, on the other hand, capitalises on recent research results in order to bridge the gap between the research state-of-the-art and the state-of-practice in the construction of 3D models from 2D paintings.
Textured 3D models constitute a new and exciting way for perceiving and appreciating paintings. Their viewer can experience a feeling of immersion; paintings are no longer perceived as static artifacts from a long-gone past but as living, vibrant entities. With the aid of appropriate software, the viewer can literally dive into the painting, interacting with it and observing it from various viewpoints in impressive walk-throughs and inspiring fly-bys. This enables non-specialists to step into history and experience the scene in the space and time frame perceived by the artist.
Until the beginning of the 15th century, artists lacked the knowledge of creating an illusion of the third dimension in their works, which essentially look flat and fail to represent volume. Objects and characters were typically drawn depending on their importance rather than their distance from the observer. Such drawing practices were abolished during the Renaissance. The Italian painters of the time were the first to be interested in naturalism and studied the geometry of image formation in order to rationalise the representation of space by reproducing the perspective effects in the images of the world that they were creating. Giotto di Bondone was the first painter to treat a painting as a window into space, being concerned with the third dimension, the proportions and the natural appearance of surfaces. However, it was not until the writings of Florentine architects Filippo Brunelleschi and Leon Battista Alberti that linear perspective was formalised as an artistic technique aimed at creating a systematic illusion of space behind the canvas. The comprehension of the relations of perspective to perceptual aspects of depth and space, allowed painters to take advantage of the impressive ability of the human visual system to infer 3D properties of shape from a single 2D image.
Hence, the use of perspective revolutionised the art of painting and raised it to a prestigious level among the fine arts. Renaissance masters such as Masaccio, Piero della Francesca, Leonardo da Vinci and Albrecht Dürer pushed theory to a considerably sophisticated stage, paving the way for its complete mathematical formulation. This mathematical system that allows the creation of the illusion of depth and volume on a flat surface has become to be known as linear perspective and is briefly presented next.
Recover has successfully investigated the topic of semi-automatic single view reconstruction from one perspective image. With the combined efforts of all partners, a prototype system capable of interactively extracting textured VRML 3D models from images has been developed. To the best of our knowledge, this system possesses a set of characteristics that render it unique among the set of existing commercial software products that assist the recovery of 3D models/measurements from images. For instance, most of the existing such products (e.g. Eos PhotoModeler, Realviz ImageModeler, Vexcel FotoG) require multiple images of the same scene, whereas a single image suffices for RECOVER. Other such products primarily focus on aiding the user perform measurements directly on images (e.g. iPhotoMEASURE) rather than producing 3D models. The class of software products that focus on strictly one image is quite limited and consists of products that are currently either discontinued (e.g. Metacreations Canoma, GeoTango SilverEye) or not yet fully available (e.g. Freewebs Fotowoosh). The RECOVER system also strives to be flexible, offering assumptions that restrict it to a particular class of images (e.g. aerial/satellite images such as those targeted by SilverEye). Another advantage offered by RECOVER is that it lets the user get involved in the reconstruction process, thus exploiting the remarkable capability of the human brain to interpret 3D structure from a single image.
The primary envisaged use of the RECOVER system is to employ it for creating 3D models that will serve as digital content for developing interactive multimedia applications related to cultural heritage. Evidently, such usage has the potential for important societal implications related to improved accessibility and visibility of European cultural resources. Furthermore, RECOVER technology can have a broad spectrum of possible practical applications ranging from the study of art history and assistive technologies for people with special needs to video games, 3D photography, visual metrology, digital visualisation, architectural photogrammetry, urban visualisation and planning, forensics, guidance and e-learning. From a more abstract perspective, a 3D model reconstructed from digital content in the form of an image, can be thought of as a means of annotating the latter with metadata. Such metadata can improve content reuse, personalisation, searchability, interchange and management.
RECOVER encompasses a versatile set of techniques for achieving single view reconstruction from a single image. These techniques support a set of functionalities that place RECOVER at a unique position among similar systems. More specifically, RECOVER offers techniques for single view intrinsic camera calibration from a variety of cues, geometrically constrained 3D reconstruction with minimal user interaction, texture mapping as well as manipulation and 3D model completion and editing. The developed techniques are coupled with an interaction model targeted to single view reconstruction, which describes the effects of user actions to the system model and vice versa. The core reconstruction techniques have been implemented in ANSI C and have been packaged into a MS Windows DLL accessible through an API. The user interface has been developed as a plug-in for the Blender open-source 3D modeller, using Python scripting.
Widespread deployment of information and telecommunications technologies has triggered an ever increasing demand for digital content to support a variety of applications. In particular, advances in three-dimensional (3D) model rendering and visualisation have emphasised the need for 3D digital content to be employed in computer graphics, mixed reality and communication. This in turn, has created a tremendous potential for techniques capable of producing digital 3D models corresponding to scenes and objects, transforming the relevant research into a hot topic for several years. Equally important to the production of digital content is the capture of the content's semantics with appropriate metadata, aiming to improve content reuse, personalisation, searchability, interchange and management.
Being non-intrusive and cheap in terms of the required equipment, approaches that are based on the processing of images provide a particularly attractive paradigm for achieving 3D reconstruction and at the same time adding geometric semantics to images. According to the current state-of-practice, fully manual reconstruction techniques based on the use of CAD and 3D modelling tools for reconstructing paintings are quite tedious and labour-intensive, therefore time-consuming and costly. Laser scanning techniques cannot be applied due to the fact that the canvas used for painting is 2D. Conventional photogrammetric approaches and multi-view geometry vision techniques are also inapplicable due to their need for several images acquired from different viewpoints. Recover's approach, on the other hand, capitalises on recent research results in order to bridge the gap between the research state-of-the-art and the state-of-practice in the construction of 3D models from 2D paintings.
Textured 3D models constitute a new and exciting way for perceiving and appreciating paintings. Their viewer can experience a feeling of immersion; paintings are no longer perceived as static artifacts from a long-gone past but as living, vibrant entities. With the aid of appropriate software, the viewer can literally dive into the painting, interacting with it and observing it from various viewpoints in impressive walk-throughs and inspiring fly-bys. This enables non-specialists to step into history and experience the scene in the space and time frame perceived by the artist.
Until the beginning of the 15th century, artists lacked the knowledge of creating an illusion of the third dimension in their works, which essentially look flat and fail to represent volume. Objects and characters were typically drawn depending on their importance rather than their distance from the observer. Such drawing practices were abolished during the Renaissance. The Italian painters of the time were the first to be interested in naturalism and studied the geometry of image formation in order to rationalise the representation of space by reproducing the perspective effects in the images of the world that they were creating. Giotto di Bondone was the first painter to treat a painting as a window into space, being concerned with the third dimension, the proportions and the natural appearance of surfaces. However, it was not until the writings of Florentine architects Filippo Brunelleschi and Leon Battista Alberti that linear perspective was formalised as an artistic technique aimed at creating a systematic illusion of space behind the canvas. The comprehension of the relations of perspective to perceptual aspects of depth and space, allowed painters to take advantage of the impressive ability of the human visual system to infer 3D properties of shape from a single 2D image.
Hence, the use of perspective revolutionised the art of painting and raised it to a prestigious level among the fine arts. Renaissance masters such as Masaccio, Piero della Francesca, Leonardo da Vinci and Albrecht Dürer pushed theory to a considerably sophisticated stage, paving the way for its complete mathematical formulation. This mathematical system that allows the creation of the illusion of depth and volume on a flat surface has become to be known as linear perspective and is briefly presented next.
Recover has successfully investigated the topic of semi-automatic single view reconstruction from one perspective image. With the combined efforts of all partners, a prototype system capable of interactively extracting textured VRML 3D models from images has been developed. To the best of our knowledge, this system possesses a set of characteristics that render it unique among the set of existing commercial software products that assist the recovery of 3D models/measurements from images. For instance, most of the existing such products (e.g. Eos PhotoModeler, Realviz ImageModeler, Vexcel FotoG) require multiple images of the same scene, whereas a single image suffices for RECOVER. Other such products primarily focus on aiding the user perform measurements directly on images (e.g. iPhotoMEASURE) rather than producing 3D models. The class of software products that focus on strictly one image is quite limited and consists of products that are currently either discontinued (e.g. Metacreations Canoma, GeoTango SilverEye) or not yet fully available (e.g. Freewebs Fotowoosh). The RECOVER system also strives to be flexible, offering assumptions that restrict it to a particular class of images (e.g. aerial/satellite images such as those targeted by SilverEye). Another advantage offered by RECOVER is that it lets the user get involved in the reconstruction process, thus exploiting the remarkable capability of the human brain to interpret 3D structure from a single image.
The primary envisaged use of the RECOVER system is to employ it for creating 3D models that will serve as digital content for developing interactive multimedia applications related to cultural heritage. Evidently, such usage has the potential for important societal implications related to improved accessibility and visibility of European cultural resources. Furthermore, RECOVER technology can have a broad spectrum of possible practical applications ranging from the study of art history and assistive technologies for people with special needs to video games, 3D photography, visual metrology, digital visualisation, architectural photogrammetry, urban visualisation and planning, forensics, guidance and e-learning. From a more abstract perspective, a 3D model reconstructed from digital content in the form of an image, can be thought of as a means of annotating the latter with metadata. Such metadata can improve content reuse, personalisation, searchability, interchange and management.
RECOVER encompasses a versatile set of techniques for achieving single view reconstruction from a single image. These techniques support a set of functionalities that place RECOVER at a unique position among similar systems. More specifically, RECOVER offers techniques for single view intrinsic camera calibration from a variety of cues, geometrically constrained 3D reconstruction with minimal user interaction, texture mapping as well as manipulation and 3D model completion and editing. The developed techniques are coupled with an interaction model targeted to single view reconstruction, which describes the effects of user actions to the system model and vice versa. The core reconstruction techniques have been implemented in ANSI C and have been packaged into a MS Windows DLL accessible through an API. The user interface has been developed as a plug-in for the Blender open-source 3D modeller, using Python scripting.
