Skip to main content
European Commission logo print header

Virtual Image-processing System for Intelligent Reconstruction of 3D Environments

Objective

The objective of the project is to develop innovative computer vision technologies to be applied for the reconstruction of 3D environments with great accuracy and realistic appearance. Rather than spend hours of expert modelling jobs trying to build synthetic worlds, the aim of the project is to develop techniques that allow to extract 3D geometry and textures directly from a video stream. Due to automatic mapping of textures obtained from the video stream; the result would be a trustworthy 3D reconstruction of the scenario with the most realistic appearance current technology allows. The pilot application proposed to demonstrate the technology will consist of the simulation of a Virtual Museum in 3D. The reconstruction process of the museum will be based on a video capture session using conventional camcorders, just in the same way a home user would take a domestic video shot. No expensive devices or complex calibration are required.

Objectives:
The goal of current proposal is to push the limits of current computer vision technologies enabling the possibility to obtain realistic 3D reconstruction of scenarios from video streams using conventional camcorders.
Based on captured images, the computer vision algorithms will generate a simulation of the environment in 3D. The geometry and textures of the scenario will be directly extracted from the video stream through stereovision and applied to the 3D synthetic world for improved realism.
The final result will be a real-like 3D environment with interactive capabilities. The user will be able to navigate, observe the scene from different positions, zoom-in in details and interact with the objects in the scene by means of hot spots. The 3D simulation of a famous museum has been selected as the pilot application to demonstrate the possibilities of the technology. The pilot application itself will become a marketable outcome.

Work description:
The project involves research and development activities in three areas: Computer vision, 3D visualisation and interactive user interfaces. All these contribute to build a novel data reduction tool where 3D geometry and texture information present in the real world is transferred to digital format by means of 3D computer scenarios with realistic appearance.
It is important to mention that the 3D models created by the system are not complex man-made 3D models generated by graphics Cad tools. The models obtained by the system are the result of a sophisticated Computer Vision process on real video streams that reflects with reliability the real aspect of the environment to be reconstructed.
The models will not show computer-like appearance or machine-like lighting. They will not have the appearance of brand new objects usually found in traditional computer graphic simulations but will demonstrate the real texture obtained from the video images. This means the 3D-reconstruction process will show most trustworthy state of the objects being reconstructed including possible time degradation or broken parts.
Computer vision activities play a crucial role in the project. They will be based on current expertise on related subjects, but an important research effort is required in areas such as: stereo vision, 3D view registration, feature extraction, tracking, auto-calibration, fusion of range and intensity data and structure from motion. Maybe one of the most innovative aspects of the project lies in the research effort required to extract, analyse and compute 3D geometry from moving video data. Being a demanding task, motion analysis is a basic component for a successful implementation of project objectives.
But the success of the project will relay not only on the computer vision algorithms but also on the 3D reconstruction procedure itself. The basic idea is to avoid the time consuming effort required to model by hand each element of a 3D scenario. The objective is rather to devise a semiautomatic method that will directly translate the 3D information contained in a 2D video stream acquired using conventional camcorders into useful digital data.
The VISIRE project will comprise the following activities: Image acquisition, feature extraction, feature tracking, calibration, image registration, geometry correction, Euclidean 3D reconstruction, texture mapping, image based rendering, user interface and production of the contents of the museum pilot application.
The foreseen sequence of activities to be performed for a 3D reconstruction session is as follows:
The user takes one or several free hand video shots around the scenario he desires to reconstruct.
The acquired images are taken to the laboratory where the 3D reconstruction tools builds the real-like synthetic 3D environment. Images from the video stream are analysed, the 3D geometry information of the objects appearing in the scenario is recovered and textures are mapped after appropriate geometry correction.
As a result of processing the video data, the 3D modelisation of the site is obtained.
The target application for the project will consist of the simulation of a famous museum, but the techniques to be developed within the project can also be applied with little extra effort to a great variety of applications like 3D simulations of buildings, archaeological sites, public works (bridges, motorways), topographic 3D reconstruction of terrain, tele-shopping services.
The final result will consist of a synthetic real-like 3D world where any computer manipulation will be possible. The 3D model of the museum will be digital and will support all the possibilities of digitally built environments but it will also support real-like textures taken from the original images.
The user will be able to navigate through the different parts of the museum, watch the exhibition cabinets, focus his attention on special details, choose his poin.

Milestones:
The project is expected to provide the following results:
The 3D reconstruction tools. A complete software environment that allows Multimedia professionals to build realistic 3D models of the interior of buildings. The tools use as input a set of video sequences and produces the finished 3D models in VRML format.
The simulation of a famous museum in Florence. As part of the work foreseen in the project the 3D reconstruction tools will be tested by building the 3D simulation of a famous museum in Florence. The resulting multimedia production will be used for promotion and dissemination. It could be considered also the possibility to commercialize the finished model. Current alternatives for the museum include Museum of Science in Florence, Museum of Technology in Milan, Uffizi Gallery and Palazzo Vechio.
New computer vision algorithms that will hopefully advance the state of the art in the field.

Call for proposal

Data not available

Coordinator

EPTRON, S.A.
EU contribution
No data
Address
CALLE JUAN VIGON 3
28003 MADRID
Spain

See on map

Total cost
No data

Participants (4)