Objectif
VANGUARD is a project that aims at flexible methods for 3D scene modelling. The first central issue is the user-friendly acquisition of models of real objects and their surroundings. Traditionally, such modelling is based on stereo vision or active triangulation. Both methods require a level of investment that is not straightforward when large groups of users are to be considered. Stereo vision would require at least two cameras with a specially designed rig to position them. Active triangulation requires special projection apparatus. In both cases, painstaking calibration processes are involved, to obtain the required precision of reconstruction. This calibration is not straightforward and renders the approaches unattractive for the user. VANGUARD will replace these techniques by reconstruction from uncalibrated image sequences. What this means is that a single camera can be carried around without further requirements from the user who wants to model a scene. Reconstructions are formed from the sequence, although no information on camera parameters or the camera motion is available. The theoretical foundations underlying this plan have been laid out by the consortium already.
A second focus is the realistic rendering of the real data from arbitrary viewpoints, i.e. including those not seen by the camera that acquired the data. Work on this topic includes the extraction of realistic surface reflectance models from the sequence (e.g. different colouring of diffuse and specular reflections), the creation of shadows if light sources are virtually moved, the introduction of artificial field of view and unsharpness, etc.
A third topic is the integration of real and synthetic data. Again, issues such as shadowing, interreflections, etc. are raised, but also natural type of interactions between real and synthetic shapes are to be investigated. Artificial gravity will allow users to "drop" synthetic objects into the scene, where they will obtain a stable pose, to "drag" them over flat surfaces, to constrain these motions such as not to violated impermeability constraints, etc.
It is expected that VANGUARD will generate novel, generic technologies for three-dimensional scene reconstruction. An important advantage over traditional techniques will be easier operation by the user, who no longer has to take care of precisely calibrating cameras or measuring camera motion. Also GUI's will be developed that allow users to "play around" with the acquired 3D descriptions, to merge them or let objects blend into environments where they have never been.
Actual achievements include:
automatic generation of VRML 3D models from image sequences for polyhedral shape.
theoretical foundations for uncalibrated reconstruction.
algorithms for software steadycam.
visualisation of generated 3D models on autostereoscopic display.
automatic facial feature extraction from dynamic 3D face reconstructions.
submission to MPEG4-SNHC for 3D geometry-based texture mapping.
software for image mosaicing.
Expected Impact
VANGUARD hopes to produce the kind of 3D reconstruction techniques that would bring them to virtually anybody's fingertips. Anybody who has a camcorder would be in a position to supply the reconstruction algorithms with the necessary input. This would bring 3D acquisition into the realm of the consumer market. From there on, things can start developing very quickly! Also note that 3D reconstructions could be generated from existing video footage. This would be difficult with existing techniques.
Main contributions to the programme objectives:
Main deliverables
The construction and rendering of 3D models based on multiple video images of the same scene.
Contribution to the programme
The advancement of imaging technology for consumer market applications which will bring life to the virtual world.
Technical Approach
A substantial part of the work carried out under VANGUARD has to do with the development of new theoretical insights on how scenes can be reconstructed from uncalibrated video data. The main mathematical vehicle is geometry, although issues of robust implementation, computational efficiency, and numerical stability take a fair share of the time. VANGUARD exploits fully the latest insights developed by the computer vision community in the realm of geometry, such as the role of constraints that govern the position of features in several images or the existence of certain fixed structures in images such as the absolute conic. All these concepts are new compared to the principles that have been exploited previously for calibrated reconstruction.
The project is designed as to gradually evolve towards higher levels of automation in the processing, on the one hand, and more complicated types of shapes and scenes, on the other.
Summary of Trial
VANGUARD plans the following demonstrations:
Stereo visualisation
The objective is to take a monocular video sequence and generate a stereo sequence, forming the images that would have been seen had a stereo camera platform moved along the same path. The result has greater utility than simply using a 'stereo camera', since new viewpoints can be generated and "look around" is possible. The stereo data will be shown on a stereo colour display with such capabilities (and without the requirement to wear special glasses).
Hybrid real/virtual scene visualisation
This demo will show an augmented reality system which integrates real-world models and graphics. It combines 3D interactive graphics, distributed event handling, real-world models for seamless interaction between real and virtual objects for users at remote sites. The planned area for demonstration is a "tele-auction", where potential buyers would be able to have a 3D view of the objects. The users will be able to manipulate the objects themselves.
This facility can naturally be performed using models and scenes at different sites, with the information transmitted in a high level form (e.g. using VRML) across a telecommunications network (e.g. The InterNet).
Tele-exhibition
The manipulation of the 3D world from 2D imagery includes the capability to generate synthetic images of the real world. Key frames from a video sequence are stored and are later manipulated for synthesising the tour - or even another tour whose route can differ from the original route. The applications of this basic paradigm include visualisation, telecommunications and Internet applications.
The visualisation aspect includes the possibility for remote display, e.g. viewing museum artifacts (where the original is valuable/fragile) or distribution on the WWW. The telecommunications aspect includes the capability to move in a real world by sending only few parameters describing the motion of the virtual camera. In this paradigm, both the sender and the receiver have the set of key frames stored on disk. The Internet applications follow immediately from the telecommunication aspect.
Augmented reality.
Key Issues
Of course, a lot of ground work remains to be done at the level of developing efficient and effective reconstruction algorithms from uncalibrated image sequences. VANGUARD strives to obtain metric reconstructions from as few images as possible, for instance, in order to safeguard a maximal practicality.
Other issues are the extraction of good reflectance models from real data in combination with the available knowledge on object shapes. Also getting at realistic interactions between objects that have never really shared the same space is a timely objective.
Champ scientifique
- natural sciencescomputer and information sciencesinternet
- engineering and technologyelectrical engineering, electronic engineering, information engineeringelectronic engineeringsensorsoptical sensors
- natural sciencescomputer and information sciencesartificial intelligencecomputer vision
- natural sciencesmathematicspure mathematicsgeometry
- natural sciencescomputer and information sciencessoftwaresoftware applicationssimulation software
Thème(s)
Appel à propositions
Data not availableRégime de financement
CSC - Cost-sharing contractsCoordinateur
3001 Heverlee
Belgique