Research objectives and content
For virtual studio TV production, MPEG-4 video object coding, and multi-site videoconference applications using a virtual meeting room, synthetic and natural image sequences are mixed to generate a synthetic/natural hybrid image sequence. If the natural and the synthetic objects are illuminated differently and non-diffusely, e.g. by spot lights at different positions, the shading of the natural objects do not match that of the synthetic objects leading to annoying artefacts in the synthetic/natural hybrid image sequence.
In this project, the shading effects in the natural image sequence will be compensated such that natural and synthetic objects appear as beeing illuminated by the same illumination. For first time for shading compensation, the illumination of the natural objects will be described physically by illumination parameters defining two point light sources and ambient diffuse light. Using available prior knowledge on 3D shape and motion of the natural objects, the illumination parameters will be automatically derived from a set of images of the natural image sequence by maximum-likelihood estimation. For first time, the estimator will take into account errors in a) given 3D shape and motion, b) image noise and quantization, and c) model failures like specular reflections. The proposed illumination estimation and shading compensation method will be integrated in a multisite videoconference system developed by the host organization INRIA.
Training content (objective, benefit and expected impact)
By this training, the TMR applicant will achieve scientific and methodical skills. Scientifically, he will benefit from experience of the host organization INRIA in fields related to the TMR project like statistical modelling and robust analysis of image sequences as well as physics-based illumination models for computer graphics. Methodically, at INRIA he will learn from working in an environment of collaborative applied research with industrial and European partners.
Links with industry / industrial relevance (22)
In this project, the natural objects will be described by the VRML/MPEG-4 syntax for synthetic/natural hybrid coding (SNHC). By using this forthcoming world-wide standard, industrial relevance is achieved, because results of this project will be compatible with other VRML/MPEG-4 applications.