New tools to improve video post-production
Inpainting, the process of reconstructing or altering videos, is a valuable tool in video post-production, particularly in our ever-evolving digital world. The EU-funded INPAINTING (Inpainting Tools for Video Post-production: Variational theory and fast algorithms) looked at improving the removal of an object from a scene in a video. It sought to replace this manual and time-consuming process with a more automated technology. This is an effort which requires enhanced depth estimation, motion estimation, image inpainting and image propagation in video. To achieve this aim the project team investigated new mathematical models and frameworks in order to develop the required computer algorithms. Looking at computing similarities between image regions with focus on their size and shape, the team elaborated a theoretical framework that can analyse similarities at different scales. It then outlined a new ‘affine covariant metric’ that specifies the shape of the area in question, considering as well the image structure. INPAINTING then applied this to fill a region in the image with missing information (e.g. in the case of object removal). Its advanced algorithms were used to copy pieces of the known part of the image and assemble them coherently to cover the unknown region. Another project challenge involved modifying the texture of an object in a video such as adding a logo or hiding a marker. To overcome this, it edited just one frame and then automatically propagated the effect throughout the video seamlessly. The team also devised a continuous model and numerical methods that enable image propagation without motion restrictions, facilitating as well the handling of occlusions and illumination issues. Striving to enhance optical flow when inpainting is applied, the team developed algorithms that decrease computation time and improve rotational movements, encouraging smoothness, accuracy and a more realistic effect. On another front the project team worked on improving depth image in order to simplify post-production of video material and facilitate the production of 3D content. It developed a way to increase the resolution of the depth map using the colour image as a guide, as well as enhancing relative depth estimation from a single image. If these enhancements are commercialised, digital video editing could benefit significantly, from removing unnecessary information to enhancing the quality of old film material.
Keywords
Video, post-production, inpainting, image propagation, affine covariant
