Skip to main content

Inpainting-based Compression of Visual Data

Periodic Reporting for period 2 - INCOVID (Inpainting-based Compression of Visual Data)

Reporting period: 2019-04-01 to 2020-09-30

Generating huge amounts of visual data, be it images or videos, has
never been easier than today. This creates a growing demand for lossy
codecs (coders and decoders) that produce visually convincing results
also for very high compression rates. Popular transform-based codecs
such as JPEG and JPEG 2000 have reached a state where one cannot expect
significant improvements anymore. To go beyond their limitations,
fundamentally different ideas are needed.

Inpainting-based codecs can change this situation. They store only a
small, carefully optimised part of the data. In the decoding step, the
missing information is filled in with a suitable inpainting mechanism.
A successful realisation of inpainting-based codecs can offer decisive
advantages over transform-based codecs: The stored information is more
intuitive and closer to the mechanisms of human perception. Moreover,
the concept is very flexible: It allows to integrate a number of
different features and can be tailored towards dedicated applications.
Most importantly, the higher the compression rate, the larger are the
qualitative advantages over transform-based codecs.

However, the potential of these codecs is widely unexplored so far,
since difficult fundamental problems must be solved first. This includes
optimisation of the data and the inpainting process, sophisticated data
coding, and the design of real-time capable sequential and parallel
numerical algorithms. We are committed to addressing all these
challenges in an integrated approach: We cover the entire spectrum
from its theoretical foundations over highly efficient numerical
algorithms to codecs for specific applications, and a real-time 4K
video player as demonstrator.

This will lift inpainting methods from a visually pleasant image
editing tool to a fundamental paradigm in coding. If these research
results enter forthcoming coding standards, they will also have an
impact on everybody’s daily life.
- On the theoretical side, we have introduced a general framework
that unifies and extends two important classes of inpainting methods.
These approaches can be handled with a single novel algorithm now,
which allows comprehensive and fair evaluations.

- Images with rich texture pose specific problems for inpainting-based
codecs. We have shown that these problems can be addressed with nonlocal,
patch-based inpainting methods in combination with novel data selection
strategies. This has led to sparse reconstruction results in a quality
that was not possible before.

- We have developed various extensions and adaptations of inpainting-based
codecs. This includes dedicated models for piecewise smooth images, motion
fields, and audio signals. Also various features beyond greyvalues have
been investigated. These variants, which can offer superior performance,
illustrate the generality and flexibility of the inpainting-based
compression paradigm.

- Different algorithmic accelerations have been established that result
in more efficient data inpaintings. Together with novel data selection
strategies that require a much smaller number of inpaintings, substantial
speed-up are possible.

The project is on track.
Progress beyond the state of the art:

- unification of existing inpainting methods

- novel codecs allowing better quality for richly textured images

- well-performing dedicated codecs for specific applications

- various algorithmic accelerations

Expected results:

- progress towards a more complete optimisation model that also
incorporates estimates of the coding costs.

- further quality improvements through dedicated codecs,
e.g. codecs exploiting triangulations, and methods that automatically
select and combine different features.

- faster algorithms / surrogate models for important inpainting methods,
perhaps also involving deep learning

- demonstrator: real-time 4K video player
original test image
data kept
reconstruction by nonlocal inpainting