Inpainting-based Compression of Visual Data

Project Information

INCOVID

Grant agreement ID: 741215

DOI 10.3030/741215

Closed project

EC signature date 8 September 2017

Start date 1 October 2017

End date 30 September 2022

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost € 2 460 000,00

EU contribution € 2 460 000,00

Coordinated by

UNIVERSITAT DES SAARLANDES
Germany

Periodic Reporting for period 4 - INCOVID (Inpainting-based Compression of Visual Data)

Reporting period: 2022-04-01 to 2022-09-30

Summary of the context and overall objectives of the project

Generating huge amounts of visual data, be it images or videos, has
never been easier than today. This creates a growing demand for lossy
codecs (coders and decoders) that produce visually convincing results
also for very high compression rates. Popular transform-based codecs
such as JPEG and JPEG 2000 have reached a state where one cannot expect
significant improvements anymore. To go beyond their limitations,
fundamentally different ideas are needed.

Inpainting-based codecs can change this situation. They store only a
small, carefully optimised part of the data. In the decoding step, the
missing information is filled in with a suitable inpainting mechanism.
A successful realisation of inpainting-based codecs can offer decisive
advantages over transform-based codecs: The stored information is more
intuitive and closer to the mechanisms of human perception. Moreover,
the concept is very flexible: It allows to integrate a number of
different features and can be tailored towards dedicated applications.
Most importantly, the higher the compression rate, the larger are the
qualitative advantages over transform-based codecs. However, the
potential of these codecs was widely unexplored, since difficult
fundamental problems had to be solved first.

The INCOVID project has addressed these challenges in an integrated
approach that covers a wide spectrum of aspects. We have established
important theoretical foundations, introduced better data selection
strategies, developed dedicated codecs for specific applications,
implemented highly efficient numerical algorithms, and created a
demonstrator that offers real-time performance in 4K resolution. These
achievements show that inpainting methods are far more than a visually
pleasant image editing tool: They have become the key component of a
well-founded and powerful alternative paradigm in coding. It can play
an important role in future codecs that we will use in our daily life.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

- On the theoretical side, we have introduced a general framework
that unifies pseudodifferential inpainting and interpolation with
radial basis functions. We have also established a rigorous existence
theory for edge-enhancing anisotropic diffusion which offers
state-of-the-art performance for general imagery. Moreover, we have
laid the theoretical foundations of inpainting-based codecs in terms
of sparsification and quantisation scale-spaces.

- Many alternative approaches for sparse inpainting have been studied
that are either novel or have not been introduced into inpainting-based
codecs so far. This includes pseudodifferential inpainting, Shepard
interpolation, inpainting with smoothed particle hydrodynamics,
Euler's elastica, diffusion-shock inpainting, and sparse exemplar-based
inpainting. The latter allowed to reconstruct texture in a quality
that exceeds previous approaches by a large margin.

- We have developed various extensions and adaptations of inpainting-based
codecs. This includes dedicated models for piecewise smooth images, motion
fields, and audio signals. Also novel features beyond grey or colour values
have been introduced, e.g. local averages. These variants, which can offer
superior performance in individual applications, illustrate the generality
and flexibility of the inpainting-based compression paradigm.

- Improved data selection strategies have been developed that are more
efficient and allow reconstructions of higher quality. This includes
densification strategies which use error map dithering and do not need
many inpaintings, as well as deep neural network approaches that do not
require any inpaintings at all. Most importantly, we have established
a general framework for the simultaneous incorporation of different
feature types for inpainting-based image representations.

- To encode these optimised data in a compact way, we have performed a
systematic evaluation of coding strategies for sparse inpainting masks.
This allowed us to find and adapt the most powerful ones.

- Various algorithmic accelerations have been implemented that lead
to fast data inpaintings. We have shown how one can speed up
exemplar-based inpainting with space-filling curves, and we have
demonstated the advantages of finite element methods that exploit
adaptive triangulations. An approach that applies the discrete cosine
transform within block decompositions allowed us to perform real-time
video decoding in Full HD resolution already with a purely CPU-based
implementation. Last but not least, exploring the parallelisation
capabilities of domain decomposition methods on a GPU, we were able
to inpaint almost 40 colour images in 4K resolution in one second.
This constitutes the envisioned 4K real-time demonstrator and
concludes the INCOVID project.

These achievements have been presented in invited keynote talks at
leading conferences in the fields of data compression (DCC 2018), applied
mathematics (GAMM 2018), and imaging science (SIAM IS 2022), ensuring
a wide dissemination within these scientific communities.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Basically all the above mentioned achievements have advanced the
state of the art in inpainting-based compression of visual data,
either from a theoretical, algorithmic, or application perspective.
The following four highlights illustrate the progress:

- The theoretical foundation of inpainting-based codecs as sparsification
and quantisation scale-spaces has established close ties between two
hitherto fairly unconnected scientific communities: the coding
community and the scale-space community (Cardenas et al. 2019,
Peter et al. 2021).

- We have pioneered data optimisation for exemplar-based inpainting.
The work of Karos et al. (2018) introduces densification by dithering
of error maps, and it allows an inpainting-based sparse representation
of highly textured images in unprecented quality.

- We have achieved a breakthrough in the simultaneous incorporation
of different feature types for inpainting-based image representations
(Jost et al. 2023). This approach is surprisingly simple, fairly
general, and it incorporates both spatial and tonal optimisation.
Moreover, it has allowed us to establish local averages as a novel
feature type.

- In terms of highly efficient algorithms, the results of Kämper and
Weickert (2022) are most significant. They show how one can inpaint
almost 40 colour images in 4K resolution in one second on a contemporary
GPU. This real-time demonstrator constitutes an acceleration by two
orders of magnitude compared to the state of the art before the onset
of INCOVID.

original test image

data kept

reconstruction by nonlocal inpainting