Going Deep and Blind with Internal Statistics

Project Information

DeepInternal

Grant agreement ID: 788535

DOI

10.3030/788535

Project closed

EC signature date 24 April 2018

Start date 1 May 2018

End date 31 October 2023

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 2 466 940,00

EU contribution

€ 2 466 940,00

Coordinated by

WEIZMANN INSTITUTE OF SCIENCE
Israel

Periodic Reporting for period 4 - DeepInternal (Going Deep and Blind with Internal Statistics)

Reporting period: 2022-11-01 to 2023-10-31

Summary of the context and overall objectives of the project

In the past decade, since the revival of Deep Neural-Networks (DNNs), there has been unprecedented progress and breakthrough results in Computer Vision, both in high-level and low-level vision tasks. Nevertheless, most of this impressive performance stems from the ability to train DNNs on huge amounts of training data (often tediously hand-labelled). This restricts the applicability of current Deep-Learning methods to specific problems and domains where enough training data exist. This limitation renders them inapplicable to domain areas where very little or no training data are available, or where data labeling (manual or automatic) is ill-defined.

In the past few years, during this project, my students and I have shown how very few training data can be used to train DNNs; often no prior training examples whatsoever! We have shown that DNNs can be trained on examples extracted directly from the single available test image. We have shown how to combine the power of unsupervised Internal Data Recurrence with the sophistication and inference-power of Deep Learning, to obtain the best of both worlds. This self-supervised learning approach gives rise to true “Zero-Shot” Learning, and has already made an impact on the scientific community (both Computer Vision and Deep learning), as well as in other domains (e.g. Reconstruction of data from brain activity), and is likely to have far reaching applications for the society. Some of these are detailed next.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During this project, we have developed new approaches and theories for Self-Supervised Deep Learning, by exploiting the internal redundancy inside a single natural image/video. We coined it “Deep Internal Learning”. The strong recurrence of information inside a single natural image/video provides powerful internal examples which suffice for training Deep Networks, without any prior examples or training data. This new “Deep Internal Learning” paradigm gives rise to true “Zero-Shot Learning”. We have demonstrated the power of this approach to a range of problems, including super-resolution (in images – CVPR’2018; in videos – ECCV’2020), image-segmentation, transparent layer separation, blind image-dehazing (CVPR’2019), image-retargeting (ICCV’2019), blind super-resolution (NeurIPS’2019), diverse image & video generations (CVPR’2022 & ECCV’2022), diverse video interpolation and extrapolation (ICML’2023), and more. We have also shown how such self-supervision can be used for reconstructing images from brain recordings (fMRI) with very few external training data (NeurIPS’2019), for image classification from fMRI brain activity (NeuroImage’2022), and for video reconstruction from fMRI (arXiv’2022).

More recently we have shown that the notion of “Internal Learning” can further be applied to recover the training data of a trained classifier, directly from the parameters of the network (NeurIPS’2022, ICLRworkshop’2023, NeurIPS’2023). Our findings have serious negative implications on Data Privacy in Deep Learning.

During the project the team published 18 research papers summarizing different aspects of the project goals, 15 of which were peered reviewed and published in the leading international journals and/or conferences including: Nature Communications, NeuroImage, NeurIPS, CVPR, ICCV, ECCV, ICML. These papers present excellent progress compared to the original plan, and beyond. We have made the code and data of the published papers available (where applicable). We further have a few new papers currently in preparation or under review (not published yet), which contain additional new exciting breakthroughs.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

• We developed the theory of “Deep Internal Learning” to image and video data, thus enabling totally unsupervised Deep-Learning when no training data whatsoever are available.

• We showed the applicability of “Deep Internal Learning” to a wide range of inference tasks, and to a wide variety of network architectures. We further showed that these types of networks often lead to SOTA (state-of-the-art) results on out-of-distribution data for a variety of tasks.

• We showed that deep learning can further be done with modest amounts of supervised training data by exploiting self-supervision.

• We developed state-of-the-art (SOTA) Image & Video Reconstruction from fMRI brain activity. This was done using a very small amount of supervised training data.

• We have developed the first-ever large-scale Image-Classification (to more than 100 classes!) from fMRI brain activity. This was applied to classification of never-before-seen-classes during training time.

• We have developed the first-ever method for reconstructing training data directly from the parameters of trained neural-net, without any prior assumptions or any additional side information. Our findings have serious negative implications on Data Privacy in Deep Learning. As such, this line of work, although new, is already drawing a lot of attention in the Deep Learning community.

A GAN is trained on a single input image, generates many new images of the same patch distribution.