Project description
New models for effective video representations
Computer vision has made significant strides in applying deep learning (DL) to images, but progress in video analysis has been slower due to the complexity and diversity of video data, necessitating larger training datasets than images. Raw video data is typically high-dimensional, making processing entire video pixel volumes at scale costly. While video-specific models possess fundamental properties, they are constrained by the low-level information in videos, impacting their capabilities, applicability, and robustness. The ERC-funded OmniVideo project seeks to develop Neural OmniVideo Models by integrating two approaches to capture the dynamics of videos using DL-based frameworks and external models. This integration aims to introduce methodologies for video analysis and synthesis, resulting in fundamentally new and effective video representations.
Objective
The field of computer vision has made unprecedented progress in applying Deep Learning (DL) to images. Nevertheless, expanding this progress to videos is dramatically lagging behind, due to two key challenges: (i) video data is highly complex and diverse, requiring order of magnitude more training data than images, and (ii) raw video data is extremely high dimensional. These challenges make the processing of entire video pixel-volumes at scale prohibitively expensive and ineffective. Thus, applying DL at scale to video is restricted to short clips or aggressively sub-sampled videos.
On the other side of the spectrum, video-specific models—a single or a few neural networks trained on a single video—exhibit several key properties: (i) facilitate effective video representations (e.g. layers) that make video analysis and editing significantly more tractable, (ii) enable long-range temporal analysis by encoding the video through the network, and (iii) are not restricted to the distribution of training data. Nevertheless, the capabilities, applicability and robustness of such models are hampered by having access to only low-level information in the video
We propose to combine the power of these two approaches by the new concept of Neural OmniVideo Models: DL-based frameworks that effectively represent the dynamics of a given video, coupled with the vast knowledge learned by an ensemble of external models. We are aimed at pioneering novel methodologies for developing such models for video analysis and synthesis tasks. Our approach will have several important outcomes:
• Give rise to fundamentally novel effective video representations.
• Go beyond state-of-the-art in classical video analysis tasks that involve long-range temporal analysis.
• Enhance the perception of our dynamic world through new synthesis capabilities.
• Gain profound understanding of the internal representation learned by state-of-the-art large-scale models, and unveil new priors about our dynamic.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
- engineering and technology electrical engineering, electronic engineering, information engineering information engineering telecommunications telecommunications networks data networks
You need to log in or register to use this function
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.1.1 - European Research Council (ERC)
MAIN PROGRAMME
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-ERC - HORIZON ERC Grants
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) ERC-2023-STG
See all projects funded under this callHost institution
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
7610001 Rehovot
Israel
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.