CORDIS
EU research results

CORDIS

English EN
Methods for Managing Audiovisual Data: Combining  Automatic Efficiency with Human Accuracy

Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy

Objective

Audiovisual media content created and used in films and videos is key for people to communicate and entertain. It has also become an essential resource of modern history, since a large portion of memories and records of the 20th and 21st centuries are audiovisual. To fully benefit from this asset, fast and effective methods are needed to cope with the rapidly growing audiovisual big data that are collected in digital repositories and used internationally. MeMAD will provide novel methods for an efficient re-use and re-purpose of multilingual audiovisual content which revolutionize video management and digital storytelling in broadcasting and media production. We go far beyond the state-of-the-art automatic video description methods by making the machine learn from the human. The resulting description is thus not only a time-aligned semantic extraction of objects but makes use of the audio and recognizes action sequences. While current methods work mainly for English, MeMAD will handle multilingual source material and produce multilingual descriptions and thus enhance the user experience. Our method interactively integrates the latest research achievements in deep neural network techniques in computer vision with knowledge bases, human and machine translation in a continuously improving machine learning framework. This results in detailed, rich descriptions of the moving images, speech, and audio, which enable people working in the Creative Industries to access and use audiovisual information in more effective ways. Moreover,the intermodal translation from images and sounds into words will attract millions of new users to audiovisual media, including the visually and hearing impaired. Anyone using audiovisual content will also benefit from these verbalisations as they are non-invasive surrogates for visual and auditory information, which can be processed without the need of actually watching or listening, matching the new usage of video consumption on mobile devices.

Coordinator

AALTO KORKEAKOULUSAATIO SR

Address

Otakaari 1
02150 Espoo

Finland

Activity type

Higher or Secondary Education Establishments

EU Contribution

€ 752 207,50

Participants (8)

Sort alphabetically

Sort by EU Contribution

Expand all

HELSINGIN YLIOPISTO

Finland

EU Contribution

€ 397 398,75

EURECOM

France

EU Contribution

€ 401 598,75

UNIVERSITY OF SURREY

United Kingdom

EU Contribution

€ 370 680

YLEISRADIO OY

Finland

EU Contribution

€ 354 850

LIMECRAFT NV

Belgium

EU Contribution

€ 633 080

LINGSOFT OY

Finland

EU Contribution

€ 157 717,50

INSTITUT NATIONAL DE L'AUDIOVISUEL

France

EU Contribution

€ 212 528,75

LINGSOFT LANGUAGE SERVICES OY

Finland

EU Contribution

€ 151 532,50

Project information

Grant agreement ID: 780069

Status

Ongoing project

  • Start date

    1 January 2018

  • End date

    31 December 2020

Funded under:

H2020-EU.2.1.1.

  • Overall budget:

    € 3 431 593,75

  • EU contribution

    € 3 431 593,75

Coordinated by:

AALTO KORKEAKOULUSAATIO SR

Finland