Skip to main content

Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy

Deliverables

Evaluation report, intermediate version

The intermediate version of the report presents findings of the evaluation of the second iteration of the prototype system, against which the established test scenarios are executed again, along with an evaluation of a first set of quantitative and qualitative criteria. This version includes a final set of recommended improvements to be included in the final MeMaD prototype system.

Data interchange format specification, final version.

This final version of the interchange format specification incorporates feedback from the first and second prototype evaluation cycles and defines the final set of criteria and interchange specifications that that final prototype needs to conform to and will be tested against.

Evaluation report, initial version

The initial version of the prototype system evaluation report, based on feedback provided by a select group of end users. This report will present the first user feedback after execution of test scenarios and will recommend an initial set of improvements to the system specifications and evaluation criteria.

Summary of dissemination and communication activities

Summary of dissemination and communication activities taken by partners, highlighting successful generation of new research and commercial projects and collaborations.

Specification of the data interchange format, initial version

The initial version of the data interchange format will define functional and non-functional requirements of the MeMaD prototype system, based on input concerning the tools developed in WP2, WP3, WP4 and WP5. The requirements are laid out in reference to user requirements and are documented with test scenarios and evaluation criteria.

TV programme annotation model

Report on an initial annotation model for TV programming as well as on the method enabling to go from script and automatic transcription to true subtitles respecting the time and spaces captioning constraints.

Report on discourse-aware machine translation for audio-visual data

A report on neural machine translation models with contextual features beyond sentence boundaries.

Report on cross-lingual content retrieval based on automatic translation

A report on the use of machine translation in cross-lingual retrieval of audio-visual content.

Report on multimodal machine translation

A report on models with multimodal input and initial evaluations of their quality.

Best Practice Guide for Video Description

This practical guide will outline principles and models of video description drawing on insights about multimodal translation of audiovisual content and the empirical analysis conducted in this WP.

Specification of the data interchange format, intermediate version

This iteration of the data interchange format updates the specification and future evaluation criteria with feedback and improvements from the first prototype system development and evaluation report.

Evaluation report, final version

This revision of the report will contain a final evaluation of the MeMaD prototype system, performed using a complete set of quantitative and qualitative test criteria for each of the project’s use cases.

Report on comparative analysis of human and machine video description

his deliverable will report the main findings from the comparative analysis of human descriptions of audiovisual content with corresponding machine-based descriptions generated in WP2.

Data management plan

Report on the initial data management life cycles for the data to be collected, processed and generated during the project.

Data management plan, update 1

Updated version of DMP that covers significant changes in project datasets and data policies that arise during the project.

Data management plan, update 2

Final version of DMP that covers significant changes in project datasets and data policies that arise during the project.

TV moments detection and linking, final version

Updated implementations of the back-end microservices of MeMAD. These will cover the full span of functionalities foreseen in WP3: annotations with a broader set of entity types, diverse enrichments triggered by a more accurate moment detection. The deliverable will include a report describing the microservices and their evaluation results.

Implementations of methods adapted to enhanced human inputs

Further improvements and additions to the tools and libraries contained in D2.1.  These will include documentations and actual methods for speaker segmentation and diarization as well as for visual content analysis of video footage. Contains a report.

Collection of annotations and / or video data resulting from the project

A collection of datasets and media corpora that will act as the project legacy datasets. These will be stored to relevant data repositories following the project DMP. Contains a report that describes the collection.

Multimodally annotated dataset of described video

This deliverable will provide a) transcriptions of a set of audiovisual materials that have audio description and subtitles in at least one project language and b) annotations of relevant visual, auditory and verbal elements, aligned with the corresponding information in the audio description and subtitles. Contains a report that describes the transcriptions and annotations.

TV moments detection and linking, initial version

The first implementation version of the back-end microservices developed in MeMAD. They will cover initial annotation and enrichment services attached to simple highlight moments extracted from programmes. Contains a report that describes the microservices.

Software and demonstration of human-like content description generation

Final versions of the developed visual and aural tools for multimodal content description, combined with standalone demonstrations and documentation in a report. The methods aim at referring to the recurrent objects and persons in the described media contents in human-like intelligent ways.

Libraries and tools for multimodal content analysis

A joint collection of tools, libraries and their documentations from Aalto, Eurecom, Lingsoft, LLS and INA.  These are needed in the continuation of this work package and also in the task T6.2 Prototype implementation. Contains a report.

Tools and models for multimodal, multilingual and discourse-aware MT

A release of tools and pre-trained models described in D4.1 and D4.2 with a report containing the documentation and user guidelines.

Setup of website with presentation of project and consortium partners

Website with presentation of project and consortium partners, initial setup.

Final website with presentation of project and consortium partners

Final version of Website with presentation of project and consortium partners.

Publications

MEMAD Project: End User Feedback on AI in the Media Production Workflows

Author(s): Lauri Saarikoski, Dieter Van Rijsselbergen, Maija Hirvonen, Maarit Koponen, Umut Sulubacak, Kaisa Vitikainen
Published in: Proceedings of IBC 2020, 2020

MT for Subtitling : Investigating professional translators’ user experience and feedback

Author(s): Maarit Koponen, Umut Sulubacak, Kaisa Vitikainen, Jörg Tiedemann
Published in: Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020 : 1st Workshop on Post-Editing in Modern-Day Translation, 2020, Page(s) 79-92

Deep Contextual Attention for Human-Object Interaction Detection

Author(s): Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, Jorma Laaksonen
Published in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, Page(s) 5693-5701
DOI: 10.1109/iccv.2019.00579

Named Entity Recognition for Spoken Finnish

Author(s): Dejan Porjazovski, Juho Leinonen, Mikko Kurimo
Published in: Proceedings of the 2nd International Workshop on AI for Smart TV Content Production, Access and Delivery, 2020, Page(s) 25-29
DOI: 10.1145/3422839.3423066

The University of Helsinki Submissions to the WMT19 News Translation Task

Author(s): Aarne Talman, Umut Sulubacak, Raúl Vázquez, Yves Scherrer, Sami Virpioja, Alessandro Raganato, Arvi Hurskainen, Jörg Tiedemann
Published in: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), 2019, Page(s) 412-423
DOI: 10.18653/v1/w19-5347

VIREO-EURECOM @ TRECVID 2019: Ad-hoc Video Search (AVS)

Author(s): Phuong Anh Nguyen, Jiaxin Wu, Chong-Wah Ngo, Francis Danny, Benoit Huet
Published in: TRECVID 2019, 23rd International Workshop on Video Retrieval Evaluation, 12-13 November 2019, Gaithersburg, MD, USA, 2019

Fusion of Multimodal Embeddings for Ad-Hoc Video Search

Author(s): Danny Francis, Phuong Anh Nguyen, Benoit Huet, Chong-Wah Ngo
Published in: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019, Page(s) 1868-1872
DOI: 10.1109/iccvw.2019.00233

End-to-end and HMM/DNN ASR in an equal data setting: A Finnish case study

Author(s): Aku Rouhe, Astrid Van Camp, Mittul Singh, Hugo Van Hamme, Mikko Kurimo
Published in: Proceedings of Interspeech, 2021

Attention-Based End-To-End Named Entity Recognition From Speech

Author(s): Dejan Porjazovski, Juho Leinonen, Mikko Kurimo
Published in: Text, Speech, and Dialogue - 24rd International Conference, TSD 2020, Brno, Czech Republic, 2021

Using Artificial Intelligence to Preserve Audiovisual Archives - New Horizons, More Questions

Author(s): Jean Carrive
Published in: Proceedings of the 27th ACM International Conference on Multimedia, 2019, Page(s) 1-2
DOI: 10.1145/3343031.3349583

L-STAP: Learned Spatio-Temporal Adaptive Pooling for Video Captioning

Author(s): Danny Francis, Benoit Huet
Published in: Proceedings of the 1st International Workshop on AI for Smart TV Content Production, Access and Delivery - AI4TV '19, 2019, Page(s) 33-41
DOI: 10.1145/3347449.3357484

PicSOM and EURECOM Experiments in TRECVID 2019

Author(s): Hector Laria Mantecon, Jorma Laaksonen, Danny Francis, Benoit Huet
Published in: Proceedings of TRECVID 2019, 2019

INA’s MIREX 2018 music and speech detection system

Author(s): David Doukhan, Eliott Lechapt, Marc Evrard, Jean Carrive
Published in: 14th Music Information Retrieval Evaluation eXchange (MIREX), September 2018, Paris, France., 2018

Spherediar: An Effective Speaker Diarization System for Meeting Data

Author(s): Tuomas Kaseva, Aku Rouhe, Mikko Kurimo
Published in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, Page(s) 373-380
DOI: 10.1109/asru46091.2019.9003967

Using Fan-Made Content, Subtitles and Face Recognition for Character-Centric Video Summarization

Author(s): Ismail Harrando, Alison Reboud, PasqualeLisena,Raphaël Troncy, Jorma Laaksonen, Anja Virkkunen, Mikko Kurimo
Published in: Proceedings of the TRECVID 2020 Workshop, 2020

Cognate-aware morphological segmentation for multilingual neural translation

Author(s): Grönroos, Stig-Arne; Virpioja, Sami; Kurimo, Mikko
Published in: Third Conference on Machine Translation (WMT18); Brussels, Belgium, 2018, Page(s) 390-397

The WMT’18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English

Author(s): Franck Burlot, Yves Scherrer, Vinit Ravishankar, Ondřej Bojar, Stig-Arne Grönroos, Maarit Koponen, Tommi Nieminen, François Yvon
Published in: Proceedings of the Third Conference on Machine Translation: Shared Task Papers, 2018, Page(s) 546-560
DOI: 10.18653/v1/w18-6433

Two-Stream Part-Based Deep Representation for Human Attribute Recognition

Author(s): Rao Muhammad Anwer, Fahad Shahbaz Khan, Jorma Laaksonen
Published in: 2018 International Conference on Biometrics (ICB), Issue Proceedings - 2018 International Conference on Biometrics, ICB 2018, 2018, Page(s) 90-97
DOI: 10.1109/ICB2018.2018.00024

The MeMAD Submission to the IWSLT 2018 Speech Translation Task

Author(s): Sulubacak, Umut; Tiedemann, Jörg; Rouhe, Aku; Grönroos, Stig-Arne; Kurimo, Mikko
Published in: Proceedings of the International Workshop on Spoken Language Translation, 2018, Page(s) 89-94

The MeMAD Submission to the WMT18 Multimodal Translation Task

Author(s): Stig-Arne, Grönroos; Huet, Benoit; Kurimo, Mikko; Laaksonen, Jorma; Merialdo, Bernard; Pham, Phu; Sjöberg, Mats; Sulubacak, Umut; Tiedemann, Jörg; Troncy, Raphaël; Vázquez Carrillo, Juan Raúl
Published in: Proceedings of the Third Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, 2018, Page(s) 609-617

The Aalto system based on fine-tuned AudioSet features for DCASE2018 task2 - general purpose audio tagging

Author(s): Zhicun Xu, Peter Smit, and Mikko Kurimo
Published in: Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018), 2018, Page(s) 24-28

Deep Multimodal Features for Movie Genre and Interestingness Prediction

Author(s): Olfa Ben-Ahmed, Benoit Huet
Published in: 2018 International Conference on Content-Based Multimedia Indexing (CBMI), 2018, Page(s) 1-6
DOI: 10.1109/cbmi.2018.8516504

EURECOM participation in TrecVid VTT 2018

Author(s): Danny Francis, Benoit Huet, Bernard Merialdo
Published in: TRECVID 2018, 22nd International Workshop on Video Retrieval Evaluation, November 13-15, 2018, Gaithersburg, USA, 2018

PicSOM Experiments in TRECVID 2018

Author(s): Mats Sjöberg, Hamed R. Tavakoli, Zhicun Xu, Hector Laria Mantecon, Jorma Laaksonen
Published in: TRECVID 2018, 22nd International Workshop on Video Retrieval Evaluation, November 13-15, 2018, Gaithersburg, USA, 2018

Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models

Author(s): Wang, Tzu-Jui Julius; Pehlivan, Selen; Laaksonen, Jorma
Published in: Proceedings of the British Machine Vision Conference (BMVC), 2020

EURECOM at TRECVid AVS 2019

Author(s): Danny Francis, Phuong Anh Nguyen, Benoit Huet, Chong-Wah Ngo
Published in: TRECVID 2019, 23rd International Workshop on Video Retrieval Evaluation, 12-13 November 2019, Gaithersburg, MD, USA, 2019

The University of Helsinki Submission to the IWSLT2020 Offline SpeechTranslation Task

Author(s): Raúl Vázquez, Mikko Aulamo, Umut Sulubacak, Jörg Tiedemann
Published in: Proceedings of the 17th International Conference on Spoken Language Translation, 2020, Page(s) 95-102
DOI: 10.18653/v1/2020.iwslt-1.10

Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

Author(s): Aku Rouhe, Tuomas Kaseva, Mikko Kurimo
Published in: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, Page(s) 7064-7068
DOI: 10.1109/icassp40776.2020.9053998

The MeMAD Submission to the WMT18 Multimodal Translation Task

Author(s): Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, Raúl Vázquez
Published in: Proceedings of the Third Conference on Machine Translation: Shared Task Papers, 2018, Page(s) 603-611
DOI: 10.18653/v1/w18-6439

Explainable Zero-Shot Topic Extraction Using a Common-Sense Knowledge Graph

Author(s): Harrando, Ismail; Troncy, Raphaël
Published in: LDK 2021, 3rd Conference on Language, Data and Knowledge, 1-3 September 2021, 2021

Geometry-aware Relational Exemplar Attention for Dense Captioning

Author(s): Tzu-Jui Julius Wang, Hamed R. Tavakoli, Mats Sjöberg, Jorma Laaksonen
Published in: 1st International Workshop on Multimodal Understanding and Learning for Embodied Applications - MULEA '19, 2019, Page(s) 3-11
DOI: 10.1145/3347450.3357656

Predicting Media Memorability with Audio, Video, and Text representation

Author(s): Reboud, Alison; Harrando, Ismail; Laaksonen, Jorma; Troncy, Raphaël
Published in: Working Notes Proceedings of the MediaEval 2020 Workshop, Issue 3, 2020

AI4TV 2020 - 2nd International Workshop on AI for Smart TV Content Production, Access and Delivery

Author(s): Raphaël Troncy, Jorma Laaksonen, Hamed R. Tavakoli, Lyndon Nixon, Vasileios Mezaris, Mohammad Hosseini
Published in: Proceedings of the 28th ACM International Conference on Multimedia, 2020, Page(s) 4756-4757
DOI: 10.1145/3394171.3421894

Paragraph-length image captioning using hierarchical recurrent neural networks

Author(s): Arturs Polis
Published in: Master's thesis, 2019

Spherediar – an efficient speaker diarization system for meeting data

Author(s): Tuomas Kaseva
Published in: Master's thesis, 2019

Visual Storytelling: Captioning of Image Sequences

Author(s): Aditya Surikuchi
Published in: Master's thesis, 2019

Audio Event Classification Using Deep Learning Methods

Author(s): Zhicun Xu
Published in: Master's thesis, 2018

Deep Reinforcement Sequence Learning for Visual Captioning

Author(s): Héctor Laria Mantecón
Published in: Master's thesis, 2019

Semantic representations of images and videos

Author(s): Danny Francis
Published in: 2019

VIREO @ Video Browser Showdown 2020

Author(s): Phuong Anh Nguyen, Jiaxin Wu, Chong-Wah Ngo, Danny Francis, Benoit Huet
Published in: MultiMedia Modeling - 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II, Issue 11962, 2020, Page(s) 772-777
DOI: 10.1007/978-3-030-37734-2_68

Comparing human and automated approaches to visual storytelling

Author(s): Sabine Braun, Kim Starr, Jorma Laaksonen
Published in: Innovation in Audio Description Research, 2020, Page(s) 159-196
DOI: 10.4324/9781003052968-9

Easy Web API Development with SPARQL Transformer

Author(s): Pasquale Lisena, Albert Meroño-Peñuela, Tobias Kuhn, Raphaël Troncy
Published in: The Semantic Web – ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II, Issue 11779, 2019, Page(s) 454-470
DOI: 10.1007/978-3-030-30796-7_28

A Novel Ensemble Method for Named Entity Recognition and Disambiguation Based on Neural Network

Author(s): Lorenzo Canale, Pasquale Lisena, Raphaël Troncy
Published in: The Semantic Web – ISWC 2018 - 17th International Semantic Web Conference, Monterey, CA, USA, October 8–12, 2018, Proceedings, Part I, Issue 11136, 2018, Page(s) 91-107
DOI: 10.1007/978-3-030-00671-6_6

Multi-stream Convolutional Networks for Indoor Scene Recognition

Author(s): Rao Muhammad Anwer, Fahad Shahbaz Khan, Jorma Laaksonen, Nazar Zaki
Published in: Computer Analysis of Images and Patterns - 18th International Conference, CAIP 2019, Salerno, Italy, September 3–5, 2019, Proceedings, Part I, Issue 11678, 2019, Page(s) 196-208
DOI: 10.1007/978-3-030-29888-3_16

Advances in subword-based HMM-DNN speech recognition across languages

Author(s): Peter Smit, Sami Virpioja, Mikko Kurimo
Published in: Computer Speech & Language, Issue 66, 2021, Page(s) 101158, ISSN 0885-2308
DOI: 10.1016/j.csl.2020.101158

Multimodal machine translation through visuals and speech

Author(s): Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, Jörg Tiedemann
Published in: Machine Translation, Issue 34/2-3, 2020, Page(s) 97-147, ISSN 0922-6567
DOI: 10.1007/s10590-020-09250-0

Taking a Cue From the Human

Author(s): Kim Linda Starr, Sabine Braun, Jaleh Delfani
Published in: Journal of Audiovisual Translation, Issue 3/2, 2020, ISSN 2617-9148
DOI: 10.47476/jat.v3i2.2020.138

Transfer learning and subword sampling for asymmetric-resource one-to-many neural translation

Author(s): Stig-Arne Grönroos, Sami Virpioja, Mikko Kurimo
Published in: Machine Translation, Issue 34/4, 2020, Page(s) 251-286, ISSN 0922-6567
DOI: 10.1007/s10590-020-09253-x

Finding the Right Words

Author(s): Sabine Braun, Kim Starr
Published in: Journal of Audiovisual Translation, Issue 2/2, 2019, Page(s) 11-35, ISSN 2617-9148
DOI: 10.47476/jat.v2i2.103

MediaEval 2018: Predicting Media Memorability

Author(s): Cohendet, Romain; Demarty, Claire-Hélène; Duong, Ngoc Q.K.; Sjöberg, Mats; Ionescu, Bogdan; Do, Thanh Toan
Published in: CEUR Workshop Proceedings, Issue 2283, 2018, ISSN 1613-0073

FaceRec: An Interactive Framework for Face Recognition in Video Archives

Author(s): Pasquale Lisena; Jorma Laaksonen; Raphael Troncy
Published in: Ceur Workshop Proceedings, 2021, ISSN 1613-0073
DOI: 10.5281/zenodo.4764633

Effective video hyperlinking by means of enriched feature sets and monomodal query combinations

Author(s): Mohammad Reza Kavoosifar, Daniele Apiletti, Elena Baralis, Paolo Garza, Benoit Huet
Published in: International Journal of Multimedia Information Retrieval, Issue 9/3, 2020, Page(s) 215-227, ISSN 2192-6611
DOI: 10.1007/s13735-019-00173-y

Machine translation and fair access to information

Author(s): Mary Nurminen, Maarit Koponen
Published in: Translation Spaces, Issue 9/1, 2020, Page(s) 150-169, ISSN 2211-3711
DOI: 10.1075/ts.00025.nur

Describing Gender Equality in French Audiovisual Streams with a Deep Learning Approach

Author(s): David Doukhan, Géraldine Poels, Zohra Rezgui, Jean Carrive
Published in: VIEW Journal of European Television History and Culture, Issue 7/14, 2019, Page(s) 103, ISSN 2213-0969
DOI: 10.18146/2213-0969.2018.jethc156

Big Data Analytics for Large‐Scale Multimedia Search

Author(s): Stefanos Vrochidis, Benoit Huet,Edward Chang, Ioannis Kompatsiaris
Published in: 2019
DOI: 10.1002/9781119376996

Innovation in Audio Description Research

Author(s): Sabine Braun, Kim Starr
Published in: 2020
DOI: 10.4324/9781003052968

Datasets

Speech recognition alignments for Finnish parliament data

Author(s): Virkkunen, Anja; Mansikkaniemi, André; Kurimo, Mikko
Published in: Zenodo

Speech recognition alignments for Finnish parliament data

Author(s): Virkkunen, Anja; Mansikkaniemi, André; Kurimo, Mikko
Published in: Zenodo

MeMAD Knowledge Graph

Author(s): Harrando, Ismail; Troncy, Raphael
Published in: Zenodo

MeMAD Knowledge Graph

Author(s): Harrando, Ismail; Troncy, Raphael
Published in: Zenodo

MeMAD data for automatic cross-lingual retrieval experiments

Author(s): Umut Sulubacak; Jorma Laaksonen
Published in: Zenodo

doclevel-MT-benchmark-discoMT2019

Author(s): Tiedemann, Jörg; Scherrer, Yves
Published in: Zenodo

doclevel-MT-benchmark-discoMT2019

Author(s): Tiedemann, Jörg; Scherrer, Yves
Published in: Zenodo

Software

MeMAD subtitle translation

Author(s): Tiedemann, Jörg
DOI: 10.5281/zenodo.4556121; 10.5281/zenodo.4556120
Publisher: Zenodo