Multimedia Content Annotations for Rapid Exploitation in Multi-Screen Environments

Periodic Reporting for period 1 - MECANEX (Multimedia Content Annotations for Rapid Exploitation in Multi-Screen Environments)

Berichtszeitraum: 2015-01-01 bis 2016-06-30

Every day, the need for enriched multimedia content is becoming ever more prevalent. Modern audiences appear willing to immerse into more interactive and personalised viewing experiences, through second-screen environment and enriched video content. With production houses, broadcasters, advertising agencies and online publishing companies seeking to better reach their audience with relevant, highly targeted content, the need for innovative tools that aid them in this task is a requirement rather than a desire. The adaption of enriched multimedia content to multi-screen environments, which enable the automatic porting to different target platforms, such as regular web pages, mobile pages and mobile apps, as well as TV applications, is a significant asset for the global market of enriched multimedia content. The objective of MECANEX project is to provide the necessary solutions to these problems, to devise tools for automatic annotation and editorial support of multimedia content, extraction of personalized information and adaption of enriched multimedia content in multi-screen environments. Furthermore, the major novelty of MECANEX is the creation of SaaS (REST based) toolkit that provides the aforementioned functionalities by bringing together complementary expertise and technology of different European companies and research institutes, in order to provide a cost effective multi-screen video metadata enrichment solution for SMEs. Such tools will be adopted by production and post-production companies, as well as by content providers or relevant service oriented companies and will be adapted into the creation process. MECANEX will offer SMEs the ability to semi-automatically enrich video collections for multi-screen application, as well rapid prototype multi-screen concepts and test related interaction models that eventually will improve the chance for SMEs to develop more solid business models, marketing and advertising campaigns. Finally, the developed social and personalization mechanisms, allows for the collection of consumers’ preferences and provide feedback and input to the aforementioned tools. In a nutshell, automatic annotation, editorial, multi-screen and social and personalization tools have been integrated into a working MECANEX toolkit ready for the production of multimedia content, as well as the advertising agencies.

The strategic key objective of MECANEX is to provide innovative tools and methods, towards:
➢ enabling the automatic annotation and editing during the creation, production and post-production of multimedia content that will allow SMEs, such as production houses, broadcasters, advertising agencies, online publishing companies, etc. to create and enrich multimedia content ready for multi-screen environments as well as content ready for marketing campaigns.
➢ supporting search and retrieval mechanisms from libraries with existing enriched multimedia content that could be used as building blocks of a fast innovative creative process.
➢ pairing created metadata with multimedia content allowing easy creation and access to relevant educational and recreational content, e.g. wikipedia pages of landmarks appeared in a video, images of actors or staff participating, relevant trivia, historic events affecting the story etc.
➢ providing a multi-screen toolkit for the development of prototypes that enable an automatic porting to different target platforms, such as regular web pages, mobile pages and mobile apps as well as TV applications.
➢ gathering from end users/consumers, processing and delivering personalized information, which will facilitate the production and editing process, while at the same time will allow advertising & marketing brokers to automatically ingest relevant ads and marketing data via coupling user preferences with the details of the playing multimedia content. End users participate within this process providing feedback towards both the enrichment of metadata by manually annotating parts of the content, as well as expressing their views related both to the experience and the recommended content itself.

In order to implement the above strategic objective the following operational objectives have been targeted and met through respective activities, during the project lifetime:
1. Determine user and investor requirements: This objective refers to defining the framework and acts as a reference point for the development of the different tools that will be created as part of the project. This has been achieved by aligning the user requirements with the technological and the technical capabilities of the toolkits envisioned by the technical project partners.
2. Development of automatic annotation and editing tool: Providing the enhanced flexibility to the production process, the advertising agencies and the online publishing companies to automatically annotate, edit and enrich multimedia content is of great necessity in the real production and advertising world.
3. Development of multi-screen tool for rapid prototyping: Multi-screen environment provides flexibility and advanced functionalities in production process, editing and advertising area compared to single-screen applications. MECANEX exploited multi-screen environments / applications towards providing functionalities to the corresponding SMEs to synthesize and preview the enriched multimedia content with the respective annotations and metadata.
4. Development of social and personalization tool: End users’ personalized information is gathered based on relevance feedback, personalization and social recommendation mechanisms. This tool provides feedback to the creation, production, post-production, automatic annotation and editing process of multimedia content, in order to be enriched based on users’ preferences and needs, and improve the experience of users with respect to the recommended content.
5. Development of an overall working integrated MECANEX toolkit: MECANEX aspired to provide a working integrated toolkit, consisting of automatic annotation, editing, multi-screen and social and recommendations tools. The MECANEX toolkit will be available for integration in existing production infrastructures, while indicative relevant demonstrators illustrate the prominent benefits of the corresponding SMEs in the overall process.
6. Evaluation, testing and benchmarking of MECANEX tool results: The proof of concept and the performance of the developed MECANEX tool has been evaluated, tested and benchmarked by checking conformance with user and investor requirements and the overall project goals, as well as against state of the art methodologies.

The final developed MECANEX prototype provides an open RESTful toolkit packed as a SaaS that enables: i) the automation of the costly and time-consuming process of multimedia annotation, ii) the efficient search,categorization and retrieval of enriched content, iii) the fast and dynamic creation of multimedia footage and personalized clips based on existing productions, iv) incorporation of calls to action within multimedia content thus directing and engaging end users during playback, v) the interconnection of media streams with
users social presence and preferences favoring customizable personalized services, vii) the provisioning of insights with regard to users’ actions and interactions within the enrich content. The latter verify the multi parametric innovations, features and unique characteristics of the MECANEX tools, favoring its placement to the market and providing competitive advantages over competition. Extensive evaluations and benchmarking of the different tools and the overall integrated prototype, with carefully designed experiments and benchmarking procedures - including both sicientific/analytic peformance evaluations, as well as real tests by end users, professional users and relevant industry stakeholders through online questionnaire filling, guided interviews and strategic benchmarking with potential investors - demonstrtaed and verified that the tools reach or supersede the state-of- the-art performance of related algorithms, while possess (either individually or integrated as a whole system) unique feature set and are technologically advanced compared to already available tool sets.

The results of MECANEX project are expected to have a significant impact on the efficiency of the use and exploitation of multimedia content, regarding every phase of the life cycle of the latter, from the production to the final consumption, via delivering a set of innovative tools and methods that will facilitate SMEs a) to proficiently harvest the benefits and b) minimize the existing gap in the use and exploitation of multimedia content. Automatic multimedia content annotation based on landmarks, individuals, thematic areas, events, place, etc. may lead to the creation of enriched content with multimodal information driving the creation of novel multi-screen applications, as well as innovative marketing campaigns. Moreover, efficient search and retrieval mechanisms on the above content will facilitate multimedia categorization and organization, pairing the generated metadata with the actual content, thus creating libraries filled with enriched content directly usable as building blocks of a fast innovative creative process by SME creative players. Educational or reacreation content and/or accompanying applications can be developed towards engaging users via a large variety of new emerging experiences and thus, attracting new customers. Advertisement & marketing brokers can automatically ingest relevant ads and marketing data coupling user preferences with the details of the multimedia content. Moreover, end-users can participate within this process providing feedback towards both the enrichment of metadata by manually annotating parts of the content, as well as expressing their
views related both to the experience and the content itself.

The concept of the MECANEX emanates from the need of relevant SMEs to slim line the production both of annotation streams and personalization data towards the rapid development of new products as well as the enhancement of existing ones. Towards this direction our efforts focused on producing a toolkit collection of the appropriate tools allowing easy integration and exploitation of MECANEX results from the participating SMEs in the near future. We have developed the MECANEX concept into five parallel pillars as follows:

A. Multimedia content analysis, automatic annotation tools and mechanisms, aiming into developing REST based web services for the automatic annotation of existing or even newly created content in a general way, as well as in in specific way for multi-screen context. Such tools allow the adaptation of automatic annotation processes into the creative process from Production and Post-Production Companies as well as Content Providers or relevant service oriented companies. Search and retrieval tools aim to build upon the automatic annotation tools by providing a REST based web service for the search and retrieval of content. Content can be retrieved based on specific criteria characteristics of the video allowing the fast creation of informational and demo clips based on existing productions. The Automatic Annotation tool enables the analysis and automatic annotation of multimedia content that was collected within the project from various partners' sources (archiving multimedia content). Advanced automatic annotation algorithms constitute the Automatic Annotation tool (AAT), which enable the enrichment of multimedia content with automatic annotations, related text, as well as multimodal information.

B. Editorial tool provides the enhanced flexibility to edit the already enriched metadata via supporting functionalities like accepting, ignoring, extending them. The editorial tool ensures the quality of the newly created meta-data. The output of editorial tool provides feedback to the multimedia content analysis and automatic annotation processes. This tool supports mainly the needs of online publishing companies and also production houses and broadcasters by its provided feedback.

C. Multi-screen tool provides a decision engine, which takes the enriched metadata, advertisements, live detected social data and information of the current context and turn it into a scenario to be displayed on the available screens at any moment. This is an active cycle, so the multi-screen tool sends signals of usage back to the social and personalization tool in order to be enhanced further while watching and interacting with the content and other viewers.The Multi-screen tool performs synthesizing and previewing of the enriched multimedia content with the respective annotations and metadata in multi-screen environments.

D. Personalization data aggregation and relevance feedback, aims to provide a framework for connecting users’ responses and comments with content streams and annotation data. The provided tools allow easy and fast interconnection of media streams with users clusters and preferences allowing SMEs to provide customizable personalized services through promotions, advertisements and marketing campaigns as well as personalized multi-screen services. Moreover, social and personalization tool provides feedback to the automatic annotation, search and retrieval processes, as well as to the multi-screen tool. The Social and Personalization (SP) tool is designed towards assessing users’ experience with the enriched multimedia content, as well as its success in the corresponding advertising and marketing campaign. User-generated tags, personal information provided by the end user and relevance feedback data are collected towards constructing users’ profiles and facilitating a novel personalization mechanism. Two basic workflows of the SP tool are designed which are differentiated based on the type of end user, i.e. Business-to-Business and Business-to-Customer. Advanced algorithms are designed for the construction of user profiles and their update via relevance feedback, while recommendation and clustering algorithms are examined as part of the SP tool.

E. The last toolkit is a light-weight integration module, focusing on its seamless and fast deployment of already existing infrastructure, tools and services. The core of this SaaS-based toolkit is to define how data flows from and to the different SaaS elements, allowing SMEs to fine-tune, add filters and keep track of the flow from source to the multi-screen applications. Each license deal between content providers and SMEs leads to technical contract that is defined inside this tool.

The main part of the MECANEX multimedia repository is provided by LUCE and consists of grayscale videos of standard and high definition. For the Automatic Annotation Tool the functionalities of face detection and recognition, object detection and landmark recognition are necessary to analyze these content. The face detection uses of a recent face representation algorithm which provides state-of-the-art results in order to boost performance. For the generic object detection, the AAT was trained a deep convolutional neural network using the grayscale converted images of Imagenet. The landmark detection/recognition in the MECANEX project provides automated location estimation and landmark recognition based on the state-of-the-art technologies Bag-of-Words, RANSAC and Avrithis.
The results of the three independent modules of the Automatic Annotation Tool for every frame are unified, creating a single output file for each analyzed video. The Editorial Tools goal in the MECANEX Platform is to provide an easy-to-use editor (B2C) and an API (B2B). The Content Enrichment component uses semantic web technologies to make powerful enrichments and to easily connect to other semantic ontologies. To prevent errors that may occur during the enrichment process e.g. detecting wrong semantic enrichment, the editor is supported by easy-to-use UI which will become part of the workflow to approve and correct enrichments if necessary. The output of the annotation is consumed in the Editorial Tool. The tool also supports the editor while correcting the enrichment by offering an autocompletion in the UI which makes the process for the editor faster and easier. The developed SP tool consists of a novel algorithm devised for building users’ profile and its update via implicit and explicit relevance feedback processes, along with a novel algorithm for providing recommendations to users. The proposed recommendation algorithm is of hybrid nature incorporating features from content filtering that is based on the target user’s profile itself and from collaborative filtering that accounts other, similar to the target users’ profiles. The incorporation of content and collaborative filtering features within the hybrid scheme is tunable towards tha one or the other direction, based also on the specific target of each application. Content filtering emphasizes more on the target user’s preferences itself, while collaborative filtering achieves the introduction of novelty and diversity in recommendations for evolving and enhancing the users’ experience and interests. The novelty of the proposed SP tool algorithms lies mainly in the interconnection, enhancement and adaptation of existing user profiling, relevance feedback and recommendation schemes to MECANEX multimedia content that consists of videos paired with enrichments and advertisements and to the Business-to-Business and Business-to-Customer application scenarios. All the aforementioned tools have been intrgrated into a single prototype. The MECANEX platform forms the backbone of the system and consists out of several components. The foundation is build upon the open source Springfield framework. The Management dashboard exposes the data stored in the platform and allows professional users to use the different MECANEX services (e.g. automatic annotation, editorial, content enrichment, etc.). Finally enriched data is made available to a (multiscreen) player that can interact with other tools, such as the Advertisement Tool for displaying advertisements and is linked with the SP Tool for play out filtering. The platform uses JSON as a communication format between the different tools developed by the MECANEX partners. The tools are all available as online services that communicate through HTTP REST calls. The main advantage is that this approach allows the platform to integrate with other, additional, external services without too much hassle. By means of this we keep the platform as versatile as possible and allow for integration with other web services if desired.Demonstrations of the application and incorporation of the MECANEX platform within the workflows of Production and Post-Production environments, Marketing and Advertising Agencies Infrastructures and Online Publishing Companies are provides. A critical part and prerequisite with respect to the vailidation of the work performed and the proposed final MECANEX toolset included the execution of the following actions: a) to collect investor and end-user/consumer requirements for the deployment of the tool set, taking into account both the production, editorial and advertising process requirements (i.e. investor requirements), as well as end-users needs, b) to propose thematic units of multimedia content to be used in the testing of the proposed tools, based also on user and investor requirements, and finally d) to evaluate the different developed components with carefully designed experiments and benchmarking procedures. With reference to the latter, extensive evaluations and benchmarking of the different tools and the overall integrated prototype, with carefully designed experiments and benchmarking procedures - including both sicientific/analytic peformance evaluations, as well as real tests by end users, professional users and relevant industry stakeholders through online questionnaire filling, guided interviews and strategic benchmarking with potential investors - demonstrtaed and verified that the tools reach or supersede the state-of- the-art performance of related algorithms, while possess (either individually or integrated as a whole system) unique feature set and are technologically advanced compared to already available tool sets. Significant part of the efforts devoted and activities performed by the consortoum partners were dedicated to the proper and detailed analysis and evaluation of the corresponding and potential market. In that respect the project partners: a) identified exploitable products of the MECANEX toolkit and highlighted the main features and benefits of each technology derived, with emphasis on the innovative aspects; b) for each of the exploitable products state of the art analysis with respect to potential competitors in the market of interest has been performed; c) Identified and assessed the features of the target market: size, growth rate, share that the technology/product could reach, driving factors likely to change the market, technical and commercial barriers, other technologies likely to emerge in the near future; d) actively looked for for cooperation with decision makers and key players of the industry, from the very beginning of the project, identifying specific channels for potential exploitation thus creating a core group of stakeholders interested in exploiting MECANEX results.

Since its beginning, the MECANEX project has been a high-interest, high-priority venture for the consortium partners. As such, it has been imperative to raise general, public awareness throughout the project duration and publicly promote its objectives and progress so as to ensure wide adoption upon completion. This, of course, required strong dissemination in both the academic and industrial sectors, as well as public communication actions in line with the project’s open access policy. The work conducted by the MECANEX Consortium was presented in multiple events organised by high profile organisations in various cities. Thus, the MECANEX results have been presented by partners of the consortium in fora and conferences, as well as in exhibitions where there was a MECANEX dedicated stand. Additionally, MECANEX papers have been published in high profile journals, magazines and conferences. Finally, the MECANEX partners have contacted members of other European or national projects as well as groups, networks and institutions pursuing research and innovation goals in areas related to the MECANEX domains, resulting to a number of collaborative synergies. In a nutshel the following activities have been realized:

a) Public Reports/Deliverables: The MECANEX consortium has attempted to maximize the visibility and exploitation of the project and its long-term impact, by providing as many as possible publicly available results that can be easily discovered and re-used. The public deliverables of the project are available on the MECANEX website.

b) MECANEX Flyer: MECANEX consortium designed a good visual, easy to read through and customer oriented flyer in order to ensure that it will excite the reader and will inspire him enough to act. Vivid colors were used in order to make the flyer stand-out. The text content was brief but to the point: it highlights the main message of the project without bulking up the reader with many technical details. It clearly states the objectives of MECANEX and demonstrates its main tools in order to acquaint the public with its features.

c) MECANEX Poster: The MECANEX consortium has designed and produced a representative poster that has already been used in the public events where the project participated at. Experienced designers worked on the poster in order to reach the desirable outcome. The poster carries the following information: i) Summary of the Scope and Objectives of the MECANEX project; ii) Graphical analysis of the overall architecture of the toolkit; iii) Overview of the toolkit’s components, namely the Automatic Annotation Tool, the Multi-screen Tool and the Social & Personalization Tool. The poster also expanded on a use case for the toolkit so that members of the audience could better understand its potential applications while grasping some of the technical details involved in its development. The effective use of graphics, colors and fonts according to best practices resulted in the creation of a poster that was both interesting and visually slick, so that it attracted the maximum amount of viewers to MECANEX booths.

d) Papers at Journals & Magazines, Conferences & Workshops: MECANEX has published three articles presenting its research and innovation results in international journals, conferences and workshops.

e) MECANEX Online Presence: i) The MECANEX website (http://www.MECANEX.eu) has been created since the early beginning of the project, providing the external community with up-to-date information on the progress of the project. The goal of the public part of the web site is to foster project awareness by presenting MECANEX’s technical concept, scope and objectives and the planned approach; ii) MECANEX has been present in Facebook since the project start (https://www.facebook.com/Mecanex-539954919478659/). Its Facebook account is active and it is currently used for dissemination purposes of the events that MECANEX members participate in and to notify registered users with regards to the published deliverables or dissemination material; iii) The project’s Twitter account is currently active under the account name: @Mecanex_. This account contributes to attracting the audience closer to the project. Through Twitter the consortium shares the news, the ideas and the progress of MECANEX. Furthermore, Twitter gives the opportunity to contemplate what competitors are tweeting about
or search for industry keywords to listen and gather market intelligence; iv) The MECANEX project has released platform software and tools developed within the project as open source. The project has established an open source repository in github (world’s largest open source hosting platform), where software that was developed by the project has been uploaded frequently. Anonymous read access is allowed for external parties.

f) Representation of MECANEX at Dissemination & Communication Events: MECANEX has been (re)presented in 8 dissemination & communication events with interdisciplinary audiences that are elaborated upon in this section. These include Exhibitions, Conferences, Workshops, Symposia, Hackathons, etc.

g) Dissemination & Communication Events organised by MECANEX: MECANEX has organized one public dissemination & communication event, with focus on "Creative Audiovisual industry: A smartly booming sector". The objective of this one-day public workshop, showcase and dissemination event during the last month of the project where the project technology and developments are mature enough, was to demonstrate the technologies that enable new services offering in the multimedia content area sector at project end. Various stakeholders, potential partners and industry related audience were invited in order to disseminate the results of MECANEX activities and improve its visibility.

h) MECANEX Synergies & Collaboration Activities: This synergies have focused on two dimensions. First, MECANEX has targeted key EU clusters that are related to its technological objectives and has exploited collaboration synergies, technical concertation and contribution to specific working groups. Secondly, MECANEX has targeted collaboration with numerous EU ICT research projects that are launched under the H2020 or the 7th Framework Programme and that focus on areas similar or complementary to the MECANEX technology domains.

Key advances contributed by MECANEX are briefly summarized in the following technological areas and categories:

• Multimedia Content Analysis
Multimedia content analysis functionalities will allow “small pieces” of information from different modalities to complement each other at various levels of the content analysis process in order to provide reliable input for reasoning at the highest level. Moreover, these functionalities will support the combination of features from different modalities in multi-modal content models via extracting content-related information from multimedia data. These functionalities will facilitate the production process within production houses and broadcasters, as well as will allow the fast and easy creation of marketing campaigns by the advertising agencies.

• Automatic Annotation
Automatic annotation functionalities will enable the enrichment of the multimedia content with supplementary information, based on advanced algorithms, techniques and methodologies. The multimedia content will be fully annotated, thus providing the ability to the production houses, the broadcasters and the advertising agencies to maximize their profit and enhance their production and advertising process, as well as the end users to exploit and utilized the annotations in order to improve their experience.

• Search and Retrieval of Multimedia Content
Search and retrieval functionalities will support the fast and easy creation of new multimedia content by the production houses and the broadcasters, as well as the creation of marketing campaigns by the advertising agencies. Based on these functionalities the existing enriched multimedia content by the annotation process will act as building block in order to create new content. Therefore, the production process will be facilitated and accelerated, thus providing great benefits and profit to the related industrial area.

• Editorial selection and enhancement of meta-data
Having editorial tools as part of the enrichment workflow will bridge the gap between fully automatic systems that are nice for research but can’t be trusted by content creators to be taken into production and manual enrichment tools that have a too high a overhead for content creators. The novel element is having a manual (editorial) phase that is much quicker by having the automatic system act as a helper not a replacement for editors.

• Multi-screen Environments
The developed toolkit does not need to be adapted for different output devices but adapts dynamically to them. The novelty is that it also adapts to the available devices during the consuming of the content. It allows for the first time to create multi-screen applications that react to changes in their environment like people joining/leaving with their phones, tablets and other devices.

• Relevance Feedback,Personalization and Social Recommendation
MECANEX provides a social and personalization tool, which will support the advanced mechanisms of relevance feedback, personalization, and social recommendations. The output of this tool will be personalized information, which will be utilized by production houses’, broadcasters’, advertising agencies’ and annotation and online publishing companies in order to enhance the functionalities and the whole process during the production and editorial process within multi-screen environments. The above mentioned type of companies and the corresponding market, where these companies target to, will have great benefits and profit by exploiting the personalized provided information by the SP tool, due to the fact that they will be able to target to the specific needs, preferences and habits of users.

Overall MECANEX is expected to contribute to improving innovation capacity and integration of new knowledge in European creative industries, strengthening their competitiveness in the European and international markets. Specifically:
a) reinforces the position of the European ICT and digital media industry by opening wider market opportunities.
b) provides effective contribution to global technology in the area of multimedia content archiving, indexing and retrieval that is still at its infancy.
c) provides greater creativity to all interested actors, stimulated through technologies and tools, possibly also to be used in immersive and interactive applications.
d) enforces competitiveness of European companies in the still very early stages of multimedia content archiving and exploitation.
e) project's multi-characteristic results can be directly exploited in the TV, cinema, gaming, social media and online media domains.
f) provides tools and algorithms which are able to detect multimedia content especially in targeted audiences and display sizes where the content will be shown.
g) incurs cost savings to creative SMEs either by its SaaS nature eliminating the need for infrastructure, either by the major reduction of operational costs due to the introduction of automatic annotation mechanisms.

The MECANEX toolkit is mainly a B2B solution even though B2C applications have been identified as well. The target markets of the Mecanex solution that have been identified and have already been analyzed in the following sectors are:
• The marketing & advertising market
• The Multimedia Publishing and Broadcasting market
• The Online Services and Software Development market

The potential Mecanex customers differ in terms of content ownership as well as in the degree that they want to pursue the entire end-to-end workflow from content production to content distribution. Content owners will most probable be interested in enhancing their content with web information. The automatic annotation support that the Mecanex toolkit provides will lower annotation costs for broadcasters and production houses. Furthermore the advanced search & retrieval technologies will a) reduce the time / cost for finding relevant pieces of content and b) will improve the quality level of content for re-use. The fully annotated multimedia content will provide the ability to the production houses, the broadcasters and the advertising agencies to maximize their profit and enhance their production and advertising process, as well as the end users to exploit and utilized the annotations in order to improve their experience. Advertising companies will be interested in providing targeted advertisement to the consumers by exploiting multimedia annotations towards designing and providing sophisticated user-engaging content- and context-aware personalized advertising services that will maximize users’ experience and eventually increase brand awareness and conversion rates. Furthermore software developers will be able to create innovative applications giving a rapid prototyping environment for rapid and low-cost pilots which can be used to evaluate decision towards large scale development of both content-productions as the multiscreen playout environment.

Value chain of content business and Project impact

Public Event

Workflow description

MECANEX Poster

MECANEX Flyer

MECANEX Platform

Periodic Reporting for period 1 - MECANEX (Multimedia Content Annotations for Rapid Exploitation in Multi-Screen Environments)

Herunterladen Den Inhalt der Seite herunterladen