Community Research and Development Information Service - CORDIS


MixedEmotions Report Summary

Project ID: 644632
Funded under: H2020-EU.

Periodic Reporting for period 1 - MixedEmotions (Social Semantic Emotion Analysis for Innovative Multilingual Big Data Analytics Markets)

Reporting period: 2015-04-01 to 2016-03-31

Summary of the context and overall objectives of the project

Emotion analysis is central to tracking customer and user behavior and satisfaction, which can be observed from user interaction in the form of explicit feedback through email, call center interaction, social media comments, etc., as well as implicit acknowledgment of approval or rejection through facial expressions, speech or other non-verbal feedback. In Europe specifically, but increasingly also globally, an added factor is that user feedback can be in multiple languages, in text as well as in speech and audio-visual content. This implies different cultural backgrounds and thus different ways to produce and perceive emotions in everyday interactions, beyond the fact of having language specific methods for encoding and decoding emotions.

Making sense of accumulated user interaction from different (‘mixed’) data sources, modalities and languages is challenging and has not yet been explored in fullness in an industrial context. Commercial solutions exist but do not address the multilingual aspect in a robust and large-scale setting and do not scale up to big data volumes that need to be processed, or the integration of emotion analysis observations across data sources and/or modalities on a meaningful level, i.e. keeping track of entities involved as well the connections between them (who said what? to whom? in the context of which event, product, service?)
MixedEmotions develops an integrated Big Linked Data platform that provides:

- Large-scale emotion analysis and fusion on heterogeneous, multilingual, text, speech, video and social media data streams, leveraging open access and proprietary data sources, and exploiting social context by leveraging social network graphs.

- Semantic-level emotion information aggregation and integration through robust extraction of social semantic knowledge graphs for emotion analysis.

- The MixedEmotions platform is developed and evaluated in the context of three Pilot Projects that are representative of a variety of data analytics markets: Social TV, Brand Reputation Management, and Call Centre Operations. Each of the companies involved in the pilot projects have specific innovation objectives as follows:

- Pilot Project I (Social TV): Expert System S.p.A. aims at extending its current Expert System COGITO API suite with added features for Social Multimedia Analytics namely in the COGITO Media API. MixedEmotions enables the inclusion of entity-level emotion and sentiment analysis (program, actor, sportsperson), related contents analytics (liked content, mentioned entities), multimedia media analytics (emotion and sentiment analysis on online videos), and Social TV (related social networks activity while watching TV programs).

- Pilot Project II (Brand Reputation Management): Paradigma Tecnológico is able to enhance its Online Reputation Tool in three areas: real-time capabilities, report generation, and multilingual data analysis. Together, these innovations will increase internationalization of the product and provide better reputation management services to customers.

- Pilot Project III (Call Centre Operations): Phonexia offers solutions for call centre operations that benefit from the MixedEmotions platform through advanced extraction of emotions from speech, and enhanced multilingual processing.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the current reporting period the following achievements were met in implementing the objectives as outlined above (corresponding to the MixedEmotions Milestone 1 “Startup” and 2 “Pilot Delivery - Cycle 1”):

Milestone 1 (M3) - Startup

- D1.6 Quality management plan delivered:

D1.6 implements the objective of defining the quality management plan for the project by month M1. The Quality Plan (QP) defines the general approach to quality assurance and procedures to be followed for beneficiary communication, documentation, deliverable production and research activity / software development. It describes:
Communication procedures (between beneficiaries).
How to produce reports, cost statements and deliverables as well as naming and versioning conventions along with formats to adopt.
How to review and distribution of the various types of deliverables: reports, prototypes and demonstrators.
How to apply technical quality control, risk assessment and contingency strategies
How to carry out decision-making and conflict resolution.
A general approach to software and hardware development quality standards.
The QP is intended to provide procedures, which support the objectives of:
On time and within work programme specification quality deliverables
Risk identification and work plan deviation
Taking preventative or corrective action at the earliest possible stage
Overall the QP represents an important project task as it is instrumental in monitoring and reporting on the project's achievements. Project quality assurance is the joint responsibility of all partners and will be applied to all levels of project activities.

- D1.7 IP management plan delivered:

D1.7 implements the objective of defining an IP management plan by month M3. The IP Management Plan (IPMP) defines the general approach to defining, implementing and monitoring the actions taken within the strategy for the knowledge management and protection from the beginning of the project to ensure the effective exploitation of MixedEmotions innovations. This deliverable states the original status of the different consortium partners’ contributions in terms of intellectual property (IP), licensing, and rights to use, and then states the different options in terms of property and licensing of the project results, keeping in line with the intended open source nature of the MixedEmotions project. Since it is still too soon to determine the exact modules and results that will be produced, this plan does not intend to establish a final list of MixedEmotions software components, with their corresponding IP and licenses; it is oriented to present the possible different interests and options that the industrial and non industrial consortium partners may have, and the plan to discuss options and reach a common agreement in terms of licensing and IP. The innovation management is very relevant for the success of the project and it will play an important role in the management structure and decision-making processes of MixedEmotions.

- D1.9 Data Management plan, initial version delivered:
D1.9 implements the objective of defining a data management strategy by month M3. The Data Management Plan (DMP) describes the data management life cycle for all data sets that will be collected, processed or generated by the MixedEmotions project. It outlines how research data will be handled during the project, and even after it is completed, describing what data will be collected, processed or generated and following what methodology and standards, whether and how this data will be shared and/or made open, and how it will be curated and preserved. As the DMP is not a fixed document, it will evolve and gain more precision and substance during the lifespan of the project; therefore the first versions will be necessarily incomplete.

- D2.1 Common methodology and infrastructure specification delivered:
D2.1 implements the objective of defining a common methodology and infrastructure specification by month M3. This deliverable defines a common playground for the specification of the three project pilots: Social TV, Brand Reputation Management, Call Center Operations. Each pilot was analyzed in a dedicated section providing a very preliminary description of: pilot use case(s), data availability, pilot requirements from MixedEmotions platform, evaluation strategies in the overall MixedEmotions project context. This deliverable is mainly based on the outcome of the project kick-off meeting in Galway and subsequent interactions with SMEs responsible for pilot development and technology providers. This document aims to provide a methodology and drafts an extended questionnaire (see Appendix I) to collect information needed to elicit, from different use cases analysis, a common design methodology to cover user requirements, software specifications, reference testing data and MixedEmotions platform requirements and interactions. This document defines a preliminary requirements template that has been instantiated and extended in deliverable D2.2.

- D2.2 Business scenarios and data selection for pilots delivered:
D2.2 implements the objective of defining a business scenario and the selection of data needed for the three pilots by month M3. This deliverable defines initial requirements and delivers specifications for the implementation of pilots in WP6 and requirements for the specification of the MixedEmotions platform in WP3 and for the realization of the infrastructure for emotion analysis in WP4. This deliverable starts from a pilot questionnaire outcomes and interactions among partners involved in pilot design and development and MixedEmotions platform technology providers during the June 2015 technical meeting at NUIG, as well as several dedicated conference calls and a bilateral meeting dedicated to Pilot I between ES and DW.

- D5.1 Initial data modelling for Social Semantic Knowledge Graph delivered:
D5.1 implements the objective of defining a model for the social semantic knowledge graph that covers multilinguality, multimodality and different types of emotions by M4. It is a fundamental characteristic of the MixedEmotions approach that it is powered by a live updated large-scale social and semantic web fed knowledge base (or “Social Semantic Knowledge Graph”), which is a semantically integrated representation of entities (e.g. people, events, locations, shows, products, services, etc.) extracted across content in different languages, enriched with social context of these entities (e.g. based on extraction/inclusion of social graphs) and emotion/sentiment on/by these entities. This deliverable is the initial report on the data modelling for the Social Semantic Knowledge Graph, which is meant to be the foundation for the integration during the first iteration of the project.

- D7.1 MixedEmotions Website Online:
D7.1 implements the objective of launching a website for the project by M1. This deliverable includes also the development of the project logo. Both are major elements of the communication strategy that has been described in full detail in D7.2 Communication Plan. The website has been launched on May 4th 2015 at The Mixed Emotions website is the main online communication channel for the project. It has been designed as a central information place with key information about recent developments and achievements of the project and news about the emotion technologies market. The website has been well maintained and is continuously updated with attended events and new publications. More than 2300 visits were registered throughout the course of the year, coming from all over Europe but also the United States, South America and Asia. The data on unique visitors per day shows that we are reaching a professional audience that is visiting the website primarily during office hours. Over weekends the usage of the website is significantly lower. Towards the fourth quarter of the year, the frequency of publications on the website has increased with monthly articles written by a partner and proofread and formatted by the communication leader DW.

- D7.2 Communication Plan delivered:
Deliverable 7.2 presents a strategy of how MixedEmotions is aiming to reach a relevant audience in business, academia and the general public. The purpose of this deliverable has been to provide an initial project communication plan by highlighting target groups and defining internal communication procedures and means. This includes online and offline media. MixedEmotions is a “lab to market” project with a runtime of two years. Therefore a special focus is set to reach a business audience, specifically within the industries covered by the three pilots: Social TV, Brand Reputation Management and Call Centre Operations.

- D7.5 Dissemination Plan delivered:
The dissemination activities described in D7.5 aim at: disseminating the results of the project at technology fairs, scientific conferences and publications; collaborating with other projects and providing contributions to selected standardisation bodies. The dissemination strategy followed in MixedEmotions consists of three fundamental steps, also referred to as independent strategies: Awareness, in which promotion of the project and its aims has to be pursued; Engagement, a compound of activities (e.g. presence on Social Coding sites) that aim at involving more developers and researchers in the development of the different tools and schemas of MixedEmotions, as well as getting them to use them and provide feedback; and finally Demonstration, whose intention is to show specific results of the project. The difference between Demonstration and Awareness is that the focus of the former is on results and dissemination of outcomes.

- D7.8 Exploitation Plan delivered:
The Exploitation Plan from Deliverable 7.8 presents the MixedEmotions Platform market landscape, briefly outlines the Pilots vertical markets, and assesses the project expected outcomes. We present a SWOT analysis of the MixedEmotions Platform and an AS IS - TO BE analysis to outline the impact of the project on SMEs current offerings. We provide a preliminary exploitation plan, based on an open source strategy, for Mixedemotions as an integrated Big Data platform for emotion analysis composed by a suite of existing tools and services provided by the partners and new tools developed ad-hoc for the use cases and pilots is provided. Moreover, we provide individual exploitation plans for industry and academic partners:
Industrial partners (mainly SMEs) will improve their content analytics solutions and emotion analysis services and roll them out to market.
Academic partners will address the “scientific market” as an attractive exploitation target and will identify also technology transfer activities and potential creation of spin-offs.
A dedicated section is devoted to industrial partners’ exploitation definition methodologies based on the CANVAS methodology.

- D7.11 Detailed training activities plan delivered:
D7.11 implements the objective of defining a clear plan for the planned training activities by month M3. The purpose of this deliverable has been to detail a plan of training activities that enables knowledge transfer from academy to industry. The training activities have been designed for the following users: industrial users, service developers and researchers. The plan includes two webinars to be organized after the main project milestones, M12 and M21. These webinars will cover the scientific and technological approaches to emotion recognition across languages and modalities, as well as their efficient large-scale integration into commercial offerings. In addition, the plan includes F2F training activities (summer schools, academic courses and tutorials).

Milestone 2 (M12) - Pilot Delivery, cycle 1

- D3.1 Architecture Specification and Platform Implementation, Initial Version delivered (M9):
D3.1 implements the objective of detailing the design of the MixedEmotions platform. This document deals with integration strategies, parallelization in Spark, integration strategies, a proposed format for module interoperability and a description of the modules included or used by the platform. Furthermore, it presents the pilots that are being developed in conjunction with the platform. It is worth noting that the proposed architecture aims to be compatible with each of the defined pilots in order to demonstrate the use of the MixedEmotions platform. Hence, as the main challenge of this task, this architecture addresses the integration of the modules provided by the project partners.

- D4.1 Emotion Recognition from Multilingual Audio Content, initial Version delivered (M9):
D4.1 implements the objective of detecting emotion from audio content. The approach is twofold: first we investigated and compared cross-lingual emotion recognition for “the same language”, “within language family”, and “between language families”; second, we proposed a transfer learning method to enhance emotion recognition between different languages (corpora) in order to reuse the knowledge gained for one language and apply it to another language.

- D4.3 Emotion Recognition from Multilingual Textual Content, initial version delivered (M9):
D4.3 reviews the state of the art in emotion recognition from text as well as emotion representation schemes. Completed sentiment analysis modules in multiple languages are presented as well as initial investigations into supervised and unsupervised emotion detection from text where a lack of quality annotated data was identified as a barrier to better performance. Initial efforts to obtain new quality annotated data for emotion detection models are described. Several approaches to the application of machine translation for cross-lingual model transfer and assessment methodologies for these approaches are presented as well.

- D4.5 Emotion Recognition from Image and Video Content delivered (M9) :
D4.5 reviews available tools, methods and datasets for extracting high-level information from images and videos which are relevant to emotion recognition. It targets two scenarios: interview-like situation in which a person is captured by a frontal camera, and analysis of general media content (specifically violence detection). The document focuses on face detection, facial part localization, head orientation estimation, and upper body part localization. It further analyzes an estimation of personal information (e.g. gender, age, race, and possibly personality), appearance-related information (e.g. facial hair, glasses), facial expressions and emotions. A novel deceit detection dataset is proposed and a pilot scenario together with pilot recordings are presented.

- D4.7 Adaptive Multilingual, Multimodal Fusion for Emotion Recognition, initial version delivered (M9):
D4.7 describes the initial version of the adaptive multimodal emotion recognition. It comprises the description of data, methods and results for i) multi-modal (audio/visual/physiology based) emotion recognition and ii) multi-modal (audio/video/text) sentiment recognition in videos with a speaker. Although the physiology modality is not part of MixedEmotions, the applied methodologies for the fusion could be transferred for the considered modalities (e.g., text).

- D4.9 Social Context Analysis for Emotion Recognition delivered (M9):
This deliverable describes the context of interactions in social networks and its analysis, with an emphasis on its use for Emotion Recognition tasks. It presents the architecture of a social context module that is used to analyse such contexts of emotions in social media, and its implementation, which includes state-of-the-art metrics of social context aspects,
such as influence. We describe a preliminary experiment with real data captured from Twitter that illustrates the extraction of temporal patterns of emotions in social media. Lastly, we touch on the main challenges to solved in the upcoming months, which will lead to the final version of this deliverable (D4.10).

-D3.4 Acceleration of Large-scale Emotion Analysis Methods, initial version delivered (M12):
This deliverable describes the strategies adopted for the optimization of the modules composing the MixedEmotions platform. To this end, an exhaustive analysis of the modules has been performed, revealing two different groups of platform components depending on their nature and specific attributes: non-distributed and distributed modules. Hence, distinct approaches have been designed to each of them in order to optimize their use (i.e., minimization of allocated resources and latency). For those modules in the former group, the integration in the platform has been proposed to optimize their use. This integration involves the definition of a standard format to exchange information in the platform, which is given by the JSON-LD format. In case of distributed modules, parallelization is adopted as the main acceleration strategy. This parallelization is implemented either by map-reduce programming or by software replication among the machines of the cluster. In both scenarios, Spark is employed as the tool to orchestrate the management of the data and the tasks to be executed on them.

- D5.2 Data Modelling for the Social Semantic Knowledge Graph, final version delivered (M12):
Deliverable D5.2 provides a refinement of deliverable D5.1. The core characteristic of the MixedEmotions approach is that at the base it is powered by a live updated large scale web fed knowledge base which we call “Social Semantic Knowledge Graph”. With this definition we mean the ability for the infrastructure inside MixedEmotions to deal with a semantically integrated representation of entities (e.g. people, events, locations, shows, products, services, etc.) extracted across content in different languages and modalities (i.e. audio/speech/image/video/text), enriched with social context of these entities (e.g. based on extraction/inclusion of social graphs) and emotion/sentiment on/by these entities. Moreover, and more importantly this is a stream of data, not only a “Static Snapshot” so the infrastructure, e.g. as reflected by the D5.3 deliverable, as well as the data models here illustrated were selected for their ability to address changing information. First, we go through the vocabularies as extracted from the data sources (Wikipedia) and as actually used in the pilots that are being built. Then we focus on the core innovation of the second part of the project - in terms of data structures and big data infrastructure which has been the steering from more academic “RDF only” tools to a blend of others, JSON and Elasticsearch being the most important that are considerably popular among developers and provide a very solid backbone in term of Scalability and “on the fly operations” compared to previous systems. This deliverable therefore details efforts to make the “Data Model” the conceptual centerpiece of Elasticsearch/JSON solutions again via mixed consideration on how to build a “knowledge graph” from a set of indexes and how the statistical/on the fly tools available within Elasticsearch (the Elasticsearch graph capabilities) contribute to the MixedEmotions goals.

- D5.3 Social Semantic Knowledge Graph Infrastructure and API, initial version delivered (M12):
This is the first deliverable of WP5 in describing the API and approach toward providing Semantic Graph capabilities and Socially derived data in the platform of Mixedemotions. The core part of the contribution is the description of how we extended Elasticsearch - which would out of the box only handle typically homogeneous streams of data - into being able to operate on “relationally” and ultimately “graph” data models. By achieving this, we reach a very powerful objective: advanced relational/graph data analytics and search while operating on an ultra scalable, big data cluster infrastructure. To this end we have implemented two core APIs. The Siren Join API - a methodology and language for very high performance relational joins across Elasticsearch indexes. This is core to answering questions like “What are the core emotions associated with comments related to politicians of party X?” (where indexes of comments and indexes of data about politicians are distinct). The Kibi Gremlin interface - this is an implementation of the Gremlin Knowledge graph query language on top of the Kibi infrastructure and therefore capable of questions like “What is the closest path - e.g. fact of news - that connects different emotions”

- D6.1 Social TV Pilot Implementation, initial version delivered (M12):
D6.1 describes the technical and functional features of the first implementation of Pilot I on Social TV. It starts from the guidelines defined in D2.2 Business Scenario Development & Data selection, that defines initial requirements and delivers specifications for the implementation of pilots in WP6 and requirements for the specification of the MixedEmotions platform. Although in this first year several results were analyzed, the requirements of the first months have undergone a significant evolution to follow new business scenarios. In order to define the objectives of the first version of Pilot 1 in the first section of this deliverable a Graphical User Interface for Pilot I is described. Similarly, in the second section of the deliverable, the back-end for Pilot I is shown with the architecture and pipelines, in order to clarify its structure and the interaction with modules of the Mixedemotions Platform.

- D6.3 Brand Reputation Management Pilot Implementation, initial version delivered (M12):
D6.3 covers the first version of the Brand Reputation Management pilot, which focuses on analyzing the emotions around mentions of target brands in various internet sources and social media. This pilot relies on emotion extraction and sentiment analysis to generate graphical reports about how the brand is perceived online, giving the brand owner the insight needed to evaluate its online presence. The deliverable introduces the implementation, functionality and visualization for this pilot.

- D6.5 Call Centre Operations Pilot Implementation, initial version delivered (M12):
D6.5 describes the Pilot 3 scenario which deals with Call Centre Operations. The deliverable describes the Phonexia Platform for Call Centres, the Speech Analytic Platform (SPAS), including an introduction to its functionality, implementation and visualization of emotion recognition modules. It also addresses related issues, like a REST server for calling specific modules, gender and age recognition for better analysis and conditioning the output and Speech To Text advances in Phonexia.

- D7.3 Communication Report, initial Version delivered (M12):
D7.3 describes the communication activities and results of the MixedEmotions project of the first year. Besides the development of a corporate design which is the basis for all produced promotional material (Logo, Website, Flyer, Poster, Roll-up banner) these activities - amongst others - include the frequent population of the website and the twitter channel with project news, articles, events and results. Considering the variety of MixedEmotions activities and the received positive feedback it can be stated that the communication planning from M3 enabled a successful and effective project communication campaign which was geared to inform stakeholders about the project and its goals.

- D7.6 Dissemination Report, initial Version delivered (M12):
This deliverable covers the dissemination activities taken in the first half of the project. These activities follow the plan established in D7.5 Dissemination Plan. So far, the awareness and engagement parts of the plan have been carried out, with multitude of scientific events and publications in which MixedEmotions has been represented by one or more of the partners. The rest of the activities will be included in D7.7. This upcoming phase will focus on the dissemination of results obtained in the project, such as the open source big data platform for emotion analysis.

- D7.9 Exploitation Report, initial Version delivered (M12):
This deliverable presents the status of the exploitation related activities at the end of the first year of the project. It reports the outcomes of the pilots’ SME vertical market analysis and describes the current status of exploitable results, performing a SWOT analysis on the MixedEmotions Platform and an As-Is & To Be analysis on Pilot outcomes to determine the impact of each of them on the current offering of the SME partners in the project. Moreover, it presents the updated and revised mid-term exploitation plan at consortium level and the updated individual exploitation plans for industrial and academic partners.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The MixedEmotions project, by performing a seamless technology transfer and industrialization process on social semantic, multilingual and multimodal research outcomes and prototypes, makes available in the market several innovative functionalities presently not available except through the MixedEmotions platform. By doing so, it will enable the European content analytics industry and above all SMEs that are a driving force of this market and generally cannot afford to finance foundational research on such topics to build Big Data emotion analysis solutions for European content. Moreover, the MixedEmotions platform will enable service and product innovation across current offerings of the consortium SMEs by allowing a fine-grained and semantically integrated large-scale analysis of emotion in customer feedback and other relevant content streams, across different European languages, content modalities (audio, video, text, social media) and including open and linked data sources.

Emotion detection in digital media is a challenging area with many open research questions. The MixedEmotions project has extended the state-of-the-art in many aspects of emotion detection. For acoustic emotion recognition we developed an end-to-end deep neural network based approach where the feature extraction step is omitted from the classification process. Furthermore, using canonical correlation analysis to make the distribution of different corpora similar to each other, we improved classification accuracy for multilingual audio emotion recognition. Additionally, through fusion of audio and video, either at the feature level or fusion level, we could yield higher performance for emotion recognition.

For emotion detection from video, we extended the state of the art in several detection tasks relevant to emotion detection. These include face detection, facial landmarks, facial expressions (neutral, smile, sad, angry, surprise), face alignment, head orientation, upper body part localization, personal traits (e.g. age, gender, race, facial hair, hair style, eye wear), head movement, body pose, and body movement. Improvements in these areas has been achieved by applying an assortment of classifiers including random forests, AdaBoost and convolutional neural networks (CNNs). These tasks form the building blocks for deceit and emotion analysis from video. We achieved beyond state-of-the-art performance in deceit and emotion analysis by their combination.

For emotion detection from text, we have developed methodologies for building and extending linguistic resources for poorly resourced languages, including the transfer of emotion lexicons to those languages, enabling a basic level of automated emotion detection where this was previously not possible. Emotion detection from text in general suffers from a lack of ground truth data for evaluation and model training, which we addressed as well through a substantial emotion annotation effort. In addition to emotion detection, MixedEmotions has developed novel suggestion mining methods and tools.

Related information

Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top