CORDIS

EU research results

CORDIS

Social Semantic Emotion Analysis for Innovative Multilingual Big Data Analytics Markets

Fact Sheet

Reporting

Results

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Ireland

Objective

MixedEmotions will develop innovative multilingual multi-modal Big Data analytics applications that will analyze a more complete emotional profile of user behavior using data from mixed input channels: multilingual text data sources, A/V signal input (multilingual speech, audio, video), social media (social network, comments), and structured data. Commercial applications (implemented as pilot projects) will be in Social TV, Brand Reputation Management and Call Centre Operations. Making sense of accumulated user interaction from different data sources, modalities and languages is challenging and has not yet been explored in fullness in an industrial context. Commercial solutions exist but do not address the multilingual aspect in a robust and large-scale setting and do not scale up to huge data volumes that need to be processed, or the integration of emotion analysis observations across data sources and/or modalities on a meaningful level. MixedEmotions will implement an integrated Big Linked Data platform for emotion analysis across heterogeneous data sources, different languages and modalities, building on existing state of the art tools, services and approaches that will enable the tracking of emotional aspects of user interaction and feedback on an entity level. The MixedEmotions platform will provide an integrated solution for: large-scale emotion analysis and fusion on heterogeneous, multilingual, text, speech, video and social media data streams, leveraging open access and proprietary data sources, and exploiting social context by leveraging social network graphs; semantic-level emotion information aggregation and integration through robust extraction of social semantic knowledge graphs for emotion analysis along multidimensional clusters.

Field of Science

/humanities/languages and literature/linguistics/phonetics

/humanities/languages and literature/languages - general

/natural sciences/computer and information sciences/data science/big data

/natural sciences/computer and information sciences/data science/data analysis

Programme(s)

H2020-EU.2.1.1.4. - Content technologies and information management: ICT for digital content, cultural and creative industries

Topic(s)

ICT-15-2014 - Big data and Open Data Innovation and take-up

Call for proposal

H2020-ICT-2014-1

See other projects for this call

Funding Scheme

IA - Innovation action

Coordinator

NATIONAL UNIVERSITY OF IRELAND GALWAY

Address

University Road
H91 Galway

Ireland

Activity type

Higher or Secondary Education Establishments

EU Contribution

€ 557 775

Website

Contact the organisation

Participants (9)

Sort alphabetically

Sort by EU Contribution

Expand all

UNIVERSIDAD POLITECNICA DE MADRID

Spain

EU Contribution

€ 291 125

UNIVERSITAT PASSAU

Germany

EU Contribution

€ 343 687,50

EXPERT SYSTEM SPA

Italy

EU Contribution

€ 279 354

PARADIGMA DIGITAL SL

Spain

EU Contribution

€ 296 625

VYSOKE UCENI TECHNICKE V BRNE

Czechia

EU Contribution

€ 353 637,50

SINDICE LIMITED

Ireland

EU Contribution

€ 331 852,50

DEUTSCHE WELLE

Germany

EU Contribution

€ 341 035

PHONEXIA SRO

Czechia

EU Contribution

€ 241 818,50

MILLWARD BROWN CZECH REPUBLIC SRO

Czechia

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Ireland

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Ireland

Periodic Reporting for period 2 - MixedEmotions (Social Semantic Emotion Analysis for Innovative Multilingual Big Data Analytics Markets)

Reporting period: 2016-04-01 to 2017-03-31

Summary of the context and overall objectives of the project

Effective automated emotion analysis enables a better understanding of user and customers’ behavior and needs, allowing businesses to adapt to real demand and usage, with great potential for real economic gains and improved user experience. Emotions can be detected and tracked through user interaction in a large variety of forms, such as explicit feedback through email, call center interaction, social media comments, etc., and can include implicit acknowledgment of approval or rejection through facial expressions, speech or other non-verbal feedback. The information can be found in speech, text, or audio-visual content.
One ambitious goal of MixedEmotions is to tackle these multiple “mixed” data source and modality challenges. Another ambition is to provide services in multiple languages, to open up to a wider audience. A third important challenge is to handle large amounts of data as is often required in customer service and user feedback industries. Different aspects and facets of the technologies developed in MixedEmotions will thus serve different targets and users of those technologies will be able to build a wide range of applications that serve diverse use cases, in diverse industries.
MixedEmotions produced an Integrated Big Linked Data Platform for Emotion Recognition, tackling all the challenges mentioned above, which are not currently fully addressed by the commercial solutions on the market. The MixedEmotions Platform is free, open source, and accessible online via http://mixedemotions.insight-centre.org/.
The MixedEmotions platform has been developed and evaluated in the context of three Pilot Projects that are representative of a variety of data analytics markets: Social TV, Brand Reputation Management, and Call Centre Operations. In each case, the relevant industry partner having reached their specific innovation objectives.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

MixedEmotions project and platform have been developed around three use cases for which business scenarii were established in the early stages of the project through consultation between industry and academic partners.
A flexible Micro-services architecture was chosen to be followed for the MixedEmotions platform, allowing it to act both as a toolbox in which components could be used separately, and as a Big Data analysis tool, with the ability to distribute computing resources across many physical machines and to manage sophisticated data analysis pipelines.
After agreeing on which representations for the concept of emotion would be used, modules were developed for each modality (text, audio, video, social media).
A two dimensional emotion scheme (arousal and valence) was used for emotion detection in audio. Cross-lingual recognition was included in the module as well as detection of age, gender and personality. For text, modules are provided for recognition of sentiment and emotion for several languages. Both categorical and dimensional emotion schemes are used, and a crowd-sourced emotion annotated corpus was collected. In parallel, a multilingual WordNet-Affect lexicon in 23 European languages was developed, providing baseline emotion detection capabilities in those languages.
In the image and video modalities, face-based classifiers, including emotional facial expressions, were implemented with capability to detect multiple faces as well as age and gender.
Finally, a fusion model has been built for multimodal emotion recognition.
Tools and interfaces for search and analytics on mixed data enriched with emotions have been developed, providing big data analysis and semi-structured knowledge graph capabilities and APIs. The social semantic knowledge graph carries structured information about named entities enriched with emotions. A platform of analysis for social media and social context using graph analytics has also been developed.
The three pilots have been successfully implemented through the project. Two use cases were identified in the Social TV pilot: providing an emotion-based recommender system in Smart TV, and an Editorial Dashboard in which emotions associated with DW RSS News can be visualized alongside their corresponding tweets. The Brand Reputation management pilot can now monitor brands in social networks, including in videos, provide a multimodal emotion recognition analysis, and a visualizations of the results. As for the third pilot, dealing with Call Centre Operations, the project allowed Phonexia to highlight potentially problematic calls through the detection of emotional valence and arousal variation in the speech and provide rich feedback on customer satisfaction.
The communication and dissemination strategies were conducted successfully, and MixedEmotions was present and advertised through a wide range of material (video, posters, flyers, etc.), media (website, Twitter, LinkedIn, etc.), and events (webinars, tutorial, conferences, etc.). Concerning exploitation of project outcomes, the components made available by the platform provide additional technologies that enabled the industry partners (and by extension other SMEs) to improve their business processes by providing emotion analysis decision support tools, enhancing their offering and targeting new markets and customers. The industrial partners finalized their market analyses for the vertical markets associated to each pilot by selecting early customers installations and commercial piloting.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The MixedEmotions project now enables the European content analytics industry to build emotion analysis solutions. This is especially true for SMEs, which are a driving force of this market and generally cannot afford to finance core research and development in the area.
Moreover, the MixedEmotions platform enabled innovations in services and products currently offered in the consortium SMEs, giving them access to fine-grained and semantically integrated large-scale analysis of emotion in customer feedback and other relevant content streams, across different European languages, content modalities (audio, video, text, social media) and including open and linked data sources.
Although MixedEmotions is an innovation action, most components integrated into the platform haven been extended with capabilities exceeding the state of the art in their respective fields.
For acoustic emotion recognition, a specific end-to-end deep neural network based approach new to the audio processing domain was developed. Furthermore, using canonical correlation analysis, we improved classification accuracy for multilingual audio emotion recognition.
For emotion detection from video, the state of the art was extended in several detection tasks relevant to emotion detection, including facial expressions (neutral, smile, sad, angry, surprise). We achieved beyond state-of-the-art performance in deceit and emotion analysis by the combination of these tasks.
For emotion detection from text, we have developed methodologies for building and extending linguistic resources for poorly resourced languages, including the transfer of emotion lexicons to those languages. The lack of ground truth data for evaluation and model training was addressed through a substantial emotion annotation effort. In addition to emotion detection, MixedEmotions has developed novel suggestion mining methods and tools.
Finally, through fusion of text, audio and video analysis, we could yield higher performance for emotion recognition.

Basic Architecture of the MixedEmotions Platform

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Ireland

Project information

MixedEmotions

Grant agreement ID: 644632

Project website

Start date

1 April 2015
End date

31 March 2017

Funded under:

H2020-EU.2.1.1.4.

Overall budget:

€ 3 529 617,14
EU contribution

€ 3 036 910

Coordinated by:

NATIONAL UNIVERSITY OF IRELAND GALWAY

Download

PDF

XML

Last update: 24 June 2019

Record number: 194226

Permalink: https://cordis.europa.eu/project/rcn/194226/en

CORDIS

Project information

Objective

Coordinator

Participants (9)

Project information

Project information

Periodic Reporting for period 2 - MixedEmotions (Social Semantic Emotion Analysis for Innovative Multilingual Big Data Analytics Markets)

Project information

Project information

Deliverables

Publications

Project information

Project information

Project information

Download

CORDIS

Project information

Objective

Coordinator

Participants (9)

Project information

Project information

Periodic Reporting for period 2 - MixedEmotions (Social Semantic Emotion Analysis for Innovative Multilingual Big Data Analytics Markets)

Project information

Project information

Deliverables

Publications

Project information

Project information

Project information

Share this page

Download