MixedEmotions project and platform have been developed around three use cases for which business scenarii were established in the early stages of the project through consultation between industry and academic partners.
A flexible Micro-services architecture was chosen to be followed for the MixedEmotions platform, allowing it to act both as a toolbox in which components could be used separately, and as a Big Data analysis tool, with the ability to distribute computing resources across many physical machines and to manage sophisticated data analysis pipelines.
After agreeing on which representations for the concept of emotion would be used, modules were developed for each modality (text, audio, video, social media).
A two dimensional emotion scheme (arousal and valence) was used for emotion detection in audio. Cross-lingual recognition was included in the module as well as detection of age, gender and personality. For text, modules are provided for recognition of sentiment and emotion for several languages. Both categorical and dimensional emotion schemes are used, and a crowd-sourced emotion annotated corpus was collected. In parallel, a multilingual WordNet-Affect lexicon in 23 European languages was developed, providing baseline emotion detection capabilities in those languages.
In the image and video modalities, face-based classifiers, including emotional facial expressions, were implemented with capability to detect multiple faces as well as age and gender.
Finally, a fusion model has been built for multimodal emotion recognition.
Tools and interfaces for search and analytics on mixed data enriched with emotions have been developed, providing big data analysis and semi-structured knowledge graph capabilities and APIs. The social semantic knowledge graph carries structured information about named entities enriched with emotions. A platform of analysis for social media and social context using graph analytics has also been developed.
The three pilots have been successfully implemented through the project. Two use cases were identified in the Social TV pilot: providing an emotion-based recommender system in Smart TV, and an Editorial Dashboard in which emotions associated with DW RSS News can be visualized alongside their corresponding tweets. The Brand Reputation management pilot can now monitor brands in social networks, including in videos, provide a multimodal emotion recognition analysis, and a visualizations of the results. As for the third pilot, dealing with Call Centre Operations, the project allowed Phonexia to highlight potentially problematic calls through the detection of emotional valence and arousal variation in the speech and provide rich feedback on customer satisfaction.
The communication and dissemination strategies were conducted successfully, and MixedEmotions was present and advertised through a wide range of material (video, posters, flyers, etc.), media (website, Twitter, LinkedIn, etc.), and events (webinars, tutorial, conferences, etc.). Concerning exploitation of project outcomes, the components made available by the platform provide additional technologies that enabled the industry partners (and by extension other SMEs) to improve their business processes by providing emotion analysis decision support tools, enhancing their offering and targeting new markets and customers. The industrial partners finalized their market analyses for the vertical markets associated to each pilot by selecting early customers installations and commercial piloting.
