Community Research and Development Information Service - CORDIS

H2020

SEWA Report Summary

Project ID: 645094
Funded under: H2020-EU.2.1.1.4.

Periodic Reporting for period 1 - SEWA (Automatic Sentiment Estimation in the Wild)

Reporting period: 2015-02-01 to 2016-01-31

Summary of the context and overall objectives of the project

The overall aim of the SEWA project is to enable computational models for machine analysis of facial, vocal, and verbal behaviour in the wild. This is to be achieved by capitalising on the state-of-the-art methodologies, adjusting them, and combining them to be applicable to naturalistic human-centric human-computer interaction (HCI) and computer-mediated face-to-face interaction (FF-HCI). The target technology uses data recorded by a device as cheap as a web-cam and in almost arbitrary recording conditions including semi-dark, dark and noisy rooms with dynamic change of room impulse response and distance to sensors. It represents a set of audio and visual spatiotemporal methods for automatic analysis of human spontaneous (as opposed to posed and exaggerated) patterns of behavioural cues including analysis of rapport, mimicry, and sentiment such as liking and disliking.

In summary, the objectives of the SEWA project are:
(1) development of technology comprising a set of models and algorithms for machine analysis of facial, vocal and verbal behaviour in the wild,
(2) collection of the SEWA database being a publicly available benchmark multilingual dataset of annotated facial, vocal and verbal behaviour recordings made in-the-wild representing a benchmark for efforts in automatic analysis of audio-visual behaviour in the wild,
(3) deployment of the SEWA results in both mass-market analysis tools based on automatic behaviour-based sentiment analysis of users towards marketed products and a sentiment-driven recommendation engine, and
(4) deployment of the SEWA results in a novel social-network-based FF-HCI application – sentiment-driven Chat Social Game.

The SEWA project is expected to have many benefits. Technologies that can robustly and accurately analyse human facial, vocal and verbal behaviour and interactions in the wild, as observed by webcams in digital devices, would have profound impact on both basic sciences and the industrial sector. They could open up tremendous potential to measure behaviour indicators that heretofore resisted measurement because they were too subtle or fleeting to be measured by the human eye and ear. They would effectively lead to development of the next generation of efficient, seamless and user-centric human-computer interaction (affective multimodal interfaces, interactive multi-party games, and online services). They would have profound impact on business (automatic market research analysis would become possible, recruitment would become green as travels would be reduced drastically), and they could enable next generation healthcare technologies (remote monitoring of conditions like pain, anxiety and depression), to mention but a few examples.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

"WP1 – SEWA DB collection, annotation and release
• Obtained ethical approval for the SEWA experiment.
• Designed the SEWA experiment protocol and implemented the data collection website.
• Conducted 199 successful data recording sessions using the aforementioned website. A total of 398 participants from 6 different cultural backgrounds (British, German, Hungarian, Serbian, Greek and Chinese) were recorded, resulting in more than 44 hours of audio-visual corpus covering a wide range of spontaneous expressions of emotions and sentiment.
• Extracted the low-level acoustic features (ComParE and GeMAPSv01a) from all SEWA recordings.
• Automatically tracked the 49 facial landmarks in all SEWA recordings. These results will be further refined through semi-automatic correction.
• Identified a total of 540 representative segments (high/low arousal, high/low valence, and liking/disliking; in total 90 segments per culture group) from the SEWA corpus. These segments – titled “the core SEWA dataset” – will be annotated fully in terms of facial landmarks, vocal and verbal cues, facial action units (FAUs), continuously valued emotion dimensions (valence and arousal), mimicry, sentiment, rapport, and template behaviours.
• Released the SEWA database version 0.1 internally, as according to the data management plan, and prepared the web-portal and EULA for subsequent public database release.

WP2 – Low-level Feature Extraction
• Implementation and evaluation of a software tool openWord to generate Bag-of-Audio-Words (BoAW) representations from acoustic low-level descriptors (LLDs) for robust acoustic features.
• Feature enhancement by deep neural networks to improve acoustic features computed from noisy speech signals.
• Cross-corpus emotion analysis, i.e., testing models for emotion analysis on languages which are not included in the training data.
• Implementation of the incremental in-the-wild face alignment method for automatic facial landmark localisation.
• Generation of multi-lingual dictionaries for BoAW representations with multi-databases in different languages.
• Application of the state-of-the-art of linguistic features employed in text retrieval to the sentiment analysis task.

WP3 – Mid-level feature extraction
• Existing, state-of-the-art tracking algorithm has been used for extracting the features such as facial landmarks, 3D head pose, nods and tilts (WP3.1).
• Three methods for Facial Action Unit (AU) detection and intensity estimation have been developed (WP3.2).
• The methods for AU detection were trained and tested on two publicly available datasets of naturalistic facial behaviour coded in terms of AU intensity. On both datasets the proposed methods improved the state-of-the-art in automatic AU detection and intensity estimation.

WP4 – Continuous Affect and Sentiment Sensing in the Wild
• Work on WP4 will start in M15 (April 2016).

WP5 – Behaviour Similarity in the Wild – start M12 end M30
• Work on WP5 will start at the end of M12 (from 1st February 2016).

WP6 – Temporal Behaviour-Patterning and Interpersonal Sentiment in the Wild
• Work on WP6 started just in M12 (January 2016).

WP7 – Integration, Applications and Evaluation

PlayGen have advanced the definition and design of the Chat Social Game.
• Refined and tested the concept underlying the Chat Social Game so that it is focused on a practical application with potential social and financial benefit.
• Clarified target user group and signed up 3 universities as partners to support user recruitment.
• Carried out 2 focus groups to define user needs.
• Developed initial game design concepts and mockups.
• Implemented initial prototype two-player chat-based game for debating called Sumobate.
• Progressed core technical functionality and advanced technical integration discussions.
• Planned evaluation approach.

RealEyes has focused on three major activities. First, they redefined targeted fields and the potential use of recommender engine with sentiment analysis support. Second, they worked on the computational framework that allows for testing ideas and methods for linking sentiment and emotion analysis with ad placement recommendation. Third, they built connections with different industrial players who could benefit from the SEWA results. In particular, they:
• Analysed the business interest in the use of sentiment analysis enhanced recommendations and found that the market of recommender systems is quite populated.
• Identified an already strong and increasing interest in media inventory optimization and online advertising.
• Initiated collaboration with potential future partners to help define target groups and clarify their needs.
• Obtained advert performance data from their partners which, in conjunction with social media performance and user rating, will be used for the work required to build and test the recommender engine. These profiles constitute an important part of the concept for using sentiment and emotion analysis for recommendation in an effective way.
• Implemented a first version for validating correspondences between emotion and (future) sentiment analysis and the quality of advertisements.
• Initial studies were conducted on modelling and clustering user and advert emotion profiles.
• Ran the first set of statistical analysis on signals derived from behavioural observations.
• Progressed core technical functionality and advanced technical integration discussions.
• Together with Imperial College London lead the organization of the Valorisation Board.

WP8 – Dissemination, Ethics, Communication and Exploitation
Workshops:
• The 300-VW 2015 (300 Videos in the Wild) – Facial Landmark Tracking in-the-wild Challenge & Workshop was organised as a satellite event of the IEEE International Conference on Computer Vision (ICCV 2015) in Santiago, Chile, in December 2015. SEWA sponsored the Prizes for the two winners of the challenge (USD 200 per winner; in total USD 400). (http://ibug.doc.ic.ac.uk/resources/300-VW/).
• The IBM-SEWA Cognitive Workshop 2015 was organised as a joint event between the IBM and the SEWA consortium with the aim to cross-fertilise the ideas on the state of affairs in cognitive computing and the future of it. The workshop was held in conjunction with the SEWA plenary meeting and the SEWA Valorisation Board meeting in October 2015, in London. (http://sewaproject.eu/ibmsewa15).
• The FERA 2015 (Facial Expression Recognition and Analysis Challenge) in facial action unit and facial expression detection in unconstrained images and videos was organized for the IEEE 11th International Conference on Automatic Face and Gesture Recognition in Ljubljana, Slovenia, in May 2015. SEWA sponsored 3 best paper prizes. (EUR 150 per winner, in the line with the prizes awarded at 2011 edition of FERA; in total EUR 450) (http://sspnet.eu/fera2015/).
• The WASA 2015 Workshop on Automatic Sentiment Analysis in the Wild was organized as a satellite event at the AAAC/IEEE 6th International Conference on Affective Computing and Intelligence Interaction (ACII 2015) in Xi'an, China, in September 2015. SEWA sponsored the Keynote Speaker (Jeffrey Cohn, USD 1000) and the Best Paper Award (USD 100). (http://sewaproject.eu/wasa15).
• The AV+EC 2015 (Audio/Visual + Emotion Challenge and Workshop) in the field of audio-visual behaviour understanding in the wild was organized for the ACM International Conference in Multimedia in Brisbane, Australia, in October 2015. This was the 5th AV+EC workshop so far and it includes physiological data for the first time. (http://sspnet.eu/avec2015/). SEWA sponsored the workshop by helping the data annotation.

Publications:
• “Sparkle: Adaptive Sample Based Scheduling for Cluster Computing”, C. Hao, J. Shen, H. Zhang, X. Zhang, Y. Wu, M. Li. In Proceedings of the 5th International Workshop on Cloud Data and Platforms (CloudDP), satellite event to EuroSys 2015.
• “Neural Conditional Ordinal Random Fields for Agreement Level Estimation”, N. Rakicevic, O. Rudovic, S. Petridis and M. Pantic. In Proceedings of the 1st International Workshop on Automatic Sentiment Analysis in the Wild (WASA), satellite event to ACII 2015.
• “Sentiment Apprehension in Human-Robot Interaction with NAO”, J. Shen, O. Rudovic, S. Cheng and M. Pantic. In Proceedings of the 1st International Workshop on Automatic Sentiment Analysis in the Wild (WASA), satellite event to ACII 2015.
• “Cross-Language Acoustic Emotion Recognition: An Overview and Some Tendencies”, S. Feraru, D. Schuller, and B. Schuller. In Proceedings of the 6th Bi-annual Conference on Affective Computing and Intelligent Interaction (ACII) 2015.
• “Detection of Negative Emotions in Speech Signals Using Bags-of-Audio-Words”, F. Pokorny, F. Graf, F. Pernkopf and B. Schuller. In Proceedings of the 1st International Workshop on Automatic Sentiment Analysis in the Wild (WASA), satellite event to ACII 2015.
• “Face Reading from Speech – Predicting Facial Action Units from Audio Cues”, F. Ringeval, E. Marchi, M. Mehu, K. Scherer, and B. Schuller. In Proceedings of INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association (ISCA), 2015.
• “Modelling User Affect and Sentiment in Intelligent User Interfaces”, B. Schuller. In Proceedings of the 20th ACM International Conference on Intelligent User Interfaces (IUI), 2015.
• “The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data”, F. Ringeval, B. Schuller, M. Valstar, S. Jaiswal, E. Marchi, D. Lalanne, R. Cowie, and M. Pantic. In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge (AVEC), satellite event to ACM-MM 2015.
• “The 5th International Audio/Visual Emotion Challenge and Workshop”, F. Ringeval, B. Schuller, M. Valstar, R. Cowie, and M. Pantic. In Proceedings of the 23rd ACM International Conference on Multimedia (ACM-MM), 2015.
• “Speech Analysis in the Big Data Era”, B. Schuller. In Proceedings of the 18th International Conference on Text, Speech and Dialogue (TSD), satellite event of INTERSPEECH 2015, Lecture Notes in Artificial Intelligence (LNAI), Springer, 2015.
• “Variable-state Latent Conditional Random Fields for Facial Expression Recognition and Action Unit Detection”, R. Walecki, O. Rudovic, V. Pavlovic, M. Pantic. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2015.
• “Latent Trees for Estimating Intensity of Facial Action Units”, S. Kaltwang, S. Todorovic, M. Pantic. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
• “Second Facial Expression Recognition and Analysis Challenge”, Michel F. Valstar, Timur Almaev, Jeffrey M. Girard, Gary McKeown, Marc Mehu, Lijun Yin, Maja Pantic and Jeffrey F. Cohn. In Proceedings of the Automatic Face and Gesture Recognition (FG), 2015.
• “Fast and Exact Bi-Directional Fitting of Active Appearance Models”, J. Kossaifi, G. Tzimiropoulos, M. Pantic. In Proceedings of the Facial Expression Recognition and Analysis Challenge (FERA), satellite event to FG 2015.
• “Robust Statistical Face Frontalization”, C. Sagonas, Y. Panagakis, S. Zafeiriou, M. Pantic. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
• “Multi-conditional Latent Variable Model for Joint Facial Action Unit Detection”, S. Eleftheriadis, O. Rudovic, M. Pantic. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
• “Prediction based audio-visual fusion for classification of non-linguistic vocalisations”, S. Petridis and M. Pantic. IEEE Transactions on Affective Computing, accepted for publication, 2015.
• “Probabilistic Slow Features for Behavior Analysis” L. Zafeiriou, M. A. Nicolaou, S. Zafeiriou, S. Nikitidis, M. Pantic. IEEE Transactions on Neural Networks and Learning Systems, accepted for publication, 2015.
• “Robust Correlated and Individual Component Analysis”, Y. Panagakis, M.A. Nicolaou, S. Zafeiriou, M. Pantic. IEEE Transactions on Pattern Analysis & Machine Intelligence, accepted for publication, 2015.
• “Discrimination Between Native and Non-Native Speech Using Visual Features Only”, C. Georgakis, S. Petridis, M. Pantic. IEEE Transactions on Man and Cybernetics, accepted for publication 2015.
• “Doubly Sparse Relevance Vector Machine for Continuous Facial Behavior Estimation”, S. Kaltwang, S. Todorovic, and M. Pantic. IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted for publication 2015.

Public Presentations:
• (Jan 2016) SEWA coordinator speaking at Nature Magazine’s Ideas Lab at WEF in Davos.
https://www.youtube.com/watch?v=ZHxsRpd0XjI&index=1&list=PL7m903CwFUgkv0-OOrqJaZNgO4_AhWJWG
• (Jan 2016) SEWA coordinator speaking at WEF in Davos
Prof. Maja Pantic spoke at World Economic Forum in Davos on Emotional AI.
https://webcasts.weforum.org/widget/1/davos2016?p=1&pi=1&hl=english&id=76123
• (Dec 2015) Dr Ognjen Rudovic and Dr Jie Shen (ICL) present SEWA technology at the Science Fair in Belgrade Serbia (a public event attended by 20,000+ children from elementary and high schools):
http://festivalnauke.org/Program/Beogradski-sajam/Humaniji-nego-sto-mislite
• (Mar 2015) SEWA coordinator speaking at Royal Academy
Prof. Maja Pantic spoke of SEWA and iBUG research at Royal Academy Event.
https://www.royalacademy.org.uk/event/ra-schools-spring-symposium

Scientific Talks:
• (Nov' 2015) SEWA UP PI, Björn Schuller speaking at LIRIS Workshop on Emotion, Ecole Centrale de Lyon/Technicolor in an invited talk on the topic “Sound Affects”.
• (Nov' 2015) SEWA coordinator speaking at ACPR 2015 as a Keynote speaker.
http://acpr2015.org
• (Oct' 2015) SEWA UP PI, Björn Schuller speaking at “Orange Hour”, GfK, Hamburg, in an invited talk on the topic “Emotions in the Voice: Making them Accessible for Consumer Research”.
• (Oct' 2015) SEWA UP PI, Björn Schuller speaking at Cognitive Computing Workshop 2015 as a Keynote speaker.
http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/engineering/computing/eventssummary/event_16-9-2015-12-8-6
• (Sep' 2015) SEWA coordinator speaking at ACII 2015 as a Keynote speaker.
http://www.acii2015.org
• (Sep' 2015) SEWA UP PI, Björn Schuller speaking at 18th International Conference on Text, Speech and Dialogue (TSD 2015) as a Keynote speaker.
http://www.kiv.zcu.cz/tsd2015/index.php?page=speakers
• (Sep' 2015) SEWA coordinator speaking at ECMR 2015 as a Keynote speaker.
https://lcas.lincoln.ac.uk/ecmr15/?q=node/1
• (Sep' 2015) SEWA UP PI, Björn Schuller speaking at Annual Meeting of the New Champions 2015 (AMNC, ""Summer Davos""), BetaZone Session, World Economic Forum (WEF) as a Keynote speaker.
https://agenda.weforum.org/news/world-economic-forum-honours-its-2015-young-scientists-community-at-annual-meeting-of-the-new-champions/
• (Sep' 2015) SEWA UP PI, Björn Schuller speaking at International Symposium on Companion-Technology (ISCT 2015, Expert Workshop of the SFB-TR 62) as a Keynote speaker.
https://isct2015.informatik.uni-ulm.de/wp-content/uploads/2015/09/ISCT2015Program.pdf
• (Avg' 2015) SEWA UP PI, Björn Schuller speaking at SMART School on Computational Social and Behavioral Sciences as a Keynote speaker.
http://www.smart-labex.fr/SMART_School_on_Computational_Social_and_Behavioral_Sciences.html
• (Jul' 2015) SEWA coordinator speaking at ICME 2015 as a Keynote speaker.
http://www.icme2015.ieee-icme.org
• (Jun' 2015) SEWA UP PI, Björn Schuller speaking at 4th Machine Learning for Interactive Systems Workshop (MLIS 2015) held at the International Conference on Machine Learning (ICML'15).
http://scuba.usc.edu/?q=workshops
• (Jun' 2015) SEWA UP
PI, Björn Schuller speaking at UK SPEECH 2015 - 4th Meeting of the UK and Irish Speech Science and Technology Research Community as a Keynote speaker.
http://www.ukspeech.org.uk/workshop/UKSpeech2015-programme.pdf
• (Jun' 2015) SEWA UP PI, Björn Schuller speaking at invited inaugural lecture, University of Passau.
http://www.fim.uni-passau.de/fileadmin/files/dekanat/Veranstaltungshinweise/Antrittsvorlesung_Schuller_Wirth_Handschuh.pdf
• (May' 2015) SEWA UP PI, Björn Schuller speaking at invited tele-talk, Applied Signal Processing in Mental Health Workshop, University of Southern California as a Keynote speaker. http://scuba.usc.edu/?q=workshops
• (Apr' 2015) SEWA UP PI, Björn Schuller speaking at ""Orange Hour"" as a Keynote speaker.http://www.gfk-verein.org/en/events/orange-hour
• (Apr' 2015) SEWA UP PI, Björn Schuller speaking at 2015 International Symposium on Computational Psychophysiology as a Keynote speaker.http://www.taf.sdnu.edu.cn/enligsh.htm
• (Mar' 2015) SEWA UP PI, Björn Schuller speaking at NII Shonan Meeting: The Future of Human-Robot Spoken Dialogue: from Information Services to Virtual Assistants, Seminar 059 as a Keynote speaker. http://shonan.nii.ac.jp/shonan/blog/2013/12/10/the-future-of-human-robot-spoken-dialogue-from-information-services-to-virtual-assistants/

Press Coverage:
• (Jan 2016) SEWA coordinator interview for TV Serbia (RTS) on 4th Industrial Revolution
http://www.rts.rs/page/tv/sr/story/20/RTS+1/2182120/Svetska+ekonomija+u+slobodnom+padu.html
• (Jan 2016) SEWA coordinator interview for France24 on Future AI
http://www.france24.com/en/20160128-people-profit-davos-wef-global-economy-digital-revolution-robots-france-reform
• Wall Street Journal http://blogs.wsj.com/digits/2015/04/17/emotion-tracking-startup-gets-eu-funding-boost/
• Marketing http://www.marketingmagazine.co.uk/article/1343366/european-commission-issues-€36m-grant-tech-measures-content-likeability
• Campaign http://www.campaignlive.co.uk/youtube/article/1343366/european-commission-issues-36m-grant-tech-measures-content-likeability/
• Media Week http://www.mediaweek.co.uk/article/1343366/european-commission-issues-€36m-grant-tech-measures-content-likeability
• Brand Republic http://www.brandrepublic.com/article/1343366/european-commission-issues-€36m-grant-tech-measures-content-likeability
• The Drum http://www.thedrum.com/news/2015/04/17/company-tracks-human-emotions-webcams-awarded-26m-grant-eu-commission
• Research Live http://www.research-live.com/news/realeyes-eu-grant-for-measuring-ad-likeability/4013199.article
• DotRising http://www.dotrising.com/2015/04/20/european-commission-awards-2-6m-grant-to-find-tech-to-measure-content-likeability/
• Tech Investor News http://www.techinvestornews.com/Enterprise/Latest-Enterprise-News/realeyes-receives-grant-to-develop-likeability-tracking-webcam
• Computer Business Review http://www.cbronline.com/news/big-data/hardware/realeyes-receives-grant-to-develop-likeability-tracking-webcam--4557214
• (Sep' 2015) Computer lernt menschliches Verhaltenhttp://www.pnp.de/nachrichten/heute_in_ihrer_tageszeitung/bayern/1801562_Computer-lernt-menschliches-Verhalten.html
• (Jun' 2015) SEWA project to develop methods for automatic behavioural analysishttp://www.uni-passau.de/en/bereiche/press/press-releases/news/detail/sewa-project-to-develop-methods-for-automatic-behavioural-analysis/
• (Jun' 2015) Projekt SEWA entwickelt Methoden zur automatischen Verhaltensanalysehttp://www.uni-passau.de/bereiche/presse/pressemeldungen/meldung/detail/projekt-sewa-entwickelt-methoden-zur-automatischen-verhaltensanalyse/ (Feb' 2015) Björn Schuller: ""Grosse Gefühle - Robotik und Emotionen"",
radio interview broadcast on radiobremen/Nordwestradio ,
""Glauben und Wissen"", Tina Würfel.

Ethics
SEWA consortium had arranged for an Ethical Advisory Board, which consists of experts on various fields of ethics that concern the SEWA project. The members of the Ethical Advisory Board are Prof. Laurence Devillers of the Paris-Sorbonne IV University in France and Prof. Jean-Gabriel Ganascia of the University Pierre et Marie Curie in France. The Ethical Advisory Board meets at most once a year with the PMC. The first meeting was held in conjunction with the SEWA kick-off meeting on 12-13 February 2015, in London, UK. The recommendations made by the Ethical Advisory Board have been discussed by the PMC, adopted by the project, and are forwarded to the Commission as part of deliverable D8.2. The Ethical Advisory Board will be consulted in all ethical issues as they arise in the course of the work in the various research lines.

Communication
• Internet Presence: The consortium has set up a web site: www.sewaproject.eu as a dissemination tool to be maintained during the project and beyond. It has been set up with general information about SEWA, members of the consortium, the objectives and results of the project, up-to-date news about all dissemination efforts, including the information about project presence in conferences, fairs, exhibitions, etc. The website facilitate subscription to project news and events, download of public deliverables, download of publications related to the project, download of released software tools and data, download of demonstration videos, and download of video-lectures recorded during the project-related events. It also facilitate a project-private space for filing deliverables and internal reports.
• Press: The full list of press coverage is listed in 1.2.8.
• Industry Fairs: ICL took part in The Festival Nauke in Belgrade 3 to 6 December 2015. http://festivalnauke.org/Program/Beogradski-sajam/Humaniji-nego-sto-mislite
The aim was to promote the technology advances developed by the SEWA project to elementary and high-school children and to stimulate their interest in science, in general, and Artificial Intelligence and Human-Computer Interaction, in particular.
• Workshops: The full list of SEWA workshops is listed in 1.2.8.

Exploitation
• Realeyes’ involvement in SEWA project has had a positive impact on its market position. It has strengthened its relationship with existing customers (especially those who took a role in the Valorisation Advisory Board, e.g., IPSOS). It also attracted attention of new customers. Of those, the most important is a significant commercial contract with one of the world’s largest media agencies – Mediacom. This development had wide press coverage.
http://www.mrweb.com/drno/news22038.htm
http://digiday.com/agencies/facial-coding-saves-clients-millions-not-running-campaigns/
http://realbusiness.co.uk/article/32747-marketing-firm-mediacom-to-track-consumer-emotions-with-uk-tech-startup-realeyes
• Realeyes has also partnered up with one of the Valorisation Board members, Xaxis. The goal of the partnership is to establish the links between emotional audience profiling and impact on the ad tech industry. This is a part of an ongoing collaboration and it is too early to disclose any results. But collaboration itself already paves the way for further commercial partnership between the two companies in the future.
• Through the course of exploration of possibilities for emotional profiling of ads and users, Realeyes has built a training and evaluation framework, which can be used for effective data analysis and predictive model training. A pilot study on their emotion-based predictive models suggests a direct link between emotions and effectiveness of an advertising video content. This is significant in the area of market research and, if these results are confirmed in subsequent studies, it would enable Realeyes to further strengthen its position in the market research industry. The pilot study in question was made possible in part thanks to the work of Realeyes on the SEWA ad recommendation engine.
• PlayGen’s participation in the Valorisation board meeting and follow up conversations, lead to identifying a number of commercial opportunities, including jobs market, employability skills, dating and sales training.
• PlayGen organised a SEWA internal workshop and a set of meetings with project partners to explore potential business opportunities combining video chat, games and emotion detection technologies. It was decided to focus on exploration of potential market opportunities within employability skills for young people, in particular the potential role of emotion detection in enhancement of communication skills.

WP9 – Project co-ordination and management
• Creation of website: www.sewaproject.eu
• Creation of mailing list: sewa_formal@googlegroups.com
• Managed and submitted deliverables
• Project coordinator with ICL Research Contracts handled signing of the consortium agreement
• Project manager worked with consortium for pre-financing calculation
• Project manager highlighted policy and procedure changes from FP7 to H2020
• List of project meetings during this period:
12 and 13 February 2015 – Kick off meeting – ICL, London, UK.
6 and 20 March, 22 May, 19 June 2015 – Phone meetings.
16 July 2015 – Plenary meeting – ICL, London, UK.
4 September 2015 – Phone meeting.
30 September and 1 October – Plenary and Valorisation Board meetings – ICL, London, UK.
1 December 2015 and 11 January 2016 – Phone meetings.
"

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

WP1 – SEWA DB collection, annotation and release
• A total of 398 participants from 6 different cultural backgrounds (British, German, Hungarian, Serbian, Greek and Chinese) were recorded in the wild, resulting in more than 44 hours of audio-visual corpus covering a wide range of spontaneous expressions of emotions and sentiment during both video-watching and computer-mediated face-to-face communication sessions. The data will be annotated in terms of facial landmarks, vocal and verbal cues, facial action units (FAUs), continuously valued emotion dimensions (valence and arousal), mimicry, sentiment, rapport, and template behaviours. The SEWA database will be publicly released in a web-based searchable form. The SEWA database is the very first to contain in-the-wild recordings of people’s reactive and interactive behaviours, to be demographically balanced (equal number of male and female subjects, equal number of subjects from each age group 20-30-40-50-60, in each age group there is at least one of the following interactive dyads male-male, male-female, female-female), having all subjects who are native speakers, to be annotated in terms of all visual, vocal, and verbal cues, affective dimensions, sentiment, and social signals such as rapport and mimicry.

WP2 – Low-level Feature Extraction
• Proven that Bag-of-Audio Words is able to predict emotions in terms of arousal and valence in a better way than all other known approaches and published results.
• The deteriorating effect of noise on acoustic features is overcome using de-noising auto-encoders (feature enhancement).
• Development of a hybrid system combining BoAW (acoustic features) and BoW (Bag-of-Words, linguistic features) with different feature fusing schemes.
• Implemented the incremental in-the-wild face alignment method for automatic facial landmark localisation. The tracker is capable of accurately tracking the 49 facial landmarks in real-time and is robust against illumination change, partial occlusion and head movements.

WP3 – Mid-level feature extraction
• Showed that by modelling AU segments in videos using a mixture of nominal and ordinal states improves the AU segmentation/detection over the state-of-the-art conditional random field models that employ either type of the states (i.e., nominal or ordinal).
• The proposed extension of the CORF (Conditional Ordinal Random Field) model, by defining its feature functions by means of Neural Networks, resulted in better estimation of AU intensities. The model was also applied to the task of agreement level estimation from the MAHNOB database – and it outperformed the existing methods applicable to the target task.

WP4 – Continuous Affect and Sentiment Sensing in the Wild
• Work on WP4 will start in M15 (April 2016).

WP5 – Behaviour Similarity in the Wild
• Work on WP5 will start at the end of M12 (from 1st February 2016).

WP6 – Temporal Behaviour-Patterning and Interpersonal Sentiment in the Wild
• No findings so far as WP6 started just in M12 (January 2016).

WP7 – Integration, Applications and Evaluation
• Social Chat Game
Through a series of discussions with the Valorisaton Board and the project partners, it has been concluded that the application of the SEWA technology for Chat Social Game should represent a new approach to communication skills training utilising emotion detection technologies developed in SEWA, together with validation methodologies. The application is targeting young people aged 18+, who are either in educational institutions or have recently completed education, who are shortly embarking on a career, and who’d benefit from a light touch, fun and meaningful way of practicing negotiation and discussions that are part of the everyday working life (e.g. job interview, dealing effectively with customers, negotiating a reduction in rent with a landlord or being more effective at dealing with work colleagues). Feedback from employers and end-user focus groups has demonstrated that this has potential to develop students’ negotiation and persuasion skills and increase the chances of their employability and their subsequent performance in what is usually their first job, in a cost-effective way. In addition to the gaps in influencing skills identified by employers and end-users, the focus on this aspect of communication appears to be particularly well suited for digital games, as there are clear objectives and possible ways of assessing the goals, as well as being a good fit with automatic emotion detection, as emotions play a significant role in interpersonal communication. Additionally as supported by the European Commission, this approach contributes to the promotion of recognition and validation of knowledge and skills acquisition through non-formal learning. Therefore this application aims to contribute to employability of young people, with obvious positive societal impacts.
In order to pre-empt integration, a simple two person chat social game was developed utilising on of the existing platforms being utilised in the platform. This provided both experience of the specific software as well as helped to identify future issues with respect to integration, delivery and evaluation.
• Advert Recommender System
The Advert Recommender System aims to integrate and validate a novel approach to ad placement optimization by analysing nonverbal behavioural cues of the users/customers. The great advantage of this method over traditional approaches is that ad placement can be made more personal, informative, effective and less annoying thus significantly improving the usefulness for both the advertisers and the potential customers. To this aim Realeyes did the following:
i) Built a framework to record audio, video, and questionnaire data online in response to video content. The framework has been used to collect user responses for over 150 advert videos (among which are the 4 advert videos used as stimuli material in the SEWA data collection). Each of the 150 videos is associated with a performance measure (i.e., advert is successful/ unsuccessful), which was provided to us by our clients. This information and the behavioural responses by users constitute a unique dataset, which can be analysed for patterns of users’ behaviour being predictive of the ultimate performance of an advert.
ii) A new framework has been designed and is being built that allows for quick statistical testing of “emotional profiles of ads” (which, in essence, determine which people would react how on a specific ad) and their use for predicting ads’ performance.
iii) A pilot study has been carried out on the possible use of the collected data for building emotional profiles of adverts. Initial findings are positive and confirm our expectations, namely, that emotional reactions of users can be predictive of ads’ performance and that they can be used for better targeting. Once we receive approval of our commercial partners to share the findings, we plan to publish these results in academic and market research papers.

WP8 – Dissemination, Ethics, Communication and Exploitation

(i) Dissemination & Communication
SEWA partners have increased the interest of general public in the field of automatic emotion recognition and corresponding applications. There are several evidences for this: a major article on SEWA in German national press (Passauer Neue Presse), a TV report on emotional agents partly filmed at the chair of PI Björn Schuller at University of Passau, invitation to the SEWA coordinator to speak on SEWA and related technologies at the World Economic Forum in Davos in January 2016, invitation to SEWA partners to present SEWA technology at the Science Fair in Belgrade (Serbia), and two TV interviews with the SEWA coordinator on emotional robots/technology and their role in the 4th Industrial Revolution (see 1.2.8 for details).

The awareness of the scientific community about the importance of research focus on automatic analysis of human behaviour observed in the wild and automatic audio-visual sentiment analysis has been raised by means of both the 1st International Workshop on Automatic Sentiment Analysis in the Wild (WASA’15), which has has been organised by SEWA partners, and a large number of Keynotes given by the SEWA partners at which SEWA project and its aims have been explained (for the complete list, see 1.2.8).

(ii) Exploitation, Socio-Economic impact, and Social Implications of the project
While socio-economic impact of the project is clear, relating to the significant technological leverage that the industrial partners of the SEWA project benefit from, as explained in 1.3.7, social implications of the project are less clear, though possibly profound. Let us explain this in more detail.

According to the European Commission report in 2014, 14 million young people in the EU are not in employment, education and training, contributing over €162 billion in annual economic loss as well as additional long term personal and social costs. Since reliable training through education to secure employment no longer exists, governments in Europe including the UK government are investing in a range of schemes such as apprenticeships to help young people get a foothold on the job ladder. In the UK alone some 500,000 individual apprenticeships were offered in the year 2014/15 representing a 14% annual growth in number of apprenticeship placements. With each apprentice receiving support from the government including communication training by specialist organisations, companies offering apprenticeship placements consistently cite lack of soft skills such as ability to communicate well as a critical barrier for young people to succeed.

In addition to half a million young people seeking apprenticeship, in the UK, over 2 million people graduate from universities and higher education establishment per year, the figure in Europe for tertiary education is over 20 million individuals, the great majority of whom will then be embarking on a career which requires amongst other skills, to have the communication skills necessary to succeed in job interviews as well as in the majority of instances to then fit into working environment through effective communication skills.

With a potential European market size of some 30 million individual young people annually, a large proportion of who appear to benefit from better communication skills, the strategies for exploitation of the outcome of SEWA will explore a number of avenues. In the first instance the focus will be in the UK market, since PlayGen is in the UK, it will exploit the good links it has with local and national government agencies including the department of education as well as a large number of UK universities, to develop a commercial offering centred around improving student employability, increasing apprenticeship success rates, highlighting the benefits in reduction of training cost and long term success of individuals. From a given potential addressable market of some 2.5 million in the UK, and armed with scientific evidence that SEWA delivers better communication skills through non-formal setting it may be estimated that even a modest uptake of some 5% could potentially generate revenues of €1.2m annually. Of course this only represents the UK, extending the solution to the English speaking nations and further leveraging the languages supported by SEWA, it is feasible to imagine far larger revenue in due course.

Whilst the current focus is on employability skills, the core technologies and game at a meta-level can be extended for a wide range of applications and communication skills, from sales training to customer support training to dating applications. We will continue to monitor and explore potential markets as we develop, evaluate and validate the approach from both technology and commercial aspects, for instance exploring freemium and tiered models aimed both at B2C and B2B markets in addition to the strategies highlighted above.

Related information

Record Number: 186633 / Last updated on: 2016-07-14
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top