The project built on models trained on a new dataset of multimodal communicative human behavior - obtained throughout rigorous recording of natural interactions in multiparty settings session carried out in the scope of the project (in total 4 hours and 50 human participants involved) - combined with state-of-the-art techniques in multimodal communication and language technology, leading to entirely fresh views on human-machine interaction.
The project has also innovated in the field of affective computing and behavioral analytics, through Investigation of perception and automatic detection of psychological variables, group leadership and emotion-related features in group interaction through exploitation of linguistic, acoustic and visual features.
The work performed in the project can be summarised in the following activities:
- Experimental design, implementation and collection of a multimodal corpus of three-party interactions.
- Conversational dominance quantification and detection.
- Personality detection from audio and text.
- Measuring engagement from linguistic repetitions and turn-taking elements.
- Dialogue laughter classification and modeling.
- Speech pause analytics for the detection of the next speaker in multi-party interaction.
- Development of an innovative scoring system to measure collaboration and task success in small groups from voice, facial expressions, turn-taking, and personality features.
Research carried out in the project resulted in 9 publications and the release of the project’s dataset, the MULTISIMO corpus.
The Fellow’s paper “Quantifying dominance in the Multisimo corpus” presented at the 9th IEEE Conference on Cognitive Infocommunications, received a best paper award.
To disseminate the project results and to achieve maximum outreach, the Fellow participated in more than 10 conferences and other networking events, gave two invited talks, and was actively involved in the European Researchers’ Night events in 2016 and 2017.
To create and maintain a vibrant community around the area of multimodal interaction analysis, the Fellow co-organised two scientific tracks in international conferences.
As part of her career development plan, and in order to maximise the impact of the research results of the project, the researcher fully exploited training opportunities provided by the host institution and the EU, including academic training, research development and exploitation of research results.
The Fellow also demonstrated efficacy in directing researchers through contributing to the supervision of both final year undergraduates and postgraduates, and a measure of the success of this is that such projects have culminated in peer reviewed publications. She also contributed to the teaching mission of the host institution through training on research project supervision activities, and through providing guest lectures to modules that are part of the core curriculum of both undergraduate and post-graduate courses.
