Skip to main content

Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing

Objective

"The ""Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing"" (HOL-DEEP-SENSE) project aims at augmenting affective machines such as virtual assistants and social robots with human-like acumen based on holistic perception and understanding abilities.

Social competencies comprising context awareness, salience detection and affective sensitivity present a central aspect of human communication, and thus are indispensable for enabling natural and spontaneous human-machine interaction.
Therefore, with the aim to advance affective computing and social signal processing, we envision a ""Social Intelligent Multi-modal Ontological Net"" (SIMON) that builds on technologies at the leading edge of deep learning for pattern recognition.
In particular, our approach is driven by multi-modal information fusion using end-to-end deep neural networks trained on large datasets, allowing SIMON to exploit combined auditory, visual and physiological analysis.

In contrast to standard machine learning systems, SIMON makes use of task relatedness to adapt its topology within a novel construct of subdivided neural networks. Through deep affective feature transformation, SIMON is able to perform associative domain adaptation via transfer and multi-task learning, and thus can infer user characteristics and social cues in a holistic context.

This new unified sensing architecture will enable affective computers to assimilate ontological human phenomena, leading to a step change in machine perception. This will offer a wide range of applications for health and wellbeing in future IoT-inspired environments, connected to dedicated sensors and consumer electronics.

By verifying the gains through holistic sensing, the project will show the true potential of the much sought-after emotionally and socially intelligent AI, and herald a new generation of machines with hitherto unseen skills to interact with humans via universal communication channels."

Call for proposal

H2020-MSCA-IF-2017
See other projects for this call

Funding Scheme

MSCA-IF-GF - Global Fellowships

Coordinator

TECHNISCHE UNIVERSITAET MUENCHEN
Address
Arcisstrasse 21
80333 Muenchen
Germany
Activity type
Higher or Secondary Education Establishments
EU contribution
€ 193 184

Participants (1)

UNIVERSITAET AUGSBURG

Participation ended

Germany
EU contribution
€ 6 644,20
Address
Universitaetsstrasse 2
86159 Augsburg
Activity type
Higher or Secondary Education Establishments

Partners (1)

MASSACHUSETTS INSTITUTE OF TECHNOLOGY
United States
Address
Massachusetts Avenue 77
02139 Cambridge
Activity type
Higher or Secondary Education Establishments