Skip to main content
European Commission logo
English English
CORDIS - EU research results
CORDIS
CORDIS Web 30th anniversary CORDIS Web 30th anniversary

Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing

Objective

"The ""Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing"" (HOL-DEEP-SENSE) project aims at augmenting affective machines such as virtual assistants and social robots with human-like acumen based on holistic perception and understanding abilities.

Social competencies comprising context awareness, salience detection and affective sensitivity present a central aspect of human communication, and thus are indispensable for enabling natural and spontaneous human-machine interaction.
Therefore, with the aim to advance affective computing and social signal processing, we envision a ""Social Intelligent Multi-modal Ontological Net"" (SIMON) that builds on technologies at the leading edge of deep learning for pattern recognition.
In particular, our approach is driven by multi-modal information fusion using end-to-end deep neural networks trained on large datasets, allowing SIMON to exploit combined auditory, visual and physiological analysis.

In contrast to standard machine learning systems, SIMON makes use of task relatedness to adapt its topology within a novel construct of subdivided neural networks. Through deep affective feature transformation, SIMON is able to perform associative domain adaptation via transfer and multi-task learning, and thus can infer user characteristics and social cues in a holistic context.

This new unified sensing architecture will enable affective computers to assimilate ontological human phenomena, leading to a step change in machine perception. This will offer a wide range of applications for health and wellbeing in future IoT-inspired environments, connected to dedicated sensors and consumer electronics.

By verifying the gains through holistic sensing, the project will show the true potential of the much sought-after emotionally and socially intelligent AI, and herald a new generation of machines with hitherto unseen skills to interact with humans via universal communication channels."

Coordinator

TECHNISCHE UNIVERSITAET MUENCHEN
Net EU contribution
€ 193 184,00
Address
Arcisstrasse 21
80333 Muenchen
Germany

See on map

Region
Bayern Oberbayern München, Kreisfreie Stadt
Activity type
Higher or Secondary Education Establishments
Links
Total cost
€ 193 184,00

Participants (1)

Partners (1)