Skip to main content

Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing

Objective

"The ""Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing"" (HOL-DEEP-SENSE) project aims at augmenting affective machines such as virtual assistants and social robots with human-like acumen based on holistic perception and understanding abilities.

Social competencies comprising context awareness, salience detection and affective sensitivity present a central aspect of human communication, and thus are indispensable for enabling natural and spontaneous human-machine interaction.
Therefore, with the aim to advance affective computing and social signal processing, we envision a ""Social Intelligent Multi-modal Ontological Net"" (SIMON) that builds on technologies at the leading edge of deep learning for pattern recognition.
In particular, our approach is driven by multi-modal information fusion using end-to-end deep neural networks trained on large datasets, allowing SIMON to exploit combined auditory, visual and physiological analysis.

In contrast to standard machine learning systems, SIMON makes use of task relatedness to adapt its topology within a novel construct of subdivided neural networks. Through deep affective feature transformation, SIMON is able to perform associative domain adaptation via transfer and multi-task learning, and thus can infer user characteristics and social cues in a holistic context.

This new unified sensing architecture will enable affective computers to assimilate ontological human phenomena, leading to a step change in machine perception. This will offer a wide range of applications for health and wellbeing in future IoT-inspired environments, connected to dedicated sensors and consumer electronics.

By verifying the gains through holistic sensing, the project will show the true potential of the much sought-after emotionally and socially intelligent AI, and herald a new generation of machines with hitherto unseen skills to interact with humans via universal communication channels."

Field of science

  • /natural sciences/computer and information sciences/artificial intelligence/computational intelligence
  • /natural sciences/computer and information sciences/artificial intelligence/pattern recognition
  • /engineering and technology/electrical engineering, electronic engineering, information engineering/electronic engineering/signal processing
  • /natural sciences/computer and information sciences/artificial intelligence/machine learning/deep learning
  • /natural sciences/computer and information sciences/artificial intelligence/machine learning

Call for proposal

H2020-MSCA-IF-2017
See other projects for this call

Funding Scheme

MSCA-IF-GF - Global Fellowships

Coordinator

UNIVERSITAET AUGSBURG
Address
Universitaetsstrasse 2
86159 Augsburg
Germany
Activity type
Higher or Secondary Education Establishments
EU contribution
€ 199 828,20

Partners (1)

MASSACHUSETTS INSTITUTE OF TECHNOLOGY
United States
Address
Massachusetts Avenue 77
02139 Cambridge
Activity type
Higher or Secondary Education Establishments