Affective Computing Models: from Facial Expression to Mind-Reading

Project Information

ACMod

Grant agreement ID: 101130271

DOI

10.3030/101130271

EC signature date 5 December 2023

Start date 1 March 2024

End date 29 February 2028

Funded under

• Marie Skłodowska-Curie Actions (MSCA)

Total cost

No data

EU contribution

€ 846 400,00

Coordinated by

OULUN YLIOPISTO
Finland

Periodic Reporting for period 1 - ACMod (Affective Computing Models: from Facial Expression to Mind-Reading)

Reporting period: 2024-03-01 to 2026-02-28

Summary of the context and overall objectives of the project

The ACMod project addresses one of the key challenges in human communication and human-computer interaction: understanding facial expressions and emotional states across different cultures. While facial expression analysis technologies have made significant progress in recent years, current systems still struggle to interpret emotions reliably in realistic and culturally diverse environments. In particular, differences in how emotions are expressed and perceived across cultures challenge the idea of universal facial expressions and limit the development of truly adaptive intelligent systems.

ACMod aims to advance the understanding and modelling of naturalistic facial behaviours by combining expertise from psychology, affective computing, computer vision, and artificial intelligence. The project focuses on reconstructing facial macro- and micro-expressions, analysing physiological and behavioural signals related to emotions, and developing autonomous 4D facial models capable of representing dynamic emotional states. To support this work, the consortium is building multimodal datasets combining facial data, depth information, and physiological signals collected in cross-cultural contexts.

The project brings together European and East Asian academic and industrial partners through the MSCA Staff Exchanges programme, promoting interdisciplinary collaboration and long-term knowledge exchange. Through secondments, workshops, joint research activities, and shared datasets, ACMod strengthens international research networks and supports the training and career development of researchers.

By improving the modelling of emotional behaviours and realistic facial interactions, ACMod aims to contribute to the next generation of human-centred technologies, including intelligent virtual agents, social robotics, affective computing systems, and augmented or virtual reality applications. The project also contributes to ongoing discussions on ethical and responsible use of emotion-aware AI technologies, including privacy, digital identity, and trustworthy human-machine interaction.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

ACMod aims to improve how intelligent systems understand and reproduce human emotions and facial behaviours in realistic and culturally diverse environments. To achieve this, the project combines expertise from psychology, artificial intelligence, computer vision, and affective computing to study how emotions are expressed through facial movements, physiological responses, and social interactions.

A central objective of the project is the development of multimodal approaches that combine facial images, dynamic 3D facial information, and physiological signals such as heart activity or muscle responses. By bringing these different sources of information together, ACMod seeks to create more reliable and human-centred models of emotion understanding than those currently used in conventional facial expression analysis systems.

The project also explores new methods for reconstructing and generating realistic 3D and 4D facial behaviours, including expressive virtual avatars capable of displaying natural emotional reactions. These developments contribute to the long-term goal of improving interaction with intelligent agents, social robots, and virtual or augmented reality systems, making them more adaptive, expressive, and socially aware.

Beyond the technological aspects, ACMod promotes extensive international collaboration and interdisciplinary knowledge exchange between European and East Asian research partners. Through shared datasets, workshops, scientific exchanges, and joint research activities, the project contributes to the development of a broader research community working on emotion-aware artificial intelligence and human-centred digital technologies.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

ACMod introduces a new interdisciplinary approach to facial behaviour modelling by combining psychological theories of emotion with advanced artificial intelligence and computer vision techniques. Rather than focusing only on visible facial expressions, the project studies how emotions are linked to both external facial behaviours and internal physiological responses, enabling a more complete understanding of human affective states.

Current facial analysis systems are often limited to standard image-based approaches and may struggle in realistic social situations or across different cultural contexts. ACMod moves beyond these limitations by integrating multiple types of information, including dynamic 3D facial data, physiological signals, and behavioural analysis. This allows the project to explore more realistic, robust, and culturally aware models of emotion recognition and expression.

The project also advances the development of realistic digital humans and expressive virtual avatars capable of reproducing subtle facial movements and emotional behaviours. Such technologies could support future applications in human-computer interaction, social robotics, healthcare, education, virtual communication, and immersive digital environments.

At the same time, ACMod places ethical and societal considerations at the very centre of the project. This is particularly important given the highly sensitive nature of facial, behavioural, and physiological data, as well as the growing capability of AI systems to infer emotional states from everyday consumer technologies. By integrating ethical reflection, responsible data governance, and external ethical oversight directly into the research process, ACMod promotes a new model for the development of trustworthy and human-centred AI systems, where technological innovation and societal safeguards evolve together.