Periodic Reporting for period 1 - TAMED (Tensor-bAsed Machine learning towards genEral moDels of affect)
Reporting period: 2020-07-01 to 2022-06-30
The action “TAMED: Tensor-bAsed Machine learning towards genEral moDels of affect” run at the Institute of Digital Games, University of Malta. It focused on devising methods and algorithms for realising aspects of general emotional intelligence, one of the core long-term goals of Artificial Intelligence (AI) and artificial psychology. In other words, TAMED focused on developing computational tools (models of affect) for detecting and recognising users’ emotional states while performing a task. The term “general” emphasises that the developed tools should not be restricted to specific users or tasks. Instead, they should be general to effectively capture affective patterns across different contexts.
Since Picard’s seminal paper in 1995 (see R. Picard, “Affective Computing,” Technical Report (MIT Media Laboratory Perceptual Computing Section, Nov. 1995).), the Affective Computing (AC) field has advanced the study of modelling human affect substantially. Most modern approaches, however, indicate that any success of AC depends heavily on the domain, the task at hand, and the context in general. This specificity limitation is detrimental both to the scientific value and to the practical applicability of the methods developed and studied in the AC field. Deriving general “context-free” affect models is a crucial step towards understanding the inner workings of emotional intelligence, advancing AI agents and building better human-computer and human-robot interaction systems affecting the everyday lives of millions of people.
Based on the discussion above, the main objective of TAMED was to derive general models of affect and investigate the degree to which the construction of such models is possible. Towards this direction, research efforts focused on the entire cycle of affective modelling, starting with the definition of general input-output representations of context and affect, moving towards developing novel Machine Learning models, and validating the derived affect models on complex real-world problems.
Since Picard’s seminal paper in 1995 (see R. Picard, “Affective Computing,” Technical Report (MIT Media Laboratory Perceptual Computing Section, Nov. 1995).), the Affective Computing (AC) field has advanced the study of modelling human affect substantially. Most modern approaches, however, indicate that any success of AC depends heavily on the domain, the task at hand, and the context in general. This specificity limitation is detrimental both to the scientific value and to the practical applicability of the methods developed and studied in the AC field. Deriving general “context-free” affect models is a crucial step towards understanding the inner workings of emotional intelligence, advancing AI agents and building better human-computer and human-robot interaction systems affecting the everyday lives of millions of people.
Based on the discussion above, the main objective of TAMED was to derive general models of affect and investigate the degree to which the construction of such models is possible. Towards this direction, research efforts focused on the entire cycle of affective modelling, starting with the definition of general input-output representations of context and affect, moving towards developing novel Machine Learning models, and validating the derived affect models on complex real-world problems.
To achieve the main objective above, we split the work into four technical work packages (WP2-WP5). WP2 focused on context’s input representations. We investigated the impact of different information modalities on the performance of affect models, including information modalities that correspond to direct users’ measurements, such as physiology and facial images, but also modalities that do not correspond to direct users’ measurements, such as the content of the interaction (e.g. recorded gameplay footage). The latter is very important to mitigate privacy issues inherently related to affect modelling. Finally, we also investigated the degree to which processing information to construct hand-crafted features affect the generality of the derived models of affect. WP3 built on the fact that emotions are subjective and, thus, within any computational process, they should be treated as ordinal variables in an attempt to mitigate subjectivity bias (specificity limitation). We investigated the effect of subjectivity bias and the degree to which it can be mitigated by treating emotions both as nominal (continuous and discrete) and ordinal variables (relative) in affect modelling. WP4 dealt with associating the outcomes of WP2 and WP3 via machine learning. Within this WP we developed models of affect based on different learning paradigms (classification, regression and preference learning) and protocols (supervised, self-supervised). During WP5, we evaluated the performance of affect models based on the outcomes of WP2-WP4. We evaluated the models on real-world as well as digital games affective corpora. Moreover, in collaboration with Massive Entertainment AB, we collected the first large-scale affect corpus of users interacting with commercial standard games. The work conducted during WP2-WP5 resulted in two peer-reviewed journal publications plus one more under revision, eight peer-reviewed conference publications plus one more under review, and one technical report.
During WP6, related to dissemination and communication activities, the Fellow 1) was the main organiser of the first “What’s Next In Affect Modelling?” workshop that took place within the International Conference on Affect Modelling and Intelligent Interaction (ACII 2021 – one of the most important annual meetings for the Affective Computing community). Also, after the successful first event, the Fellow will organise the second workshop in the series within ACII 2022, 2) served as Keynote Speaker at the 1st International Conference on Novelties in Intelligent Digital Systems (NIDS 2021), 3) delivered an invited talk at the University of West Attica on Machine Learning for High-Order Data Analysis, 4) served as a panel member for the UM Grants Week organised by the University of Malta (UM), the Malta Council for Science and Technology (MCST), the European University of the Seas (SEA-EU) and supported by EU Commission, 5) featured in Press Releases in Malta Business Weekly and THINK magazine, 6) delivered lectures for postgraduate IDG courses, and 7) co-supervised two early-stage researchers (PhD students). Finally, he published two journal papers and eight peer-reviewed conference papers under the “Green” open access model.
During WP6, related to dissemination and communication activities, the Fellow 1) was the main organiser of the first “What’s Next In Affect Modelling?” workshop that took place within the International Conference on Affect Modelling and Intelligent Interaction (ACII 2021 – one of the most important annual meetings for the Affective Computing community). Also, after the successful first event, the Fellow will organise the second workshop in the series within ACII 2022, 2) served as Keynote Speaker at the 1st International Conference on Novelties in Intelligent Digital Systems (NIDS 2021), 3) delivered an invited talk at the University of West Attica on Machine Learning for High-Order Data Analysis, 4) served as a panel member for the UM Grants Week organised by the University of Malta (UM), the Malta Council for Science and Technology (MCST), the European University of the Seas (SEA-EU) and supported by EU Commission, 5) featured in Press Releases in Malta Business Weekly and THINK magazine, 6) delivered lectures for postgraduate IDG courses, and 7) co-supervised two early-stage researchers (PhD students). Finally, he published two journal papers and eight peer-reviewed conference papers under the “Green” open access model.
The TAMED action has pushed the frontiers of affect modelling in two main ways. First, during TAMED we proposed, for the first time, an affect modelling methodology that does not rely on direct users’ measurements. Specifically, we made and validated the hypothesis that the emotional states of users’ are embedded into and can be recovered by general-purpose representations of the interactive content such as gameplay footage. The validation of this hypothesis opens new avenues for AC research since it releases affect modelling from the requirement of direct users’ measurements (e.g. physiological signals, facial data etc.) and results in user-agnostic models. Second, we introduced the concept of privileged information into affect modelling during the action. Exploiting the Learning Under Privileged Information framework (see Pechyony, Dmitry, and Vladimir Vapnik. “On the theory of learning with privileged information.” Advances in neural information processing systems 23 (2010).), we proved that affect models developed in a lab can be efficiently transferred into the real world. This work tries to bridge the gap between in-vitro (lab), and in-vivo (in the wild) affect modelling and provides the means for developing models of affect that operate in unconstrained real-world conditions. Besides the impact on affect modelling, TAMED’s fundamental research has also had an impact on other disciplines. The tensor-based machine learning models developed and theoretically validated during the action were exploited in Remote Sensing, Human Pose Recognition and Cultural Heritage applications producing state-of-the-art results.
Finally, from the fellow’s perspective, during the grant, the Fellow 1) enhanced his professional network, 2) acquired teaching, mentoring and managerial skills, and 3) boosted his curriculum vitae with high-quality publications. All the activities he performed and the skills he acquired during the action were essential for his appointment as a Lecturer at the University of Malta (starting on September 1st 2022) and, thus, crucial for his career development.
Finally, from the fellow’s perspective, during the grant, the Fellow 1) enhanced his professional network, 2) acquired teaching, mentoring and managerial skills, and 3) boosted his curriculum vitae with high-quality publications. All the activities he performed and the skills he acquired during the action were essential for his appointment as a Lecturer at the University of Malta (starting on September 1st 2022) and, thus, crucial for his career development.