Periodic Reporting for period 1 - AMELIA (Emotional modelling to enhance learning in games)
Período documentado: 2023-09-01 hasta 2026-02-28
The mission of this research is to study interactions between game mechanics and emotions as a non-linear dynamical system and observe the impact of such interactions between diverse learners’ cognition and learning outcomes with GBLEs. In this European Fellowship titled, emotionAl Modelling to Enhance Learning In gAmes (AMELIA), we propose to conduct a mixed multimodal methods study to observe relations between game mechanic interactions and in-game emotions, cognition, and learning outcomes by collecting multiple data channels ranging from think-aloud recordings to neurofunctional measures between gender dimensions. AMELIA will be carried out at Tampere University (host) with supervisor Kristian Kiili and co-supervisor Manuel Ninaus (secondment) at the University of Graz. AMELIA is implemented in collaboration with an international game company called Psyon Games. The training-through-research during the fellowship will propel the fellow, Elizabeth Cloude, toward a successful academic career, enabling her to obtain a tenure-track position, apply for an ERC Starting Grant, and build an interdisciplinary, intersectoral, and international research network to study inclusive, equitable, and high-quality
GBLEs. Three research objectives (ROs) with corresponding work packages (WPs) are outlined below:
RO1. Conduct a mixed methods study and collect multimodal data (logfiles, facial expressions, heart rate, electrodermal data, performance, neuroimaging, and think-/emote-alouds) on emotions before, during, and after interacting with game mechanics in a GBLE called Antidote COVID-19 between diverse learners (i.e. gender dimension) [WP2].
RO2. Leverage non-linear dynamical analyses to assess the temporal emergence of emotions before, during, and after diverse learners interact with game mechanics using multimodal data [WP3].
RO3. Identify key relations between game mechanic interaction, emotion dynamics, cognition, and learning outcomes with Antidote COVID-19 using multimodal data between diverse learners [WP4].
The dissemination of the key project results will produce 3 scientific publications with high (open access) impact: European conference for Technology-enhanced Learning, International Conference on Games and learning Alliance, and the International journal of Serious Games. Currently, we are preparing a journal article to be submitted to the British journal of Educational Technology (April 2025). Our project will advance the scientific meaning of emotions in game-based learning and who benefit from learning with game-based learning environments. Target audience include researchers, learners, academics, teachers/instructors, and game developers and companies. The exploitation of the key project results was done first at the beneficiary where the novel approach was applied. Through open access, the key results were made available to faculty, staff, postdocs, and PhD/undergraduate students to encourage collaboration, development, and optimization of the findings beyond the scope of the project. Our results contributed to collaborative research projects such as applying findings to other game-based learning environments developed by supervisor Kiili’s lab (e.g. MediaWatch) and other TAU’s research groups that focus on games and gamification (e.g. Dr. Muhterem Dindar in the Gamification group with the BirdBreeder Simulation game). Presently, we are planning a visit to local secondary school in Michigan where the fellow is located through the CREATE for STEM institute.
GBLEs. Three research objectives (ROs) with corresponding work packages (WPs) are outlined below:
RO1. Conduct a mixed methods study and collect multimodal data (logfiles, facial expressions, heart rate, electrodermal data, performance, neuroimaging, and think-/emote-alouds) on emotions before, during, and after interacting with game mechanics in a GBLE called Antidote COVID-19 between diverse learners (i.e. gender dimension) [WP2].
RO2. Leverage non-linear dynamical analyses to assess the temporal emergence of emotions before, during, and after diverse learners interact with game mechanics using multimodal data [WP3].
RO3. Identify key relations between game mechanic interaction, emotion dynamics, cognition, and learning outcomes with Antidote COVID-19 using multimodal data between diverse learners [WP4].
The dissemination of the key project results will produce 3 scientific publications with high (open access) impact: European conference for Technology-enhanced Learning, International Conference on Games and learning Alliance, and the International journal of Serious Games. Currently, we are preparing a journal article to be submitted to the British journal of Educational Technology (April 2025). Our project will advance the scientific meaning of emotions in game-based learning and who benefit from learning with game-based learning environments. Target audience include researchers, learners, academics, teachers/instructors, and game developers and companies. The exploitation of the key project results was done first at the beneficiary where the novel approach was applied. Through open access, the key results were made available to faculty, staff, postdocs, and PhD/undergraduate students to encourage collaboration, development, and optimization of the findings beyond the scope of the project. Our results contributed to collaborative research projects such as applying findings to other game-based learning environments developed by supervisor Kiili’s lab (e.g. MediaWatch) and other TAU’s research groups that focus on games and gamification (e.g. Dr. Muhterem Dindar in the Gamification group with the BirdBreeder Simulation game). Presently, we are planning a visit to local secondary school in Michigan where the fellow is located through the CREATE for STEM institute.
Objective 1: conduct a mixed methods study and collecting multimodal data on emotions before, during, and after interacting with a game mechanics in a GBLE called Antidote COVID-19 between diverse learners
We collected a total of 81 participants multimodal data at the host institutions. An important note is that we did not collect neuro-imaging data during verbal protocols at the host institution, or during the secondment, which was the initial research intention. After further discussion with the research team at the secondment institution, concerns were raised regarding the risk of participants’ verbalizing and moving their face/face/head creating measurement errors in the neuro-imaging data during the game-based learning task (NIRS system). To overcome this, we eliminated the verbal protocol from the study at the University of Graz (where the neuro-imaging data were intended to be collected) and instead administered validated instruments to gather self-reported emotions during the learning activity. Self-reported items were administered immediately after each game-based learning activity (approximately 12 times during the game task).
A sample of 11 participants was collected during the secondment period at the secondment institution by the beneficiary. The initial sample size was intended for 60 participants; however, due to a number of factors, including 1) neuro-imaging methods require participants to adhere to strict eligibility criteria, and 2) data collection took place over the 3-month period, the beneficiary was not able to achieve the desired number of 60 participants. To overcome this barrier, arrangements between the beneficiary and secondment institution to continue data collection at the University of Graz, with aims to finish in Summer 2025. The data collection is being led by a master’s student enrolled at the University of Graz, and the beneficiary is serving as their co-supervisor. The master’s student will utilize the data collected during the study for analysis in their master’s thesis research.
The gender dimension was considered in our approach by collecting gender identify information on participants before engaging in all game-based learning tasks. These dimensions were considered and reported in our analyses and publications.
Objective 2: leveraging non-linear dynamical analyses to assess the temporal emergence of emotions before, during, and after diverse learners interact with game mechanics using multimodal data
Presently, we have conducted a series of non-linear analyses on several data channels: 1) facial recognition, 2) heart rate, 3) skin conductance response, 4) self-report data, and 5) think-/emote-aloud data. A large portion of time was dedicated to transcribing and coding the think- and/emote-aloud data following objective 1 at the host institution. The beneficiary was successful in recruiting and training a team of graduate students to facilitate qualitative coding of the verbalizations collected during the game-based learning activities. The coding took place over a period of five months in which the team coded 64 participants’ verbalizations (some participants were removed due to technical errors or very low-quality verbalizations). Now that the codes are available for analysis, we are continuing to explore different variables and data channels currently.
Presently, we have conducted several non-linear dynamical methods on the following data channels:
Facial expressions, heart-rate variability, skin conductance response, and think-/emote-aloud codes. Specifically, we have applied the following:
• Recurrence Quantification Analysis: Examined the degree of recurring values in facial expressions, skin conductance responses, and think-/emote-alouds;
• Cross-wavelet transformation Analysis: Examined the degree of synchronization (or lack thereof) between facial expressions and heart-rate variability;
• Transfer Entropy Analysis: Examined the degree of cross-channel interactions between think-/emote-alouds and skin conductance responses, think-/emote-alouds and facial expressions, and skin conductance responses and facial expressions; and,
• Sequence Analysis: Examined the degree of transitions among different think- and emote-aloud codes.
Objective 3: identify key relations between game mechanic interaction, emotion dynamics, cognition, and learning outcomes with Antidote COVID-19 using multimodal data between diverse learners
Currently, we have explored relationships between multiple variables generated from non-linear dynamical methods applied in Objective 2 to explore their relations with learning outcomes. Specifically, the beneficiary was successful in calculating statistical models using variables extracted from analyses in Objective 2. Presently, we have conducted several different methods listed below:
• Mixed Growth Modelling: Evaluated whether changes in different self-reported emotions collected during game-based learning were associated with learning outcomes after the game-based learning activity;
• Correlation: Evaluated associations between the degree of synchrony between facial expressions and heart-rate variability capture during the game-based learning activity; and,
• Hierarchical Regression and Cross-validation: Evaluating the mathematical relations between sequence analysis, recurrence quantification analysis, and transfer entropy variables in explaining variability in learning outcomes.
Generally, there are key associations between emotion variables with learning outcome measures. We evaluated whether there were significant differences in learning outcomes between participants who identified as different genders. Our results showed there were no significant differences, which were always included in our publications.
We collected a total of 81 participants multimodal data at the host institutions. An important note is that we did not collect neuro-imaging data during verbal protocols at the host institution, or during the secondment, which was the initial research intention. After further discussion with the research team at the secondment institution, concerns were raised regarding the risk of participants’ verbalizing and moving their face/face/head creating measurement errors in the neuro-imaging data during the game-based learning task (NIRS system). To overcome this, we eliminated the verbal protocol from the study at the University of Graz (where the neuro-imaging data were intended to be collected) and instead administered validated instruments to gather self-reported emotions during the learning activity. Self-reported items were administered immediately after each game-based learning activity (approximately 12 times during the game task).
A sample of 11 participants was collected during the secondment period at the secondment institution by the beneficiary. The initial sample size was intended for 60 participants; however, due to a number of factors, including 1) neuro-imaging methods require participants to adhere to strict eligibility criteria, and 2) data collection took place over the 3-month period, the beneficiary was not able to achieve the desired number of 60 participants. To overcome this barrier, arrangements between the beneficiary and secondment institution to continue data collection at the University of Graz, with aims to finish in Summer 2025. The data collection is being led by a master’s student enrolled at the University of Graz, and the beneficiary is serving as their co-supervisor. The master’s student will utilize the data collected during the study for analysis in their master’s thesis research.
The gender dimension was considered in our approach by collecting gender identify information on participants before engaging in all game-based learning tasks. These dimensions were considered and reported in our analyses and publications.
Objective 2: leveraging non-linear dynamical analyses to assess the temporal emergence of emotions before, during, and after diverse learners interact with game mechanics using multimodal data
Presently, we have conducted a series of non-linear analyses on several data channels: 1) facial recognition, 2) heart rate, 3) skin conductance response, 4) self-report data, and 5) think-/emote-aloud data. A large portion of time was dedicated to transcribing and coding the think- and/emote-aloud data following objective 1 at the host institution. The beneficiary was successful in recruiting and training a team of graduate students to facilitate qualitative coding of the verbalizations collected during the game-based learning activities. The coding took place over a period of five months in which the team coded 64 participants’ verbalizations (some participants were removed due to technical errors or very low-quality verbalizations). Now that the codes are available for analysis, we are continuing to explore different variables and data channels currently.
Presently, we have conducted several non-linear dynamical methods on the following data channels:
Facial expressions, heart-rate variability, skin conductance response, and think-/emote-aloud codes. Specifically, we have applied the following:
• Recurrence Quantification Analysis: Examined the degree of recurring values in facial expressions, skin conductance responses, and think-/emote-alouds;
• Cross-wavelet transformation Analysis: Examined the degree of synchronization (or lack thereof) between facial expressions and heart-rate variability;
• Transfer Entropy Analysis: Examined the degree of cross-channel interactions between think-/emote-alouds and skin conductance responses, think-/emote-alouds and facial expressions, and skin conductance responses and facial expressions; and,
• Sequence Analysis: Examined the degree of transitions among different think- and emote-aloud codes.
Objective 3: identify key relations between game mechanic interaction, emotion dynamics, cognition, and learning outcomes with Antidote COVID-19 using multimodal data between diverse learners
Currently, we have explored relationships between multiple variables generated from non-linear dynamical methods applied in Objective 2 to explore their relations with learning outcomes. Specifically, the beneficiary was successful in calculating statistical models using variables extracted from analyses in Objective 2. Presently, we have conducted several different methods listed below:
• Mixed Growth Modelling: Evaluated whether changes in different self-reported emotions collected during game-based learning were associated with learning outcomes after the game-based learning activity;
• Correlation: Evaluated associations between the degree of synchrony between facial expressions and heart-rate variability capture during the game-based learning activity; and,
• Hierarchical Regression and Cross-validation: Evaluating the mathematical relations between sequence analysis, recurrence quantification analysis, and transfer entropy variables in explaining variability in learning outcomes.
Generally, there are key associations between emotion variables with learning outcome measures. We evaluated whether there were significant differences in learning outcomes between participants who identified as different genders. Our results showed there were no significant differences, which were always included in our publications.
The project activities related to achieving included conducting interdisciplinary research, specifically by conducting a mixed methods study using multimodal data and non-linear dynamical analyses. Measurable indicators of our success are highlighted in several ways. First, we have published 2 high-quality research publications (so far). The publications report findings on the application of non-linear dynamical systems analysis to multiple data channels of emotional processes during game-based learning and its relation to learning outcomes. Presently, the beneficiary is preparing a manuscript that assesses relations between the think-/emote-aloud data and multimodal signals to a Special Issue on Non-linear Dynamical systems application in education to the British Journal of Educational Technology (to be submitted in Spring 2025).
The progress of the project toward delivering scientific impact including 1) creating high-quality knowledge regarding a deeper understanding of the role of emotion in game-based learning across different gender dimensions (2 peer-reviewed, open-access publications); 2) strengthening human capital in research and innovation via collaboration, workshops (3 workshops), presentations (2 conference presentations), mentorship, and transfer of knowledge and skills between the host an secondment institutions with the beneficiary; and 3) contributed toward developing and validating multimodal research approaches for combing sensor data with qualitative data during game-based learning. Specifically, we demonstrated the utility of leveraging complex systems theory and non-linear dynamical analyses to model complex dynamics and interactions among multimodal indicators of emotion components. The potential users of the project results will likely be game designers, educational psychologists, learning scientists, and instructors.
We tested the effect of different feedback scaffold designs on participants’ emotions and motivational states during game-based learning. This finding gave us insight into the role of feedback design promoting emotions and motivation beneficial for game-based learning outcomes. The implications of this research inform ways to design different feedback scaffolds to prompt or intervene when participants demonstrate the need for intervention
The progress of the project toward delivering scientific impact including 1) creating high-quality knowledge regarding a deeper understanding of the role of emotion in game-based learning across different gender dimensions (2 peer-reviewed, open-access publications); 2) strengthening human capital in research and innovation via collaboration, workshops (3 workshops), presentations (2 conference presentations), mentorship, and transfer of knowledge and skills between the host an secondment institutions with the beneficiary; and 3) contributed toward developing and validating multimodal research approaches for combing sensor data with qualitative data during game-based learning. Specifically, we demonstrated the utility of leveraging complex systems theory and non-linear dynamical analyses to model complex dynamics and interactions among multimodal indicators of emotion components. The potential users of the project results will likely be game designers, educational psychologists, learning scientists, and instructors.
We tested the effect of different feedback scaffold designs on participants’ emotions and motivational states during game-based learning. This finding gave us insight into the role of feedback design promoting emotions and motivation beneficial for game-based learning outcomes. The implications of this research inform ways to design different feedback scaffolds to prompt or intervene when participants demonstrate the need for intervention