Periodic Reporting for period 2 - TrackingMinds (Tracking the decisions of others with the own mind)
Reporting period: 2022-07-01 to 2023-12-31
A central research question in cognitive neuroscience, psychology, and economics is to understand how humans interpret the behavior of others. Despite much research on humans’ Theory of Mind, we still lack a computationally and neurobiologically plausible model. In the ERC project TrackingMinds, we propose to fill this gap by testing the hypothesis that humans utilize their own decision-making system to observe and understand actions by simulating them with their own mind.
The multi-modal and mathematically rigorous approach of TrackingMinds will advance our understanding of human mentalizing abilities profoundly. This improved knowledge can lead to many societal benefits. First, it can help to devise better interactive AI systems that take not only decision outcomes but also process information such as response times or eye movements into account. Second, it can have strong implications for equilibrium predictions in economic theory, because not only which but also how a decision is made can reveal critical information about underlying beliefs, preferences, and knowledge. Third, it can foster new interventions of social mental disorders such as autism and social anxiety, which might be characterized by a limited ability to simulate others’ minds.
The objectives of TrackingMinds are (1) to investigate whether people take process information of decisions such as response times and eye movements into account when inferring others’ preferences, and if so, whether their inferences and predictions of others align with our computational framework. These predictions will also be tested on the neural level, with a focus on connecting decision neuroscience with research on the neural basis of time perception. Furthermore, we seek (2) to test whether humans simulate others’ decisions multiple times to improve their predictions. This research question will be assessed on both the behavioral as well as the neural level. Finally, we seek (3) to devise a multi-agent artificial intelligence (AI) system that takes process information into account to improve human-AI interactions in coordination tasks and cooperative settings.
The multi-modal and mathematically rigorous approach of TrackingMinds will advance our understanding of human mentalizing abilities profoundly. This improved knowledge can lead to many societal benefits. First, it can help to devise better interactive AI systems that take not only decision outcomes but also process information such as response times or eye movements into account. Second, it can have strong implications for equilibrium predictions in economic theory, because not only which but also how a decision is made can reveal critical information about underlying beliefs, preferences, and knowledge. Third, it can foster new interventions of social mental disorders such as autism and social anxiety, which might be characterized by a limited ability to simulate others’ minds.
The objectives of TrackingMinds are (1) to investigate whether people take process information of decisions such as response times and eye movements into account when inferring others’ preferences, and if so, whether their inferences and predictions of others align with our computational framework. These predictions will also be tested on the neural level, with a focus on connecting decision neuroscience with research on the neural basis of time perception. Furthermore, we seek (2) to test whether humans simulate others’ decisions multiple times to improve their predictions. This research question will be assessed on both the behavioral as well as the neural level. Finally, we seek (3) to devise a multi-agent artificial intelligence (AI) system that takes process information into account to improve human-AI interactions in coordination tasks and cooperative settings.
Since the start of the project, we have set up the computational framework of understanding others’ actions and performed behavioral, online, and EEG studies. We could show in a behavioral study that people are able to infer social preferences of others by only observing how fast they make decisions. Remarkably, the speed with which they predicted others’ decisions matched those decisions, even if their own social preferences were dissimilar. This finding provides strong support for the simulation hypothesis, that is, the assumption that humans simulate others’ actions with their own mind. Results from an online study lent further support to our theory, as we were able to manipulate how people predict the risky decisions of another person by adapting their own decision-making system to specific risk environments. Both projects are currently followed up on by EEG studies that aim to delineate the underlying neural mechanisms of predicting and learning about others’ decisions. Finally, we have started to devise an interactive decision-making (bargaining) task, in which two persons take on the roles of seller and buyer, respectively. We seek to demonstrate that the seller can learn the buyer’s hidden preferences by observing their decisions and response times, and that this learning is further improved when the seller has access to the buyer’s eye movements.
To the best of our knowledge, we are the first to show that humans can infer hidden social preferences from observing the speed of decisions alone, that is, without having access to the decision itself. Furthermore, we are not only showing the effect but also providing an explanatory computational framework to capture the learning dynamics. As a next step, we seek to relate this ability to infer information from decision speed with research in time perception, which bears remarkable similarities and overlap in terms of both neural mechanisms as well as computational models. We also expect an ongoing EEG study to help us better understand the neural basis of predicting the decisions of similar and dissimilar other people. Furthermore, we will test the hypothesis that people perform multiple simulations of others to improve their predictions, using computational modeling as well as representational similarity analyses of fMRI data. Initial behavioral analyses are consistent with this multiple-simulation idea. Finally, we will apply EEG hyperscanning in the context of our interactive bargaining task to study improved neural alignment in reaching agreements in this task. Our insights from this task on different levels (behavior, eye movements, brain activity) will then be used to devise a multi-agent AI system that makes effective use of the rich body of decision process information to coordinate better with a human partner.