Periodic Reporting for period 3 - HOMEOSTASIS (The mental economy: testing a dynamic trade-off between internal storage and external sampling)
Berichtszeitraum: 2023-09-01 bis 2025-02-28
While interacting with the external world, the brain can only represent very little of this world in working memory (WM). WM is therefore generally referred to as a limited-capacity system. This limitation is not a problem in daily life, however, because the external world typically remains available and can be accessed relatively easily. The current dominant theory of WM does not explain how the brain balances between internal storage and external sampling, as this theory exclusively relates to situations in which the remembered information is no longer physically present. The HOMEOSTASIS project is motivated by the idea that WM should be studied in interaction with the world that is still within view.
HOMEOSTASIS will develop a new theoretical model of WM based on an internal mental economy: we hypothesize that WM maintains a perceptual homeostasis by dynamically trading the costs of accurate internal storage against external sampling of the external visual world. Whereas current research on WM has a strong focus on its maximum capacity, this capacity may hardly be used as observers prefer to minimize internal storage due to the effortful nature of WM storage.
We rigorously test the model’s theoretical basis using novel experimental paradigms in which WM is studied in interaction with the physically present environment. To decode the current content of WM, we adopt state-of-the-art electroencephalographic decoding techniques. Finally, we investigate patients with restricted deficits to specific components of the model and use machine learning techniques to discover biometric signatures in eye movements. This new model of WM will open a new window to diagnose WM disorders and for understanding how we interact with computer-manipulated virtual environments in an increasingly computer-dominated world.
The world that we consciously experience is an internal representation of the world, rather than the physical world itself. Through the different stages of perception, observers construct a model of the physical world by internalizing certain aspects of the visual information that is presented to our retina. Although research on visual perception has made great leaps forward in understanding which aspects of the visual world are selected by the brain for further processing, the HOMEOSTASIS project will fundamentally advance vision research: what determines which visual information is maintained in internal memory after it has been selected? This research program has various potential future applications:
First, the project will result in clear hypotheses about which properties of the environment determine the amount of information that will be internalized (e.g. reliability and familiarity). As the amount of internalized information is an index of the current memory load, this knowledge will be of great assistance in designing efficient (work) environments in which memory load is minimized.
Second, our clinical work will result in the development of a novel screening tool for specific dysfunctions in VWM by revealing the oculomotor features (biomarkers) that best reflect deficits in visual memory. Given the rise of easy-to-use and low-cost eye trackers, this approach is promising and feasible. In the near future, the cameras in mobile devices, such as tablets and smartphones, will be used to record eye movements, making such data widely available. In future studies, this tool could be extended to other neurological and developmental disorders associated with working memory deficits, such as Alzheimer’s disease and ADHD.
Third, future research programs can build on the HOMEOSTASIS project by studying how the conscious perception of the world is modulated by brain damage (e.g. hallucinations), during episodes of drug use (e.g. LSD), and during psychiatric episodes (e.g. psychosis). In these situations, it is sometimes unclear to observers whether an experience is real or false. These conditions represent extreme cases of a world that is (permanently) unreliable for an observer.
HOMEOSTASIS will develop a new theoretical model of WM based on an internal mental economy: we hypothesize that WM maintains a perceptual homeostasis by dynamically trading the costs of accurate internal storage against external sampling of the external visual world. Whereas current research on WM has a strong focus on its maximum capacity, this capacity may hardly be used as observers prefer to minimize internal storage due to the effortful nature of WM storage.
We rigorously test the model’s theoretical basis using novel experimental paradigms in which WM is studied in interaction with the physically present environment. To decode the current content of WM, we adopt state-of-the-art electroencephalographic decoding techniques. Finally, we investigate patients with restricted deficits to specific components of the model and use machine learning techniques to discover biometric signatures in eye movements. This new model of WM will open a new window to diagnose WM disorders and for understanding how we interact with computer-manipulated virtual environments in an increasingly computer-dominated world.
The world that we consciously experience is an internal representation of the world, rather than the physical world itself. Through the different stages of perception, observers construct a model of the physical world by internalizing certain aspects of the visual information that is presented to our retina. Although research on visual perception has made great leaps forward in understanding which aspects of the visual world are selected by the brain for further processing, the HOMEOSTASIS project will fundamentally advance vision research: what determines which visual information is maintained in internal memory after it has been selected? This research program has various potential future applications:
First, the project will result in clear hypotheses about which properties of the environment determine the amount of information that will be internalized (e.g. reliability and familiarity). As the amount of internalized information is an index of the current memory load, this knowledge will be of great assistance in designing efficient (work) environments in which memory load is minimized.
Second, our clinical work will result in the development of a novel screening tool for specific dysfunctions in VWM by revealing the oculomotor features (biomarkers) that best reflect deficits in visual memory. Given the rise of easy-to-use and low-cost eye trackers, this approach is promising and feasible. In the near future, the cameras in mobile devices, such as tablets and smartphones, will be used to record eye movements, making such data widely available. In future studies, this tool could be extended to other neurological and developmental disorders associated with working memory deficits, such as Alzheimer’s disease and ADHD.
Third, future research programs can build on the HOMEOSTASIS project by studying how the conscious perception of the world is modulated by brain damage (e.g. hallucinations), during episodes of drug use (e.g. LSD), and during psychiatric episodes (e.g. psychosis). In these situations, it is sometimes unclear to observers whether an experience is real or false. These conditions represent extreme cases of a world that is (permanently) unreliable for an observer.
- We have made several important steps towards understanding how visual working memory is employed In naturalistic settings in which the information remains physically present. We showed for example that when it is possible to re-inspect relevant information in the world, humans often do so. Moreover, we showed that in such situations not all of the information that is memorized is used, some information remains in memory unused. We developed a theoretical model that can capture such behavior in naturalistic tasks. Currently we are testing our model to see how well it generalized to various other naturalistic contexts (e.g. when actions have consequences, when one needs to perform under pressure) and thereby improving the model. What is becoming increasingly apparent is the importance of the various cognitive strategies humans employ in various settings.
- We also made great strides in assessing the costs of sampling information in the environment and in uncovering how distinct stages of visual working memory can be studied in neurotypical and clinical populations. For instance, we were able to set up collaborations with several clinical settings, and managed to collect data in patients with Korsakoff’s syndrome, patients that were referred from an outpatient memory clinic, and patients that suffered from stroke. Where earlier studies investigating the trade-off between sampling externally and storing internally focused on what happens when sampling costs are increased, we were able to investigate what happens when storing is impaired. We identified differences in eye-movement patterns between patients with Korsakoff’s syndrome and controls, and are now aiming to investigate whether and how subjective rather than objective memory impairments influence sampling behavior.
- We adopted pupil size as a physiological measure to gain insights into these questions. In our recent work, we have shown that a) planning saccades is more costly than shifting attention covertly and that b) distinct pupillary signals inform about different stages of visual working memory use (i.e. encoding, maintenance and selection).
- We use EEG multivariate pattern analysis to quantify the representational strength of task-relevant features in anticipation of a change-detection task. We find that task-relevant (attended) memorized features are more strongly represented than irrelevant (unattended) features. More importantly, we find that task-relevant features evoke significantly weaker representations when they are perceptually available compared to when they are unavailable. These findings demonstrate that, contrary to what subjective experience suggests, vividly perceived stimuli elicit weaker neural representations (in terms of detectable multivariate information) than the same stimuli maintained in visual working memory. Apparently, an efficient visual system spends little of its limited resources on the internal representation of information that is externally available anyway.
- We also made great strides in assessing the costs of sampling information in the environment and in uncovering how distinct stages of visual working memory can be studied in neurotypical and clinical populations. For instance, we were able to set up collaborations with several clinical settings, and managed to collect data in patients with Korsakoff’s syndrome, patients that were referred from an outpatient memory clinic, and patients that suffered from stroke. Where earlier studies investigating the trade-off between sampling externally and storing internally focused on what happens when sampling costs are increased, we were able to investigate what happens when storing is impaired. We identified differences in eye-movement patterns between patients with Korsakoff’s syndrome and controls, and are now aiming to investigate whether and how subjective rather than objective memory impairments influence sampling behavior.
- We adopted pupil size as a physiological measure to gain insights into these questions. In our recent work, we have shown that a) planning saccades is more costly than shifting attention covertly and that b) distinct pupillary signals inform about different stages of visual working memory use (i.e. encoding, maintenance and selection).
- We use EEG multivariate pattern analysis to quantify the representational strength of task-relevant features in anticipation of a change-detection task. We find that task-relevant (attended) memorized features are more strongly represented than irrelevant (unattended) features. More importantly, we find that task-relevant features evoke significantly weaker representations when they are perceptually available compared to when they are unavailable. These findings demonstrate that, contrary to what subjective experience suggests, vividly perceived stimuli elicit weaker neural representations (in terms of detectable multivariate information) than the same stimuli maintained in visual working memory. Apparently, an efficient visual system spends little of its limited resources on the internal representation of information that is externally available anyway.
We have shown how we can gain evidence on VWM usage from eye movements, for instance by showing that certain aspects of eye movements are different between neurotypical controls and patients with traumatic brain injury, and between various states of arousal (such as low or high mental effort). One of the major upcoming aims will be to investigate whether we can dissociate VWM usage between neurotypicals and various patient populations. If successful, we will develop a novel screening tool for specific dysfunctions in VWM by revealing the oculomotor features (biomarkers) that best reflect deficits in visual memory.
In the coming years, we will also adopt virtual reality (VR) as a research technique as it allows observers to navigate in a world that is completely created and controlled by computers. In contrast to the physical environment, we can use VR to systematically manipulate the reliability and familiarity of the environment during natural behavior and objectify how these properties of the environment influence the trade-off between internal storage and external sampling. We are currently developing a novel set of experiments in which this new technology is used to its full potential to understand the fundamental functioning of the human brain.
At the end of the project, we will have developed a theoretical framework that establishes VWM as an embodied system that aims to establish perceptual homeostasis by only storing visual information about the external world when the need arises. In this framework, we will have tested the influence of the 1) cost of sampling, 2) cost of internal storage, 3) familiarity of the environment, and 4) reliability of the external world. Experiments are currently ongoing regarding all four of these factors.
In the coming years, we will also adopt virtual reality (VR) as a research technique as it allows observers to navigate in a world that is completely created and controlled by computers. In contrast to the physical environment, we can use VR to systematically manipulate the reliability and familiarity of the environment during natural behavior and objectify how these properties of the environment influence the trade-off between internal storage and external sampling. We are currently developing a novel set of experiments in which this new technology is used to its full potential to understand the fundamental functioning of the human brain.
At the end of the project, we will have developed a theoretical framework that establishes VWM as an embodied system that aims to establish perceptual homeostasis by only storing visual information about the external world when the need arises. In this framework, we will have tested the influence of the 1) cost of sampling, 2) cost of internal storage, 3) familiarity of the environment, and 4) reliability of the external world. Experiments are currently ongoing regarding all four of these factors.