Periodic Reporting for period 1 - COLOUR (Colourful predictions? Unraveling the resolution and neural mechanisms of colour expectation)
Okres sprawozdawczy: 2022-10-01 do 2025-03-31
Whether watching TV or navigating through city traffic, our brain constantly tries to anticipate what will happen next. This mechanism is vital, since processing of the incoming visual information and the initiation of an adequate action takes time. Predictive processing theories suggest that our brain therefore constantly predicts incoming sensory information. The level of granularity and concrete sensory content of these predictions is, however, less clear. Specifically, while it has been shown that the mere expectation of a stimulus can trigger the formation of a template of this stimulus in sensory visual areas, so far we do not know whether such template contains color information. However, color expectations and the resolution at which they operate are especially crucial for real life. Color does not only play a huge role in every day visual search (looking for traffic signs at the street or for tomatoes in a grocery store), it also facilitates face and object recognition, and supports visual memory and non-verbal communication (e.g. flushed cheeks). Most importantly, what we expect will strongly influence what we perceive and how fast we can react to it. Hence, knowing which colors can be distinguished in expectation templates would help us, e.g. to better decide on the color of warning signals or traffic signs. The overall objective of the project was to unravel for the first time the neural mechanisms and resolution underlying color expectations. Where and when are they formed in the brain and do they contain decodable sensory color information?
To assess color expectations with high spatio-temporal resolution, I first set up simultaneous EEG/MEG (electro- and magnetoencephalography) recordings, and adjusted colors in the DKL-space to make sure that they match in luminance and saturation, such that any effect observed can in the end be unambiguously linked to color hue. I collected full EEG/MEG datasets (n=37) for decoding color from the brain and testing whether color representations emerge in periods of color anticipation. As a first step, to reconstruct color representations from brain activity, we employed a color localizer task, where human participants were shown four different colors while their electromagnetic brain responses were recorded (Figure 1, A1). Using multivariate pattern analyses, we found color-related information in activity patterns peaking around 120ms after color presentation onset in early visual regions (Figure 1, A2). As a second step, participants then viewed colored stimuli presented in an 80% predictable sequence, enabling the brain to form expectations about the upcoming color (see Figure 1, B1 for an example sequence). A neutral grey placeholder was shown 400ms before the actual color stimulus to evoke a measurable visual response during color expectations (“ping approach”). A decoding model trained on the color localizer data could successfully reconstruct not only the color from the post-stimulus response (Figure 1, B2, lower row) but also from the brain activity elicited by the grey placeholder (Figure 1, B2, upper row). Our findings demonstrate that human predictions carry, indeed, decodable sensory color information in early visual cortical areas. This further specifies the level of sensory detail that is present in predictive sensory templates. Understanding what sensory information is actually contained in human expectations and at which level of detail they operate will also help to better link prediction processes to perception and attentional guidance.
The results have been presented on national and international conferences and are currently written up for open-access publication. A new study leveraging color decoding techniques and applying it to a follow-up research question is currently running under my supervision.
The results have been presented on national and international conferences and are currently written up for open-access publication. A new study leveraging color decoding techniques and applying it to a follow-up research question is currently running under my supervision.
Using a multi-class decoding model I could establish robust reconstruction of color in early visual cortical areas both in EEG and MEG (Figure 1, A2), dovetailing with latest color decoding work. I could for the first time successfully retrieve sensory color information from an expectation template formed by mere statistical regularities (color order), proving that human visual expectations carry sensory color information. I could pinpoint the cortical location as well as the time course of this template (Figure 1, B2). While I did not observe an influence on subsequent color processing in the cortex (VEP amplitudes and classifier accuracies comparable for expected and unexpected colors), participants were faster to respond to a hue change when the color matched the expectation template speaking for its behavioral relevance.
In sum, the project demonstrated the feasibility of decoding non-spatial features from the brain’s predictions. It helped to better characterize human expectations and the level of detail at which they operate, which is essential when linking predictions to other cognitive processes like perception, attention, or memory. Since data were acquired with two modalities (EEG and MEG), the results can be easily transferred to and compared with the more ubiquitous EEG work.
In sum, the project demonstrated the feasibility of decoding non-spatial features from the brain’s predictions. It helped to better characterize human expectations and the level of detail at which they operate, which is essential when linking predictions to other cognitive processes like perception, attention, or memory. Since data were acquired with two modalities (EEG and MEG), the results can be easily transferred to and compared with the more ubiquitous EEG work.