Periodic Reporting for period 2 - COLOURCODE (The Mind's Eye: Decoding Colour Experience)
Reporting period: 2022-06-01 to 2023-11-30
The most extraordinary products of the human mind are personal, subjective phenomenal experiences such as the qualitative experience of the redness of red. Phenomenal experiences define how we as humans experience the world, so the question of how they arise (seemingly intractable ever), is a key part of understanding ourselves. The COLOURCODE project aims to use colour, as simple and archetypal subjective experiences, as a model system in neuroscience as it has been used in philosophy to make inroads into the underpinnings of subjective experience. COLOURCODE tackles this research question from three angles. First, by better characterising cortical colour representations to narrow the gap between what we know about the colour representations in the brain that must contribute to subjective colour experience and the mechanisms of the experiences themselves. Second, by investigating the neural basis of the phenomenon of illusory flicker-induced colour percepts, which isolate subjective experiences of colour from coloured stimuli in the external world. Third, by using individual differences as a tool to investigate how the interaction between neural hardware and colours in the external world constrains subjective colour experiences. Using these approaches COLOURCODE aims to (i) characterise cortical colour representations and measure the properties of cortical colour mechanisms, (ii) understand what colour information is represented in neural oscillations and (iii) investigate how colour experience is constrained by the number and type of retinal receptors and by colour information in the external world.
At the mid-point of the project we have made progress towards all three main aims. We employ diverse methods in neuroimaging, computational modelling and psychophysics to try to understand subjective colour experience, its variation and its neural underpinnings.
Towards our first aim of using neuroimaging to understand cortical colour representation, we have analysed a very high quality functional magnetic resonance imaging (fMRI) dataset where whole brain responses were measured to 30,000 images of natural scenes in 8 participants. We have developed a toolbox to analyse the colour properties of the images, and have used it to characterise brain responses to colour in images of natural scenes. This research has led to our discovery of the ventral food stream, a series of cortical brain areas that are selective for food objects. Our findings, published in Current Biology, support the idea that there are brain areas specialised for representing the visual properties of food, in which colour has a key role.
Using the same high quality fMRI dataset We have developed a new method for identifying brain regions that have distinct functions. We create ‘average’ images preferred by individual voxels in the fMRI measurements. When we plot the colours of the average images at the locations of the voxels on a brain map, boundaries between brain regions coloured differently indicate functional distinctions between areas that respond to different visual features that are correlated with different colours in the image set. Our method provides a data-driven approach for finding clusters of voxels similar visual preferences and could be used to find and define regions of interest for research.
We have conducted a series of electroencephalography (EEG) experiments where we induce steady-state visually-evoked potentials (SSVEPs), in a novel method to measure the properties of cortical colour mechanisms. Our results reveal broadly tuned neural populations (that respond to a relatively large range of hues) and that unexpectedly represent opponent (opposite) colours, e.g. blue and yellow.
Towards our second aim of investigating the involvement of oscillatory brain activity in subjective colour experience, we have pursued three research directions. First, we have used machine learning to decode colour from oscillations in magnetoencephalography (MEG) and EEG data that used coloured stimuli. Second, we are currently writing a review paper on the complex phenomenon of illusory colours provoked from black and white flickering light. Third, we are building a stroboscope to produce precisely timed black and white flicker with which to research these illusory colours during the second half of the project.
Towards our third aim of using individual differences in colour perception to answer questions about subjective colour experience, we have pursued several research avenues. We have written a comprehensive review paper on individual differences in colour perception, outlining what is known about individual differences in different domains of colour perception and their causes, and identifying gaps in knowledge to be addressed. In the review we showed that individual differences in the subjective experience of yellow can be predicted by an interaction between observer-specific receptors and experience with natural illuminations. We are continuing this work to predict subjectively pure colours from an interaction between cone receptors and the broader properties of colours in natural scenes.
We have investigated individual differences in the neural representation of colours simultaneously in the retina (using electroretinography, EEG) and in the cortex (using EEG). Preliminary results show that colour representations of people with normal colour vision and people with colour vision deficiency differ in the retina but converge in the cortex, implying that people with colour vision deficiency amplify weakened retinal colour signals to relatively normalise subjective colour experience. In related work we have shown that an illusory colour after effect is enhanced in people with colour vision deficiency, and that the pattern of results can be explained by amplification. We have also investigated the effect of filters that claim to enhance colour experience in anomalous trichromacy. In both modelling and empirical studies our work has been the first to show that they can be effective.
We have experimentally induced individual differences in colour perception by manipulating the colour properties of the scenes observers are currently immersed in. Results have shown that colour discrimination, perception of colour variance and aesthetic preferences are all influenced by the current colour environment. During the COVID-19 pandemic when research with human participants was restricted, we investigated individual differences in colour representations inside deep neural nets, specifically how representations are affected by the training task demands. Our work could help us understand how differences in human colour representations are influenced by the tasks involving colour that individuals must complete in their daily lives or early in life.
Towards our first aim of using neuroimaging to understand cortical colour representation, we have analysed a very high quality functional magnetic resonance imaging (fMRI) dataset where whole brain responses were measured to 30,000 images of natural scenes in 8 participants. We have developed a toolbox to analyse the colour properties of the images, and have used it to characterise brain responses to colour in images of natural scenes. This research has led to our discovery of the ventral food stream, a series of cortical brain areas that are selective for food objects. Our findings, published in Current Biology, support the idea that there are brain areas specialised for representing the visual properties of food, in which colour has a key role.
Using the same high quality fMRI dataset We have developed a new method for identifying brain regions that have distinct functions. We create ‘average’ images preferred by individual voxels in the fMRI measurements. When we plot the colours of the average images at the locations of the voxels on a brain map, boundaries between brain regions coloured differently indicate functional distinctions between areas that respond to different visual features that are correlated with different colours in the image set. Our method provides a data-driven approach for finding clusters of voxels similar visual preferences and could be used to find and define regions of interest for research.
We have conducted a series of electroencephalography (EEG) experiments where we induce steady-state visually-evoked potentials (SSVEPs), in a novel method to measure the properties of cortical colour mechanisms. Our results reveal broadly tuned neural populations (that respond to a relatively large range of hues) and that unexpectedly represent opponent (opposite) colours, e.g. blue and yellow.
Towards our second aim of investigating the involvement of oscillatory brain activity in subjective colour experience, we have pursued three research directions. First, we have used machine learning to decode colour from oscillations in magnetoencephalography (MEG) and EEG data that used coloured stimuli. Second, we are currently writing a review paper on the complex phenomenon of illusory colours provoked from black and white flickering light. Third, we are building a stroboscope to produce precisely timed black and white flicker with which to research these illusory colours during the second half of the project.
Towards our third aim of using individual differences in colour perception to answer questions about subjective colour experience, we have pursued several research avenues. We have written a comprehensive review paper on individual differences in colour perception, outlining what is known about individual differences in different domains of colour perception and their causes, and identifying gaps in knowledge to be addressed. In the review we showed that individual differences in the subjective experience of yellow can be predicted by an interaction between observer-specific receptors and experience with natural illuminations. We are continuing this work to predict subjectively pure colours from an interaction between cone receptors and the broader properties of colours in natural scenes.
We have investigated individual differences in the neural representation of colours simultaneously in the retina (using electroretinography, EEG) and in the cortex (using EEG). Preliminary results show that colour representations of people with normal colour vision and people with colour vision deficiency differ in the retina but converge in the cortex, implying that people with colour vision deficiency amplify weakened retinal colour signals to relatively normalise subjective colour experience. In related work we have shown that an illusory colour after effect is enhanced in people with colour vision deficiency, and that the pattern of results can be explained by amplification. We have also investigated the effect of filters that claim to enhance colour experience in anomalous trichromacy. In both modelling and empirical studies our work has been the first to show that they can be effective.
We have experimentally induced individual differences in colour perception by manipulating the colour properties of the scenes observers are currently immersed in. Results have shown that colour discrimination, perception of colour variance and aesthetic preferences are all influenced by the current colour environment. During the COVID-19 pandemic when research with human participants was restricted, we investigated individual differences in colour representations inside deep neural nets, specifically how representations are affected by the training task demands. Our work could help us understand how differences in human colour representations are influenced by the tasks involving colour that individuals must complete in their daily lives or early in life.
Our research on the representation of colour in the ‘colour biased regions’ of the ventral visual pathway unexpectedly revealed that the regions were most activated by the presence of food in the images presented to participants. After intensive investigation of the responses of these regions we concluded that they are selective for food. This was a significant advance on the current state of the art. A visual pathway specialised for processing food images makes evolutionary sense, and our work has shown that colour representation is intimately tied with food representation in the colour biased regions. Our work has accelerated the direction of travel of current thinking about representation of colour in these brain regions, that it is “utility-based”, and tied statistically and functionally to other relevant visual properties of objects.
Our work on voxel preferred images has produced unexpectedly clear boundaries between brain areas which must give functionally distinct responses to images of natural scenes. The novel method we have developed is relatively simple and very effective, and could be applied to other large fMRI datasets. It offers the potential to investigate new functionally defined brain regions and to match them between participants. The method is beyond the current state-of-the-art and has potential as a tool to enhance the field of functional brain mapping.
In the second half of the project we will continue to progress the work we have established on colour representations in the cortex, and on individual differences in the subjective experience of colour. We will work intensively on research scheduled for the second half of the project, investigating illusory colours induced by black and white flicker and how they arise in the brain.
Our work on voxel preferred images has produced unexpectedly clear boundaries between brain areas which must give functionally distinct responses to images of natural scenes. The novel method we have developed is relatively simple and very effective, and could be applied to other large fMRI datasets. It offers the potential to investigate new functionally defined brain regions and to match them between participants. The method is beyond the current state-of-the-art and has potential as a tool to enhance the field of functional brain mapping.
In the second half of the project we will continue to progress the work we have established on colour representations in the cortex, and on individual differences in the subjective experience of colour. We will work intensively on research scheduled for the second half of the project, investigating illusory colours induced by black and white flicker and how they arise in the brain.