Periodic Reporting for period 1 - TIRESIA (Technology for visual Impairments Rehabilitation on Early-life through Social Information Augmentation)
Periodo di rendicontazione: 2021-09-01 al 2023-08-31
Considering that vision is an extremely important sensory modality in humans, visual impairments (VI) influence most instrumental and social activities of daily living, affecting overall quality of life. In the vast majority of cases, both if the visual impairments are caused by peripheral or cerebral lesions of the visual system, individuals with VI retain some degree of vision. Though visual disorders are less frequent in children than in adults, it is especially important to timely assess the presence of a vision’s impairment in order to program an early intervention. Indeed, problems in visual functioning affect the child’s overall development. Therefore, early planning of a visual rehabilitation therapy is crucial to improve the child’s functional vision and facilitate the development in all areas right when the brain plasticity is maximal and key competencies are developing.
The aim of this action is to contribute to connect advances in neuroscience and machine learning to study the impact of VI on key functional competencies and improve treatment strategies. Specific objectives are focusing on exploiting machine learning techniques both to investigate how visual impairments at an early age affect the development of social skills, and to develop novel tools for their training, in the context of visual rehabilitation. The ultimate goal is that of generating knowledge and data useful to develop technologies that enhance residual skills of visually impaired people and facilitate their social inclusion, independence and equal opportunities in the society.
The aim of this action is to contribute to connect advances in neuroscience and machine learning to study the impact of VI on key functional competencies and improve treatment strategies. Specific objectives are focusing on exploiting machine learning techniques both to investigate how visual impairments at an early age affect the development of social skills, and to develop novel tools for their training, in the context of visual rehabilitation. The ultimate goal is that of generating knowledge and data useful to develop technologies that enhance residual skills of visually impaired people and facilitate their social inclusion, independence and equal opportunities in the society.
The work carried out during the outgoing phase of the action took place at the Infolab, within the Computer Science and Artificial Intelligence Lab (CSAIL) of the Massachusetts Institute of Technology (MIT), and allowed to reach the following exploitable results:
• Definition of the characteristics of groups participating to the studies. We selected 3 groups of subjects (both children and adults): typically developing (TYP), with visual impairments (VI), and with Autism Spectrum Disorders (ASD). The aim is to characterize the influences of sensory perceptual issues (peripheral visual impairments) and cognitive issues (neurodevelopmental disorders such as ASD) on visual attention patterns in the context of social interactions.
Although several studies already exploited eye-tracking technology to assess visual attention in the context of ASD, they mainly used static rather than dynamic stimuli, and they typically did not consider complex social interactions among agents/humans. Furthermore, only few pioneer studies explored the possibility of applying eye-tracking to the visual assessment of children with VI, and almost no datasets from populations of impaired subjects are publicly available. This research activity will contribute to advance the state of the art by generating new open-access eye-tracking (ET) datasets suitable to characterize and model the influences of both sensory low-level issues and high-level cognitive issues on the perception of visual information that is useful for social interaction.
• Design of a dataset of dynamic (video) visual social stimuli, suitable to be used both for studies with human subjects and to train computer vision models. The dataset is suitable on one hand to characterize the development of the human ability to understand social behaviours (specifically to classify interactions between agents, distinguishing among help, hinder or no interaction). On the other hand, it can be used to train computer vision models to classify social interactions, exploiting a human-like mechanism of attributing saliency to the visual input.
• Collection of behavioural and eye-tracking data from visually impaired and healthy subjects, to characterize visual social attention. Specifically, we collected: 1) an eye-tracking dataset on static images, during a preferential looking task (social versus object) from VI children (considering both low visual acuity and nystagmus) and sighted children; 2) a behavioural dataset of answers related to recognition of facial expressions with and without face masks from sighted and VI children (including children with low visual acuity, nystagmus and saccadic issues); 3) an eye-tracking dataset (including behavioural answers) collected from sighted adults, who were asked to classify videos of social interactions (help, hinder or no interaction) among simple agents performing goal-oriented actions.
All the above-mentioned datasets will contribute to characterize how social skills develop on typical or VI individuals, and they will be made publicly available upon publication of results. The eye-tracking datasets of visual stimuli and human gaze data will be specifically annotated to make them exploitable to train computer vision models, towards the goal of developing human-like saliency models and computational models of human visual attention.
• Setup a benchmark of computational models of visual attention, to predict images/videos’ social saliency based on human-gaze data. The benchmark models will be applied to the dataset of social stimuli introduced at point 2, and it will provide a baseline for developing human-like computational models of visual attention. This result is relevant for applications of machine learning tools supporting the rehabilitation of visual impairments. Furthermore, the output from this research activity will be likely to be exploitable beyond VI, in the context of assessment and rehabilitation of other developmental disorders affecting social behaviours, such as ADHD (attention deficit hyperactivity disorder) and ASD.
• Definition of the characteristics of groups participating to the studies. We selected 3 groups of subjects (both children and adults): typically developing (TYP), with visual impairments (VI), and with Autism Spectrum Disorders (ASD). The aim is to characterize the influences of sensory perceptual issues (peripheral visual impairments) and cognitive issues (neurodevelopmental disorders such as ASD) on visual attention patterns in the context of social interactions.
Although several studies already exploited eye-tracking technology to assess visual attention in the context of ASD, they mainly used static rather than dynamic stimuli, and they typically did not consider complex social interactions among agents/humans. Furthermore, only few pioneer studies explored the possibility of applying eye-tracking to the visual assessment of children with VI, and almost no datasets from populations of impaired subjects are publicly available. This research activity will contribute to advance the state of the art by generating new open-access eye-tracking (ET) datasets suitable to characterize and model the influences of both sensory low-level issues and high-level cognitive issues on the perception of visual information that is useful for social interaction.
• Design of a dataset of dynamic (video) visual social stimuli, suitable to be used both for studies with human subjects and to train computer vision models. The dataset is suitable on one hand to characterize the development of the human ability to understand social behaviours (specifically to classify interactions between agents, distinguishing among help, hinder or no interaction). On the other hand, it can be used to train computer vision models to classify social interactions, exploiting a human-like mechanism of attributing saliency to the visual input.
• Collection of behavioural and eye-tracking data from visually impaired and healthy subjects, to characterize visual social attention. Specifically, we collected: 1) an eye-tracking dataset on static images, during a preferential looking task (social versus object) from VI children (considering both low visual acuity and nystagmus) and sighted children; 2) a behavioural dataset of answers related to recognition of facial expressions with and without face masks from sighted and VI children (including children with low visual acuity, nystagmus and saccadic issues); 3) an eye-tracking dataset (including behavioural answers) collected from sighted adults, who were asked to classify videos of social interactions (help, hinder or no interaction) among simple agents performing goal-oriented actions.
All the above-mentioned datasets will contribute to characterize how social skills develop on typical or VI individuals, and they will be made publicly available upon publication of results. The eye-tracking datasets of visual stimuli and human gaze data will be specifically annotated to make them exploitable to train computer vision models, towards the goal of developing human-like saliency models and computational models of human visual attention.
• Setup a benchmark of computational models of visual attention, to predict images/videos’ social saliency based on human-gaze data. The benchmark models will be applied to the dataset of social stimuli introduced at point 2, and it will provide a baseline for developing human-like computational models of visual attention. This result is relevant for applications of machine learning tools supporting the rehabilitation of visual impairments. Furthermore, the output from this research activity will be likely to be exploitable beyond VI, in the context of assessment and rehabilitation of other developmental disorders affecting social behaviours, such as ADHD (attention deficit hyperactivity disorder) and ASD.
Results from this action will help us to better evaluate how humans process socially relevant visual information. In particular, our hypothesis is that there is a significant difference in human visual attention patterns when observing a socially relevant vs a non-relevant visual input, and that this difference can be modeled computationally. Furthermore, we hypothesize that both sensory perceptual low-level and neurological high-level issues affect the resulting social saliency. Indeed, our results show that VI can significantly impact the attention towards socially relevant visual information. With this project, we aim to understand the social abilities of people with ASD and VI using a model-driven approach. We have a model that for the first time agrees with human subjects when viewing videos of social interactions: we want to investigate whether this model is sensitive enough to recognize impairments that people with ASD and VI have. The expected results from this approach are two-fold: 1) exploiting data from human subjects with ASD and VI as a ground truth we can discover what in the model’s architecture is contributing to differences in social perception between these populations; 2) quantifying with an objective model the extent to which different groups have more limited social perception, and under what conditions, could lead to a much better understanding of these conditions and to better planning of interventions. Comparing data from the groups of individuals with ASD and VI could lead to findings that can change our understanding of both.