Periodic Reporting for period 1 - MOVES (MOdelling Vocal Expression in Schizophrenia)
Reporting period: 2021-02-01 to 2023-01-31
Can computational and machine learning approaches to speech analysis (e.g. natural language processing (NLP), speech signal technology (SST)) be used in a diagnostic and prognostic fashion to support clinicians in the diagnosis and monitoring of neuropsychiatric condition and their clinical symptoms?
Individuals with neuropsychiatric disorders present speech and language atypicalities often associated with clinical symptoms. The assessment, monitoring, and treatment of these disorders in psychiatric practice is deeply rooted in human communication: the use of SST and NLP can therefore assist clinicians in the diagnosis and monitoring of these disorders by providing them with quantitative measures of clinical features related to speech. However, our current understanding of speech and language atypicalities in neuropsychiatric conditions is very poor and is limited by the lack of a comprehensive and systematic approach, little consideration of the heterogeneity of disorders and the replicability and generalizability of previous findings. These are reasons that have so far prevented the development of effective clinical applications of these technologies in neuropsychiatric conditions.
The MOVES project aimed to provide the first comprehensive account of the mechanisms underlying voice and speech atypicalities in schizophrenia, assess their impact on clinical evaluation, and lay the ground for more reliable and evidence-based screening tools. The first goal of the project was to explicitly assess how well previous speech and NLP findings generalize across a large cross-linguistic and heterogeneous dataset of voice recordings from patients with schizophrenia and controls. The second major goal of the study was to determine how various factors, i.e. sociodemographic, clinical, contextual (e.g. speech task), and cultural factors, interact in affecting speech and language production in schizophrenia. In addition, the project aimed to identify and overcome other important gaps in the current literature, such as the need for a stronger collaborative effort aimed to collect larger shared multilingual corpus representative of the heterogeneity of the schizophrenia spectrum, the need for a cumulative approach able to build on previous findings, and critical reflection on the potentialities and risks of speech-based clinical applications.
Individuals with neuropsychiatric disorders present speech and language atypicalities often associated with clinical symptoms. The assessment, monitoring, and treatment of these disorders in psychiatric practice is deeply rooted in human communication: the use of SST and NLP can therefore assist clinicians in the diagnosis and monitoring of these disorders by providing them with quantitative measures of clinical features related to speech. However, our current understanding of speech and language atypicalities in neuropsychiatric conditions is very poor and is limited by the lack of a comprehensive and systematic approach, little consideration of the heterogeneity of disorders and the replicability and generalizability of previous findings. These are reasons that have so far prevented the development of effective clinical applications of these technologies in neuropsychiatric conditions.
The MOVES project aimed to provide the first comprehensive account of the mechanisms underlying voice and speech atypicalities in schizophrenia, assess their impact on clinical evaluation, and lay the ground for more reliable and evidence-based screening tools. The first goal of the project was to explicitly assess how well previous speech and NLP findings generalize across a large cross-linguistic and heterogeneous dataset of voice recordings from patients with schizophrenia and controls. The second major goal of the study was to determine how various factors, i.e. sociodemographic, clinical, contextual (e.g. speech task), and cultural factors, interact in affecting speech and language production in schizophrenia. In addition, the project aimed to identify and overcome other important gaps in the current literature, such as the need for a stronger collaborative effort aimed to collect larger shared multilingual corpus representative of the heterogeneity of the schizophrenia spectrum, the need for a cumulative approach able to build on previous findings, and critical reflection on the potentialities and risks of speech-based clinical applications.
The MOVES project followed the following workflow:
1) First, we systematically reviewed and meta-analyzed the evidence for distinctive speech and language patterns in schizophrenia. The meta-analyses (MA) indicated a plausible speech and language profile associated with schizophrenia, i.e. reduced speech production, pitch variability, and semantic coherence. However, the MA also identified some critical issues, such as: i) small sample size of previous studies ii) limited attention to the heterogeneity of the disorder iii) limited attempts at theory-driven research that directly addresses the mechanisms underlying atypical language patterns iv) no cross-linguistic studies assessing the role of language and cultural diversity.
2) We thus collected a large cross-linguistic dataset of audio recordings of patients with schizophrenia (n=437) and controls (n=445) in four different languages (Danish, German, Mandarin-Chinese, Japanese). This corpus allowed us to: (a) systematically assess the generalizability of results from MA across different languages and samples for the first time in schizophrenia, thus better accounting for cross-linguistic differences2,3; b) provide a comprehensive assessment of patients' symptomatology and clinical profile by modeling the relationships between acoustic features and pharmacotherapy, relevant clinical aspects, demographic and social features; c) test the cross-linguistic generalizability of voice-based machine learning models (ML) for predicting the diagnosis of schizophrenia.
3) Overall, we found little evidence for a universal speech and language profile characterizing schizophrenia. Speech and language patterns were highly heterogeneous, with different sources of heterogeneity interacting at different levels. We found that only two results from the previous MA could be generalized across all the different languages of our corpus (lower pitch variability and lower sentence-level coherence), whereas most speech and language features showed different patterns across languages or were inconsistent with results from MA. Cross-linguistic generalization of voice-based ML models for predicting schizophrenia was generally low too, with performance close to change when a model was trained on a language (e.g. Danish) and tested on a different language (e.g. Chinese). These results raise some questions about the generalizability of previous findings and the possibility of building on them cumulatively. However, they also point to where future attempts might be directed: larger shared multilingual corpora representative of the heterogeneity of the schizophrenia spectrum, a more systematic assessment of the generalizability of results under more varied conditions to account for heterogeneity of disorders, more collaborative work and multi-sites studies.
4) To address this issue, the project contributed to establish and foster an international collaborative network to study the mechanisms underlying speech and language markers in schizophrenia. This network included several partners, such as the DISCOURSE consortium (https://discourseinpsychosis.org/) and it is now contributing to: a) promote large-scale world-wide cross-linguistic data collection c) homogenize the speech data collection procedure across sites c) create an open data repository d) seek concrete opportunities for clinical trials application and research funding.
5) The project produced 7 scientific papers published in international journals and 7 conference abstracts. The work carried during the project has been disseminated in many different ways, including social networks, conferences, workshops and fairs. A special topic in Frontiers in Psychology has been published, and a workshop titled “Voice- and speech-based markers of neuropsychiatric conditions: assessing methodological foundations for clinical application” has been held at the host institution.
1) First, we systematically reviewed and meta-analyzed the evidence for distinctive speech and language patterns in schizophrenia. The meta-analyses (MA) indicated a plausible speech and language profile associated with schizophrenia, i.e. reduced speech production, pitch variability, and semantic coherence. However, the MA also identified some critical issues, such as: i) small sample size of previous studies ii) limited attention to the heterogeneity of the disorder iii) limited attempts at theory-driven research that directly addresses the mechanisms underlying atypical language patterns iv) no cross-linguistic studies assessing the role of language and cultural diversity.
2) We thus collected a large cross-linguistic dataset of audio recordings of patients with schizophrenia (n=437) and controls (n=445) in four different languages (Danish, German, Mandarin-Chinese, Japanese). This corpus allowed us to: (a) systematically assess the generalizability of results from MA across different languages and samples for the first time in schizophrenia, thus better accounting for cross-linguistic differences2,3; b) provide a comprehensive assessment of patients' symptomatology and clinical profile by modeling the relationships between acoustic features and pharmacotherapy, relevant clinical aspects, demographic and social features; c) test the cross-linguistic generalizability of voice-based machine learning models (ML) for predicting the diagnosis of schizophrenia.
3) Overall, we found little evidence for a universal speech and language profile characterizing schizophrenia. Speech and language patterns were highly heterogeneous, with different sources of heterogeneity interacting at different levels. We found that only two results from the previous MA could be generalized across all the different languages of our corpus (lower pitch variability and lower sentence-level coherence), whereas most speech and language features showed different patterns across languages or were inconsistent with results from MA. Cross-linguistic generalization of voice-based ML models for predicting schizophrenia was generally low too, with performance close to change when a model was trained on a language (e.g. Danish) and tested on a different language (e.g. Chinese). These results raise some questions about the generalizability of previous findings and the possibility of building on them cumulatively. However, they also point to where future attempts might be directed: larger shared multilingual corpora representative of the heterogeneity of the schizophrenia spectrum, a more systematic assessment of the generalizability of results under more varied conditions to account for heterogeneity of disorders, more collaborative work and multi-sites studies.
4) To address this issue, the project contributed to establish and foster an international collaborative network to study the mechanisms underlying speech and language markers in schizophrenia. This network included several partners, such as the DISCOURSE consortium (https://discourseinpsychosis.org/) and it is now contributing to: a) promote large-scale world-wide cross-linguistic data collection c) homogenize the speech data collection procedure across sites c) create an open data repository d) seek concrete opportunities for clinical trials application and research funding.
5) The project produced 7 scientific papers published in international journals and 7 conference abstracts. The work carried during the project has been disseminated in many different ways, including social networks, conferences, workshops and fairs. A special topic in Frontiers in Psychology has been published, and a workshop titled “Voice- and speech-based markers of neuropsychiatric conditions: assessing methodological foundations for clinical application” has been held at the host institution.
The MOVES project contributed to create the foundation for a better science of speech and NLP markers of schizophrenia by a) systematically identifying the main issues hampering progress in the field, b) attempting to overcome some of these limitations, by collecting a large cross-linguistic corpus to test the generalizability of previous findings and applying a rigorous methodology and state-of-the-art computational techniques c) by identifying future steps that need to be taken to advance acoustic and NLP clinical applications in neuropsychiatry.
Future potential developments of the project are: a) establishing concrete collaborations with private and public stakeholders (e.g. industry partners) to design, translate, and implement in clinical trials speech and language-based tools for actual clinical use 2) Strengthen and expand the collaborative network with the DISCOURSE consortium to collect a large open corpus and address the clinical and ethical challenges required to develop speech and NLP markers-based clinical decision support systems c) Build a data-analysis platform to run different types of speech and language analyses apt for neuropsychiatric research.
Future potential developments of the project are: a) establishing concrete collaborations with private and public stakeholders (e.g. industry partners) to design, translate, and implement in clinical trials speech and language-based tools for actual clinical use 2) Strengthen and expand the collaborative network with the DISCOURSE consortium to collect a large open corpus and address the clinical and ethical challenges required to develop speech and NLP markers-based clinical decision support systems c) Build a data-analysis platform to run different types of speech and language analyses apt for neuropsychiatric research.