Periodic Reporting for period 2 - ENRICH (Enriched communication across the lifespan)
Reporting period: 2018-10-01 to 2020-09-30
ENRICH (Enriched Communication across the Lifespan) is a EU-funded Marie Sklodowska Curie European Training Network made up of universities, research institutes, clinics and technology companies whose objectives are to better understand how listeners process different styles of speech, to determine what aspects of speech make one form more intelligible or easy to process than another, to characterise how different types of listener are affected by distinct speech styles and to design effective algorithms that are capable of enriching speech, making it easier to understand and less demanding to process.
ENRICH has trained 14 early-stage researchers with backgrounds in psychology, linguistics, engineering and computer science, acquiring skills in disciplines required to make novel contributions in speech communication, as well as training in complementary areas including entrepreneurship, technical writing, scientific conduct and public dissemination. ENRICH has led to new insights into how listeners respond to different forms of speech and has built on these findings, and from observations of talkers, to generate new algorithms that have improved upon the state of the art in near-end listening enhancement. Findings have been published in journals and international conferences, disseminated to scientific peers and industry groups across Europe and beyond, and communicated to interested citizens online and at public events.
While synthetic speech has been shown to be less intelligible than natural speech, studies at the University of Edinburgh and the University of the Basque Country examined pupil response measures to computer-generated speech, finding that it is also more effortful to process. Using EEG, researchers at Fraunhofer IDMT found that speech enhancement algorithms reduce correlates of listening effort even when intelligibility cannot be improved further.
Other studies have demonstrated how listening effort varies across different listener groups. In collaboration with Sonova, researchers at UCL measured the extent to which older, hearing-impaired listeners suffer more from fast speech and reverberant speech. Other lister cohorts show enhanced capabilities: a study at the University Medical Centre Groningen showed that musicians outperform non-musicians in tasks involving identifying words from one talker in the presence of a competing talker, in part by attenting more to durational information in speech.
Speakers modify the way they produce speech in challenging conditions, and insights from studying exactly what they do in such scenarios can feed into better algorithms. Scientists at Radboud University Nijmegen collected a large corpus of speech produced in noise by both native and non-native talkers, showing that natural enrichment strategies are common across the two types of talkers and equally beneficial for listeners. Studies at Horzentrum examined the role of visual information, head orientation and gaze changes in complex multi-talker listening tasks, leading to recommendations for future hearing aid strategies.
Scientists from ENRICH have also provided open source software tools that enable measurement of listener preferences for arbitrary speech modifications, a toolkit for pupillometry and a number of new audio and audiovisual speech corpora for other researchers to use.
Results have been disseminated in 60 papers, 48 talks, 66 posters and 21 demonstrations. ENRICH organised a major public understanding event at the UK's Royal Institution (London, 2020), an Industry Event at the International Congress on Acoustics (Aachen, 2019), and a (virtual) Show and Tell event at the International Conference on Acoustics, Speech and Signal Processing (Barcelona, 2020). ENRICH also organised a large-scale international blind evaluation of intelligibility-enhancing speech modifications, the Hurricane 2.0 Challenge, with listener panels in three European countries; results were disseminated at the (virtual) International Conference on Speech Communication, Interspeech (Shanghai, 2020). ENRICH Early Stage Researchers have won Best Paper and Best Poster prizes at several international conferences. A youtube video describing ENRICH has been viewed more than 1100 times.
The impact of ENRICH is threefold. First, ENRICH has played a role in raising societal awareness of the need to understand not just how well the information content of a message is received but also how much effort is required to process it. Second, in the socio-economic sphere ENRICH has developed the groundwork for technological solutions that will impact all situations where live, pre-recorded, or synthetic speech output is used, including public address and early-warning systems, classroom audio and domestic voice assistants. Third, ENRICH has significantly expanded the community of young scientists with a multidisciplinary training in both human and machine perception, gained in academia and in industry, creating a lasting network of expertise that will spread out within Europe and beyond with the inventiveness to generate new science and new wealth-creation in a field that impacts all our lives.