Enriched communication across the lifespan

Project Information

ENRICH

Grant agreement ID: 675324

DOI

10.3030/675324

Project closed

EC signature date 11 August 2016

Start date 1 October 2016

End date 30 September 2020

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 3 593 601,36

EU contribution

€ 3 593 601,36

Coordinated by

UNIVERSIDAD DEL PAIS VASCO/ EUSKAL HERRIKO UNIBERTSITATEA
Spain

Periodic Reporting for period 2 - ENRICH (Enriched communication across the lifespan)

Reporting period: 2018-10-01 to 2020-09-30

Summary of the context and overall objectives of the project

Communication using speech is natural, ubiquitous, and efficient, so a reduced capacity in speaking or hearing creates huge barriers to social inclusion. Trends in health, technology and social mobility create challenges for listeners. Age-related hearing loss affects half of all Europeans by the age of 80. Listeners are faced with artificial speech from voice assistants in the home, at work, and while travelling, often in noisy environments. We are exposed to speech spoken non-natively, or spoken in a first language other than our own. These factors reduce intelligibility or make speech more demanding to process. Technological solutions such as hearing aids and speech synthesis can help, yet their use often imposes even greater effort for listeners.

ENRICH (Enriched Communication across the Lifespan) is a EU-funded Marie Sklodowska Curie European Training Network made up of universities, research institutes, clinics and technology companies whose objectives are to better understand how listeners process different styles of speech, to determine what aspects of speech make one form more intelligible or easy to process than another, to characterise how different types of listener are affected by distinct speech styles and to design effective algorithms that are capable of enriching speech, making it easier to understand and less demanding to process.

ENRICH has trained 14 early-stage researchers with backgrounds in psychology, linguistics, engineering and computer science, acquiring skills in disciplines required to make novel contributions in speech communication, as well as training in complementary areas including entrepreneurship, technical writing, scientific conduct and public dissemination. ENRICH has led to new insights into how listeners respond to different forms of speech and has built on these findings, and from observations of talkers, to generate new algorithms that have improved upon the state of the art in near-end listening enhancement. Findings have been published in journals and international conferences, disseminated to scientific peers and industry groups across Europe and beyond, and communicated to interested citizens online and at public events.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

ENRICH tackles many distinct forms of speech including healthy, disordered, non-native, casual, and computer-generated speech. Improvements in message understanding and cognitive effort due to enrichment have been quantified in ENRICH using physiological measures (pupil size or brain activity measured by EEG), dual-task designs, subjective judgements, identification scores, and task completion rates. Using advanced virtual reality software to create well-controlled but highly-realistic simulations, speech modifications have been tested with normal-hearing and hearing-impaired listeners, non-native listeners and participants with enhanced listening skills.

While synthetic speech has been shown to be less intelligible than natural speech, studies at the University of Edinburgh and the University of the Basque Country examined pupil response measures to computer-generated speech, finding that it is also more effortful to process. Using EEG, researchers at Fraunhofer IDMT found that speech enhancement algorithms reduce correlates of listening effort even when intelligibility cannot be improved further.

Other studies have demonstrated how listening effort varies across different listener groups. In collaboration with Sonova, researchers at UCL measured the extent to which older, hearing-impaired listeners suffer more from fast speech and reverberant speech. Other lister cohorts show enhanced capabilities: a study at the University Medical Centre Groningen showed that musicians outperform non-musicians in tasks involving identifying words from one talker in the presence of a competing talker, in part by attenting more to durational information in speech.

Speakers modify the way they produce speech in challenging conditions, and insights from studying exactly what they do in such scenarios can feed into better algorithms. Scientists at Radboud University Nijmegen collected a large corpus of speech produced in noise by both native and non-native talkers, showing that natural enrichment strategies are common across the two types of talkers and equally beneficial for listeners. Studies at Horzentrum examined the role of visual information, head orientation and gaze changes in complex multi-talker listening tasks, leading to recommendations for future hearing aid strategies.

Scientists from ENRICH have also provided open source software tools that enable measurement of listener preferences for arbitrary speech modifications, a toolkit for pupillometry and a number of new audio and audiovisual speech corpora for other researchers to use.

Results have been disseminated in 60 papers, 48 talks, 66 posters and 21 demonstrations. ENRICH organised a major public understanding event at the UK's Royal Institution (London, 2020), an Industry Event at the International Congress on Acoustics (Aachen, 2019), and a (virtual) Show and Tell event at the International Conference on Acoustics, Speech and Signal Processing (Barcelona, 2020). ENRICH also organised a large-scale international blind evaluation of intelligibility-enhancing speech modifications, the Hurricane 2.0 Challenge, with listener panels in three European countries; results were disseminated at the (virtual) International Conference on Speech Communication, Interspeech (Shanghai, 2020). ENRICH Early Stage Researchers have won Best Paper and Best Poster prizes at several international conferences. A youtube video describing ENRICH has been viewed more than 1100 times.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

ENRICH has progressed the state of the art scientifically, by producing a clearer understanding of the changes speakers make when talking in adverse conditions, the effects of cognitive effort during speech processing on different types of listeners, and the impact of different speech styles on listening effort. ENRICH has also led to quantitative improvements in the technical arena. A competition-winning speech modification algorithm from the University of Edinburgh outperformed the previous state of the art in boosting intelligibility of natural and synthetic speech in noise and reverberation, while the University of Crete demonstrated that machine learning techniques can improve speech quality while preserving the intelligibility gains of earlier signal-processing approaches.

The impact of ENRICH is threefold. First, ENRICH has played a role in raising societal awareness of the need to understand not just how well the information content of a message is received but also how much effort is required to process it. Second, in the socio-economic sphere ENRICH has developed the groundwork for technological solutions that will impact all situations where live, pre-recorded, or synthetic speech output is used, including public address and early-warning systems, classroom audio and domestic voice assistants. Third, ENRICH has significantly expanded the community of young scientists with a multidisciplinary training in both human and machine perception, gained in academia and in industry, creating a lasting network of expertise that will spread out within Europe and beyond with the inventiveness to generate new science and new wealth-creation in a field that impacts all our lives.

MSCA-ITN-ETN-675324 ENRICH (logo)