How Hands Help Us Hear

Projektinformationen

HearingHands

ID Finanzhilfevereinbarung: 101040276

DOI

10.3030/101040276

EK-Unterschriftsdatum 11 Juli 2022

Startdatum 1 September 2022

Enddatum 31 August 2027

Finanziert unter

• European Research Council (ERC)

Gesamtkosten

€ 1 499 988,00

EU-Beitrag

€ 1 499 988,00

Koordiniert durch

STICHTING RADBOUD UNIVERSITEIT
Netherlands

Periodic Reporting for period 1 - HearingHands (How Hands Help Us Hear)

Berichtszeitraum: 2022-09-01 bis 2025-02-28

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

When we have a conversation, we do not only produce sound but we also move to the rhythm of our speech. This is so inherent to human communication that we even gesture when speaking over the phone (while the receiver cannot actually see our hand movements). However, what these simple rhythmic movements contribute to spoken communication remains unclear. In other words, does the receiver benefit from seeing even simple beat gestures when trying to work out what’s being said?

The HearingHands program tests this question, viewing the timing of simple gestures as a form of multimodal prosody. We hypothesize that the temporal alignment of hand gestures to speech prosody (so-called ‘gesture-speech coupling’) directly influences what we hear. Our objectives are [WP1] to chart the PREVALENCE of the use of gesture-speech coupling as a multimodal prominence cue in production and perception across a typologically diverse set of languages; [WP2] to capture the VARIABILITY in production and perception of gesture-speech coupling in both neurotypical and atypical populations; [WP3] to determine the CONSTRAINTS that govern gestural timing effects in more naturalistic communicative settings. These objectives will be achieved through cross-linguistic comparisons of gesture-speech production and perception, testing multimodal integration in autistic and neurotypical individuals, and psychoacoustic tests of gestural timing effects employing eye-tracking and virtual reality. Outcomes are expected to reveal that even the simplest flicks of the hands can guide the listener in spoken word recognition and speech segmentation.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

We have successfully demonstrated that:
- beat gestures serve as a cue to lexical stress in both Dutch and Spanish, distinguishing CONtent from conTENT, and CANto from canTÓ;
- this effect of beat gestures takes place in real time, temporally anchored to the beat apex, and arising as the word is still unfolding;
- this effect of beat gestures can be reliably detected in a mini-test of under 10 min;
- it can also be triggered by a human-like artificially-generated moving avatar;
- beat gestures can even have a lasting impact on spoken word recognition, shaping subsequent audio-only speech perception through recalibration;
- in Mandarin (a lexical tone language), gestures time to vowel onset, not pitch peaks (unlike stress languages);
- in Mandarin, producing a gesture raises the f0 across the entire lexical tone contour.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

We are currently exploring the gesture-speech integration in autistic as compared to neurotypical participants. This may reveal relevant distinctions in multisensory speech processing in autism. Another avenue with potential applied impact concerns the use of virtual avatars to enrich the acoustic speech signal, especially in challenging listening conditions.