Periodic Reporting for period 3 - CoAct (Communication in Action: Towards a model of Contextualized Action and Language Processing)
Okres sprawozdawczy: 2021-09-01 do 2023-02-28
Language is fundamental to human sociality. While the last century has given us many fundamental insights into how we use and understand it, core issues that we face when doing so within its natural environment—face-to-face conversation—remain untackled. When we speak we also send signals with our head, eyes, face, hands, torso, etc. How do we orchestrate and integrate all this information into meaningful messages? CoAct will lead to a new model with in situ language processing at its core, the Contextualized Action and Language (CoALa) processing model. The defining characteristic of in situ language is its multimodal nature. Moreover, the essence of language use is social action; that is, we use language to do things—we question, offer, decline etc. These social actions are embedded in conversational structure where one speaking turn follows another at a remarkable speed, with millisecond gaps between them. Conversation thus confronts us with a significant psycholinguistic challenge. While one could expect that the many co-speech bodily signals exacerbate this challenge, CoAct proposes that they actually play a key role in dealing with it. It tests this in three subprojects that combine methods from a variety of disciplines but focus on the social actions performed by questions and responses as a uniting theme: (1) ProdAct uses conversational corpora to investigate the multimodal architecture of social actions with the assumption that they differ in their ‘visual signatures’, (2) CompAct tests whether these bodily signatures contribute to social action comprehension, and if they do so early and rapidly, (3) IntAct investigates whether bodily signals play a facilitating role also when faced with the complex task of comprehending while planning a next social action. Thus, CoAct aims to advance current psycholinguistic theory by developing a new model of language processing based on an integrative framework uniting aspects from psychology , philosophy and sociology.
The aim of this project is to identify visual patterns in multimodal signalling, as well as to investigate binding and prediction mechanisms underlying the processing of multimodal utterances during comprehension. In investigating these aspects, we treat human language as a joint action, and as a situated, multimodal phenomenon shaped by the communicative intentions of the interacting agents. In terms of methods, we use a mix of approaches, involving both quantitative corpus studies and experimentation, as well as empirical paradigms from psycholinguistics, neuroscience, psychology, and linguistics. In addition to traditional tools, we also use techniques such as Virtual Reality, motion capture and physiological measures.
So far, the project has led to a large corpus of casual conversations including audio, video and kinematic recordings. This corpus is in the process of being annotated for many behaviours of interest to the project, including manual and head gestures, facial signals, speech and prosody. These data will be analysed to answer various questions relating to how people communicate intentions multimodally in social interaction, and how these multimodal signals feed into the process of reaching mutual understanding and alignment. Further, the corpus data will serve to test specific hypotheses about multimodal intention communication in experimental settings, both in terms of producing communicative acts (with a focus on speech acts/social actions) and how these are processed on a cognitive and neural level. Some experimental studies have already begun, including cutting edge methods such as EEG-hyperscanning and motion tracking. Experimental studies using Virtual Reality stimuli motivated by the corpus analyses are currently being prepared.
So far, the project has led to a large corpus of casual conversations including audio, video and kinematic recordings. This corpus is in the process of being annotated for many behaviours of interest to the project, including manual and head gestures, facial signals, speech and prosody. These data will be analysed to answer various questions relating to how people communicate intentions multimodally in social interaction, and how these multimodal signals feed into the process of reaching mutual understanding and alignment. Further, the corpus data will serve to test specific hypotheses about multimodal intention communication in experimental settings, both in terms of producing communicative acts (with a focus on speech acts/social actions) and how these are processed on a cognitive and neural level. Some experimental studies have already begun, including cutting edge methods such as EEG-hyperscanning and motion tracking. Experimental studies using Virtual Reality stimuli motivated by the corpus analyses are currently being prepared.
It is the project's aims to shed light on how people use their bodies in conjunction with speech to encode meaning in conversational interaction, and how these complex multimodal constructs are processed by the brain. Considering the project uses an interdisciplinary approach in terms of methodological approaches, we expect the project to significantly advance our understanding of the communication of intentions (speech acts/social actions) on a behavioural, cognitive, neural and physiological level, using an innovative combination of techniques.