Periodic Reporting for period 2 - IDDISC (Individualized Interaction in Discourse)
Reporting period: 2022-08-01 to 2024-01-31
Humans adapt the content and form of their utterances to different interlocutors (students vs. colleagues vs. granny), and monitor the level of understanding in their conversational partner. Today's NLP systems are however largely blind with respect to individual variation in language comprehension, which in turn leads to misunderstandings and lack of naturalness in the interaction.
The vision of project IDDISC is to enable individualized language interaction with computer systems, such that information or explanations generated by a system will fit the user and the situation, by explicitly modelling their state of understanding. This project aims to break new ground by addressing individual differences in comprehension at the pragmatics and discourse level, i.e. with respect to the inferred meaning that goes beyond the literal meaning of an utterance. This project thus holds the promise to reduce the risk of misunderstandings, and enable adaptation of automatically generated language (e.g. explanations, summaries) to specific users.
To address these goals, we conduct empirical experimental studies on how humans interpret language, and more specifically on differences between different individuals in how they interpret language. We additionally aim to relate the observed differences to individual cognitive properties such as working memory capacity, language experience, theory of mind or reasoning abilities. In a second step, we then design computational models that capture these individual differences and aim to account for the observed differences in interpretation. These can then be used to benefit downstream natural language generation tasks by adapting language to individuals.
We also develop new statistical methods and crowd-sourcing paradigms as part of this project, which we hope will open the door to other researchers for investigating individual differences in all areas of language processing.
The vision of project IDDISC is to enable individualized language interaction with computer systems, such that information or explanations generated by a system will fit the user and the situation, by explicitly modelling their state of understanding. This project aims to break new ground by addressing individual differences in comprehension at the pragmatics and discourse level, i.e. with respect to the inferred meaning that goes beyond the literal meaning of an utterance. This project thus holds the promise to reduce the risk of misunderstandings, and enable adaptation of automatically generated language (e.g. explanations, summaries) to specific users.
To address these goals, we conduct empirical experimental studies on how humans interpret language, and more specifically on differences between different individuals in how they interpret language. We additionally aim to relate the observed differences to individual cognitive properties such as working memory capacity, language experience, theory of mind or reasoning abilities. In a second step, we then design computational models that capture these individual differences and aim to account for the observed differences in interpretation. These can then be used to benefit downstream natural language generation tasks by adapting language to individuals.
We also develop new statistical methods and crowd-sourcing paradigms as part of this project, which we hope will open the door to other researchers for investigating individual differences in all areas of language processing.
The project has conducted research in the areas of experimental research on individual differences, modelling of pragmatic reasoning and application of the results towards a summarization task.
Our experimental studies showed that pragmatic reasoning tends to be stable within an individual across time -- if one individual is given a specific task, and then asked a few months later to do a similar task, we find that their tendency for what kinds of inferences they draw in the task are highly correlated. This indicates that there might be specific differences in cognition between different people what cause downstream differences in pragmatic inferencing.
We have also investigated which cognitive abilities are the main drivers affecting pragmatic inferencing. For example, we found that in spatial perspective taking abilities determine how listeners interpret ambiguous instructions like "give me the book to the left" in a context where conversational partners are seated opposite each other, such that "to the left" could be either interpreted as the left from the perspective of the listener or the perspective of the speaker. In a different study, we found that the ability of listeners to adapt their pragmatic inferences to a specific speaker (e.g. whether a speaker tends to exaggerate) is correlated with a listener's cognitive abilities in working memory updating. We also showed across several experiments that reasoning abilities are a main factor in predicting pragmatic reasoning abilities, and that theory of mind or working memory seemed to play a lesser role.
An important step in our work consists of building computational models which can predict the experimental results. Our models comprise Bayesian models and rational speech act models, the parameters of which we estimate from related individual differences tasks. To date, we have proposed RSA model designs for several of the phenomena that we have investigated experimentally.
A core goal of the project is to transfer these findings into better models for human-computer communication. Our experimental results show however that people do adapt their pragmatic behaviour to the interaction partner, and that they behave differently with computational partners compared to humans. Importantly, this also depends on the perceived competence of the computer partners. In interaction with modern AI agents, we even found that people behaved in a way that suggested that they thought of these AI partners to take stronger load in the conversation than a human partner (while for old less competent computer partners, results are the opposite).
The advent of large language models such as ChatGPT has necessitated an adaptation of our research methods. We decided to critically assess the ability of current LLMs to adapt to different user groups. Our recent study into ChatGPT revealed that while the system has the ability to adapt to different user groups, it does so less successfully than human-authored text.
Our experimental studies showed that pragmatic reasoning tends to be stable within an individual across time -- if one individual is given a specific task, and then asked a few months later to do a similar task, we find that their tendency for what kinds of inferences they draw in the task are highly correlated. This indicates that there might be specific differences in cognition between different people what cause downstream differences in pragmatic inferencing.
We have also investigated which cognitive abilities are the main drivers affecting pragmatic inferencing. For example, we found that in spatial perspective taking abilities determine how listeners interpret ambiguous instructions like "give me the book to the left" in a context where conversational partners are seated opposite each other, such that "to the left" could be either interpreted as the left from the perspective of the listener or the perspective of the speaker. In a different study, we found that the ability of listeners to adapt their pragmatic inferences to a specific speaker (e.g. whether a speaker tends to exaggerate) is correlated with a listener's cognitive abilities in working memory updating. We also showed across several experiments that reasoning abilities are a main factor in predicting pragmatic reasoning abilities, and that theory of mind or working memory seemed to play a lesser role.
An important step in our work consists of building computational models which can predict the experimental results. Our models comprise Bayesian models and rational speech act models, the parameters of which we estimate from related individual differences tasks. To date, we have proposed RSA model designs for several of the phenomena that we have investigated experimentally.
A core goal of the project is to transfer these findings into better models for human-computer communication. Our experimental results show however that people do adapt their pragmatic behaviour to the interaction partner, and that they behave differently with computational partners compared to humans. Importantly, this also depends on the perceived competence of the computer partners. In interaction with modern AI agents, we even found that people behaved in a way that suggested that they thought of these AI partners to take stronger load in the conversation than a human partner (while for old less competent computer partners, results are the opposite).
The advent of large language models such as ChatGPT has necessitated an adaptation of our research methods. We decided to critically assess the ability of current LLMs to adapt to different user groups. Our recent study into ChatGPT revealed that while the system has the ability to adapt to different user groups, it does so less successfully than human-authored text.
For the remaining duration of the project, we will continue our work on experimental and statistical methods for individual differences research. We additionally plan to explore using the cognitive architecture ACT-R for modelling our experimental findings. ACT-R models hold the promise of being more strictly grounded in what we know about how the brain works.
We also plan to propose better adaptation mechanisms for NLG tasks within the new LLM methodological framework.
We also plan to propose better adaptation mechanisms for NLG tasks within the new LLM methodological framework.