Periodic Reporting for period 4 - DREAM (Distributed dynamic REpresentations for diAlogue Management)
Reporting period: 2024-03-01 to 2025-05-31
Our ability to communicate using language in conversation is considered the hallmark of human intelligence and the holy grail of AI. In recent years, artificial conversational agents such as chatbots and personal assistants have become increasingly present in our everyday lives. Systems like ChatGPT have millions of daily users, many company websites host customer service chatbots, and dialogue agents are also being used in education and health coaching. Interacting with these systems already feels a lot more natural and is less frustrating for the user than ten or even five years ago. Nevertheless, the degree to which these systems help us understand the skills underlying language use is limited. In DREAM, we combine insights from linguistic and cognitive theories of human dialogue with machine learning techniques to develop data-driven models of language use that deepen our understanding of this basic human activity and inspire the design of more trust-worthy and human-like artificial conversational agents.
The work carried out in DREAM can be summarised in terms of the following 5 main achievements:
1) A suit of implemented conversational agent models. In a series of publications, we have developed machine learning models that generate referring expressions that are discriminative, exploit the conversational common ground, and are adapted to the expertise of the dialogue partner.
2) A comprehensive analysis and evaluation of multimodal systems. In another series of publications, we have provided evidence that vision-language models learn better semantic representations than language-only models for concepts that are visually grounded; we have shed light on how and when these models perform multimodal integration, and have uncovered patterns of alignment and miss-alignment between their internal representations and human cognitive signals.
3) An in-depth analysis of linguistic priming and interactive alignment in humans and language models. In another series of publications, we have carried out empirical corpus-based analyses of human-human dialogue in first- and second-language acquisition setups and shown that large language models exhibit structural priming effects that are similar to those observed in humans.
4) A framework based on notions from Information Theory to analyse and model language use in conversation. In another series of publications, we have shown that information content tends to decrease over the course of a dialogue rather than remain constant as argued by well-known theories and proposed a novel model of utterance processing effort that relies on alternative sampling operationalised with large language models.
5) An analysis of model uncertainty and its practical consequences. In yet another series of publications, we have studied model uncertainty and its relation to human production variability, proposed methods to exploit uncertainty to generate clarification questions, and expose the dangers of unrecognised ambiguity and lack of uncertainty awareness.
In addition to the scientific publications, the project's output also includes public code repositories and datasets, which are available from the project's website.
1) A suit of implemented conversational agent models. In a series of publications, we have developed machine learning models that generate referring expressions that are discriminative, exploit the conversational common ground, and are adapted to the expertise of the dialogue partner.
2) A comprehensive analysis and evaluation of multimodal systems. In another series of publications, we have provided evidence that vision-language models learn better semantic representations than language-only models for concepts that are visually grounded; we have shed light on how and when these models perform multimodal integration, and have uncovered patterns of alignment and miss-alignment between their internal representations and human cognitive signals.
3) An in-depth analysis of linguistic priming and interactive alignment in humans and language models. In another series of publications, we have carried out empirical corpus-based analyses of human-human dialogue in first- and second-language acquisition setups and shown that large language models exhibit structural priming effects that are similar to those observed in humans.
4) A framework based on notions from Information Theory to analyse and model language use in conversation. In another series of publications, we have shown that information content tends to decrease over the course of a dialogue rather than remain constant as argued by well-known theories and proposed a novel model of utterance processing effort that relies on alternative sampling operationalised with large language models.
5) An analysis of model uncertainty and its practical consequences. In yet another series of publications, we have studied model uncertainty and its relation to human production variability, proposed methods to exploit uncertainty to generate clarification questions, and expose the dangers of unrecognised ambiguity and lack of uncertainty awareness.
In addition to the scientific publications, the project's output also includes public code repositories and datasets, which are available from the project's website.
The five achievements mentioned above constitute substantial progress beyond the state of the art. Through these achievements, DREAM has advanced our scientific understanding of language use in conversation and its computational modelling, as well as showing that non-symbolic artificial intelligence techniques are powerful tools for modelling linguistic knowledge and conversational skills.