The project has conducted research in the areas of experimental research on individual differences, modelling of pragmatic reasoning and application of the results towards a summarization task.
Our experimental studies showed that pragmatic reasoning tends to be stable within an individual across time -- if one individual is given a specific task, and then asked a few months later to do a similar task, we find that their tendency for what kinds of inferences they draw in the task are highly correlated. This indicates that there might be specific differences in cognition between different people what cause downstream differences in pragmatic inferencing.
We have also investigated which cognitive abilities are the main drivers affecting pragmatic inferencing. For example, we found that in spatial perspective taking abilities determine how listeners interpret ambiguous instructions like "give me the book to the left" in a context where conversational partners are seated opposite each other, such that "to the left" could be either interpreted as the left from the perspective of the listener or the perspective of the speaker. In a different study, we found that the ability of listeners to adapt their pragmatic inferences to a specific speaker (e.g. whether a speaker tends to exaggerate) is correlated with a listener's cognitive abilities in working memory updating. We also showed across several experiments that reasoning abilities are a main factor in predicting pragmatic reasoning abilities, and that theory of mind or working memory seemed to play a lesser role.
An important step in our work consists of building computational models which can predict the experimental results. Our models comprise Bayesian models and rational speech act models, the parameters of which we estimate from related individual differences tasks. To date, we have proposed RSA model designs for several of the phenomena that we have investigated experimentally.
A core goal of the project is to transfer these findings into better models for human-computer communication. Our experimental results show however that people do adapt their pragmatic behaviour to the interaction partner, and that they behave differently with computational partners compared to humans. Importantly, this also depends on the perceived competence of the computer partners. In interaction with modern AI agents, we even found that people behaved in a way that suggested that they thought of these AI partners to take stronger load in the conversation than a human partner (while for old less competent computer partners, results are the opposite).
The advent of large language models such as ChatGPT has necessitated an adaptation of our research methods. We decided to critically assess the ability of current LLMs to adapt to different user groups. Our recent study into ChatGPT revealed that while the system has the ability to adapt to different user groups, it does so less successfully than human-authored text.
