The ERC funded team has made significant contributions with respect to all planned actions.
First, we have developed a number of data-driven models of human behavior. Human activity is inherently complex, non-linear and over long timescales becomes a stochastic process. However, many applications require predictive models of human activity, for example to design better user interfaces one needs to reason about human action sequences, or in human-robot interaction the agent needs to plan with respect to the current and future state of the human in order to minimize latency. Our lab is at the forefront of data-driven user modelling efforts reaching from algorithms to predict 3D motion data into the future, to inferring cognitive load from pupil dilatation to inferring visual attention from images alone. This work has been disseminated in a significant number of top-tier publications in the computer-vision, machine-learning and HCI literature, including multiple papers at CVPR, ICCV, ECCV, 3DV, NeurIPS, ICLR and CHI. Some of this work has been recognised via honourable mention and best paper honours.
Second, we have developed several computational UI adaptation techniques that leverage the inferred user state and acquired models to adjust the type, timing and location of information that is displayed. This work is foundational to an emerging area in computational system and user interface design as it changes the way we think about and ultimately design user interfaces. For example, in a recent paper published at the International Symposium on User Interface Software and Technology (ACM UIST) we demonstrate how an estimate of cognitive load can be leveraged to continuously optimize what information is shown to the user, where it shown and how much screen real-estate is dedicated to the element in Mixed Reality applications. In the context of the OPTINT project we are developing several related algorithms and combine them with traditional User Interface methods in order to give designers better tools to build more usable interactive technologies. We have deployed these algorithms and methods not only to graphical user interfaces but also to the computational design of advanced interactive technologies that contain emergent sensing and actuation elements. This line of work has led to top-tier publications on computational design of haptic feedback mechanisms at ACM UIST (3x) and on the computational design of intelligent garments that can provide kinesthetic feedback to wearer at ACM UIST and Eurographics.
Furthermore, in the final period of the project we have established a framework for the development and modelling of online UI adaptation that is based on the (multi-agent) reinforcement learning framework. In an upcoming publication (under review at ACM SIGCHI) we have demonstrated how this framework can be leveraged to train cooperative policies. That is we learn to adapt a UI in an online fashion by training two RL agents simultaneously. One models the user and the other agents learns to adapt the UI such that the user's performance is maximised. Crucially, we show that this is possible even when the underlying task of the user is not known a-priori by the UI configuration policy but is inferred by observing the user's actions. This overcomes a significant bottleneck in existing UI adaptation techniques that rely on knowing the user's goals.
