Periodic Reporting for period 4 - OPTINT (OPTINT: Optimization-based Design of Interactive Technologies)
Reporting period: 2021-08-01 to 2022-07-31
As digital technology moves further away from the desktop setting, it is becoming increasingly clear that conventional interfaces (i.e. mice and keyboards) are no longer adequate means for interaction and that the traditional computing paradigm will be replaced or complemented by new forms of interaction with digital information. While the exact nature of future interfaces is not known, we already do know that future devices will be mobile, will be equipped with ample computational power and will utilize several sensing capabilities. Furthermore, we will interact with diverse set of devices and interfaces such as wearable computers, head-worn displays. Finally, today everybody uses computing devices and the expectation is that these are usable without any prior training or deeper understanding of the technical aspects.
Contemporary user interface (UI) design techniques have been developed for the era of the PC, but designing for these emerging interaction paradigms is a much more diverse and difficult problem and requires reasoning about i) choice of sensing technology ii) sensor placement and configuration iii) and finally reasoning about appropriate input recognition algorithms. These aspects are intertwined with usability in an inseparable fashion.
The main goal of OPTINT is to lay the foundations for the design and implementation of 21st century interactive technologies. In order to achieve this goal the ERC sponsored team works on computational approaches to model human behavior, algorithms to interpret this data and to extract semantic meaning from the low-level observations and to predict the state of the user into the future. This state includes observable aspects such as the location and movements of the user but also unobservable aspects such cognitive load and attention. With such models in place the team then develops algorithms for the computational design of interactive technologies. Such tools may then help a designer to build user interface technologies while jointly considering low-level sensing technology and usability concerns. Leveraging optimization-based algorithms, it will allow for a faster and more efficient exploration of the design space, leading to better designs of user interfaces and to reduced cost of the design process.
Contemporary user interface (UI) design techniques have been developed for the era of the PC, but designing for these emerging interaction paradigms is a much more diverse and difficult problem and requires reasoning about i) choice of sensing technology ii) sensor placement and configuration iii) and finally reasoning about appropriate input recognition algorithms. These aspects are intertwined with usability in an inseparable fashion.
The main goal of OPTINT is to lay the foundations for the design and implementation of 21st century interactive technologies. In order to achieve this goal the ERC sponsored team works on computational approaches to model human behavior, algorithms to interpret this data and to extract semantic meaning from the low-level observations and to predict the state of the user into the future. This state includes observable aspects such as the location and movements of the user but also unobservable aspects such cognitive load and attention. With such models in place the team then develops algorithms for the computational design of interactive technologies. Such tools may then help a designer to build user interface technologies while jointly considering low-level sensing technology and usability concerns. Leveraging optimization-based algorithms, it will allow for a faster and more efficient exploration of the design space, leading to better designs of user interfaces and to reduced cost of the design process.
The ERC funded team has made significant contributions with respect to all planned actions.
First, we have developed a number of data-driven models of human behavior. Human activity is inherently complex, non-linear and over long timescales becomes a stochastic process. However, many applications require predictive models of human activity, for example to design better user interfaces one needs to reason about human action sequences, or in human-robot interaction the agent needs to plan with respect to the current and future state of the human in order to minimize latency. Our lab is at the forefront of data-driven user modelling efforts reaching from algorithms to predict 3D motion data into the future, to inferring cognitive load from pupil dilatation to inferring visual attention from images alone. This work has been disseminated in a significant number of top-tier publications in the computer-vision, machine-learning and HCI literature, including multiple papers at CVPR, ICCV, ECCV, 3DV, NeurIPS, ICLR and CHI. Some of this work has been recognised via honourable mention and best paper honours.
Second, we have developed several computational UI adaptation techniques that leverage the inferred user state and acquired models to adjust the type, timing and location of information that is displayed. This work is foundational to an emerging area in computational system and user interface design as it changes the way we think about and ultimately design user interfaces. For example, in a recent paper published at the International Symposium on User Interface Software and Technology (ACM UIST) we demonstrate how an estimate of cognitive load can be leveraged to continuously optimize what information is shown to the user, where it shown and how much screen real-estate is dedicated to the element in Mixed Reality applications. In the context of the OPTINT project we are developing several related algorithms and combine them with traditional User Interface methods in order to give designers better tools to build more usable interactive technologies. We have deployed these algorithms and methods not only to graphical user interfaces but also to the computational design of advanced interactive technologies that contain emergent sensing and actuation elements. This line of work has led to top-tier publications on computational design of haptic feedback mechanisms at ACM UIST (3x) and on the computational design of intelligent garments that can provide kinesthetic feedback to wearer at ACM UIST and Eurographics.
Furthermore, in the final period of the project we have established a framework for the development and modelling of online UI adaptation that is based on the (multi-agent) reinforcement learning framework. In an upcoming publication (under review at ACM SIGCHI) we have demonstrated how this framework can be leveraged to train cooperative policies. That is we learn to adapt a UI in an online fashion by training two RL agents simultaneously. One models the user and the other agents learns to adapt the UI such that the user's performance is maximised. Crucially, we show that this is possible even when the underlying task of the user is not known a-priori by the UI configuration policy but is inferred by observing the user's actions. This overcomes a significant bottleneck in existing UI adaptation techniques that rely on knowing the user's goals.
First, we have developed a number of data-driven models of human behavior. Human activity is inherently complex, non-linear and over long timescales becomes a stochastic process. However, many applications require predictive models of human activity, for example to design better user interfaces one needs to reason about human action sequences, or in human-robot interaction the agent needs to plan with respect to the current and future state of the human in order to minimize latency. Our lab is at the forefront of data-driven user modelling efforts reaching from algorithms to predict 3D motion data into the future, to inferring cognitive load from pupil dilatation to inferring visual attention from images alone. This work has been disseminated in a significant number of top-tier publications in the computer-vision, machine-learning and HCI literature, including multiple papers at CVPR, ICCV, ECCV, 3DV, NeurIPS, ICLR and CHI. Some of this work has been recognised via honourable mention and best paper honours.
Second, we have developed several computational UI adaptation techniques that leverage the inferred user state and acquired models to adjust the type, timing and location of information that is displayed. This work is foundational to an emerging area in computational system and user interface design as it changes the way we think about and ultimately design user interfaces. For example, in a recent paper published at the International Symposium on User Interface Software and Technology (ACM UIST) we demonstrate how an estimate of cognitive load can be leveraged to continuously optimize what information is shown to the user, where it shown and how much screen real-estate is dedicated to the element in Mixed Reality applications. In the context of the OPTINT project we are developing several related algorithms and combine them with traditional User Interface methods in order to give designers better tools to build more usable interactive technologies. We have deployed these algorithms and methods not only to graphical user interfaces but also to the computational design of advanced interactive technologies that contain emergent sensing and actuation elements. This line of work has led to top-tier publications on computational design of haptic feedback mechanisms at ACM UIST (3x) and on the computational design of intelligent garments that can provide kinesthetic feedback to wearer at ACM UIST and Eurographics.
Furthermore, in the final period of the project we have established a framework for the development and modelling of online UI adaptation that is based on the (multi-agent) reinforcement learning framework. In an upcoming publication (under review at ACM SIGCHI) we have demonstrated how this framework can be leveraged to train cooperative policies. That is we learn to adapt a UI in an online fashion by training two RL agents simultaneously. One models the user and the other agents learns to adapt the UI such that the user's performance is maximised. Crucially, we show that this is possible even when the underlying task of the user is not known a-priori by the UI configuration policy but is inferred by observing the user's actions. This overcomes a significant bottleneck in existing UI adaptation techniques that rely on knowing the user's goals.
The team has made significant contributions to the state of the art in a variety of important sub-areas including data driven user modelling, human motion prediction, gaze and attention estimation and has leveraged these models in optimization-based design tools. The work has lead to several top-tier publications at the premier venues for computer vision (CVPR, ICCV, ECCV and 3DV), machine learning (NeurIPS, ICLR) and human computer interaction (CHI UIST).