Periodic Reporting for period 3 - OPTINT (OPTINT: Optimization-based Design of Interactive Technologies)
Reporting period: 2020-02-01 to 2021-07-31
Contemporary user interface (UI) design techniques have been developed for the era of the PC, but designing for these emerging interaction paradigms is a much more diverse and difficult problem and requires reasoning about i) choice of sensing technology ii) sensor placement and configuration iii) and finally reasoning about appropriate input recognition algorithms. These aspects are intertwined with usability in an inseparable fashion.
The main goal of OPTINT is to lay the foundations for the design and implementation of 21st century interactive technologies. In order to achieve this goal the ERC sponsored team works on computational approaches to model human behavior, algorithms to interpret this data and to extract semantic meaning from the low-level observations and to predict the state of the user into the future. This state includes observable aspects such as the location and movements of the user but also unobservable aspects such cognitive load and attention. With such models in place the team then develops algorithms for the computational design of interactive technologies. Such tools may then help a designer to build user interface technologies while jointly considering low-level sensing technology and usability concerns. Leveraging optimization-based algorithms, it will allow for a faster and more efficient exploration of the design space, leading to better designs of user interfaces and to reduced cost of the design process.
First, we have developed a number of data-driven models of human behavior. Human activity is inherently complex, non-linear and over long timescales becomes a stochastic process. However, many applications require predictive models of human activity, for example to design better user interfaces one needs to reason about human action sequences, or in human-robot interaction the agent needs to plan with respect to the current and future state of the human in order to minimize latency. Our lab is at the forefront of data-driven user modelling efforts reaching from algorithms to predict 3D motion data into the future, to inferring cognitive load from pupil dilatation to inferring visual attention from images alone.
Second, we have developed several computational UI adaptation techniques that leverage the inferred user state and acquired models to adjust the type, timing and location of information that is displayed. This work is foundational to an emerging area in computational system and user interface design as it changes the way we think about and ultimately design user interfaces. For example, in a recent paper to appear at the International Symposium on User Interface Software and Technology (ACM UIST) we demonstrate how an estimate of cognitive load can be leveraged to continuously optimize what information is shown to the user, where it shown and how much screen real-estate is dedicated to the element in Mixed Reality applications. In the context of the OPTINT project we are developing several related algorithms and combine them with traditional User Interface methods in order to give designers better tools to build more usable interactive technologies.