Community Research and Development Information Service - CORDIS

ERC

COMPUTED Report Summary

Project ID: 637991
Funded under: H2020-EU.1.1.

Periodic Reporting for period 1 - COMPUTED (Computational User Interface Design)

Reporting period: 2015-04-01 to 2016-09-30

Summary of the context and overall objectives of the project

Despite extensive research on human-computer interaction (HCI), the problem of how to best design usable user interfaces (UIs) is unsolved. That is, no method exists that guarantees the optimal or even a provably good design. Instead, idea-generation in UI design is driven by heuristics and iterative improvements informed by empirical data collection. This heuristic–iterative approach is costly and at times ineffective, because UI design often involves very large design spaces with multiple objectives and constraints and complex user behavior.

COMPUTED aims to establish the foundations for solving UI design problems by combinatorial optimization methods that deploy mathematical models of user behavior as objective functions. Given objectives and constraints, a UI is automatically optimized. Previous work in UI optimization shows significant improvements to usability, but the scope has been restricted to virtual keyboards and widget layouts. COMPUTED researches methods that could vastly expand the scope and permit solutions to any well-defined UI design problem. First, objective functions are currently limited to models of sensorimotor performance. COMPUTED develops algorithmic support for acquiring more comprehensive models that cover the main human factors. Second, current work has formally defined only one UI optimization problem, the letter assignment problem. To combat a more relevant set of design problems with appropriate optimization methods, COMPUTED formally analyzes recurring design problems. Third, previous work has followed the “fire-and-forget” approach where the problem is completely predefined for an optimizer. COMPUTED develops a novel interactive UI optimization paradigm that promotes fast convergence to good results even in the face of uncertainties and incomplete preknowledge. The novel capabilities are demonstrated in four hard cases: 1) universal keyboard layout, 2) web applications, 3) hand gesture input, and 4) interactive dashboards in cars.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project has develop new methodology to support model-based combinatorial optimization. We defined and solved a new problem in interaction design: the functionality selection problem. We constructed a new integer programming–based method for this problem and constructed a visualization of the outputs. This method can support any interaction designer working with early stage concepts. We have also defined and presented first real-time optimizers for web page sketches and started to work on deep learning based optimization of web page layouts.

One fundamental problem the project has looked at is how to infer models of users from realistic data. To this end, we have worked on inverse modeling of computational rationality models. The idea is to use ABC (approximate bayesian computation) to infer model parameters from naturalistic user data, such as clickstreams. We showed that parameters of the human sensorimotor systems can be obtained like this in menu interaction. This will enable an optimizer to learn models of users without running experiments. These models can then be used to optimize or adapt the UI for that user group.

COMPUTED has developed the first zero-effort approach to interactive optimization. This means that the designer's goal is inferred automatically and used to generate a diverse set of ideas on on alternative designs. Presently, we are working on the problem of how to infer regions of a graphical UI from its image only. This will help generalizing the use of these approaches beyond sketching to, for example, web pages etc.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Results from the first period of COMPUTED show that combinatorial UI optimization offers a strong complement to existing design methods. The results have shown formalizations of design problems that allow applying combinatorial optimization methods. This can significantly increase the cost-efficiency of UI design and improve the outputs. COMPUTED has shown that these methods support even novel, ill-defined design problems, where they support designers by allowing them to delegate combinatorially hard sub-problems to a computer. It supports decision-making and offers a objective criteria that can prevent costly and futile iteration.

On the applied side, we have worked with AFNOR, a French standardization agency, to apply our methods to the redesign of the Azerty keyboard layout, and in particular to improve its performance and usability by remapping the special characters. This practical problem required advances in modelling shortcut behavior and ergonomics, and solving a large (quadratic assignment) problem. We have found further improvements to state-of-the-art QAP algorithms. The results will be presented to AFNOR at the end of December. We have also presented several breakthrough results in gestural interaction, including the first model-based approach to gesture optimization, as well as optimization approaches to real-time hand tracking. Another application worth mentioning is scatterplot optimization: we apply models of human perception to tune the design of scatterplot for data analysis tasks such as correlation estimation and cluster detection. It is the first time models of human perception are used for this purpose.
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top