Periodic Reporting for period 4 - COMPUTED (Computational User Interface Design)
Berichtszeitraum: 2019-10-01 bis 2020-03-31
COMPUTED aims to establish the foundations for solving UI design problems by combinatorial optimization methods that deploy mathematical models of user behavior as objective functions. Given objectives and constraints, a UI is automatically optimized. Previous work in UI optimization shows significant improvements to usability, but the scope has been restricted to virtual keyboards and widget layouts. COMPUTED researches methods that could vastly expand the scope and permit solutions to any well-defined UI design problem. First, objective functions are currently limited to models of sensorimotor performance. COMPUTED develops algorithmic support for acquiring more comprehensive models that cover the main human factors. Second, current work has formally defined only one UI optimization problem, the letter assignment problem. To combat a more relevant set of design problems with appropriate optimization methods, COMPUTED formally analyzes recurring design problems. Third, previous work has followed the “fire-and-forget” approach where the problem is completely predefined for an optimizer. COMPUTED develops a novel interactive UI optimization paradigm that promotes fast convergence to good results even in the face of uncertainties and incomplete preknowledge. The novel capabilities are demonstrated in four hard cases: 1) universal keyboard layout, 2) web applications, 3) hand gesture input, and 4) interactive dashboards in cars.
1. Mathematical definition of design problems and their efficient solution by known means in combinatorial optimization. In 2015-2017, we have defined several common UI problems not defined before, in particular in menu optimization, functionality selection, keyboard design, biomechanical design, and functionality selection. Some problems are defined at the level of a decision problem, which then can be solved using black-box optimizers. Others are defined using integer programming and permit the computation of guarantees for solution quality. We have also defined and presented first real-time optimizers for web page sketches and started to work on deep learning based optimization of web page layouts.
2. Interactive optimization in design. In 2015-2017, we have defined new methods that base on real-time optimization, robust optimization, and machine learning. They allow supporting the designer while avoiding overloading and providing sufficient means for steering optimization. COMPUTED has developed the first zero-effort approach to interactive optimization. This means that the designer's goal is inferred automatically and used to generate a diverse set of ideas on on alternative designs. Presently, we are working on the problem of how to infer regions of a graphical UI from its image only. This will help generalizing the use of these approaches beyond sketching to, for example, web pages etc.
3. Modeling of interactive behavior: We have advanced modeling of interactive behavior, in particular in the area of computational rationality. In this approach, we defined bounds/capacity limitations of humans and estimate the best achievable performance with a candidate design using Q-learning or other reinforcement learning approaches. This has been thus far applied in menus, information search, and visual layouts. One fundamental problem the project has looked at is how to infer models of users from realistic data. To this end, we have worked on inverse modeling of computational rationality models. The idea is to use ABC (approximate bayesian computation) to infer model parameters from naturalistic user data, such as clickstreams. We showed that parameters of the human sensorimotor systems can be obtained like this in menu interaction. This will enable an optimizer to learn models of users without running experiments. These models can then be used to optimize or adapt the UI for that user group.
4. Demonstrators: Using results from #1-#3, we have shown new results in challenging UI design problems from keyboards to menus and over to functionality selection, gestural input, web layouts, and biomechanics.