Skip to main content
European Commission logo print header

Inference and machine learning methods in human-computer interaction

Final Report Summary - INFERENCEHCI (Inference and machine learning methods in human-computer interaction)

This research project has investigated how to design new fluid and efficient user interfaces that take advantage of machine learning and other uncertain reasoning procedures. The work that was carried out can broadly be divided into two (partially over-lapping) themes: a) understanding the design space of user interfaces driven by machine learning, and b) design of novel user interfaces based on machine learning. This report presents the results for each theme separately below.

Theme 1: understanding the design space

To design efficient user interfaces based on machine learning it is critical we understand users’ capabilities and limitations. In this theme we assessed and analyzed how the design of an intelligent user interface influences users’ performance and behaviour. Many intelligent user interfaces are on some level based on users’ understanding of spatiotemporal patterns. Examples of such interfaces are handwriting recognition and various forms of gestural interfaces. We conducted a study to assess if so-called space time cube visualization could be used to aid users’ understanding of such patterns (Kristensson et al. 2009). Space time cube visualization is a method to visualize two-dimensional spatiotemporal data in 3D by visualizing time as the third dimension (figure 1). We found that users were on average twice as fast in understanding complex spatiotemporal patterns when they used space time cube visualization in comparison to a 2D baseline. However, for simple data lookups a 2D visualization is better. Hence, user interfaces that need to convey both simple and complex spatiotemporal data patterns should consider incorporating both display variants. This work was published in IEEE Transactions on Visualization and Computer Graphics in 2009. Another project in this theme was to understand the human performance of state of the art handwriting recognition. Tremendous effort has been invested into developing accurate handwriting recognizers that are capable of recognizing unconstrained handwriting that enables users to mix printed and cursive writing. However, the research literature predicted that users would write much slower using handwriting recognition than a much simpler-to-implement on-screen keyboard (about 16 wpm vs. 25 wpm). Hence it was questionable exactly how much use end-users would benefit from the research invested into creating accurate handwriting recognizers. However, we found that the research literature based its assumption on an old unreliable study that measured handwriting rather than handwriting recognition. We conducted an 11-session longitudinal user study with 12 participants to assess the learning curve of state of the art handwriting recognition in comparison to an on-screen keyboard baseline (Kristensson and Denby 2009). We found that, contrary to the prior beliefs in the literature, handwriting recognition performance was on par with an on-screen keyboard. Thus, handwriting recognition performance had been underestimated in the previous literature. This work was published in Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2009). The third work in this theme was the authoring of an essay about intelligent text entry methods – text entry methods that enable users to write faster using machine learning and other uncertain reasoning methods. This essay had two primary contributions. First, it put intelligent text entry methods into their historical perspective. Second, it highlighted five challenges in the research field: localization, error correction, editor support, feedback, and context of use. The essay was published in AI Magazine in 2009.