Periodic Reporting for period 2 - FORCE-UI (Force-responsive Deformable User Interfaces)
Reporting period: 2022-03-01 to 2023-08-31
Touch input is the universal approach for communicating intention to interactive computing systems. However, touchscreens are flat, solid surfaces that fail to reproduce the qualities of real-world object interaction (e.g. deformation, stiffness, springiness). Further, their expressivity is fundamentally bounded by a single measurable dimension: contact with a solid, touch-sensitive surface. This project will radically increase the expressivity of touchscreen-based interaction by co-locating three force modalities into a single visual display surface (force input, resistive- and kinetic- force feedback). This project will create a new paradigm for Human-Computer Interaction (HCI): Force Responsive Deformable User Interfaces (FRUI). FRUIs will allow users to ‘push through’ a display and for the display to ‘push back’.
If successful, this project has the potential for significant impact on society. FRUIs support ‘eyes free’ interaction, allowing users to exploit their sense of touch to mute a phone call or acknowledge a message while their phone is in their pocket. Equally, they provide a tactile channel for visually-impaired users to consume typically visual-only content. They also facilitate a full gamut of expressive input gestures that can be transmitted and reproduced to show affection or distaste to a social media post or used as input to graphical drawing tool.
The work in this project is broken into four core themes/objectives:
• Develop engineering approaches for integrating multi-point deformable force input, resistive force feedback, and kinetic force feedback into a single display surface.
• Empirically characterise and model fundamental user behaviour with deformable force-responsive user interfaces.
• Design conceptual models of interaction for deformable force-responsive user interfaces.
• Evaluate interaction techniques that exploit the expressive power of deformable force-responsive interfaces.
If successful, this project has the potential for significant impact on society. FRUIs support ‘eyes free’ interaction, allowing users to exploit their sense of touch to mute a phone call or acknowledge a message while their phone is in their pocket. Equally, they provide a tactile channel for visually-impaired users to consume typically visual-only content. They also facilitate a full gamut of expressive input gestures that can be transmitted and reproduced to show affection or distaste to a social media post or used as input to graphical drawing tool.
The work in this project is broken into four core themes/objectives:
• Develop engineering approaches for integrating multi-point deformable force input, resistive force feedback, and kinetic force feedback into a single display surface.
• Empirically characterise and model fundamental user behaviour with deformable force-responsive user interfaces.
• Design conceptual models of interaction for deformable force-responsive user interfaces.
• Evaluate interaction techniques that exploit the expressive power of deformable force-responsive interfaces.
Engineering Approaches:
• Developed a simple, single ‘pixel’ prototype that uses electro-mechanical braking to produce variable stiffness.
• Combined deformable force input, variable resistance feedback and a visual display using pneumatics and soft-robotics into a single system.
• Applied our knowledge of pneumatic FRUIs to construct PAWS (Physical Artefact for Well-being Support) – a handheld device targeted at digital health and well-being interventions that uses kinetic force output and shape-change to ‘physicalize’ a user’s breath.
Characterising User Behaviour:
• Begun work to understand the human bandwidth for force-input across surfaces of varying deformability.
• Used Crossmodal Correspondances explores how users relate the stiffness of a device/interface to different colours and shapes. This fundamental characterisation provides the first insights into how we should incorporate visual design into FRUIs.
• Gained insights into collaboration around these novel displays through our work on a shape-changing bar chart.
Interaction Techniques:
• Explored a range of deformable force input and resistive force feedback interactions. We investigated how well users could apply different levels of force and how well they could perceive changes in resistive force output. We also developed simple applications to demonstrate the power of these interaction techniques.
• In PAWS, we examined the use of kinetic force output and deformable force input as an approach to supporting digital well-being interventions.
• Our literature review systematically analysing experimental designs for interactions in VR has informed our experimental approaches for FRUIs.
• Developed a simple, single ‘pixel’ prototype that uses electro-mechanical braking to produce variable stiffness.
• Combined deformable force input, variable resistance feedback and a visual display using pneumatics and soft-robotics into a single system.
• Applied our knowledge of pneumatic FRUIs to construct PAWS (Physical Artefact for Well-being Support) – a handheld device targeted at digital health and well-being interventions that uses kinetic force output and shape-change to ‘physicalize’ a user’s breath.
Characterising User Behaviour:
• Begun work to understand the human bandwidth for force-input across surfaces of varying deformability.
• Used Crossmodal Correspondances explores how users relate the stiffness of a device/interface to different colours and shapes. This fundamental characterisation provides the first insights into how we should incorporate visual design into FRUIs.
• Gained insights into collaboration around these novel displays through our work on a shape-changing bar chart.
Interaction Techniques:
• Explored a range of deformable force input and resistive force feedback interactions. We investigated how well users could apply different levels of force and how well they could perceive changes in resistive force output. We also developed simple applications to demonstrate the power of these interaction techniques.
• In PAWS, we examined the use of kinetic force output and deformable force input as an approach to supporting digital well-being interventions.
• Our literature review systematically analysing experimental designs for interactions in VR has informed our experimental approaches for FRUIs.
We are currently working on the following areas:
Engineering:
• Investigating the use of smart materials to implement FRUI modalities - these have the potential to provide small form-factor, low-energy approaches to producing and sensing force.
• Implementing kinetic-force output modules - we are working on electro-mechanical approaches to understanding this force modality.
• Exploring the potential for intergrating force modalities with displays.
Characterising User Behaviour:
• Continuing investigations into cross-modal correspondances, to understand how humans relate force to different physical properties.
• More fully characterising human capabilities for provide force-based input and perceive force-based output.
Conceptual Models:
• Build a conceptual understanding of how users interact with FRUIs so that we can develop guidelines for using force in user interfaces.
Interaction Techniques:
• Systematically evaluating how we interact with deformable force input and variable stiffness.
• Developing interaction techniques for kinetic-force output.
Engineering:
• Investigating the use of smart materials to implement FRUI modalities - these have the potential to provide small form-factor, low-energy approaches to producing and sensing force.
• Implementing kinetic-force output modules - we are working on electro-mechanical approaches to understanding this force modality.
• Exploring the potential for intergrating force modalities with displays.
Characterising User Behaviour:
• Continuing investigations into cross-modal correspondances, to understand how humans relate force to different physical properties.
• More fully characterising human capabilities for provide force-based input and perceive force-based output.
Conceptual Models:
• Build a conceptual understanding of how users interact with FRUIs so that we can develop guidelines for using force in user interfaces.
Interaction Techniques:
• Systematically evaluating how we interact with deformable force input and variable stiffness.
• Developing interaction techniques for kinetic-force output.