IMSquared: Immersive Multimodal Metaphoric Interactions for Well-Being
Optimizing the body’s state (e.g. heart rate) for survival is the cornerstone of well-being. Maintaining well-being is a complex process involving the interaction of automatic subconscious processes and higher-level cognitive processes that manifests as our conscious feelings (i.e. experiences of body states). Feelings are important as they help guide our actions and decision-making acting as a REGULATORY INTERFACE. Crucially, evidence suggests that our ability to ‘tune into’ and alter perception of our bodily sensations or cultivate emotional awareness is tied to positive mental health outcomes.
To inquire into this, we leverage information about how people THINK and TALK about emotion to generate immersive, body-based multimodal metaphoric interactions (IMMIs) that can enhance emotional awareness/ intelligence and well-being. The work draws on research at the intersection of cognitive (neuro)science, artificial intelligence (AI), and human-computer interaction/ games.
According to Embodied Cognitive Linguistics (ECL), abstract concepts including complex emotion concepts are grounded in bodily interaction in the world through systematic metaphorical conceptual mappings (CMs). CMs allow us to leverage structural knowledge from embodied domains (e.g. spatial motion) to reason and talk about more abstract knowledge domains (e.g. emotion) and are embedded in larger narratives. For example, we often construe emotional events in terms of spatial motion, ‘we hit a roadblock with the proposal’ (i.e. DIFFICULTIES ARE IMPEDIMENTS TO MOTION). In this way, metaphor serves as a key cognitive mechanism that helps to bridge abstraction with embodiment, also facilitating more embodied or grounded language understanding.
Evidence suggests that CMs and their reflection in language use have important consequences for affect and cognition, often at the subconscious level. However, the nature of metaphor in the mind and how this relates to cognitive processes that facilitate our ability to use metaphor, especially in more embodied and deliberate (i.e. NOVEL) ways, to impact our emotions and reasoning/ decision-making remains far from well understood.
The KEY CONCEPT of IMSquared is that understanding metaphor and its impact on emotion and reasoning is something that needs to be understood and LEARNED IN INTERACTION. We introduce a human-AI collaborative framework (i.e. the integration of human input in AI systems) to generate/ prototype IMMIs during emotional event narration and understand metaphor and its impact on affect and cognition. The work deals with both multimodal embodied interaction design and embodied language understanding.
The design of IMMIs draws on early work in ECL on modeling and learning verb/ event semantics in stories involving a KNOWLEDGE-BASED and PARAMETRIZED approach. In this view, language and co-speech gestures index and parametrize (e.g. slow, fast) more elemental embodied knowledge structures (i.e. embodied schemata) to build an embodied mental model/ simulation of what it would actually ‘be like'/ 'feel like' to engage in the activity described (e.g. ‘hit a roadblock’). In verb-based metaphor this EMBODIED SIMULATION is used to infer features of abstract events using CMs (e.g. in ‘we hit a roadblock with the proposal’ DIFFICULTIES ARE IMPEDIMENTS TO MOTION leads to the inference ‘difficulties or challenges with the proposal’). The framework supports both language understanding and motion control/ understanding and is motivated by THEORIES OF MOTOR CONTROL/ LEARNING. It notably uses a visual NODE GRAPH SYSTEM for ease of analysis. This approach is extended to interactively model and learn verb/ event semantics in the context of an immersive virtual reality (VR) world model. This involves interactively grounding verbs/ events (that can be used metaphorically) through node-based parametrizable VR motion controllers that allow to use gestures to control agent behavior in a VR world using VR gesture learning. We model embodied world knowledge domains of CMs using a VR world model focusing on spatial motion. We prototype IMMIs by leveraging this embodied world knowledge in the context of event narration, including together with generative large language models (LLMs).
We present an ECL-based human-AI collaborative framework and toolkit to generate/ prototype IMMIs in order to understand and enhance novel metaphor use during event narration and its impact on emotion and reasoning. The objectives were met through an iterative, user-centered design approach leveraging human interaction/ feedback in the creation and evaluation of MetaphorVR.
1. Metaphor & Stories: (1) Identify metaphors/ CMs and verb phrases associated with embodied knowledge of CMs including by leveraging LLMs, also iconic gestures; (2) Establish embodied event narration task(s) including asking users to narrate memorable/ emotional events and explore prototyping IMMIs to reflect on the situation described in the narrative.
2. Implementation of IMMIs & Visual Node-based/ VR interfaces: (1) Model embodied world knowledge of CMs using a VR world model; (2) Interactively ground verbs/ events through node-based parametrizable VR motion controllers that allow to use gestures to control agent behavior in a VR world using VR gesture learning; (3) Prototype IMMIs in the context of event narration using this embodied world knowledge, including together with LLMs; (4) Provide Node-based/ VR interfaces.
3. Evaluation: (1) Evaluate grounded events and IMMIs for expected inferences; (2) Evaluate MetaphorVR for impact on affect and cognition using topographical BODY MAPS OF BODILY SENSATIONS, psychological (neurophysiological) & user experience measures.
4. Data, Optimization & Exploration: (1) Create a KNOWLEDGE BASE and METAPHORIC KNOWLEDGE (narrative) repository, collect multimodal data; (2) Use HUMAN IN THE LOOP AI to learn grounded events and optimize EXPLORATION of the METAPHORIC/ AFFORDANCE SPACE.
