Experiencing life vividly through others’ eyes
What if social media went beyond photos, videos and posts, allowing us to share real emotions and even ‘relive’ experiences? Imagine feeling the rush of a roller coaster, wandering through a museum or joining a friend’s trip abroad as if it were your own – fully immersed in the sights, sounds, and even the sense of space and time. The EU-funded EXPERIENCE(opens in new window) project sought to revolutionise how humans record, share and understand experiences through virtual reality (VR) and AI. “Our main goal was to let anyone capture their own ‘extended personal reality’ – a digital recreation of what they see, hear and feel – and share it so others could ‘relive’ those moments through VR,” notes project coordinator Gaetano Valenza.
Virtual realms exuding life
EXPERIENCE relied on AI to make it automatic and easy for everyone to create and share fully immersive virtual experiences. Such realistic virtual environments have hitherto required advanced coding or 3D modelling skills. Another technological challenge was to better understand how to make virtual experiences feel as natural and emotionally engaging as real ones. “People do not always react to virtual environments the same way they do to real-life experiences. A virtual roller coaster might not feel as thrilling as a real one, or a virtual sunset might not evoke the same emotions,” states Valenza.
Recording the full spectrum of human experience
EXPERIENCE achieved remarkable progress across technology, science and clinical validation, creating a comfortable, medical-grade wearable system. Beyond wearable sensors, the prototype combines intelligent software and AI to capture and recreate multisensory experiences. “The wearable consists of an electroencephalographic cap for capturing brain signals, a chest belt for monitoring heart activity and respiration, and cameras and microphones for recording vision and sound,” highlights Valenza. Users can tag emotionally significant moments, with all data sent via Bluetooth or Wi-Fi to a mobile app. “The app manages recordings, uploads data, and provides real-time biofeedback like heart rate variability and mood visualisations,” outlines Valenza. “Advanced AI and 3D reconstruction then create realistic digital twins of physical environments, blending emotional, physiological and behavioural data. The VR interface lets users ‘relive’ recorded experiences using haptic feedback to simulate physical sensations and allowing them to slow down or speed up time to explore moments at their own pace.”
Altered space-time perception
The team faced challenges in ensuring the virtual experience triggered genuine emotions instead of fake/forced reactions. These simulations also had to trigger unique mental, emotional and behavioural responses based on the person’s specific characteristics. “Brain and body react differently to virtual and real settings, altering physiology and spatial perception,” underlines Valenza. “Emotions influenced time perception: healthy individuals experienced flexible time shifts (expansion/contraction) based on emotional content, while those with depression showed reduced sensitivity, confirming ‘emotion context insensitivity’ theories.” Brain and heart activity were closely linked during emotional moments, yet this synchrony was disrupted in depression. Space also influenced time perception: large virtual spaces made time feel slower. Ultimately, mild electrical stimulation reduced VR-induced nausea (cybersickness).
Implications for mental health
The system holds promise for highly personalised therapy, adapting the virtual environments in real time to counterbalance a user’s emotional state. For example, if someone feels anxious, the system could create a calming environment to ease anxiety. “Its real-time mood tracking, emotion-based therapy and relapse prevention capabilities make it powerful for depression, anxiety and phobias,” concludes Valenza.