Audible emotional states
Imagine receiving feedback from a device on how your emotions and cognitive state are affecting your physiological signs and adapting responses to improve your well-being. Scientists developed such a system by integrating BNCI technology into a novel electronic musical instrument. EU funding of the project B-REACTABLE (B-reactable: Multimodal tabletop system for collaborative physiology monitoring and training) made it possible. The Reactable, previously developed at the University of Pompeu Fabra, Barcelona has a musical-tabletop tangible interface and a backlit display covering the table surface. This commercially available system enables music creation without text lists or endless configuration menus. The tangibles are different modules of an analogue synthesiser that can be put together to make music or sounds. Researchers set out to exploit the tangibles to enable users to monitor and modify their physiological states based on feedback about their cognitive and emotional states. Effective sonification of the electroencephalogram (EEG) or transformation of the EEG signal into an auditory one is the key to success. This is no simple task as the point is to translate relationships in the EEG into sounds that make the relationships comprehensible to people. The team conducted a thorough literature review of real-time EEG sonifications as well as additional experiments, leading to selection of a model-based approach to EEG sonification. A 21-channel EEG device was selected and tested. During the final project year, the scientists tested the B-REACTABLE prototype with naïve users and professional Reactable disc jockeys. In parallel, the team developed a new paradigm for collaborative neurofeedback-based compassion training. It exploits the previously developed sonifications to monitor EEG activity. The sonifications are combined with presentation of emotional videos to train users in pro-social compassionate behaviour. Extensions of the technologies include a virtual platform connecting artists and scientists working on multisensory media and physical and emotional reactions to audiovisual content. Given the explosion of audiovisual media freely available on the internet and for sale for cell phones and portable electronic devices, the market potential is huge. The B-REACTABLE system is expected to become a very useful tool for training health professionals and improving the quality of life of EU citizens. The ability to control and monitor one's own physiological signals using feedback about cognitive and emotional states could soon put 'treatment' in the hands of patients.
