An innovative touch-sound technology to provide 360-degree spatial location and content

Project Information

TouchingSpace360

Grant agreement ID: 101123434

DOI

10.3030/101123434

Project closed

EC signature date 3 August 2023

Start date 1 September 2023

End date 28 February 2025

Funded under

• European Research Council (ERC)

Total cost

No data

EU contribution

€ 150 000,00

Coordinated by

REICHMAN UNIVERSITY
Israel

Periodic Reporting for period 1 - TouchingSpace360 (An innovative touch-sound technology to provide 360-degree spatial location and content)

Reporting period: 2023-09-01 to 2025-02-28

Summary of the context and overall objectives of the project

Sight and sound dominate today’s digital experiences, yet the third pillar of perception-touch still carries almost none of the information our ears routinely handle. The gap is most pronounced in spatial hearing: even the finest headphones, hearing aids, and cochlear implants struggle to inform a listener whether a siren originates from above, below, or behind. More than 1.5 billion people already live with some degree of hearing loss, a figure projected to reach 2.5 billion by 2050; for them, limited spatial cues mean reduced safety, independence, and quality of life.
TouchingSpace360 answers this need by translating three-dimensional sound into precisely patterned vibrations delivered to the fingertips or body. A now-patented algorithm encodes direction, distance, elevation, and motion in real-time and drives compact tactile actuators with imperceptible latency, allowing users to feel where a voice, vehicle, or melody originates.
The project set four overarching objectives. First, demonstrate that full-sphere acoustic detail can be rendered through touch with near-auditory precision. Second, quantify benefits for users of hearing aids and cochlear implants, who presently lack reliable spatial information. Third, test whether tactile cues enhance immersion and performance in virtual-reality, gaming, and communication scenarios for the broader public. Fourth, lay the groundwork for commercial rollout by protecting intellectual property, engaging stakeholders, and defining a pathway to scalable manufacture.
By fusing advances in neuroscience, haptics, and human-centred design, TouchingSpace360 aims to expand inclusion, boost safety, and open entirely new creative frontiers—transforming touch from an overlooked sense into a gateway for richer, more inclusive perception.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Algorithm and software: A real-time engine was developed to integrate with spatial audio and translate source positions into multichannel vibration patterns that mimic interaural weighting. Further code was used to render audio from other software platforms, running on low-cost processors with sub-20 ms latency.
Hardware development: An ergonomic hand-held device was designed, embedding two LLRA-type actuators for each hand. The design prioritises comfort, battery life, and wireless use. Add-on modules for existing VR controllers and integration with controller to body positioning were also prototyped and tested for source positioning abilities.
Laboratory studies.
Twenty-nine normally-hearing adults located moving sound sources on a virtual sphere almost as accurately by touch as by hearing; performance in background noise improved significantly with the tactile aid.
Nineteen congenitally deaf individuals with congenital hearing impairment who used cochlear implants or hearing aids learned to track 3D motion through touch after less than 10 minutes of training, significantly outperforming their baseline aided-hearing abilities.
Thirty-two hearing participants were tested on their responses to musical experiences, revealing the effects of audio-tactile listening on emotional states and anxiety levels.

Field deployments.
Therapeutic pilot: more than 500 individuals with post-traumatic stress symptoms used audio-tactile music sessions; qualitative reports highlighted reduced anxiety and heightened body awareness.
UNESCO cave installation: Over 5,000 visitors experienced an immersive audio-visual show enhanced by handheld tactile devices, demonstrating its robustness in a public venue for two consecutive months.

Dissemination & exploitation: A patent on the 3D spatial tactile technology was granted (Amedi et al, 2024); key stakeholders in audiology, haptics, and assistive-tech sectors were engaged; and two peer-reviewed articles were published (iScience 2024a, iScience 2024b), and two more are under review (journals: Frontiers in VR, Nature Scientific Reports).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

First tactile system to convey full-sphere sound location and motion in real time. Existing haptic aids deliver only simple buzzing cues; TouchingSpace360 provides graded, dynamic information that users integrate with audio intuitively.
Evidence that spatial perception can develop through touch alone: Congenitally hearing-impaired participants achieved localisation accuracies that are shown to be unattainable even for bilateral cochlear-implant users, who tend to perform better than unilateral users. Our findings show rapid spatial abilities in the congenitally hearing-impaired individuals using various combinations of assistive devices, thus informing the discourse of nature vs nurture and challenging long-held views of sensory “critical periods”.
Scalable, field-tested platform: Successful operation in clinical and cultural settings demonstrates durability, low maintenance, and high public acceptance.

Expected benefits:
Hearing-impaired users: safer mobility, better speech-in-noise understanding, and richer social participation.
General population: enhanced immersion and situational awareness in VR, gaming, and professional training.
Healthcare: a non-invasive tool for grounding and emotion regulation and anxiety.
Culture & education: inclusive, multisensory exhibits that engage mixed-ability audiences.


Next steps include miniaturising the actuator array, integrating learning algorithms that tailor patterns to individual sensitivity, expanding clinical trials, and partnering with device manufacturers to bring the technology to market.
TouchingSpace360 thus merges neuroscience, engineering, and design to turn vibration into a new “channel” for perceiving the world, advancing accessibility, safety, and multisensory experiences for millions.

Manuscripts:
Amedi, A., Snir, A., Wald, I., & Ciesla, K. (2024). Tactile Representation of Location Characteristics and Content in 3d (20240419252). https://www.freepatentsonline.com/y2024/0419252.html#google_vignette
Snir, A., Cieśla, K., Vekslar, R., & Amedi, A. (2024). Highly compromised auditory spatial perception in aided congenitally hearing-impaired and rapid improvement with tactile technology. IScience, 27(9). https://doi.org/10.1016/j.isci.2024.110808
Snir, A., Cieśla, K., Ozdemir, G., Vekslar, R., & Amedi, A. (2024). Localizing 3D motion through the fingertips: Following in the footsteps of elephants. IScience, 27(6). https://doi.org/10.1016/j.isci.2024.109820
Snir A., Ciesla K, Amedi A., A novel tactile technology enables sound source identification by hearing-impaired individuals in a complex 3D audio environment (under review, Nature Scientific Reports)
Schwartz N.†, Snir A.†, Amedi A., Feeling the music: an investigation of tactile enhancement of music on emotional experiences (under review, Frontiers in Virtual Reality)

Design of an ergonomic hand held device with two separate vibration actuators embedded within for th

Audio-Baseline (A1), Tactile (T), and Audio-Tactile (AT) spatial motion localization scores.

The experimental facility used to test spatial capabilities of the tactile devices

Field experiences of the hand held device

Field experiences of the hand held device

The task and the results of experiment 2 with the sources moving on a semi-sphere.

The tactile setup used to research the emotional impact of audio-tactile music listening

Multisensory installation at Beit Govrin Bell Cave UNESCO World Heritage Site.

Design of an ergonomic hand held device with two separate vibration actuators embedded within for th