VRACE - Virtual Reality Audio for Cyber Environments

Project Information

VRACE

Grant agreement ID: 812719

DOI

10.3030/812719

Project closed

EC signature date 28 February 2019

Start date 1 March 2019

End date 31 August 2023

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 3 961 342,80

EU contribution

€ 3 961 342,80

Coordinated by

UNIVERSITAT FUR MUSIK UND DARSTELLENDE KUNST WIEN
Austria

Periodic Reporting for period 2 - VRACE (VRACE - Virtual Reality Audio for Cyber Environments)

Reporting period: 2021-03-01 to 2023-08-31

Summary of the context and overall objectives of the project

Many believe that virtual, augmented and mixed reality (VR/AR/XR) will not only dramatically change the gaming and entertainment industry, but will also revolutionize education, social networking, healthcare, real estate, transportation and technology. While current products are already good enough to attract and excite gamers, there are several areas where significant technological improvements are required, in order to make VR acceptable for applications beyond gaming.

The ITN project "VRACE – Virtual Reality Audio for Cyber Environments" establishes a multidisciplinary network that will train the next generation of researchers in the audio part of virtual and augmented reality. An important field of research is the audio part. The importance of sound becomes evident when considering how people orientate themselves in space. Unlike seeing, hearing allows us to perceive instantly from all angles, and plays a leading role in giving us clues where to look at. To support natural orientation in VR, the visual and auditory information has to closely match, as otherwise the illusion is shattered and the experience is not convincing.

With an estimated revenue of $80bn by 2025 ("Virtual & Augmented Reality: Understanding the Race for the Next Computing Platform", Goldman Sachs, Equity Research, Jan. 13th 2016) the demand for trained VR audio experts will increase rapidly. Besides advancing methodologies in this cutting-edge technology, VRACE will train 15 Early Stage Researchers who will multiply and spread this knowledge in industry and academia. VRACE thus gives European industry a competitive edge in this global race.

The main objective of this Training Network is to raise VR to a next level beyond gaming and entertainment by benefiting from the critical mass of expertise gathered in this distinguished consortium. Driven and directed by the industrial partners, the consortium strives to make significant progress on the way towards a physically correct virtual reality and thus towards a true world simulation e.g. empowering engineers to enter VR for virtual testing and engineering. Although this network aims at sound experience design, there will be intermediate simulation results having some impact on the visual part of VR. In order to study sound generation, physical modelling of dynamic systems with moving parts and non-stationary fluid motion will be performed. The predicted motion of solid bodies and the dynamic deformation of objects will serve as the source of radiated sound, but simultaneously requires visual rendering in order to become visible and make VR physically correct.

Due to the interdisciplinary and intersectoral composition of the network, different partners have different but complementing requirements depending on their interests and their fields of expertise. Musicians need convincing sound simulations, room acousticians need tools for realistic room simulations, engineers dealing with Virtual Acoustics are interested in improving the computational efficiency of the whole audio rendering process. Especially for audio rendering it is essential to take human perception into account. This will ensure that no processing power is being wasted by rendering acoustic details which cannot be heard. Furthermore, perception modelling and psychoacoustic experiments may give clues on how sounds contribute to deepen immersion and make virtual environments more convincing.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Within the first reporting period, an excellent network with strong industrial participation has been established, in order to foster an ongoing cooperation between top universities and research centres from eight countries. All 15 Early Stage Researchers enrolled for a PhD programme and submitted a report on the plan of their studies (expose). Their Personal Career Development Plans are continuously updated in consultation with their supervisors. Outreach and dissemination activities have been carried out, as permitted by the implemented covid-19 restrictive measures. Five consortium meetings have been organised in order to monitor the progress of the project, including an assessment of research goals, methodology and risk mitigation, as well as a report on the progress of the scientific training. Five training workshops (one with physical and four with virtual presence) have been organised, along with two online team-building workshops.

Within the second reporting period, research intensified, resulting in advances in the following areas:
- Sound source and sound radiation modelling and characterisation, including sound-source separation;
- Room acoustics and outdoors wave propagation modelling and auralisation, including perceptual studies;
- Measurement and modelling of Head Related Transfer Functions and ear canals, and binaural rendering;
- Perception of complex sound sources, audio-visual latency tolerances and transfer plausibility.

Another five training workshops took place, including a public tutorial course and two VRACE special sessions in academic conferences.

20 Journal papers and 10 Open Datasets have been published and more are planned. Research results have been presented in numerous scientific conferences of various fields related to Virtual Reality, Audio Engineering and Acoustics. Exploitable results include code for sound separation, binaural rendering, sound source modelling and wave propagation, HRTF models (measured and simulated) and room impulse response measurements.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Combining the fragmented specialist knowledge in acoustics, as represented by the academic and industrial VRACE partners, created a unique concentration of know-how open to the VRACE fellows and to future PhD students. An increasing need for high-quality audio in virtual environments, as well as treatment of environmental noise, indicate how the action may contribute to objectives of the European Research Area. As already discussed during the final meeting, several of the ESRs significantly advanced the state of the art. Although this network put a strong focus on possible applications in VR, training and research are equally relevant for e.g. vehicle acoustics, room acoustics, psycho-acoustics and musical acoustics, resulting in excellent career prospects for all fellows. It is therefore not surprising that, while most ESRs are still in the final phase of their PhD studies (given that most Universities adhere to a 4-year PhD curriculum) two ESRs have been already employed by Europe-based companies in the area of acoustics (which were not members of the consortium).

Involvement in the European Acoustics Association (EAA) has aided towards structuring existing training capacities for European PhD-level training, with available courses listed in the EAA website. This raises the quality of training programs due to international competition and comparability of training courses, and due to the scientific exchange and increasing international mobility. Discussions for providing an institutional framework for basic research and exchange of know-how has also been initiated.

VRACE coordinator's lab and logo