Periodic Reporting for period 1 - VRACE (VRACE - Virtual Reality Audio for Cyber Environments)
Periodo di rendicontazione: 2019-03-01 al 2021-02-28
The ITN project "VRACE – Virtual Reality Audio for Cyber Environments" establishes a multidisciplinary network that will train the next generation of researchers in the audio part of virtual and augmented reality. An important field of research is the audio part. The importance of sound becomes evident when considering how people orientate themselves in space. Unlike seeing, hearing allows us to perceive instantly from all angles, and plays a leading role in giving us clues where to look at. To support natural orientation in VR, the visual and auditory information has to closely match, as otherwise the illusion is shattered and the experience is not convincing.
With an estimated revenue of $80bn by 2025 ("Virtual & Augmented Reality: Understanding the Race for the Next Computing Platform", Goldman Sachs, Equity Research, Jan. 13th 2016) the demand for trained VR audio experts will increase rapidly. Besides advancing methodologies in this cutting-edge technology, VRACE will train 15 Early Stage Researchers who will multiply and spread this knowledge in industry and academia. VRACE thus gives European industry a competitive edge in this global race.
The main objective of this Training Network is to raise VR to a next level beyond gaming and entertainment by benefiting from the critical mass of expertise gathered in this distinguished consortium. Driven and directed by the industrial partners, the consortium strives to make significant progress on the way towards a physically correct virtual reality and thus towards a true world simulation e.g. empowering engineers to enter VR for virtual testing and engineering. Although this network aims at sound experience design, there will be intermediate simulation results having some impact on the visual part of VR. In order to study sound generation, physical modelling of dynamic systems with moving parts and non-stationary fluid motion will be performed. The predicted motion of solid bodies and the dynamic deformation of objects will serve as the source of radiated sound, but simultaneously requires visual rendering in order to become visible and make VR physically correct.
Due to the interdisciplinary and intersectoral composition of the network, different partners have different but complementing requirements depending on their interests and their fields of expertise. Musicians need convincing sound simulations, room acousticians need tools for realistic room simulations, engineers dealing with Virtual Acoustics are interested in improving the computational efficiency of the whole audio rendering process. Especially for audio rendering it is essential to take human perception into account. This will ensure that no processing power is being wasted by rendering acoustic details which cannot be heard. Furthermore, perception modelling and psychoacoustic experiments may give clues on how sounds contribute to deepen immersion and make virtual environments more convincing.
Five training workshops (one with physical and four with virtual presence) have been organised, along with two online team-building workshops.
Research performed within the first periodic report includes:
- the formulation of different numerical algorithms for physical modelling of sound sources
- the development of machine learning algorithms for spatial audio
- numerical approaches for room acoustics and auralisation enhanced by experimental measurements
- numerical and experimental acquisition of head-related-transfer-functions (also considering the ear canal)
- listening tests to analyse sound perception in virtual environments.
By exploiting existing synergies and joining or exchanging research methodology, significant advances are expected in the area of virtual reality audio. This will allow to enhance a critical component not only of virtual-reality applications, but also of standard online communication platforms. The demand for high-quality audio in virtual environments, which is the subject of the VRACE project, has been made more apparent during the covid-19 pandemic. Despite the broad, every-day use of several online platforms ranging from common video-conference applications to elaborate 3D meeting spaces, the audio quality of such solutions has been under high scrutiny. While a possibility was offered for live online communication, several shortcoming were revealed, which require immediate improvement. Indeed, it is the audio part that constitutes the limiting factor of video conferences. This involves not only audio interruptions, but also a complete elimination of spatial information, making it impossible to intuitively identify the speaker. While striving to improve spatial audio within virtual reality applications, the accumulated knowledge resulting from the VRACE project will be of utmost importance to online communication activities that are currently of high demand in our society.