Periodic Reporting for period 1 - VRACE (VRACE - Virtual Reality Audio for Cyber Environments)
Période du rapport: 2019-03-01 au 2021-02-28
Many believe that virtual, augmented and mixed reality (VR/AR/XR) will not only dramatically change the gaming and entertainment industry, but will also revolutionize education, social networking, healthcare, real estate, transportation and technology. While current products are already good enough to attract and excite gamers, there are several areas where significant technological improvements are required, in order to make VR acceptable for applications beyond gaming.
The ITN project "VRACE – Virtual Reality Audio for Cyber Environments" establishes a multidisciplinary network that will train the next generation of researchers in the audio part of virtual and augmented reality. An important field of research is the audio part. The importance of sound becomes evident when considering how people orientate themselves in space. Unlike seeing, hearing allows us to perceive instantly from all angles, and plays a leading role in giving us clues where to look at. To support natural orientation in VR, the visual and auditory information has to closely match, as otherwise the illusion is shattered and the experience is not convincing.
With an estimated revenue of $80bn by 2025 ("Virtual & Augmented Reality: Understanding the Race for the Next Computing Platform", Goldman Sachs, Equity Research, Jan. 13th 2016) the demand for trained VR audio experts will increase rapidly. Besides advancing methodologies in this cutting-edge technology, VRACE will train 15 Early Stage Researchers who will multiply and spread this knowledge in industry and academia. VRACE thus gives European industry a competitive edge in this global race.
The main objective of this Training Network is to raise VR to a next level beyond gaming and entertainment by benefiting from the critical mass of expertise gathered in this distinguished consortium. Driven and directed by the industrial partners, the consortium strives to make significant progress on the way towards a physically correct virtual reality and thus towards a true world simulation e.g. empowering engineers to enter VR for virtual testing and engineering. Although this network aims at sound experience design, there will be intermediate simulation results having some impact on the visual part of VR. In order to study sound generation, physical modelling of dynamic systems with moving parts and non-stationary fluid motion will be performed. The predicted motion of solid bodies and the dynamic deformation of objects will serve as the source of radiated sound, but simultaneously requires visual rendering in order to become visible and make VR physically correct.
Due to the interdisciplinary and intersectoral composition of the network, different partners have different but complementing requirements depending on their interests and their fields of expertise. Musicians need convincing sound simulations, room acousticians need tools for realistic room simulations, engineers dealing with Virtual Acoustics are interested in improving the computational efficiency of the whole audio rendering process. Especially for audio rendering it is essential to take human perception into account. This will ensure that no processing power is being wasted by rendering acoustic details which cannot be heard. Furthermore, perception modelling and psychoacoustic experiments may give clues on how sounds contribute to deepen immersion and make virtual environments more convincing.
The ITN project "VRACE – Virtual Reality Audio for Cyber Environments" establishes a multidisciplinary network that will train the next generation of researchers in the audio part of virtual and augmented reality. An important field of research is the audio part. The importance of sound becomes evident when considering how people orientate themselves in space. Unlike seeing, hearing allows us to perceive instantly from all angles, and plays a leading role in giving us clues where to look at. To support natural orientation in VR, the visual and auditory information has to closely match, as otherwise the illusion is shattered and the experience is not convincing.
With an estimated revenue of $80bn by 2025 ("Virtual & Augmented Reality: Understanding the Race for the Next Computing Platform", Goldman Sachs, Equity Research, Jan. 13th 2016) the demand for trained VR audio experts will increase rapidly. Besides advancing methodologies in this cutting-edge technology, VRACE will train 15 Early Stage Researchers who will multiply and spread this knowledge in industry and academia. VRACE thus gives European industry a competitive edge in this global race.
The main objective of this Training Network is to raise VR to a next level beyond gaming and entertainment by benefiting from the critical mass of expertise gathered in this distinguished consortium. Driven and directed by the industrial partners, the consortium strives to make significant progress on the way towards a physically correct virtual reality and thus towards a true world simulation e.g. empowering engineers to enter VR for virtual testing and engineering. Although this network aims at sound experience design, there will be intermediate simulation results having some impact on the visual part of VR. In order to study sound generation, physical modelling of dynamic systems with moving parts and non-stationary fluid motion will be performed. The predicted motion of solid bodies and the dynamic deformation of objects will serve as the source of radiated sound, but simultaneously requires visual rendering in order to become visible and make VR physically correct.
Due to the interdisciplinary and intersectoral composition of the network, different partners have different but complementing requirements depending on their interests and their fields of expertise. Musicians need convincing sound simulations, room acousticians need tools for realistic room simulations, engineers dealing with Virtual Acoustics are interested in improving the computational efficiency of the whole audio rendering process. Especially for audio rendering it is essential to take human perception into account. This will ensure that no processing power is being wasted by rendering acoustic details which cannot be heard. Furthermore, perception modelling and psychoacoustic experiments may give clues on how sounds contribute to deepen immersion and make virtual environments more convincing.
Within the first reporting period, an excellent network with strong industrial participation has been established, in order to foster an ongoing cooperation between top universities and research centres from eight countries. All 15 Early Stage Researchers enrolled for a PhD programme and submitted a report on the plan of their studies (expose). Their Personal Career Development Plans are continuously updated in consultation with their supervisors. Outreach and dissemination activities have been carried out, as permitted by the implemented covid-19 restrictive measures. Five consortium meetings have been organised in order to monitor the progress of the project, including an assessment of research goals, methodology and risk mitigation, as well as a report on the progress of the scientific training.
Five training workshops (one with physical and four with virtual presence) have been organised, along with two online team-building workshops.
Research performed within the first periodic report includes:
- the formulation of different numerical algorithms for physical modelling of sound sources
- the development of machine learning algorithms for spatial audio
- numerical approaches for room acoustics and auralisation enhanced by experimental measurements
- numerical and experimental acquisition of head-related-transfer-functions (also considering the ear canal)
- listening tests to analyse sound perception in virtual environments.
Five training workshops (one with physical and four with virtual presence) have been organised, along with two online team-building workshops.
Research performed within the first periodic report includes:
- the formulation of different numerical algorithms for physical modelling of sound sources
- the development of machine learning algorithms for spatial audio
- numerical approaches for room acoustics and auralisation enhanced by experimental measurements
- numerical and experimental acquisition of head-related-transfer-functions (also considering the ear canal)
- listening tests to analyse sound perception in virtual environments.
It is expected that the interdisciplinary and intersectoral scientific training, together with a broad range of transferable skills training courses will ensure excellent career opportunities for the Early Stage Researchers in a field that is getting increased attention and attraction. At the same time, a standard will be set for future PhD curricula in the field of acoustics and audio.
By exploiting existing synergies and joining or exchanging research methodology, significant advances are expected in the area of virtual reality audio. This will allow to enhance a critical component not only of virtual-reality applications, but also of standard online communication platforms. The demand for high-quality audio in virtual environments, which is the subject of the VRACE project, has been made more apparent during the covid-19 pandemic. Despite the broad, every-day use of several online platforms ranging from common video-conference applications to elaborate 3D meeting spaces, the audio quality of such solutions has been under high scrutiny. While a possibility was offered for live online communication, several shortcoming were revealed, which require immediate improvement. Indeed, it is the audio part that constitutes the limiting factor of video conferences. This involves not only audio interruptions, but also a complete elimination of spatial information, making it impossible to intuitively identify the speaker. While striving to improve spatial audio within virtual reality applications, the accumulated knowledge resulting from the VRACE project will be of utmost importance to online communication activities that are currently of high demand in our society.
By exploiting existing synergies and joining or exchanging research methodology, significant advances are expected in the area of virtual reality audio. This will allow to enhance a critical component not only of virtual-reality applications, but also of standard online communication platforms. The demand for high-quality audio in virtual environments, which is the subject of the VRACE project, has been made more apparent during the covid-19 pandemic. Despite the broad, every-day use of several online platforms ranging from common video-conference applications to elaborate 3D meeting spaces, the audio quality of such solutions has been under high scrutiny. While a possibility was offered for live online communication, several shortcoming were revealed, which require immediate improvement. Indeed, it is the audio part that constitutes the limiting factor of video conferences. This involves not only audio interruptions, but also a complete elimination of spatial information, making it impossible to intuitively identify the speaker. While striving to improve spatial audio within virtual reality applications, the accumulated knowledge resulting from the VRACE project will be of utmost importance to online communication activities that are currently of high demand in our society.