Periodic Reporting for period 1 - CROWDMERSE (CAPTURING AND REPRODUCTION OF CROWDED ACOUSTIC ENVIRONMENTS FOR IMMERSIVE APPLICATIONS)
Reporting period: 2017-10-01 to 2019-09-30
There are several applications of the immersive reproduction of crowded acoustic events. An application of considerable commercial interest is sporting events: by placing a microphone sensor in a football stadium within the fans’ bleachers, it would be possible to capture and reproduce the experience of ""being there"" to any TV or tablet user sitting in his/her living room. Going one step further, imagine a stadium populated by dozens of microphone sensors, all feeding audio signals to a central server that allows a producer, or even the end-user, to select the desired audio feed heard. Users could be offered a visual representation of the stadium, and interactively navigate the sonic landscape. Another application space of similar nature is coverage of large music concerts.
Our final objective is to derive the technology that enables the real-time delivery of such crowded acoustic events, so that the designed platform can be readily integrated with today’s High-Definition TV (HDTV) programs. CROWDMERSE focuses on spatial capturing of the sound information using multiple closely placed microphones forming a microphone array (a node of the deployed WASN) in order to offer immersive reproduction of the soundscape to the end-user in real-time. The audio content is rendered via a home entertainment system equipped with multiple loudspeakers, accompanying the audio-visual content of a typical HDTV broadcast."
The panning functions produce an identity mapping between the incident and perceived direction, e.g. the system recreates a virtual acoustic environment which is, to some degree, identical with the original sound field at the location of capturing. In our work, the most innovative point has been that we considered on a non-identity mapping between the incident and the perceived direction, so that the listener is “placed” at the center of the stadium, even though the capturing device is located far away from that center. Indeed, placing the array at the center of the stadium would be impossible in most cases without disturbing the players inside the field. Moreover, the examined formulation allows a flexible segmentation of the acoustic environment so that different arrays may be used in order to capture different segments of the stadium geometry. During the reporting period, we derived beamforming weights that can achieve the directivity patterns that have the desired effect. We also performed a thorough calibration procedure in order to identify potential sources of mismatch that might degrade the overall performance, including the microphone device (microphone array), the loudspeaker array, and the soundcard to be employed in the listening tests.
The project expected impact is the enhancement of the career of the Researcher, as well as the derivation of technological achievements that can improve Europe’s position in the particular area of the project. Regarding the Fellow, this period has been a great exposure to the industrial landscape of a dynamic SME, while the research he performed in this period formed the basis for important results that can be obtained in future research. At the same time, the research foundations laid in this period can form a basis for important research outcomes in the area especially of sport events broadcasting, which is currently under great technological advancements in terms of rendering quality and is open to such innovations that can enhance even more the quality that the viewers enjoy.