Physically Based Simulation and Rendering of Thin Shell Sound

Project Information

PhySound

Grant agreement ID: 706708

DOI 10.3030/706708

Project terminated on 5 May 2018

EC signature date 31 August 2016

Start date 1 December 2016

End date 31 May 2019

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost € 221 635,80

EU contribution € 221 635,80

Coordinated by

INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET AUTOMATIQUE
France

Periodic Reporting for period 1 - PhySound (Physically Based Simulation and Rendering of Thin Shell Sound)

Reporting period: 2016-12-01 to 2018-05-31

Summary of the context and overall objectives of the project

Sound is as important as visuals in modern media such as movies and video games. Yet, relatively little effort has been devoted to the rendering of sound from digital environments, compared to the phenomenal advances of visual rendering. While sophisticated light transport algorithms allow the photorealistic rendering of a 3D scene, sound must be added through the ad-hoc edition of recorded sounds and their manual synchronization with the visuals, yielding limited and repetitive sounds. The PhySound project addresses this problem by generating sounds from virtual environments through physically based simulation, greatly simplifying the creation of sound content, allowing perfect synchronization with the visuals, and avoiding recordings that are sometimes slow, expensive or just impossible to capture. The project focuses on a challenging family of objects: thin shells. Thin shells are notoriously difficult to simulate due to their complex vibrations, often leading to computationally expensive chaotic regimes. Familiar thin shell sounds include crumpling paper and soda cans, striking plastic bottles and metal slabs, or playing instruments such as cymbals and gongs.

This project aims at digitally reproducing the main sources of thin shells sound: frictional contact, buckling/crumpling, and turbulence, all very distinct and characteristic sounds. The key challenge is computation time, with traditional techniques yielding accurate but prohibitively slow algorithms. We aim at making simulations tractable first, and real time afterwards.

This project considerably widens the number of real life object sounds that can be digitally generated, and contributes to the young research field of physically based sound rendering, which has the potential of becoming the next key technology of the media industry and revolutionize the way we create content just like graphics rendering did in the past. This project provides automatic sound content creation algorithms for better media production, and faster content creation cycles by avoiding recordings and visual synchronization. In addition, the project is expected to provide insight into the physical mechanisms that produce sound, which can be of interest to other fields beyond Computer Graphics.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the period covered by the report, we have addressed the physically based synthesis of thin shell sound from nonlinear and turbulent vibrations. More specifically:
- we have designed a reduced-space simulation algorithm to reduce the size of the problem from tens of thousands of degrees of freedom to a few hundred modes spanning the audible frequency range. By computing forces directly in the subspace, we achieved speed-ups of several orders of magnitude. This required designing special bending forces that could be computed directly in the subspace.
- we have designed a method that dramatically speeds-up the computation of nonlinear modal vibrations without quality loss. We split the frequency range in two sets of modes: a small set (around 50) encompassing low frequencies, and a large set (from 150 to 400) encompassing high frequencies. The low frequencies were simulated in a fully nonlinear way, mitigating computational costs thanks to the small size of the set. The high frequencies were simulated after a time-varying linearization, mitigating costs thanks to the formulation of a linear problem instead of a nonlinear one.
- we have designed a method that enriches simulated sounds with wave turbulent details. We sidestepped the prohibitively expensive computation of chaotic high-frequency dynamics by generating the turbulent spectrograms through the diffusion of energy in the frequency domain, phenomenologically capturing the behavior of turbulence.
- we generated results for a variety of thin shells, from water bottles and thin metal slabs to music instruments such as cymbals and gongs (see images attached to this summary).
- we qualitatively evaluated our results by comparing them to real recordings

Training was also an integral part of this project, and training tasks included:
- improvement of writing skills through the writing of a scientific article for the journal with the highest impact factor in the field. The article was accepted.
- teaching classes for Master students at Columbia University as an invited lecturer for Computer Graphics and Computer Animation courses.
- attending monthly seminars in a wide variety of subjects from Computer Graphics and Computer Science in general, through Columbia’s Distinguished Colloquium Series and guests lectures from professors visiting the lab.
- presentation of research progress and results at different scientific venues, such as the reference ACM SIGGRAPH Asia conference or the more local Tri-State Workshop on Imaging and Graphics.
- supervision of masters students on an independent research project and collaboration with PhD students in ongoing projects.
- participation in committees for PhD Candidacy examination.

Exploitation/dissemination:
- a journal paper accepted at ACM Transactions on Graphics, the highest impact journal in the field, and its corresponding presentation at ACM SIGGRAPH 2018, the largest venue in Computer Graphics.
- release of source code of a reference implementation to speed up adoption by the industry and encourage further research by other groups in academia.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

This project allowed to advance the state of the art in the area of physically based synthesis of sounds. Existing techniques either failed at capturing nonlinear audible phenomena, or provided very slow solutions inadequate for Computer Graphics. In any case, none of the existing techniques provided a tractable solution for sound produced by wave turbulence.

The impact of these results are many-fold:
- some of the algorithms and mathematical tools that were developed can be used outside of the scope of this project, and even in other fields of science. Namely, the formulation of forces for subspace simulation, which can be used in Computer Animation, and the frequency splitting mechanism, for the synthesis of vibrational motion in Engineering.
- we are the first to design an inexpensive approach to the synthesis of turbulent thin shell sounds, opening the door to fast adoption by the industry. By releasing the source code of the project we are further strengthening the impact of this work.
- we are expecting feedback and interest from the community in the form of potential collaborations. We have already been contacted by a French research group with reference expertise in the field of mechanical simulation of thin shells and wave turbulence to collaborate on follow-up work. We have also been contacted by the SoftBank group for an industrial project related to sound synthesis, resulting in the funding of 3 PhD students for the lab, thanks to the visibility given by our work on sound synthesis.
- Dr. Cirio has established himself as a world expert in the field of physically based sound synthesis, after two high impact publications in the reference journal ACM Transactions on Graphics. Research on this field being lately exclusively conducted in the United States, this unique transfer of knowledge positions him as the leading researcher in the field in Europe, as anticipated in Annex 1 of the Grant Agreement.

fig12.png

fig1.png

fig11.png

fig5.png