All publications referenced as (Author, year) can be found at
https://project.inria.fr/fungraph/publications/(si apre in una nuova finestra)In FUNGRAPH we developed several novel research solutions in traditional rendering, appearance capture, novel view synthesis and neural rendering and multi-view relighting.
We investigated traditional rendering algorithms using artist-generated content. Most CG images in film are rendered using the path tracing algorithm that simulates the propagation of light from the light sources to the eye along a set of paths, bouncing off surfaces with different materials. This process is often called global illumination (GI). We developed a method that pre-computes GI especially for shiny materials, allowing fast lookup at runtime (Rodriguez et al. ‘20a). We later investigated how deep learning techniques can be used to precompute GI efficiently, allowing interactive display at runtime (Diolatzis et al. ’22, Rainer et al. ’22); these are now called “neural rendering” methods for traditional assets.
We studied the problem of estimating material properties of real objects for use in traditional rendering. We use a few photos as input, providing a simple way to model materials for CG assets, that otherwise requires significant effort from trained artists. We separate photographs into layers of appearance: a “base texture", and separate layers explaining shiny appearance. We train a neural network using artist-created assets that provide “ground truth” layers. The novelty is to combine multiple copies of the network allowing use of several photos of a material patch to improve the estimate (Deschaintre et al. 19). We provide additional artistic control to allow capture at different scales (Deschaintre et al. ’20, cf image).
As an alternative to artist created assets, 3D content can be directly created from photos using Image-Based Rendering (IBR) or Novel View Synthesis (NVS). We worked on traditional IBR methods, developing solutions for the hard cases of reflections (Rodriguez et al. ‘20b) and video-based reconstruction of repetitive motion (Thonat et al. ’21). As the efficiency of deep learning approaches improved, we developed innovative solutions based on convolutional neural networks (CNN’s) for novel view synthesis (Philip et al. ’21). We also developed methods for relighting captured scenes, using synthetic data and deep learning (Philip et al. ’19, Philip et al. ’21, cf image) and more recently generative models (Poirier-Ginter et al. ’24).
FUNGRAPH introduced a new paradigm building on point-based neural rendering (Kopanas et al. ’21) demonstrating that explicit, primitive-based representations, coupled with a CNN produce novel views more efficiently and with higher quality than implicit learning-based solutions (cf image), including for reflections (Kopanas et al. ’22). These results led to the development of the most significant achievement of FUNGRAPH, i.e. 3D Gaussian Splatting (3DGS), (Kerbl, Kopanas et al. ’23,
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/(si apre in una nuova finestra) cf image). We also solved two limitations of 3DGS, treating very large scenes using thousands of photographs (Meuleman et al. ’24) and reducing the memory requirements by up to a factor of 27 (Papantonakis et al. ’24).
3DGS is a truly disruptive methodology; Our paper has been cited over 1900 times in 16 months, and many researchers have built on our software. 3DGS has unprecedented success in technology transfer. We released the code (
https://github.com/graphdeco-inria/gaussian-splatting(si apre in una nuova finestra)) as mixed license open source, free for research & evaluation and paid for commercial use. The code has been downloaded hundreds of thousands of times and Inria has sold multiple commercial licenses to companies in high-end visual effects, e-commerce, generative AI for 3D objects, social media, virtual reality & telecommunications. The method has been adopted in products from Meta (
https://bit.ly/3AUae0m(si apre in una nuova finestra)) Adobe (
https://bit.ly/4i1eKec(si apre in una nuova finestra)) Amazon (
https://bit.ly/3CAy0io(si apre in una nuova finestra)) etc.