Periodic Reporting for period 4 - CHAMELEON (Intuitive editing of visual appearance from real-world datasets)
Période du rapport: 2021-05-01 au 2023-04-30
PROBLEM BEING ADDRESSED:
Computer-generated imagery is now ubiquitous in our society, spanning fields such as games and movies, architecture, engineering, or virtual prototyping, while also helping create novel ones such as computational
materials. With the increase in computational power and the improvement of acquisition techniques, there has been a paradigm shift in the field towards data-driven techniques, which has yielded an unprecedented
level of realism in visual appearance. Unfortunately, this leads to a series of problems: First, there is a disconnect between the mathematical representation of the data and any meaningful parameters that humans understand; in other words, the captured data is machine-friendly, but not human friendly. Second, the many different acquisition systems lead to heterogeneous formats and very large datasets. And third, real-world appearance functions are usually nonlinear and high-dimensional. As a result, visual appearance datasets are increasingly unfit to editing operations, which limits the creative process for scientists, engineers, artists and practitioners in general. There is an immense gap between the complexity, realism and richness of the captured data, and the flexibility to edit such data.
IMPORTANCE FOR SOCIETY:
Simulation and editing of visual appearance is a core area in the scientific field of computer graphics, involving aspects of computer science, mathematics or physics. It is not only a fundamental aspect of digital
content creation, but an inherent part of our lives as well: Our society depends on computer-generated imagery for entertainment, education, culture, medical imaging, architecture... while many industrial processes including manufacturing, engineering or virtual prototyping depend on correct simulations to convey the desired visual information. Moreover, developing proper design and editing algorithms for visual appearance is also a key feature for the success of novel fields at the interface between engineering, physics and graphics, such as computational materials or fabrication. However, editing the visual appearance of computer generated objects is a challenging goal.
OVERALL OBJECTIVES:
1. To develop human-friendly parameter spaces for material modeling and editing, which hide the complexity of their underlying mathematical representations
2. To develop predictable editing algorithms based on such parameter spaces, so the user can use high-level commands such as "make this a bit more papery, and a tad less shiny"
3. To develop interactive feedback and efficient simulations
Computer-generated imagery is now ubiquitous in our society, spanning fields such as games and movies, architecture, engineering, or virtual prototyping, while also helping create novel ones such as computational
materials. With the increase in computational power and the improvement of acquisition techniques, there has been a paradigm shift in the field towards data-driven techniques, which has yielded an unprecedented
level of realism in visual appearance. Unfortunately, this leads to a series of problems: First, there is a disconnect between the mathematical representation of the data and any meaningful parameters that humans understand; in other words, the captured data is machine-friendly, but not human friendly. Second, the many different acquisition systems lead to heterogeneous formats and very large datasets. And third, real-world appearance functions are usually nonlinear and high-dimensional. As a result, visual appearance datasets are increasingly unfit to editing operations, which limits the creative process for scientists, engineers, artists and practitioners in general. There is an immense gap between the complexity, realism and richness of the captured data, and the flexibility to edit such data.
IMPORTANCE FOR SOCIETY:
Simulation and editing of visual appearance is a core area in the scientific field of computer graphics, involving aspects of computer science, mathematics or physics. It is not only a fundamental aspect of digital
content creation, but an inherent part of our lives as well: Our society depends on computer-generated imagery for entertainment, education, culture, medical imaging, architecture... while many industrial processes including manufacturing, engineering or virtual prototyping depend on correct simulations to convey the desired visual information. Moreover, developing proper design and editing algorithms for visual appearance is also a key feature for the success of novel fields at the interface between engineering, physics and graphics, such as computational materials or fabrication. However, editing the visual appearance of computer generated objects is a challenging goal.
OVERALL OBJECTIVES:
1. To develop human-friendly parameter spaces for material modeling and editing, which hide the complexity of their underlying mathematical representations
2. To develop predictable editing algorithms based on such parameter spaces, so the user can use high-level commands such as "make this a bit more papery, and a tad less shiny"
3. To develop interactive feedback and efficient simulations
The CHAMELEON project was structured in three Work Packages (WPs), with a large overlap between them:
WP1. Human-friendly parameter spaces
WP2. Predictable editing algorithms
WP3: Interactive feedback and efficient simulation
All of them have produced significant advances in the state of the art, even spawning and inspiring new lines of research in the international community in some cases. We showed how finding perceptual latent spaces for material appearance (which vastly differ from traditional BRDF-based specifications) lead in fact to novel and intuitive editing techniques. Apart from the particular results and edits achieved, we established a solid methodology to first design and launch reliable crowdsource experiments to gather subjective information about material appearance, then feed the data to neural architecture that learns a feature space for materials which correlate with such subjective perception of appearance. Taking this even further, we enabled material editing techniques that work in image space, with unknown information about shape or illumination. We also studied the use of Large Language Models as a new paradigm in image editing algorithms. The grand vision is that the user would simply talk to the computer telling it what to do, and the computer will be able to provide a meaningful, perceptually accurate edit to material appearance.
ERC grants are special in that risky ideas are not only allowed, but encouraged. The idea is that true breaktrhoughs are usually achieved by thinking outside the box, as opposed to following a pre-established path. In that sense, this project was also a success; a task that in principle was meant to be secondary (exploring different alternatives to express light transport in computer generated images) lead to a novel diffraction-based algorithm, which was published in Nature and which we have first applied to the problem of non-line-of-sight imaging. It has become the state of the art in NLOS methods, allowing for the first time to be able to image complex scenes hidden around corners; the intuition is that we transform any (non-mirror) wall into virtual cameras, being able to reconstruct the invisible scene by analysing the light that falls onto such walls.
In terms of exploitation and dissemination, our work has appeared in more than 20 top scientific publications (including Nature), and has included collaborations with top institutions and companies like NASA, Adobe or Meta. The PI and other project members have been invited to give talks at several international events, and some of our material acquisition methods are candidates to launch commercial endeavors.
WP1. Human-friendly parameter spaces
WP2. Predictable editing algorithms
WP3: Interactive feedback and efficient simulation
All of them have produced significant advances in the state of the art, even spawning and inspiring new lines of research in the international community in some cases. We showed how finding perceptual latent spaces for material appearance (which vastly differ from traditional BRDF-based specifications) lead in fact to novel and intuitive editing techniques. Apart from the particular results and edits achieved, we established a solid methodology to first design and launch reliable crowdsource experiments to gather subjective information about material appearance, then feed the data to neural architecture that learns a feature space for materials which correlate with such subjective perception of appearance. Taking this even further, we enabled material editing techniques that work in image space, with unknown information about shape or illumination. We also studied the use of Large Language Models as a new paradigm in image editing algorithms. The grand vision is that the user would simply talk to the computer telling it what to do, and the computer will be able to provide a meaningful, perceptually accurate edit to material appearance.
ERC grants are special in that risky ideas are not only allowed, but encouraged. The idea is that true breaktrhoughs are usually achieved by thinking outside the box, as opposed to following a pre-established path. In that sense, this project was also a success; a task that in principle was meant to be secondary (exploring different alternatives to express light transport in computer generated images) lead to a novel diffraction-based algorithm, which was published in Nature and which we have first applied to the problem of non-line-of-sight imaging. It has become the state of the art in NLOS methods, allowing for the first time to be able to image complex scenes hidden around corners; the intuition is that we transform any (non-mirror) wall into virtual cameras, being able to reconstruct the invisible scene by analysing the light that falls onto such walls.
In terms of exploitation and dissemination, our work has appeared in more than 20 top scientific publications (including Nature), and has included collaborations with top institutions and companies like NASA, Adobe or Meta. The PI and other project members have been invited to give talks at several international events, and some of our material acquisition methods are candidates to launch commercial endeavors.
We have successfully demonstrated how CG materials can be expressed in an intuitive, high-level parameter space (defined by words like "roughness", "shiny", "plasticky", etc), and how this leads to efficient editing algorithms, without the need to know the particular mathematical representations of each material. We have also leveraged Large Language Models to make the interaction even more natural, enabling free-text descriptions to query datasets of thousands of materials, removing the need for tags or keywords, as well producing natural descriptions of materials from images. Our work has inspired many related lines of research in other groups.
One important achievement has to do with the development of a novel light transport framework, published in Nature, which has the potential to extend capture and editing capabilities of real-world materials not in the line of sight of a camera.
One important achievement has to do with the development of a novel light transport framework, published in Nature, which has the potential to extend capture and editing capabilities of real-world materials not in the line of sight of a camera.