Periodic Reporting for period 2 - PDE-GIR (PDE-based geometric modelling, image processing, and shape reconstruction)
Berichtszeitraum: 2020-01-01 bis 2023-10-31
Geometric modelling, image processing, and shape reconstruction have huge applications in various sectors and greatly impact on EU’s economy, human health, security, entertainment and others. Existing problems such as big data and heavy manual operations in geometric modelling, incapacity in image processing in continuous domains, and lack of continuity and differentiability in shape reconstruction seriously affect quality, efficiency and capacity. Partial differential equation (PDE) based techniques provide effective solutions to these problems. However, current PDE-based techniques face the following problems.
The existing research on PDE-based geometric modeling only develops analytical solutions of PDEs under boundary constraints of 2-sided patches. Such 2-sided patches are incapable in creating many 3D models with branch structures that can only be tackled by 3- and 4-sided patches. Since the analytical solutions for 3- and 4-sided PDE patches are difficult to obtain, numerical methods have to be applied. It causes three problems: big data increasing storage and network transmission cost, heavy computational burden leading to low efficiency, and discrete nodes inapplicable to curved boundaries and good continuity requirement at boundaries.
Variational image-processing models, involving PDEs, energy optimization, regularization, level set functions, and numerical algorithms, offer high-quality processing capabilities for imaging. They have been widely applied in image denoising, image deblurring, image segmentation, image reconstruction, restoration of mixed noise types, and three-dimensional surface restoration. However, existing PDE-based image processing techniques suffer from expensive cost and local minimization.
Shape reconstruction based on polygon, CSG and NURBS involves big data which cause many problems such as heavy geometry processing, high data storage cost, and slow transmission over computer networks. PDE-based shape reconstruction has not been developed.
The PDE-GIR project aims to tackle these problems. Its objectives are: 1) developing 3- and 4-sided PDE patches and three PDE-based geometric modelling techniques, 2) developing variational models and fast algorithms for depth information from images and surface reconstruction from point clouds, 3) developing new static and dynamic shape reconstruction techniques from 3- and 4-sided PDE patches, variational models, and fast algorithms, and 4) organizing staff exchanges, research collaborations, knowledge transfer, dissemination and application exploitation of the new techniques.
The existing research on PDE-based geometric modeling only develops analytical solutions of PDEs under boundary constraints of 2-sided patches. Such 2-sided patches are incapable in creating many 3D models with branch structures that can only be tackled by 3- and 4-sided patches. Since the analytical solutions for 3- and 4-sided PDE patches are difficult to obtain, numerical methods have to be applied. It causes three problems: big data increasing storage and network transmission cost, heavy computational burden leading to low efficiency, and discrete nodes inapplicable to curved boundaries and good continuity requirement at boundaries.
Variational image-processing models, involving PDEs, energy optimization, regularization, level set functions, and numerical algorithms, offer high-quality processing capabilities for imaging. They have been widely applied in image denoising, image deblurring, image segmentation, image reconstruction, restoration of mixed noise types, and three-dimensional surface restoration. However, existing PDE-based image processing techniques suffer from expensive cost and local minimization.
Shape reconstruction based on polygon, CSG and NURBS involves big data which cause many problems such as heavy geometry processing, high data storage cost, and slow transmission over computer networks. PDE-based shape reconstruction has not been developed.
The PDE-GIR project aims to tackle these problems. Its objectives are: 1) developing 3- and 4-sided PDE patches and three PDE-based geometric modelling techniques, 2) developing variational models and fast algorithms for depth information from images and surface reconstruction from point clouds, 3) developing new static and dynamic shape reconstruction techniques from 3- and 4-sided PDE patches, variational models, and fast algorithms, and 4) organizing staff exchanges, research collaborations, knowledge transfer, dissemination and application exploitation of the new techniques.
WP1 has proposed a general mathematical model, constructed a unified function for 2-, 3- and 4-sided patches, obtained an approximate analytical solution of the mathematical model, used the constructed 2-, 3-, and 4-sided PDE patches to develop the three new modelling techniques of PDE patch-based geometric modelling, PDE wireframe-based geometric modelling, and PDE primitive-based geometric modelling, investigated the applications of PDE-based geometric modelling techniques in character modelling , skin deformation, facial blendshapes, optimal conversion of PDE surface-represented train heads into the NURBS representation, multi-objective aerodynamic optimization of high-speed train heads, styling design of automobiles, and developed machine learning-based techniques to solve PDEs for surface creation. The research into WP1 has led to 28 scientific publications.
WP2 has proposed a convexified global variational model for depth estimation from stereo images, investigated a fast numerical algorithm based on the augmented Lagrangian method, examined a fast direct solver for this PDE based on the fast Fourier transform and an iterative solver using the conjugate gradient iteration, for the Poisson equation in a narrow band about the solution. It has also solved the problem of surface reconstruction from point clouds by constructing the unsigned distance function and an inner product function to propose a variational model for surface construction from point clouds, implemented two software programs respectively for the two variational models and an integrated software suite in MATLAB, documented the user guide for the software, and applied the variational models, algorithms, and computer programs in the services of IDF. It also investigated machine learning-based image processing. The research into WP2 has led to 22 scientific publications.
WP3 has analyzed the methods developed in WP1 and WP2, proposed a theoretical framework of PDE-based shape reconstruction, investigated the accurate closed form solution of the theoretical framework to exactly satisfy the constraints on four boundaries of a PDE surface patch, applied the closed form solution to develop new shape reconstruction techniques for static surface and object/scene reconstruction. It has also proposed a dynamic partial differential equation, investigated its analytical solution, and used the analytical solution to develop new time-dependent shape reconstruction techniques for dynamic surface and object/scene reconstruction. In addition, it has also developed new techniques integrating artificial intelligence methods for shape reconstruction, and applied them in real-world problems, such as in the fields of entertainment (automatic skeletal motion learning routines for computer animation, virtual reality, and video games), swarm robotics (including the physical assembly of real robotic units by hardware, as well as new software libraries for programming and simulation), and in medical fields (image segmentation of medical images, e.g. human brain MRI, digital images of melanoma and other skin lesions). As a result of this work, WP3 has published 88 scientific papers.
WP4 has organized the PDE-GIR consortium to successfully complete all planned PDE-GIR secondments and project research tasks, achieve knowledge transfer, ESRs’ trainings, partners’ collaborations, publish 138 papers, disseminate research output at international conferences, organize 5 PDE-GIR workshops and 3 special issues, and exploit the applications of the developed PDE-GIR techniques in industrial and academic sectors.
During the implementation, the PDE-GIR project has always complied with the “ethics requirements” set out in WP5.
WP2 has proposed a convexified global variational model for depth estimation from stereo images, investigated a fast numerical algorithm based on the augmented Lagrangian method, examined a fast direct solver for this PDE based on the fast Fourier transform and an iterative solver using the conjugate gradient iteration, for the Poisson equation in a narrow band about the solution. It has also solved the problem of surface reconstruction from point clouds by constructing the unsigned distance function and an inner product function to propose a variational model for surface construction from point clouds, implemented two software programs respectively for the two variational models and an integrated software suite in MATLAB, documented the user guide for the software, and applied the variational models, algorithms, and computer programs in the services of IDF. It also investigated machine learning-based image processing. The research into WP2 has led to 22 scientific publications.
WP3 has analyzed the methods developed in WP1 and WP2, proposed a theoretical framework of PDE-based shape reconstruction, investigated the accurate closed form solution of the theoretical framework to exactly satisfy the constraints on four boundaries of a PDE surface patch, applied the closed form solution to develop new shape reconstruction techniques for static surface and object/scene reconstruction. It has also proposed a dynamic partial differential equation, investigated its analytical solution, and used the analytical solution to develop new time-dependent shape reconstruction techniques for dynamic surface and object/scene reconstruction. In addition, it has also developed new techniques integrating artificial intelligence methods for shape reconstruction, and applied them in real-world problems, such as in the fields of entertainment (automatic skeletal motion learning routines for computer animation, virtual reality, and video games), swarm robotics (including the physical assembly of real robotic units by hardware, as well as new software libraries for programming and simulation), and in medical fields (image segmentation of medical images, e.g. human brain MRI, digital images of melanoma and other skin lesions). As a result of this work, WP3 has published 88 scientific papers.
WP4 has organized the PDE-GIR consortium to successfully complete all planned PDE-GIR secondments and project research tasks, achieve knowledge transfer, ESRs’ trainings, partners’ collaborations, publish 138 papers, disseminate research output at international conferences, organize 5 PDE-GIR workshops and 3 special issues, and exploit the applications of the developed PDE-GIR techniques in industrial and academic sectors.
During the implementation, the PDE-GIR project has always complied with the “ethics requirements” set out in WP5.
Analytical or approximate analytical 3- and 4-sided PDE patches with any orders of continuities and using them to develop new PDE-based modelling techniques have not been investigated. How to develop variational models and fast algorithms for depth estimation from images and surface construction from point clouds is an important and active topic. The work of developing PDE-based shape reconstruction using the developed 3- and 4-sided PDE patches, variational models and fast algorithms has not been initiated.
The PDE-GIR project has brought together the experts in different fields to fuse geometric modelling, image processing and shape reconstruction, extend their knowledge, skills and research fields, enhance their potential and career development. It has enabled them to carry out research collaborations and knowledge transfer to improve research and innovation potential and develop new PDE-based techniques.
The new techniques have been applied in industrial and academic sectors to create impact and maintain and raise EU’s excellence in science and technology of geometric modelling, image processing, and shape reconstruction.
The PDE-GIR project has brought together the experts in different fields to fuse geometric modelling, image processing and shape reconstruction, extend their knowledge, skills and research fields, enhance their potential and career development. It has enabled them to carry out research collaborations and knowledge transfer to improve research and innovation potential and develop new PDE-based techniques.
The new techniques have been applied in industrial and academic sectors to create impact and maintain and raise EU’s excellence in science and technology of geometric modelling, image processing, and shape reconstruction.