Skip to main content

PDE-based geometric modelling, image processing, and shape reconstruction

Periodic Reporting for period 1 - PDE-GIR (PDE-based geometric modelling, image processing, and shape reconstruction)

Reporting period: 2018-01-01 to 2019-12-31

Geometric modelling, image processing, and shape reconstruction greatly impact on EU’s science, technology, education, and economy. Existing PDE-based modelling cannot achieve both powerful capacity and high performance, PDE-based image processing suffers from expensive cost and local minimization, and PDE-based shape reconstruction has not been developed. The PDE-GIR project aims to tackle these problems. Its objectives are: 1) developing 3- and 4-sided PDE patches and three PDE-based modelling techniques: PDE patch-based modelling, PDE wireframe-based modelling, and PDE primitive-based modelling, 2) developing variational models and fast algorithms for depth information from images and surface reconstruction from point clouds, 3) developing new static and dynamic shape reconstruction techniques from 3- and 4-sided PDE patches, variational models, and fast algorithms, and 4) organizing staff exchanges, research collaborations, knowledge transfer, dissemination and applications of the new techniques through four work packages (WP1-WP4).
WP1 aims to develop new 3- and 4-sided PDE patches and three PDE-based modelling techniques, and promote their applications. It has established a general mathematical model, constructed a unified function for 2-, 3- and 4-sided patches, obtained an approximate analytical solution of the mathematical model, examined how different design parameters affect the shape of PDE surfaces, and used the constructed 2-, 3-, and 4-sided PDE patches to develop the three new modelling techniques. The finite difference mask and machine learning-based techniques have also been developed to solve PDEs for PDE surface creation. The remaining work is to integrate the developed PDE-based geometric modelling techniques into IDF’s services and products.

WP2 aims to develop variational models for depth estimation from images, use the graph-cut and continuous max-flow techniques to solve the models, investigate variational models and fast algorithms for surface reconstruction from point clouds, integrate new depth estimation and surface reconstruction, and apply them in the industrial partner. The work carried out in the first two years has completed the plans formulated in the tasks T2.1-2.4. We have developed a convexified global variational model for depth estimation from stereo images and proposed a fast numerical algorithm based on the augmented Lagrangian method. Most time consuming part of the algorithm is solution of a boundary value problem for the 3D Poisson equation. We have developed a fast direct solver for this PDE, which is based on the fast Fourier transform. We have also developed an iterative solver, using the conjugate gradient iteration, for the Poisson equation in a narrow band about the solution. Such a solver allows us to process images of larger size. It is worth to note that the variational model for the depth estimation is a continuous analog of the graph-cut approach which proved to be very efficient for discrete models in image processing. The second problem, surface reconstruction from point clouds, is solved by constructing the unsigned distance function and an inner product function, which is similar to the signed distance function but has some important advantages for surfaces with holes and boundaries. This approach requires consistent fields of normals to the point cloud. Since we estimate the depth maps on uniform rectangular grids, the normal field is computed by rather simple formulas in contrast to the difficult case of unordered point clouds. The surface is reconstructed as a level set of the inner product function and is smoothed afterwards by the aid of a variational method with anisotropic total variation as a regularizer. We have developed a software suite in MATLAB, which first solves the depth estimation problem and determines a consistent normal field to the surface of the depth. Then the same surface is reconstructed from the point cloud consisting of the grid points on the surface. It is done in order to cover the important case when several point clouds are merged in order to obtain the surface of a 3D object. The remaining work is to apply these models and algorithms into the services of IDF and investigate possibility to use deep learning for improving some parts of the developed technology.

WP3 aims at proposing methodologies using the new techniques developed in previous WP1 and WP2 to solve the shape reconstruction problem from data points and images, and apply them to real-world problems and the industrial partner’s services and production. So far, we have analyzed the methods developed in previous WP1 and WP2, corresponding to the tasks T3.1-T3.2 and are now working on developing new techniques for surface and object/scene reconstruction, corresponding to the tasks T3.3-T3.4. We have also developed new techniques integrating artificial intelligence methods for shape reconstruction, and applying them to real-world problems, corresponding to task T3.5 which is ahead of schedule. The new techniques have been applied so far in the fields of entertainment (automatic skeletal motion learning routines for computer animation, virtual reality, and video games), swarm robotics (including the physical assembly of real robotic units by hardware, as well as new software libraries for programming and simulation), and in medical fields (image segmentation of medical images, e.g. human brain MRI, digital images of melanoma and other skin lesions). As a result of this work, in WP3 we published 37 scientific papers in only 2 years (compared to 12 planned in WP3 and 50 for the whole project in the proposal for the four years). The remaining work is to finish tasks T3.3-T3.4 (expected for end of third year) and carry out task T3.6 (expected to be finished by the end of fourth year).

WP4 is designed to effectively manage staff secondments, knowledge transfer, ESRs’ trainings, project research, coordinate partners’ collaborations, disseminate research output, and promote applications of the developed techniques. It has organized the PDE-GIR consortium to implement staff secondments for developing new PDE-based techniques, train early stage researchers, organize international conferences, workshops, and seminars, and publish research papers in international journals and conferences. The following work is to complete the remaining tasks of this work package.
Analytical or approximate analytical 3- and 4-sided PDE patches with any orders of continuities and using them to develop new PDE-based modelling techniques have not been investigated. How to develop variational models and fast algorithms for depth estimation from images and surface construction from point clouds is an important and active topic. The work of developing PDE-based shape reconstruction using the developed 3- and 4-sided PDE patches, variational models and fast algorithms has not been initiated.

The PDE-GIR project has brought together the experts in different fields to fuse geometric modelling, image processing and shape reconstruction, extend their knowledge, skills and research fields, enhance their potential and career development. It has enabled them to carry out research collaborations and knowledge transfer to improve research and innovation potential and develop new PDE-based techniques to maintain and raise EU’s excellence in science and technology of geometric modelling, image processing, and shape reconstruction.