Final Report Summary - SP-MORPH (Spectral Mesh Processing for Craniofacial Dysmorphology)
The relation between craniofacial dysmorphology and certain diseases had been already suggested many years ago. Among the most evident examples are the distinctive facial characteristics of patients with Down syndrome [Ferrario 2005], but these have also been identified in autism [Ozgen 2010], schizophrenia [Hennessy 2007], bipolar disorder [Hennessy 2010], velocardiofacial syndrome [Óskarsdóttir 2005], fetal alcohol syndrome, etc. In the latter, the unique pattern of facial anomalies is the only diagnostic feature so far that is specific to the condition and has therefore been the focus of case definition [Mutsvangwa 2010].
Based on the above evidence, craniofacial geometry has been suggested as a potential index of early developmental disturbance [Hammond 2007, Hennessy 2010, Chakravarty 2011]. Recent technological advances on 3D imaging have made it possible to analyze craniofacial shape based on indirect measurements, as opposed to the classical direct anthropometry. However, in contrast to the evident dysmorphology in diseases like Down syndrome, dysmorphology in other disorders such as schizophrenia, bipolar disorder and velocardiofacial syndrome, can be very subtle to the extent that it can hardly be identified by the human eye.
We have focused on the development of algorithms for automated and highly accurate analysis of facial surfaces in 3D, with special interest in techniques based on spectral decomposition methods. As opposed to traditional methods, based on a reduced set of landmark points, spectral mesh processing (SMP) allows analysis of the whole facial surface. Briefly speaking, SMP algorithms provide a decomposition of the geometry into its natural vibration modes [Levy 2006]. The resulting components, analogous to the Fourier Transform for 1D signals [Taubin 1995], are linked to intrinsic properties of the object, such as (a)symmetry [Ovsjanikov 2008], believed to be a crucial component of dysmorphology [Hammond 2008]. While SMP is a novel and very active trend in computer graphics and vision [Zhang 2010], it still involves a number of important technical challenges for its use in engineering applications, where input data would usually need to undergo one or more pre-processing steps, often with the need for human intervention before such spectral methods can be used.
Thus, we addressed the issues of highly accurate and fully automatic analysis of craniofacial geometry, with a special emphasis on the repeatability of results, so that large populations could be analyzed in a systematic and consistent manner. To this end, we worked on a 4-step pipeline consisting of 1) pre-processing, 2) automatic landmark localization, 3) landmark-based surface normalization, and 4) spectral analysis.
Careful inspection of the facial surfaces from the available datasets revealed an important presence of artifacts. Their impact in the input surfaces is task-dependant. On one hand, the automatic identification of landmarks should be robust to these artifacts as it constitutes the input block to the system and is aimed at allowing geometric normalization into a common reference that facilitates further analysis. In contrast, at the analysis step we aim at working with high-quality and artifact-free data.
The automatic localization of landmarks was addressed by designing an algorithm that integrated combinatorial search within the responses of feature detectors, with statistical inference. [Sukno 2012a]. A key assumption of our approach is that some landmarks might not be accurately identified by the feature detectors. While many approaches try to fight this by computing more features or analyzing more detections, we found that it is more accurate to omit the information from unreliable landmarks and infer their location statistically. This is analogous to the point-matching problem found in algorithms that search for correspondences. However, the human face is a non-rigid object and these point-matching algorithms are typically restricted to rigid transformations. We tackle this by detecting partial subsets of landmarks and inferring those that are missing so that the probability of a deformable model is maximized.
We demonstrated the accuracy of the proposed method in clinical data gathered in the context of craniofacial dysmorphology research [Hennessy 2002], where we obtained average errors of approximately 3 mm. This compares favorably with other state of the art approaches. We also proposed a new family of 3D geometry descriptors based on asymmetry patterns that not only provide higher accuracy than other popular geometric descriptors but can also be built in an incremental manner, reducing the total computational cost [Sukno 2013a, Sukno 2013c].
An interesting outcome of our research on landmark localization was to highlight the lack of consistency of the manual annotations currently available for public databases such as FRGC (Face Recognition Grand Challenge [Phillips 2005]). In contrast to traditional measures of accuracy, such as inter- and intra-observer variability, we based our analysis on the consistency of annotations by comparing the inter-landmark distances of replicates (i.e. different scans from the same individual). We objectively showed that manual annotations currently available are suboptimal and can strongly impair the accuracy of automatic models learnt therefrom. To address this issue, we presented a simple algorithm to automatically correct a set of annotations and showed that it can help to significantly improve the accuracy of the models in terms of landmark localization errors [Sukno 2013b].
While we successfully constructed a robust automatic landmarking algorithm, it was concluded that it would be highly desirable to introduce additional processing blocks to eliminate some of the artifacts present in the input data, such as singularities, non-manifold geometry, disconnected parts and holes. Hence, we have implemented solutions that can handle the majority of these issues and addressed a comparison of state-of-the-art hole-filling algorithms. Although previous comparisons of this kind existed, they were based on how visually pleasant were the resulting surfaces, without providing quantitative figures to indicate how faithful they were with respect to the hypothetical hole-less surface). In contrast, we developed a framework for quantitative evaluation of hole-filling algorithms and generated a realistic dataset of synthetic patches that we made available to serve as benchmark material [Rojas 2014].
Once the input data is free of artifacts and annotated with (automatic) landmarks, we addressed its normalization by implementing Least Squares Conformal Maps (LSCM) [Levy 2002]. Given that the face is (approximately) a genus-0 surface, it can be mapped conformaly into the 2D domain. The conformality condition ensures that the angles are locally preserved, hence minimizing mapping distortion. At least two corresponding points are needed to make the mapping unique, but additional points can be added in order to obtain a least squares solution, which would balance the localization errors of individual points [Wang 2007]. Since we can choose the target domain in 2D, it is possible to map a population of surfaces so that they are in correspondence in the mapped space. Thus, with an appropriate re-sampling and the inverse mappings from 2D to 3D, we can obtain a new representation of the input surfaces in which the whole set is in correspondence.
The above can be very powerful. On one hand, it allows us to apply standard multivariate analysis technique. But it can also help to investigate aspects such as bilateral asymmetry, since the re-sampling in the 2D domain can be designed to produce points that correspond to anatomically symmetric pairs. Recall that, in general, the human face is not strictly symmetric and, in anatomical terms, the hypothetical axis of bilateral symmetry is not contained in a plane. However, most research in facial symmetry relies on the simplification of taking the mid-sagittal plane as the plane of bilateral symmetry, which is then used to measure the deviations of one side with respect to the other [Claes 2011]. This is related to the direct use of 3D coordinates for analysis, while the facial surface can be considered a 2D manifold embedded in 3D. In contrast, the spectrum of the Laplacian relates to the structure of the manifold itself and can be used for analysis without considering reflection over the mid-sagittal plane. So far, however, computation of this spectrum is dependent on the extent of facial region that is captured, which is extremely variable. Hence, the need and utility of the mesh normalization that we perform based on LSCM.
We continue to investigate in this promising line of research. More information and updates are available in the project web-site: http://www.cipa.dcu.ie/face3d/SP_MORPH_Project.htm
[Claes 2011] P. Claes, M. Walters, D. Vandermeulen and J.G. Clemen. Spatially-dense 3D facial asymmetry assessment in both typical and disordered growth. Journal of Anatomy, 219(4):444–455, 2011.
[Chacravarty 2011] M.M. Chakravarty et al. Automated analysis of craniofacial morphology using magnetic resonance images. Plos ONE 6(5):e20241, 2011.
[Ferrario 2005] V.F. Ferrario, C. Dellavia, G. Serrao and C. Sforza. Soft tissue facial angles in Down’s syndrome subjects: a three-dimensional non-invasive study. European Journal of Orthodontics 27(4):355–62, 2005
[Hammond 2007] P. Hammond. The use of 3D face shape modelling in dysmorphology. Archives of Disease in Childhood 92:1120–1126, 2007.
[Hammond 2008] P. Hammond, C Forster-Gibson, A.E. Chudley, et al. Face–brain asymmetry in autism spectrum disorders. Molecular Psychiatry, 13, 614–623, 2008.
[Hennessy 2002] R.H. Hennessy, A. Kinsella, and J.L. Waddington (2002). 3D laser surface scanning and geometric morphometric analysis of craniofacial shape as an index of cerebro-craniofacial morphogenesis: initial application to sexual dimorphism. Biological Psychiatry, 51(6):507–514. 2002.
[Hennessy 2007] R.J. Hennessy, P.A. Baldwin, D.J. Browne, A. Kinsellac and J.L. Waddingtona. Three-dimensional laser surface imaging and geo- metric morphometrics resolve frontonasal dysmorphology in schizophrenia. Biological Psychiatry 61:1187–1194, 2007.
[Hennessy 2010] R.J. Hennessy, P.A. Baldwin, D.J. Browne, A. Kinsellac and J.L. Waddingtona. Frontonasal dysmorphology in bipolar disorder by 3D laser surface imaging and geometric morphometrics: Comparisons with schizophrenia. Schizophrenia Research, 122(1-3):63–71, 2010.
[Levy 2002] B. Levy, S. Petitjean and N. Ray et al. Least Squares Conformal Maps for Automatic Texture Atlas Generation. Proc. Int. Conf. Computer graphics and interactive techniques (SIGGRAPH), pp. 362-371, 2002.
[Levy 2006] B. Lévy. Laplace-Beltrami eigenfunctions. Towards and algorithm that ‘understands’ geometry. Proc IEEE Conf Shape Modeling and Applications, 2006.
[Mutsvangwa 2010] T.E.M. Mutsvangwa, E.M. Meintjes, D.L. Viljoen and T.S. Douglas. Morphometric analysis and classification of the facial phenotype associated with Fetal Alcohol Syndrome in 5- and 12-year-old children. American Journal of Medical Genetics Part A. 152A:32–41, 2010.
[Óskarsdóttir 2005] S. Óskarsdóttir, C. Persson, B.O. Eriksson and A. Fasth. Presenting phenotype in 100 children with the 22q11 deletion syndrome. European Journal of Pediatrics. 164(3):146–153, 2005.
[Ovsjanikov 2008] M. Ovsjanikov, J. Sun, and L. Guibas. Global intrinsic symmetries of shapes. Computer Graphics Forum, 27(5):1341–1348, 2008.
[Ozgen 2010] H.M. Ozgen, J.W. Hop, J.J. Hox, F.A Beemer and H van Engeland,. Minor physical anomalies in autism: a meta-analysis. Molecular Psychiatry, 15:300–307, 2010.
[Phillips 2005] P.J. Phillips, P.J. Flynn, T. Scruggs et al. (2005). Overview of the face recognition grand challenge. In Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, vol 1, pp. 947–954, 2005.
[Rojas 2014] [M. Rojas, F.M. Sukno, J.L. Waddington and P.F. Whelan. Quantitative Comparison of Hole Filling Methods for 3D Object Search, Proc. 7th Eurographics Workshop on 3D Object Retrieval (Accepted, In Press], 2014.
[Sukno 2012a] F.M. Sukno, J.L Waddington, and P.F. Whelan. 3D Facial Landmark Localization Using Combinatorial Search and Shape Regression. ECCV Workshop on Non-Rigid Shape Analysis and Deformable Image Alignment, LNCS vol. 7583, pp 32–41, 2012.
[Sukno 2013a] F.M. Sukno, J.L Waddington, and P.F. Whelan. Rotationally invariant 3D shape contexts using asymmetry patterns. In Proc. Int. Conf. on Computer Graphics Theory and App., pages 7–17, 2013.
[Sukno 2013b] F.M. Sukno, J.L Waddington, and P.F. Whelan. Compensating inaccurate annotations to train 3D facial landmark localization models, FG Workshop on 3D Face Biometrics Workshop, pp 1-8, 2013.
[Sukno 2013c] F.M. Sukno, J.L Waddington, and P.F. Whelan. Craniofacial Landmark Localization with Asymmetry Patterns Shape Contexts, BioPhotonics and Imaging Conference (BioPIC), Dublin, Ireland, 2013.
[Taubin 1995] G. Taubin. A signal processing approach to fair surface design. Proc SIGGRAPH, pp. 351-358, 1995.
[Wang 2007] S. Wang, Y. Wang, M. Jin et al. Conformal Geometry and Its Applications on 3D Shape Matching, Recognition, and Stitching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7):1–12, 2007.
[Zhang 2010] H. Zhang, O. van Kaick and R. Dyer. Spectral mesh processing. Computer Graphics Forum, 29(6):1865–1894, 2010.