Periodic Reporting for period 2 - H-Unique (In search of uniqueness - harnessing anatomical hand variation)
Reporting period: 2020-07-01 to 2021-12-31
H-unique will be the first multimodal automated interrogation of visible hand anatomy, through analysis and interpretation of human variation. It will be an interdisciplinary project, supported by anatomists, anthropologists, geneticists, bioinformaticians, image analysts and computer scientists. We will investigate inherent and acquired variation in search of uniqueness, as the hand retains and displays a multiplicity of anatomical variants formed by different aetiologies (genetics, development, environment, accident etc).
Hard biometrics, such as fingerprints, are well understood and some soft biometrics are gaining traction within both biometric and forensic domains (e.g. superficial vein pattern, skin crease pattern, morphometry, scars, tattoos and pigmentation pattern). A combinatorial approach of soft and hard biometrics has not been previously attempted from images of the hand. We will pioneer the development of new methods that will release the full extent of variation locked within the visible anatomy of the human hand and reconstruct its discriminatory profile as a retro-engineered multimodal biometric. A significant step change is required in the science to both reliably and repeatably extract and compare anatomical information from large numbers of images especially when the hand is not in a standard position or when either the resolution or lighting in the image is not ideal.
Large datasets are vital for this work to be legally admissible. Through citizen engagement with science, this research will collect images from over 5,000 participants, creating an active, open source, ground-truth dataset. It will examine and address the effects of variable image conditions on data extraction and will design algorithms that permit auto-pattern searching across large numbers of stored images of variable quality. This will provide a major novel breakthrough in the study of anatomical variation, with wide ranging, interdisciplinary and transdisciplinary impact.
Our key objectives are (i) To establish variability in the human hand to better understand variation. (ii) To create new algorithms to both reliably and repeatedly extract anatomical features from images. (iii) To determine the extent to which variation in hand position and image quality alters the ability to recognise features of hand anatomy. (iv) To undertake black and white box testing, to establish a hierarchy of hand biometrics. (v) To retro-engineer a multimodal biometric to represent and visualise hand variation thereby establishing uniqueness.
Hard biometrics, such as fingerprints, are well understood and some soft biometrics are gaining traction within both biometric and forensic domains (e.g. superficial vein pattern, skin crease pattern, morphometry, scars, tattoos and pigmentation pattern). A combinatorial approach of soft and hard biometrics has not been previously attempted from images of the hand. We will pioneer the development of new methods that will release the full extent of variation locked within the visible anatomy of the human hand and reconstruct its discriminatory profile as a retro-engineered multimodal biometric. A significant step change is required in the science to both reliably and repeatably extract and compare anatomical information from large numbers of images especially when the hand is not in a standard position or when either the resolution or lighting in the image is not ideal.
Large datasets are vital for this work to be legally admissible. Through citizen engagement with science, this research will collect images from over 5,000 participants, creating an active, open source, ground-truth dataset. It will examine and address the effects of variable image conditions on data extraction and will design algorithms that permit auto-pattern searching across large numbers of stored images of variable quality. This will provide a major novel breakthrough in the study of anatomical variation, with wide ranging, interdisciplinary and transdisciplinary impact.
Our key objectives are (i) To establish variability in the human hand to better understand variation. (ii) To create new algorithms to both reliably and repeatedly extract anatomical features from images. (iii) To determine the extent to which variation in hand position and image quality alters the ability to recognise features of hand anatomy. (iv) To undertake black and white box testing, to establish a hierarchy of hand biometrics. (v) To retro-engineer a multimodal biometric to represent and visualise hand variation thereby establishing uniqueness.
In the first 30 months of the project, we have established the project teams at Lancaster University and the University of Dundee. As planned, work has primarily focussed on work packages 1-3, which include the outreach, establishment of two large ground truth image datasets and 3D dataset, development of automated feature extraction algorithms. Good progress has also been made on work packages 4 and 5 in developing feature comparison algorithms to facilitate black box studies, the development of a feature hierarchy and the multimodal biometric. Although COVID-19 has resulted in delays in some aspects of the work, we have rearranged some plans to continue to make excellent progress, particularly in bringing forward development of feature comparison methodology as well as cultivating additional databases with further annotations to progress development.
We have established the infrastructure, databases, mechanisms and collection systems for the required project datasets and made very good progress with collection. This includes the development of a multi-camera image capture rig for Dataset 2 (High Quality Dataset), a web-based application to facilitate data collection for Dataset 1 (Large-Scale Citizen Science) and a multi-camera rig for photogrammetry for Dataset 3 (3D Hand Dataset). Although COVID-19 has impacted on the data collection for Datasets 2 and 3, Dataset 1 is ahead of the scheduled collection rate since it does not require face to face contact with contributors. At 2445 contributors (22,892 images), this is the largest hand database in existence and ahead of the milestone amount of contributions. We have developed and implemented our plans for continuous engagement with periodic events to ensure that submissions to our datasets continues; some in-person plans are on hold due to safety concerns relating to the coronavirus pandemic but we have worked to mitigate the negative effects of this, focusing our attention online. We have also established procedures for the annotation of datasets 1 and 2 and a considerable amount of work has been carried out on this by expert forensic anthropologists to map out the key anatomical features, with excellent progress made.
A key objective of the project and work package 2 in particular is to develop the ability to accurately extract key features from photographs of the hands, including the superficial veins, knuckle and palmar creases, pigmentation, scars and lunules. We have developed approaches for vein extraction in two modalities (colour photograph and infrared), as well as crease and pigmentation extraction. We have further developed localisation techniques to find and identify key regions of the hand from any image (regardless of scene, camera and quality) to assist in the identification, including the hand itself, knuckles and joints, punctate pigmentation, and fingernails for lunule analysis. Excellent accuracy is being achieved and this is being evaluated on our real-world Dataset 1, Dataset 2 and external datasets. Several publications have been prepared and submitted to exploit the vein feature extraction and comparison results, knuckle crease extraction and comparison and feature localisation. One of these has been published already at the International Joint Conference on Biometrics, two are currently shared on arxiv to maximise exposure and three publications are in preparation for the International Conference on Computer Vision and Pattern Recognition, the top conference on computer vision.
Work package 3 has been advanced in several directions at both Lancaster and Dundee, going beyond the original plan. We have carried out work to develop the rig for collecting multiple images of the hand for high quality 3D reconstruction, and evaluated methods of achieving this. Data collection has begun with members of the team, since face-to-face research with members of the public is currently prohibited, but this is sufficient for system evaluation in preparation for data collection with the general public, which we expect to resume in September 2021. We have also developed a technique for determining the 3D surface representation of the hand from single 2D images to assist with vein matching and have developed techniques for image quality estimation. We are currently working on refinements for the 3D reconstruction techniques, targeted towards finer features, and sub-region quality assessment with multiple characteristics; i.e. automatically detecting and evaluating the factors affecting quality.
We have established the infrastructure, databases, mechanisms and collection systems for the required project datasets and made very good progress with collection. This includes the development of a multi-camera image capture rig for Dataset 2 (High Quality Dataset), a web-based application to facilitate data collection for Dataset 1 (Large-Scale Citizen Science) and a multi-camera rig for photogrammetry for Dataset 3 (3D Hand Dataset). Although COVID-19 has impacted on the data collection for Datasets 2 and 3, Dataset 1 is ahead of the scheduled collection rate since it does not require face to face contact with contributors. At 2445 contributors (22,892 images), this is the largest hand database in existence and ahead of the milestone amount of contributions. We have developed and implemented our plans for continuous engagement with periodic events to ensure that submissions to our datasets continues; some in-person plans are on hold due to safety concerns relating to the coronavirus pandemic but we have worked to mitigate the negative effects of this, focusing our attention online. We have also established procedures for the annotation of datasets 1 and 2 and a considerable amount of work has been carried out on this by expert forensic anthropologists to map out the key anatomical features, with excellent progress made.
A key objective of the project and work package 2 in particular is to develop the ability to accurately extract key features from photographs of the hands, including the superficial veins, knuckle and palmar creases, pigmentation, scars and lunules. We have developed approaches for vein extraction in two modalities (colour photograph and infrared), as well as crease and pigmentation extraction. We have further developed localisation techniques to find and identify key regions of the hand from any image (regardless of scene, camera and quality) to assist in the identification, including the hand itself, knuckles and joints, punctate pigmentation, and fingernails for lunule analysis. Excellent accuracy is being achieved and this is being evaluated on our real-world Dataset 1, Dataset 2 and external datasets. Several publications have been prepared and submitted to exploit the vein feature extraction and comparison results, knuckle crease extraction and comparison and feature localisation. One of these has been published already at the International Joint Conference on Biometrics, two are currently shared on arxiv to maximise exposure and three publications are in preparation for the International Conference on Computer Vision and Pattern Recognition, the top conference on computer vision.
Work package 3 has been advanced in several directions at both Lancaster and Dundee, going beyond the original plan. We have carried out work to develop the rig for collecting multiple images of the hand for high quality 3D reconstruction, and evaluated methods of achieving this. Data collection has begun with members of the team, since face-to-face research with members of the public is currently prohibited, but this is sufficient for system evaluation in preparation for data collection with the general public, which we expect to resume in September 2021. We have also developed a technique for determining the 3D surface representation of the hand from single 2D images to assist with vein matching and have developed techniques for image quality estimation. We are currently working on refinements for the 3D reconstruction techniques, targeted towards finer features, and sub-region quality assessment with multiple characteristics; i.e. automatically detecting and evaluating the factors affecting quality.
Much of the work that we have carried out has improved on the state of the art (SOA), as evidenced by our publications. We have built the world's largest database of hand images and this continues to grow. In the case of vein segmentation and particularly vein map extraction, our algorithms improve considerably over the SOA in terms of accuracy. We have presented the first method for vein map extraction from colour photographs and infrared imaging, demonstrating it's improvement of the current SOA by applying it to more commonly reported tasks such as crack detection and road mapping. We have further presented the first method for simultaneously automatically detecting and labelling the regions of each of the knuckles of the hand and extended this to include fingernails, allowing us to map the skeleton of the hand and automatically break it down into key regions for analysis. We have contrasted this with a similar approach that aims to find the centre-points of the knuckles, and demonstrated that our approach also outperforms the current SOA in this task. We have also developed methods for graph matching (to facilitate identification of the vein patterns) and knuckle crease matching, both of which achieve superior results to the SOA on their respective tasks. We have further developed a method for the holistic comparison of knuckle creases.
Following on from this, we plan to further grow all of our databases and refine our 3D image reconstruction to allow for high-resolution 3D imaging, enabling us to map the minutiae of the hand. We will refine our feature extraction methodology, thoroughly evaluating our new approaches for pigmentation and lunule detection, extending these to measurements to analysing scars, tattoos and modification, while seeking to refine our vein pattern and knuckle crease algorithms that are already SOA. We will further improve on our feature comparison methodology and, very importantly, the generalisability of our approaches to facilitate analysis of all of our datasets. In this way, we will be able to map the entire hand from any photograph and thereby determine the degree of variation in the human hand. We will further refine our quality measures to build a confidence score. We will build on our holistic approaches, considering both feature- and score-level fusion. All of this will feed into a multimodal biometric, which is capable of functioning correctly regardless of the image, lighting conditions or pose.
Following on from this, we plan to further grow all of our databases and refine our 3D image reconstruction to allow for high-resolution 3D imaging, enabling us to map the minutiae of the hand. We will refine our feature extraction methodology, thoroughly evaluating our new approaches for pigmentation and lunule detection, extending these to measurements to analysing scars, tattoos and modification, while seeking to refine our vein pattern and knuckle crease algorithms that are already SOA. We will further improve on our feature comparison methodology and, very importantly, the generalisability of our approaches to facilitate analysis of all of our datasets. In this way, we will be able to map the entire hand from any photograph and thereby determine the degree of variation in the human hand. We will further refine our quality measures to build a confidence score. We will build on our holistic approaches, considering both feature- and score-level fusion. All of this will feed into a multimodal biometric, which is capable of functioning correctly regardless of the image, lighting conditions or pose.