Periodic Reporting for period 2 - D4FLY (Detecting Document frauD and iDentity on the fly)
Reporting period: 2021-03-01 to 2022-08-31
The goal of the D4FLY project was to augment the capabilities of border authorities in countering emerging threats in document and identity verification at manual and highly automated border control points and in the issuance process of genuine documents. D4FLY examined the overall identity lifecycle from document issuance based on breeder documents, like birth certificates, to document usage and identity verification in various border crossing scenarios.
The D4FLY set of tools and systems investigated various technologies to enhance the quality of document and identity verification and lay the foundation for smooth on-the-move border crossing experiences for travellers. Novel sensor hardware based on advanced light field cameras and novel algorithms were developed in the project to enhance the verification accuracy via the combined usage of multiple different biometric modalities modalities, while performing under real-time constraints.
D4FLY created a document verification system that can verify a multitude of physical and electronic security features in travel documents like passports and help to analyse and detect fraud in breeder documents. The D4FLY developments included a border control kiosk geared with enhanced enrolment, verification and detection capabilities and a continuous on-the-move corridor for biometric verification. All innovations were validated against European societal values, fundamental rights, privacy, data protection and applicable legislation. Four different border control points and one document fraud expertise centre formed the project’s testing and demonstration grounds.
The main objectives of D4FLY were to identify existing or possible future fraud methods regarding travel documents and identity and to create new technologies in automated document fraud, spoofing and impostor attack detection. Further objectives include investigation of new technologies in person identification that improve the recognition of travellers on-the-move. The research was continuously monitored to be in compliance with European societal values, fundamental rights and applicable legislation.
The D4FLY set of tools and systems investigated various technologies to enhance the quality of document and identity verification and lay the foundation for smooth on-the-move border crossing experiences for travellers. Novel sensor hardware based on advanced light field cameras and novel algorithms were developed in the project to enhance the verification accuracy via the combined usage of multiple different biometric modalities modalities, while performing under real-time constraints.
D4FLY created a document verification system that can verify a multitude of physical and electronic security features in travel documents like passports and help to analyse and detect fraud in breeder documents. The D4FLY developments included a border control kiosk geared with enhanced enrolment, verification and detection capabilities and a continuous on-the-move corridor for biometric verification. All innovations were validated against European societal values, fundamental rights, privacy, data protection and applicable legislation. Four different border control points and one document fraud expertise centre formed the project’s testing and demonstration grounds.
The main objectives of D4FLY were to identify existing or possible future fraud methods regarding travel documents and identity and to create new technologies in automated document fraud, spoofing and impostor attack detection. Further objectives include investigation of new technologies in person identification that improve the recognition of travellers on-the-move. The research was continuously monitored to be in compliance with European societal values, fundamental rights and applicable legislation.
Early in the project, high quality user needs were elicited, and the use cases have been refined in workshops with end users and stakeholders. Technical requirements have been derived and structured, based on the inputs from stakeholders and project participants. An overall system architecture was defined and implemented, specifying a modular, high performance system, using well defined communication interfaces, leveraging a distributed and heterogeneous development environment.
A novel 3D face recognition approach has been developed using images captured by innovative Raytrix cameras. Reconstruction of 3D face images from two 2D face images was researched and a new and improved approach was developed. A novel approach to thermal-to-visible face recognition was researched, where a thermal image could be captured during an enrolment process and verified against an image obtained in a capture area of a verification corridor. The research on iris-on-the-move technology using innovative Raytrix cameras led to an optimized camera setup for improved image quality and was tested in a prototype biometric corridor. A new algorithm for somatotype verification was developed and validated. All biometric components were required to perform under real-time constraints and were successfully integrated and tested in a biometric verification corridor prototype.
Alternative technologies for people identification and the application in the D4FLY scenarios have been explored. Research into techniques on continuous person verification using machine learning techniques based on smartphone sensors has led to the development of the first prototype smartphone application for enrolment and verification. Another prototype smartphone application was developed for traveller verification in confined spaces such as inside a coach, based on enrolled reference data using face recognition technologies. Potential use cases for blockchain technologies were investigated, and a prototype component was developed. A novel approach for determination of travellers’ risks based on fusion of meta data information collected from a variety of data sources was researched, and a prototype was created and discussed with end users.
A wide variety potential attacks to automated identity verification systems were analysed. Algorithms for the detection of spoofing attacks, like detection of manipulated images in travel documents, morphed images, presentation attacks and impostor fraud have been developed, integrated and tested.
New technologies in automated document fraud detection have been developed for breeder documents, as well as for travel documents (passports). Prototype software for the verification of document security elements were developed and integrated into the D4FLY document verification platform. A prototype for the analysis of breeder documents at detail level (e.g. investigation of stamps) and at tactical level has been developed and tested. Tools for travel pattern analysis based on stamp recognition in passports have been developed. A prototype for the application of blockchain technology in the context of document verification was developed.
All prototypes have been tested in field tests in the UK, Lithuania, Greece and the Netherlands. The valuable feedback from practitioners and end users was used to further improve the prototypes towards their final versions, which have been finally demonstrated in the UK (Highly automated border post and coach scenario) and in the Netherlands (Document verification).
A key part of the project work was to investigate the privacy, data protection, ethical and social issues raised in the context of the multimodal biometric systems. The ethics and data protection aspects have been continuously monitored throughout the research and development and the Ethical Impact Assessment was updated in the course of the project.
D4FLY disseminated its research and innovation results on its project web page and on multiple social media channels, through newsletters and blog posts, as well as through participation in conferences, workshops and other events.
h newsletters and blog posts, as well as through participation in conferences, workshops and other events.
A novel 3D face recognition approach has been developed using images captured by innovative Raytrix cameras. Reconstruction of 3D face images from two 2D face images was researched and a new and improved approach was developed. A novel approach to thermal-to-visible face recognition was researched, where a thermal image could be captured during an enrolment process and verified against an image obtained in a capture area of a verification corridor. The research on iris-on-the-move technology using innovative Raytrix cameras led to an optimized camera setup for improved image quality and was tested in a prototype biometric corridor. A new algorithm for somatotype verification was developed and validated. All biometric components were required to perform under real-time constraints and were successfully integrated and tested in a biometric verification corridor prototype.
Alternative technologies for people identification and the application in the D4FLY scenarios have been explored. Research into techniques on continuous person verification using machine learning techniques based on smartphone sensors has led to the development of the first prototype smartphone application for enrolment and verification. Another prototype smartphone application was developed for traveller verification in confined spaces such as inside a coach, based on enrolled reference data using face recognition technologies. Potential use cases for blockchain technologies were investigated, and a prototype component was developed. A novel approach for determination of travellers’ risks based on fusion of meta data information collected from a variety of data sources was researched, and a prototype was created and discussed with end users.
A wide variety potential attacks to automated identity verification systems were analysed. Algorithms for the detection of spoofing attacks, like detection of manipulated images in travel documents, morphed images, presentation attacks and impostor fraud have been developed, integrated and tested.
New technologies in automated document fraud detection have been developed for breeder documents, as well as for travel documents (passports). Prototype software for the verification of document security elements were developed and integrated into the D4FLY document verification platform. A prototype for the analysis of breeder documents at detail level (e.g. investigation of stamps) and at tactical level has been developed and tested. Tools for travel pattern analysis based on stamp recognition in passports have been developed. A prototype for the application of blockchain technology in the context of document verification was developed.
All prototypes have been tested in field tests in the UK, Lithuania, Greece and the Netherlands. The valuable feedback from practitioners and end users was used to further improve the prototypes towards their final versions, which have been finally demonstrated in the UK (Highly automated border post and coach scenario) and in the Netherlands (Document verification).
A key part of the project work was to investigate the privacy, data protection, ethical and social issues raised in the context of the multimodal biometric systems. The ethics and data protection aspects have been continuously monitored throughout the research and development and the Ethical Impact Assessment was updated in the course of the project.
D4FLY disseminated its research and innovation results on its project web page and on multiple social media channels, through newsletters and blog posts, as well as through participation in conferences, workshops and other events.
h newsletters and blog posts, as well as through participation in conferences, workshops and other events.
D4FLY progresses beyond state of the art in several ways and the results include:
1. Innovations in biometric recognition methods namely 3D-Face, Iris on-the-move, thermal-to-visible and somatotype verification
2. Advancements in counter spoofing and presentation attack detection technologies, especially liveliness detection technology and detection of manipulated images
3. Advancements in document verification technologies, extending the know-how on automatic detection and verification of document security elements.
4. New ways of using smartphones with apps for person identification as well as secure storage of traveller information.
Those technologies were tested in prototypes in four different scenarios at borders in four countries.
All activities were monitored and assessed for their compliance with the European data protection and privacy regulation, as well as ethical and societal impact.
1. Innovations in biometric recognition methods namely 3D-Face, Iris on-the-move, thermal-to-visible and somatotype verification
2. Advancements in counter spoofing and presentation attack detection technologies, especially liveliness detection technology and detection of manipulated images
3. Advancements in document verification technologies, extending the know-how on automatic detection and verification of document security elements.
4. New ways of using smartphones with apps for person identification as well as secure storage of traveller information.
Those technologies were tested in prototypes in four different scenarios at borders in four countries.
All activities were monitored and assessed for their compliance with the European data protection and privacy regulation, as well as ethical and societal impact.