CarMen

Project Information

CarMen

Grant agreement ID: 101168325

DOI

10.3030/101168325

EC signature date 19 July 2024

Start date 1 September 2024

End date 31 August 2027

Funded under

• Civil Security for Society

Total cost

No data

EU contribution

€ 2 998 347,88

Coordinated by

YNCREA MEDITERRANEE
France

Periodic Reporting for period 1 - CarMen (CarMen)

Reporting period: 2024-09-01 to 2026-02-28

Summary of the context and overall objectives of the project

European border management is entering a new operational era. The progressive deployment of large-scale IT systems such as the Entry/Exit System (EES), the steady growth of cross-border mobility, and the increasing sophistication of identity fraud require border authorities to reconcile three strategic imperatives: high security standards, smooth passenger flows, and strict compliance with EU fundamental rights and data protection frameworks.

Traditional border control models, largely based on stationary checkpoints and sequential identity verification, can create bottlenecks in high-throughput environments such as ferry terminals, road crossings, and coach transport hubs. In parallel, biometric technologies must remain reliable in uncontrolled field conditions (variable lighting, motion, weather, reflections).

CarMen addresses this need by developing and validating “biometrics on the move” solutions designed to operate in realistic border scenarios involving moving vehicles and pedestrian flows.

To achieve this ambition, CarMen is structured around eight specific objectives:

1: Enable border crossing checks in slowly moving vehicles using face biometrics.
Develop and validate face capture and recognition capabilities for vehicles moving at low speed (up to 20 km/h), for multiple passengers, under real-world constraints (angles, windows, motion).
2: Demonstrate the usage and benefits of Digital Travel Credentials (DTCs) at the border.
Integrate DTC-based workflows to support secure pre-enrolment, reducing reliance on time-consuming document handling.
3. Detect anomalies in biometrics on-the-move using behaviour analysis.
Introduce behavioural indicators as complementary signals to support anomaly detections.
4. Control the cost and energy consumption of biometric processing using frugal AI approaches.
Design close to edge-compatible algorithms , supporting sustainability goals and operational constraints in the field.
5. Make face recognition on the move robust and efficient.
Improve face recognition performance under motion blur, pose variations, occlusions, distance, and operational variability, while maintaining an efficient end-to-end processing pipeline.
6. Ensure robustness to any lighting thanks to heterogeneous face recognition on-the-move.
Enable reliable matching between enrolment/reference images acquired under one condition (typically visible/RGB, e.g. smartphone/DTC) and probe images captured under different sensors or spectra (e.g. NIR), bridging cross-domain gaps to maintain performance across day/night and heterogeneous illumination settings.
7.Detect presentation attacks on-the-move.
Develop and evaluate Presentation Attack Detection (PAD) methods adapted to on-the-move acquisition constraints.
8. Offer solutions compliant with border control requirements, infrastructure, and legal constraints in the field of biometrics on the move.
Ensure that the solutions can be integrated into real border processes and infrastructures, and remain compliant with legal and regulatory constraints.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the reporting period, the project focused on the definition and validation of technically realistic “biometrics on-the-move” solutions aligned with actual border operational conditions. A detailed description of use cases was established in close correspondence with field constraints, covering moving vehicles, coach environments and pedestrian flows. These use cases were not theoretical abstractions but were designed to reflect real border infrastructure layouts, environmental variability, and operational throughput requirements.

Based on these grounded use cases, the project specified and progressively deployed a modular system architecture integrating heterogeneous sensors (RGB and NIR), and a infrastructure enabling interoperability between biometric, behavioural and decision modules. The deployment of this architecture included the configuration of acquisition setups representative of real checkpoints, as well as the implementation of structured communication interfaces supporting end-to-end integration from capture to final decision.

A series of multimodal data acquisition campaigns were conducted in an iterative manner. Initial campaigns enabled the validation of sensor positioning, illumination strategies, and capture protocols. Feedback from these sessions directly informed subsequent refinements in hardware configuration, camera angles, illumination wavelength selection, and processing pipelines.

On the algorithmic side, first significant results were obtained for face detection and tracking in moving vehicle scenarios, demonstrating the feasibility of extracting usable face samples under motion and variable lighting. In parallel, early evaluation of presentation attack detection (PAD) methods for face biometrics provided an initial assessment of system robustness against spoofing attempts, including printed and display-based attacks. These first experimental results establish a technical baseline for improving detection performance under on-the-move constraints.

Overall, the activities carried out during the reporting period resulted in a field-aligned technical framework, validated acquisition infrastructures, initial operational datasets, and the first integrated performance indicators for face detection and presentation attack resilience in realistic border scenarios.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

CarMen delivers results that advance beyond the current state of the art in biometric border control systems by addressing a critical gap: reliable identity verification under dynamic, real-world operational constraints.

In the tinted vehicle scenario, specific technical measures are implemented to address the optical attenuation and spectral filtering induced by tinted or coated windshields and side windows. CarMen deploys dedicated camera streams operating at two separate wavelengths to be robust againt reflection, tinted windows and open windows. Each spectral stream is processed independently before fusion, allowing heterogeneous matching between enrolment data (typically RGB) and probe data captured in NIR conditions.

This multi-spectral acquisition strategy enables more stable facial feature extraction and improves robustness against illumination variability, reflections, and tint-induced contrast loss. By using dedicated camera flows at optimised wavelengths rather than relying on a single broadband acquisition channel, the system enhances reliability in realistic vehicle border-crossing conditions while preserving compatibility with existing infrastructure constraints.

A second key advancement concerns presentation attack detection (PAD on-the-move. Existing PAD solutions are largely benchmarked in laboratory conditions. CarMen adapts and evaluates PAD mechanisms under motion, cross-spectral acquisition, and environmental variability, including early assessment of vulnerabilities to printed and replay or injection attacks.

The project also moves beyond the state of the art through the integration of behavioural analysis as a complementary signal for anomaly detection in biometric verification. By incorporating temporal and contextual cues into the decision process, CarMen explores a more holistic reliability assessment framework, going beyond purely biometric similarity scores.