Periodic Reporting for period 1 - OPTIMAL (OPtimal Transport for Identifying Marauder Activities on LiDAR) Reporting period: 2022-01-16 to 2023-01-15 Summary of the context and overall objectives of the project Illegal excavation of archaeological sites to collect historical material culture ("looting") is a pressing problem on a global scale with strong consequences on security, economics, and society. Looting is the main source of income for terroristic groups and organised crime undermining the security and the development of the affected states. The monitoring of looting (past and ongoing) thus plays a crucial role in the protection of cultural heritage by strengthening the ability of Law Enforcement Agencies to promptly react to criminal activities. Due to the spread of the phenomenon and the impossibility of physically inspecting unreachable areas (e.g. forests covered by tick and closed canopies) or hazardous zones (e.g. turmoil in Middle East), surveillance via remote sensing is the most efficient approach to monitor looting activities. The OPTIMAL (OPtimal Transport for Identifying Marauder Activities on LiDAR) project aims to undermine the illegal excavation of cultural heritage sites by developing an efficient and principled machine learning approach, based on optimal transport, to automatically detect past and present looting directly on airborne Light Detection And Ranging (LiDAR) point cloud time-series.Moving from this overall objective, OPTIMAL specifically intended to: O1. create the first multi-temporal LiDAR dataset to train change detection methods for the identification of looting activities;O2. implement a novel change detection method, based on optimal transport, for the automatic identification and monitoring of cultural heritage looted sites directly on LiDAR point cloud time-series;O3. achieve a looting detection accuracy of 85% on two user-case scenarios relying on the ground-truthing data already collected and on the collaboration with landscape archaeologists.The project outcomes will concur to increase Europe's research profile in the current dominant discourse over the heritage safeguard by offering a powerful machine learning tool for archaeologists and stakeholders involved in the fight against marauder activities that represent a major source of income of criminal groups. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far The project set a benchmark in the use of optimal transport for the identification of looting activities directly on bi-temporal pair of airborne LiDAR point clouds collected over the same geographical area. Characterization and selection of looted sites were performed in the three months of secondment at the Centre for Cultural Heritage Technology of the Italian Institute of Technology (the European host institution). As part of the planned training activities, the fellow attended the ‘Training and Research in the Archaeological Interpretation of Lidar’ (TRAIL) in Slovenia on the 27th-28th of April 2022 to expand his knowledge on the use of LiDAR data in order to better address his research question in the field of landscape archaeology and to establish new interdisciplinary collaborations. The second output of this secondment was the collection of multi-temporal LiDAR point clouds for the identification of looting activities provided by archaeologists that collaborate to this project. In the outgoing phase at Kyoto University, the fellow designed and implemented an unbalanced optimal transport-based pipeline to identify changes related to looting activities on point clouds. The efficacy of this approach was demonstrated both on a test-bench dataset for building change detection and on the created dataset for the identification of looting activities. As intermediate result, the project enabled to automatically identify and map cultural heritage sites in wooded areas. The use of optimal transport for processing point clouds elucidated the depth of the detected looting pits, which is a crucial information to understand the state of degradation of the pillaged sites. In the first year of the project, a strategic plan for the dissemination and communication of the intrinsic interdisciplinary of OPTIMAL’s results was implemented to maximize their impact during and after the project. A dedicated website was created to serve as a hub to provide access to content suitable for both an academic and general audience. Scientific dissemination was pursued through the submission of two papers to top-tier conferences in remote sensing and computer vision. The fellow has invited speaker at a seminar organised by the Okinawa Institute of Technology to present the project’s intermediate results. Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) The outcomes of the project provide a new study basis for future research relative to the use of machine learning and remote sensing for the automatic identification and monitoring of cultural heritage looting activities. (i) Previous scholars have not taken into consideration the direct use of LiDAR point clouds to detect looting: OPTIMAL proposed for the first time the use of airborne LiDAR for monitoring and detecting looting activities by constructing and making publicly available the first multi-temporal LiDAR dataset with a ML baseline for illegal activities’ identification. This outcome provides insights on the capability of point clouds to identify the 3-D shapes of looting pits, thus fostering the development of novel methods to directly process LiDAR point clouds in archaeological research. (ii) Literature rarely focuses on the application of optimal transport to detect changes occurring in multi-temporal remote sensing data: results of the project show the potential of optimal transport theory to develop a sound computational framework to effectively deal with the change detection problem. Specifically, the developed change detection method based on unbalanced optimal transport showed superior performance over the state-of-the-art on both the only publicly available test-bench airborne LiDAR dataset for (building) change detection and on the dataset created by the fellow for looting identification. (iii) In the second year of the project, the fellow will tackle the scalability limitation that affects existing optimal transport-based methods proposing a novel approach based on the Nystrom method. The advantage of this approach is the increase of capability in processing a large number of 3D point clouds and the robustness with respect to variations in the number of available points of the two clouds captured at two successive equally spaced points in time over the same geographical location. The outcomes of this project will have an impact on future discussions on the current dominant debate on the heritage safeguard by offering a powerful machine learning tool for the archaeologists and stakeholders involved in the fight against marauder activities that represent a loss of invaluable properties as well as obliterate modern societies roots. Invited Speaker at Okinawa Institute of Technology (Japan)
