Periodic Reporting for period 1 - RePAIR (Reconstructing the Past: Artificial Intelligence and Robotics Meet Cultural Heritage)
Periodo di rendicontazione: 2021-09-01 al 2023-02-28
The physical reconstruction of shattered artworks is one of the most labour-intensive steps in archaeological research. Dug out from excavation sites are countless ancient artefacts, such as vases, amphoras and frescoes, that are damaged. The RePAIR project aims to facilitate the reconstruction process to bring ancient artworks back to life. Specifically, we are developing an intelligent robotic system that can autonomously process, match and physically assemble large fractured artefacts in a fraction of the time required by humans. In addition to the physical dimension, the level of ambition of our proposal poses several challenges that cannot be addressed with off-the-shelf technologies. Hence, we shall develop tailored solutions that will push the boundaries of research in the fields of robotics and computer vision. This new system is being tested on iconic case studies from the UNESCO World Heritage Site of Pompeii.
The work performed from the beginning of the project to the end of the first reporting period is hereafter summarized with respect to the three areas of interventions:
1) Data acquisition and 3D modelling: the pipeline for 3D acquisition of fragments has been consolidated. The focus has been on data acquisition procedures, including equipment setup and capturing pipeline, as well as post-processing pipeline for obtaining high-quality digitized representations of fresco pieces. Digital models of scanned frescoes and metadata creation have been finalized. Work has also begun on acquiring hyperspectral information from fragments. Experiments using different machine learning architectures have been conducted for fragment detection and recognition. An RGB camera placed on the robotic workbench captured pictorial and geometrical information of the fragments, providing satisfactory results in terms of detection rate. The detection model is being integrated into the robotic platform. Pose estimation is being approached as a pose regression problem using a deep neural model, with ongoing investigations. The RePAIR catalog has been set up and currently contains the 10% of the estimated total fragments.
2) Puzzle solving and reconstruction: interviews were conducted with archaeologists and restoration experts to gather knowledge and identify the main operational processes necessary for the reconstruction algorithms. The work began on the puzzle solving pipeline, which involved fragment processing, pairwise matching, and global solutions. It focused on feature extraction and virtual fragment cleaning to overcome marks made by archaeologists. Geometric features such as linear and corner features, color histograms, and boundary shape descriptors were developed. A pairwise fragment matching scheme using 3D point cloud data was then implemented, and a rendering tool was generated for photorealistic 2D images of intact surfaces. The 3D geometric puzzle solvers were explored, leveraging previous experience within the consortium and showing promising results for small puzzles using physical simulation methods. A human-in-the-loop approach was shared to allowing experts to interact with the puzzle-solving system and fix mistakes and provide helpful hints during the reconstruction process.
3) Robotic platform: fundamental requirements of the bi-manual platform have been developed and the soft end-effectors obtained. Proprioceptive sensing using a disturbance observer and motor current measurement was chosen for grasping force sensing. Various configurations of bi-manual systems were explored to maximize manipulability while ensuring kinematic feasibility and collision avoidance. Three end-effector devices based on the Pisa/IIT SoftHand technology were designed for manipulating different types of fresco fragments. Two anthropomorphic soft hand devices were created for medium/small and large fragments, while the third device had a gripper-like form with longer middle phalanges. Mechanical design improvements were made for the finger and palm parts of the third device, and soft pads were developed to enhance safety and grip when interacting with fragments. In terms of motion planning, a point-cloud completion approach was developed to generate plausible geometry for occluded parts of objects. A grasp sampling method was created to generate feasible hand poses for grasping objects, and a force-based controller was developed to improve grasping performance. A basic grasping pipeline for real-world scenarios was also developed. A full simulation integrating all platform elements was created in RViz-Gazebo environments. The sensing system, including the in-hand camera, is being integrated with the end-effectors. Work is underway to create a simulation-based dataset for bimanual grasp estimation. Additionally, a MoveIt configuration was created to control simulated arms using point control.
1) Data acquisition and 3D modelling: the pipeline for 3D acquisition of fragments has been consolidated. The focus has been on data acquisition procedures, including equipment setup and capturing pipeline, as well as post-processing pipeline for obtaining high-quality digitized representations of fresco pieces. Digital models of scanned frescoes and metadata creation have been finalized. Work has also begun on acquiring hyperspectral information from fragments. Experiments using different machine learning architectures have been conducted for fragment detection and recognition. An RGB camera placed on the robotic workbench captured pictorial and geometrical information of the fragments, providing satisfactory results in terms of detection rate. The detection model is being integrated into the robotic platform. Pose estimation is being approached as a pose regression problem using a deep neural model, with ongoing investigations. The RePAIR catalog has been set up and currently contains the 10% of the estimated total fragments.
2) Puzzle solving and reconstruction: interviews were conducted with archaeologists and restoration experts to gather knowledge and identify the main operational processes necessary for the reconstruction algorithms. The work began on the puzzle solving pipeline, which involved fragment processing, pairwise matching, and global solutions. It focused on feature extraction and virtual fragment cleaning to overcome marks made by archaeologists. Geometric features such as linear and corner features, color histograms, and boundary shape descriptors were developed. A pairwise fragment matching scheme using 3D point cloud data was then implemented, and a rendering tool was generated for photorealistic 2D images of intact surfaces. The 3D geometric puzzle solvers were explored, leveraging previous experience within the consortium and showing promising results for small puzzles using physical simulation methods. A human-in-the-loop approach was shared to allowing experts to interact with the puzzle-solving system and fix mistakes and provide helpful hints during the reconstruction process.
3) Robotic platform: fundamental requirements of the bi-manual platform have been developed and the soft end-effectors obtained. Proprioceptive sensing using a disturbance observer and motor current measurement was chosen for grasping force sensing. Various configurations of bi-manual systems were explored to maximize manipulability while ensuring kinematic feasibility and collision avoidance. Three end-effector devices based on the Pisa/IIT SoftHand technology were designed for manipulating different types of fresco fragments. Two anthropomorphic soft hand devices were created for medium/small and large fragments, while the third device had a gripper-like form with longer middle phalanges. Mechanical design improvements were made for the finger and palm parts of the third device, and soft pads were developed to enhance safety and grip when interacting with fragments. In terms of motion planning, a point-cloud completion approach was developed to generate plausible geometry for occluded parts of objects. A grasp sampling method was created to generate feasible hand poses for grasping objects, and a force-based controller was developed to improve grasping performance. A basic grasping pipeline for real-world scenarios was also developed. A full simulation integrating all platform elements was created in RViz-Gazebo environments. The sensing system, including the in-hand camera, is being integrated with the end-effectors. Work is underway to create a simulation-based dataset for bimanual grasp estimation. Additionally, a MoveIt configuration was created to control simulated arms using point control.
The RePAIR project is expected to have various impacts in different time frames. It aims to lay the groundwork for future advancements in computer science and robotics, contributing to the development of novel technologies and instruments for heritage research. By automating 3D scanning activities and accelerating the creation of cultural heritage digital repositories, RePAIR will have a significant impact on archaeology and other fields, salvaging fragmented material culture and fragile artifacts.
In addition to its technological impact, the project also focuses on the social implications of restoring historical artifacts. By re-enabling the enjoyment of lost cultural heritage items, RePAIR aims to revive collective memory and social cohesion. The project seeks to empower and reconstruct destroyed artifacts, bringing back a sense of cultural reference and belonging to communities. Through these efforts, RePAIR aims to address social conflicts caused by the loss of cultural heritage, aligning with UNESCO conventions and fostering cultural identity and social balance.
Furthermore, the project anticipates an important economic impact. The restored artwork will regain its economic value, as it becomes available to the public, experts, and connected industries. The exhibition of restored artifacts, combined with the storytelling around the cutting-edge restoration techniques used, will attract more visitors to exhibition sites and boost tourism. This, in turn, will benefit cultural institutions through increased ticket sales, enabling them to reinvest profits in conservation measures.
Moreover, RePAIR will have a positive impact on cultural tourism by attracting new visitors to restored masterpieces and exhibition locations. The project will streamline the work of conservators, reducing manual labor and allowing them to focus on analysis and conservation efforts rather than mere piece assembly. This will result in significant time savings for personnel involved in fragment fitting. Additionally, the project creates market opportunities through knowledge transfer, promoting the products and opening doors for additional market perspectives for the consortium and technology suppliers.
In addition to its technological impact, the project also focuses on the social implications of restoring historical artifacts. By re-enabling the enjoyment of lost cultural heritage items, RePAIR aims to revive collective memory and social cohesion. The project seeks to empower and reconstruct destroyed artifacts, bringing back a sense of cultural reference and belonging to communities. Through these efforts, RePAIR aims to address social conflicts caused by the loss of cultural heritage, aligning with UNESCO conventions and fostering cultural identity and social balance.
Furthermore, the project anticipates an important economic impact. The restored artwork will regain its economic value, as it becomes available to the public, experts, and connected industries. The exhibition of restored artifacts, combined with the storytelling around the cutting-edge restoration techniques used, will attract more visitors to exhibition sites and boost tourism. This, in turn, will benefit cultural institutions through increased ticket sales, enabling them to reinvest profits in conservation measures.
Moreover, RePAIR will have a positive impact on cultural tourism by attracting new visitors to restored masterpieces and exhibition locations. The project will streamline the work of conservators, reducing manual labor and allowing them to focus on analysis and conservation efforts rather than mere piece assembly. This will result in significant time savings for personnel involved in fragment fitting. Additionally, the project creates market opportunities through knowledge transfer, promoting the products and opening doors for additional market perspectives for the consortium and technology suppliers.