Periodic Reporting for period 1 - MLCULT (Machine Learning for Structural Health Monitoring of Cultural Heritage)
Reporting period: 2022-07-01 to 2024-11-30
Cultural heritage (CH) structures are generally inspected by manual visual inspections done simply by using the naked eye. In simple words, we see the damage in the building with our eyes and then decide what to do with it. With progress in computer science, researchers/inspection professionals are augmenting their capacity to inspect CH structures via analysis of digital images taken by cameras, drones, etc., using deep learning (DL) techniques. These DL techniques can even identify defects missed by human eyes and can analyse large amounts of building damage pictures once the model is trained, without much human intervention. The proposed research project aimed to develop DL-models for surface damage diagnosis by using machine learning (ML) techniques, particularly DL techniques, for reducing uncertainties in structural health diagnosis, which would enable more efficient structural interventions and repair. The objective of the MLCULT project was to use computer vision (CV) techniques to assess the surface damage condition of CH constructions for reducing condition assessment costs and assessing building safety with minimum building intrusion. CV-based models are available in concrete buildings, but in CH structures their applications are scarce. Additionally, the objective of the project was also to provide open access databases of damaged CH components to the public, as CH damage typologies are more complex and varied than modern buildings. The MLCULT project demonstrated how DL models can be trained efficiently by using proper CH damage data and then deployed in situ to automate damage identification of CH structure. Finally, the end objective was a pilot demonstration of the AI-inspect system in real case studies, basically data from the real field. The models developed provide a helping hand to CH inspectors, who can use this tool at their own advantage. For example, inspectors do not need to go to risky locations that may compromise their safety and may use drones to take pictures, which can be later fed into the DL-model to identify the damage typologies. Then, it is up to them to decide the intervention solution. The AI does not take the authority to decide the intervention and only says "these are the damages I have found" and it is for the engineer to decide the next step. Recommendations for heritage professionals using computer-vision (CV)-based models and literature reviews performed before the proposal were extended to damage assessment techniques using image processing applied to CH. The project contributes to the way we will perform visual inspections and makes our CH structures safer as CV-techniques will not miss any defects that can jeopardise the safety of structure in the long run.
We were able to build a deep learning (DL) model that classifies damage in cultural heritage (CH) structures into various classes, and the hyperparameters were tuned accordingly to work with a good accuracy. Two such DL-models were successfully developed in the MLCULT project. One was for tile-defect and a second one for crack identification in diverse backgrounds. Binary classifier (that tells if there is damage or no damage in the picture) achieved an accuracy > 99%, while the multi classifier (including damage classes as cracks, craters, glaze detachment, and tile lacunae, as well as images with no defects) in tile-related damages achieved an accuracy > 72% and lastly, for the crack dataset, average precision was in the range of 70%-95% depending on the type of material in the background.
The work performed included data collection from scratch for tile-damages and, for the crack database, the data collected by a recently joined PhD student was used to train and test the models. The Kaggle dataset (https://www.kaggle.com/datasets/nargeskarimii/various-materials-from-historic-buildings ) and Mendeley database (https://data.mendeley.com/datasets/3t3dk43bv9/1(opens in new window)) contains image data collected by the authors, which was used for training deep learning models. The main achievements also included summarising research and providing future directions in artificial intelligence assisted research area. With the help of review paper (https://doi.org/10.1016/j.culher.2024.01.005(opens in new window)) we addressed the gaps in AI-powered inspection systems, and what improvements needs to be made for the future of these technology. In concrete technology, there are many review papers in AI-powered inspection for buildings, but we have summarised the researchers in the cultural heritage area for the first time. We also pointed out that a great potential exists that can be addressed by researchers in this field of AI-assisted visual inspections by combining drones and Internet of Things (IoT) technologies, thus providing future research directions.
The work performed included data collection from scratch for tile-damages and, for the crack database, the data collected by a recently joined PhD student was used to train and test the models. The Kaggle dataset (https://www.kaggle.com/datasets/nargeskarimii/various-materials-from-historic-buildings ) and Mendeley database (https://data.mendeley.com/datasets/3t3dk43bv9/1(opens in new window)) contains image data collected by the authors, which was used for training deep learning models. The main achievements also included summarising research and providing future directions in artificial intelligence assisted research area. With the help of review paper (https://doi.org/10.1016/j.culher.2024.01.005(opens in new window)) we addressed the gaps in AI-powered inspection systems, and what improvements needs to be made for the future of these technology. In concrete technology, there are many review papers in AI-powered inspection for buildings, but we have summarised the researchers in the cultural heritage area for the first time. We also pointed out that a great potential exists that can be addressed by researchers in this field of AI-assisted visual inspections by combining drones and Internet of Things (IoT) technologies, thus providing future research directions.
Results and its potential impacts:
- The MLCULT project resulted in the development of two AI-assisted visual inspection systems, that can be deployed to inspect cultural heritage (CH) structures visually. The MLCULT project aimed to augment the use of the traditional outdated visual inspection technique in CH protection with the use of an improved deep learning (DL)-based image processing techniques. For the first time, an AI-powered visual inspection system was made that focussed on tiles with patterns going beyond state-of-the-art, as all previous studies only deal with plain tiles. The second AI-powered visual inspection system focussed on crack identification with diverse backgrounds. Both demonstrations are practical applications and can be directly used at the site. We also shared the image data of tiles with defects and crack in diverse materials, so other researchers and industry professionals can use it as a starting point to develop their AI-based inspection systems.
- These visual inspection systems can be enhanced in the future including more training databases, thus contributing to various other damage typologies. The framework for such kinds of systems will remain the same, which we have highlighted in the state-of-the-art review paper. This was the first state-of-the-art review paper in cultural heritage field as all papers focus only on modern buildings. We also pointed out that a great potential exists that can be addressed by researchers in this field of AI-assisted visual inspections by combining drones and internet of things (IoT) technologies, thus providing future research directions.
- Novel use of Generative artificial intelligence (AI) was made. We made a tool inside CHATGPT4.0 to give feedback of the damaged images without requiring any complex process, which is the first tool of its kind that may be used by practitioners. The developed GPT application is trained using tests with examples from UNI 11182 : 2006 (Cultural Heritage - Natural and Artificial Stone - Description of the Alteration - Terminology and Definition). It helps analyse and describe building material deterioration in photos using the ICOMOS, International Council on Monuments and Sites, illustrated glossary of stone deterioration patterns.
Further research needs to be done on getting instant damage notifications while inspecting structures, instead of transferring images on computers and then analysing them. Ideally, the images in CH structures must be taken by IoT-based sensor camera, transferred to cloud immediately, where it is processed via a deep-learning (DL) model, that is pre-trained to identify defects in the images and after processing, we see the defects in images instantly, while doing inspection in field itself. This is what needs to be done for future research, demonstrating it in-situ and then commercialising the AI-inspection system to be used by industry (now limited to academic research). The use of AI and IoT together will give the opportunity that traditional manual inspection solutions lack and can bridge the missing gaps to further apply AI in the CH inspection process.
- The MLCULT project resulted in the development of two AI-assisted visual inspection systems, that can be deployed to inspect cultural heritage (CH) structures visually. The MLCULT project aimed to augment the use of the traditional outdated visual inspection technique in CH protection with the use of an improved deep learning (DL)-based image processing techniques. For the first time, an AI-powered visual inspection system was made that focussed on tiles with patterns going beyond state-of-the-art, as all previous studies only deal with plain tiles. The second AI-powered visual inspection system focussed on crack identification with diverse backgrounds. Both demonstrations are practical applications and can be directly used at the site. We also shared the image data of tiles with defects and crack in diverse materials, so other researchers and industry professionals can use it as a starting point to develop their AI-based inspection systems.
- These visual inspection systems can be enhanced in the future including more training databases, thus contributing to various other damage typologies. The framework for such kinds of systems will remain the same, which we have highlighted in the state-of-the-art review paper. This was the first state-of-the-art review paper in cultural heritage field as all papers focus only on modern buildings. We also pointed out that a great potential exists that can be addressed by researchers in this field of AI-assisted visual inspections by combining drones and internet of things (IoT) technologies, thus providing future research directions.
- Novel use of Generative artificial intelligence (AI) was made. We made a tool inside CHATGPT4.0 to give feedback of the damaged images without requiring any complex process, which is the first tool of its kind that may be used by practitioners. The developed GPT application is trained using tests with examples from UNI 11182 : 2006 (Cultural Heritage - Natural and Artificial Stone - Description of the Alteration - Terminology and Definition). It helps analyse and describe building material deterioration in photos using the ICOMOS, International Council on Monuments and Sites, illustrated glossary of stone deterioration patterns.
Further research needs to be done on getting instant damage notifications while inspecting structures, instead of transferring images on computers and then analysing them. Ideally, the images in CH structures must be taken by IoT-based sensor camera, transferred to cloud immediately, where it is processed via a deep-learning (DL) model, that is pre-trained to identify defects in the images and after processing, we see the defects in images instantly, while doing inspection in field itself. This is what needs to be done for future research, demonstrating it in-situ and then commercialising the AI-inspection system to be used by industry (now limited to academic research). The use of AI and IoT together will give the opportunity that traditional manual inspection solutions lack and can bridge the missing gaps to further apply AI in the CH inspection process.