The RUTE project established a benchmark for using THz time-domain imaging with advanced computer vision and machine learning to uncover and enhance the readability of hidden text in closed books. Mockups simulating ancient manuscripts were created in collaboration with the chemistry lab of the Centre for Cultural Heritage Technology (CCHT) of the Italian Institute of Technology (host institute). The fellow completed a 13.5-hour "High Risk Safety Course at Ca' Foscari" for lab access certification.
Initially, three commercial inks were purchased, and ten inks, including iron-gall and copper-based, were synthesized using ancient recipes. To enrich the dataset, additional mockups were created with cellulose pallets and ancient-like paper, featuring text in synthesized iron-gall ink. Images of the mockups were acquired using a hyperspectral camera (HERA, Nireos) to verify if the created inks had unique spectral fingerprints in the near-infrared region. The fellow received hands-on training with the hyperspectral camera and visited the Correr Museum’s Archive in Venice to secure access to invaluable historical documents. As part of the training and dataset creation phase, the fellow conducted THz image acquisition with the Department of Molecular Sciences and Nanosystems at Ca' Foscari using a commercial THz-TDS system, TOPTICA TeraFlash Pro. Protocols for system settings, data naming, storage, and safety procedures were established. Challenges, such as paper curvature producing artifacts, were addressed by repeating scans with fixed paper and digitally correcting images with a method developed in collaboration with a colleague.
The fellow transferred knowledge on data curation and pre-processing to CCHT’s PhD students and junior postdocs. The primary outcome was a paper presented at the International Workshop on Fine Art Pattern Extraction and Recognition (FAPER) during the International Conference on Image Analysis and Processing (ICIAP, 2022). Parallelly, a novel method was designed for restoring THz time-domain images in reflection geometry, focusing on beam-shape effects removal and denoising, published in IEEE Transactions on Terahertz Science and Technology. A self-referenced method for correcting geometrical distortion due to sample surface curvature was also developed and published in the same journal.
During THz image acquisition, the challenge of optimizing step size without reducing resolution was addressed by developing a blind super-resolution approach, presented at the International Workshop on Mobile Terahertz Systems (IWMTS, 2023). Knowledge transfer led to a publication at the International Geoscience and Remote Sensing Symposium (IGARSS, 2024). Given the impracticality of training CNNs with many input images due to lengthy scanning times, a process for creating THz synthetic data was developed during a secondment at Instituto de Telecomunicações in Lisbon. This method, combining THz images' data structure with CNNs, was presented at the International Conference on Computer Vision Theory and Applications (VISAPP 2024), where the fellow also served as an oral session chair. This work revealed text from up to eight pages of a closed book and is being prepared for submission to IEEE Transactions on Image Processing.
Previously developed methods were tested on text-containing images using OCR algorithms tailored for handwritten text, with promising results being drafted for submission to the International Journal on Document Analysis and Recognition.
A strategic dissemination plan maximized RUTE’s impact. Scientific dissemination included presentations at ICIAP in Lecce, IWMTS in Bonn, and VISAPP in Rome. A dedicated website provided content for academic and general audiences. An international workshop titled “Illuminating the Past: Uncovering Hidden Layers in Cultural Heritage with Hyperspectral and THz Imaging” showcased project results and facilitated networking. Throughout the project, the fellow and the host institute maintained an active social media presence, sharing key results and updates. To explore further development, the fellow attended webinars on intellectual property, technology transfer, and gender aspects in research.
