Periodic Reporting for period 2 - CALLISTO (Copernicus Artificial Intelligence Services and data fusion with other distributed data sources and processing at the edge to support DIAS and HPC infrastructures)
Reporting period: 2022-01-01 to 2023-12-31
Europe’s sustainable economic growth and societal wellbeing can benefit from the exploitation of Earth Observation (EO) data, which is hindered by the difficulties in accessing, using and interpreting it. The generation of effective information relies on the fusion of heterogeneous sources of data.
CALLISTO's goal is to develop Artificial Intelligence (AI) solutions that, by combining data from heterogeneous sources, aim to help monitor human and environmental activity. To this purpose, EO data from the ONDA DIAS (www.onda-dias.eu) are combined with data from other distributed sources, including crowdsourced data, videos from Unmanned Aerial Vehicles (UAV) and in situ measurements, through machine learning and data fusion technologies. The outcomes are semantically-enriched and served to humans in interactive interfaces, mobile and Mixed Reality applications, creating a novel immersive environment for the Copernicus market (see Figure 1).
Four pilot use cases (PUCs) were developed. As concerns the EU’s Common Agricultural Policy (CAP) monitoring, CALLISTO overcomes the limitations of the existing EO-based approaches by using ancillary crowdsourced information to augment the scarce validation and training datasets, aiming to minimize the need for On-The-Spot Checks. In the water sector, CALLISTO can improve the ability of water industries and authorities to monitor their superficial water sources, reducing the need for physical presence and inspection. This will benefit the citizens as drinking water consumers. In the journalism sector, CALLISTO serves as a research tool for journalists, and eventually the general public, boosting awareness and usability of imagery and other datasets and supporting journalists in investigating areas and events usually hard to access. Finally, the integration of CALLISTO’s land border change detection system in SATCEN´s operational workflows is expected to support the monitoring of strategic areas such as the EU pre-frontiers, thus enabling EU governments to remain aware of the situation evolution for critical infrastructures.
The specific objectives of CALLISTO are of four different categories (see Figure 2).
CALLISTO's goal is to develop Artificial Intelligence (AI) solutions that, by combining data from heterogeneous sources, aim to help monitor human and environmental activity. To this purpose, EO data from the ONDA DIAS (www.onda-dias.eu) are combined with data from other distributed sources, including crowdsourced data, videos from Unmanned Aerial Vehicles (UAV) and in situ measurements, through machine learning and data fusion technologies. The outcomes are semantically-enriched and served to humans in interactive interfaces, mobile and Mixed Reality applications, creating a novel immersive environment for the Copernicus market (see Figure 1).
Four pilot use cases (PUCs) were developed. As concerns the EU’s Common Agricultural Policy (CAP) monitoring, CALLISTO overcomes the limitations of the existing EO-based approaches by using ancillary crowdsourced information to augment the scarce validation and training datasets, aiming to minimize the need for On-The-Spot Checks. In the water sector, CALLISTO can improve the ability of water industries and authorities to monitor their superficial water sources, reducing the need for physical presence and inspection. This will benefit the citizens as drinking water consumers. In the journalism sector, CALLISTO serves as a research tool for journalists, and eventually the general public, boosting awareness and usability of imagery and other datasets and supporting journalists in investigating areas and events usually hard to access. Finally, the integration of CALLISTO’s land border change detection system in SATCEN´s operational workflows is expected to support the monitoring of strategic areas such as the EU pre-frontiers, thus enabling EU governments to remain aware of the situation evolution for critical infrastructures.
The specific objectives of CALLISTO are of four different categories (see Figure 2).
Management: To oversee the project in accordance with the timeline and budget, internal reporting procedures were instituted. Additionally, related activities, such as the self-assessment and data management plans, were coordinated, involving the scientific and technical management to ensure effective execution.
Requirements: Use case scenarios and requirements for all four pilots were specified, refined through dedicated workshops involving end users, and reported in the appropriate deliverables. These were established with careful consideration of the software and hardware technologies integral to CALLISTO.
EO data and other distributed data sources: EO data along with various other distributed data sources are incorporated in the ONDA DIAS platform. This includes EO data derived from Copernicus services, as well as crowdsourced inputs like social media data, footage from UAVs, and direct observations such as hyperspectral reflectance data and air quality measurements.
Machine Learning: Various deep learning models like UNets and ResNets have been trained on heterogeneous, space-to-ground data sources, and used for land-based monitoring including border and agricultural surveillance. Long Short-Term Memory networks have been utilised for air quality forecasting, while Multilayer Perceptrons have been leveraged to transform Sentinel 2 multispectral imagery into hyperspectral data for water quality estimation.
In Data Fusion, the developed 3D visualisation app has enhanced the interpretation of various data sources. Complementing this, an integrated event detection module analyses geo-located social media data pinpointing event locations. MuseHash has been employed to improve the indexing of multimodal data sources, and its performance further optimised. Additionally, the UAV path planning module, also optimised, enhances data acquisition. Lastly, the Multimodal Search Engine has been developed, incorporating both the UAV data indexing and the path planning, establishing it as an all-encompassing tool for UAV missions.
Semantic Technologies: A dedicated ontology for CALLISTO has been created to represent diverse datasets and is publicly available on the VOCOREG platform. The Named Entity recognition service has been enriched with additional languages. A series of SPARQL and GeoSPARQL queries were constructed to evaluate the knowledge graph, and a user-friendly front-end has been implemented for efficient visualization.
CALLISTO Platform: Hosted on ONDA DIAS, has evolved through stages. The initial prototype catered to end-users with data visualization features. This evolved into a second prototype, enhancing tools for content creators. The final prototype focused on new integrated tools dedicated to the administrator users to maintain and operate the platform. Further, a mobile application for Galileo-enabled devices and Mixed Reality User Interface have been developed.
Pilots: Four PUCs have been identified and evaluated. Users’ feedback has been actively collected to ensure prototypes meet end-users' needs and preferences.
Dissemination/Exploitation: CALLISTO's website and social media accounts share updates about the project. Newsletters, pilot brochures, and project videos are readily available on the website. CALLISTO has organized a Joint Hackathon and the Horizon Booster initiative has recognized three of the project's EERs.
Ethics: All ethical and legal related issues such as data protection, privacy, health, safety procedures and general research ethics have been properly and timely addressed, ensuring that the project outcomes abide with the highest standards of research integrity, and especially Horizon Europe Ethics principles.
Requirements: Use case scenarios and requirements for all four pilots were specified, refined through dedicated workshops involving end users, and reported in the appropriate deliverables. These were established with careful consideration of the software and hardware technologies integral to CALLISTO.
EO data and other distributed data sources: EO data along with various other distributed data sources are incorporated in the ONDA DIAS platform. This includes EO data derived from Copernicus services, as well as crowdsourced inputs like social media data, footage from UAVs, and direct observations such as hyperspectral reflectance data and air quality measurements.
Machine Learning: Various deep learning models like UNets and ResNets have been trained on heterogeneous, space-to-ground data sources, and used for land-based monitoring including border and agricultural surveillance. Long Short-Term Memory networks have been utilised for air quality forecasting, while Multilayer Perceptrons have been leveraged to transform Sentinel 2 multispectral imagery into hyperspectral data for water quality estimation.
In Data Fusion, the developed 3D visualisation app has enhanced the interpretation of various data sources. Complementing this, an integrated event detection module analyses geo-located social media data pinpointing event locations. MuseHash has been employed to improve the indexing of multimodal data sources, and its performance further optimised. Additionally, the UAV path planning module, also optimised, enhances data acquisition. Lastly, the Multimodal Search Engine has been developed, incorporating both the UAV data indexing and the path planning, establishing it as an all-encompassing tool for UAV missions.
Semantic Technologies: A dedicated ontology for CALLISTO has been created to represent diverse datasets and is publicly available on the VOCOREG platform. The Named Entity recognition service has been enriched with additional languages. A series of SPARQL and GeoSPARQL queries were constructed to evaluate the knowledge graph, and a user-friendly front-end has been implemented for efficient visualization.
CALLISTO Platform: Hosted on ONDA DIAS, has evolved through stages. The initial prototype catered to end-users with data visualization features. This evolved into a second prototype, enhancing tools for content creators. The final prototype focused on new integrated tools dedicated to the administrator users to maintain and operate the platform. Further, a mobile application for Galileo-enabled devices and Mixed Reality User Interface have been developed.
Pilots: Four PUCs have been identified and evaluated. Users’ feedback has been actively collected to ensure prototypes meet end-users' needs and preferences.
Dissemination/Exploitation: CALLISTO's website and social media accounts share updates about the project. Newsletters, pilot brochures, and project videos are readily available on the website. CALLISTO has organized a Joint Hackathon and the Horizon Booster initiative has recognized three of the project's EERs.
Ethics: All ethical and legal related issues such as data protection, privacy, health, safety procedures and general research ethics have been properly and timely addressed, ensuring that the project outcomes abide with the highest standards of research integrity, and especially Horizon Europe Ethics principles.
CALLISTO has progressed in curating unique annotated datasets, particularly for agriculture. These datasets, hosted on the CALLISTO repository, fill a critical gap in street-level labelled data. The scientific community and the Earth System Science Data Journal have recognized the repository for its user-friendly approach. All CALLISTO-generated datasets are evaluated on their FAIRness, based on the methodology developed by the project. The tools developed and integrated to the CALLISTO platform have enabled CAP monitoring using a mix of heterogeneous data sources, including Sentinel, Very High Resolution, UAV, and street-level images.
CALLISTO has developed a 3D reconstruction algorithm for comprehensive data visualisation. These 3D models are being incorporated into a mixed reality application, which aims to allow virtual access to hard-to-reach or hazardous areas and improve users’ engagement through augmented reality.
Additionally, CALLISTO's multimodal retrieval method outperforms seven existing methods in certain use cases.
The CALLISTO Platform offers centralized and secure access to the entire environment including third-party applications (partner solutions) and access to different added-value services and visualization and analytic tools. Furthermore, the platform has been designed with an open-source development and deployment environment independent of any Cloud or data provider. It contains reusable components that can be redeployed on any cloud/DIAS or HPC infrastructure, to meet other EU platforms & initiatives.
CALLISTO has developed a 3D reconstruction algorithm for comprehensive data visualisation. These 3D models are being incorporated into a mixed reality application, which aims to allow virtual access to hard-to-reach or hazardous areas and improve users’ engagement through augmented reality.
Additionally, CALLISTO's multimodal retrieval method outperforms seven existing methods in certain use cases.
The CALLISTO Platform offers centralized and secure access to the entire environment including third-party applications (partner solutions) and access to different added-value services and visualization and analytic tools. Furthermore, the platform has been designed with an open-source development and deployment environment independent of any Cloud or data provider. It contains reusable components that can be redeployed on any cloud/DIAS or HPC infrastructure, to meet other EU platforms & initiatives.