Periodic Reporting for period 1 - CALLISTO (Copernicus Artificial Intelligence Services and data fusion with other distributed data sources and processing at the edge to support DIAS and HPC infrastructures)
Okres sprawozdawczy: 2021-01-01 do 2021-12-31
Europe’s sustainable economic growth and societal wellbeing can benefit from the exploitation of Earth Observation (EO) data, which is however hindered by the difficulties in accessing, using and interpreting it. The generation of effective information relies on the fusion of heterogeneous sources of data.
CALLISTO's goal is to develop Artificial Intelligence (AI) solutions that, by combining data from heterogeneous sources, aim to help monitor human and environmental activity. To this purpose, EO data from the ONDA DIAS (www.onda-dias.eu) are combined with data from other distributed sources, including crowdsourced data, videos from Unmanned Aerial Vehicles and in situ measurements, through machine learning and data fusion technologies. The outcomes are semantically-enriched and served to humans in interactive interfaces, mobile and Mixed Reality applications, creating a novel immersive environment for the Copernicus market (see Figure 1).
Four pilot cases are being developed. As concerns the EU’s Common Agricultural Policy (CAP) monitoring, CALLISTO overcomes the limitations of the existing EO-based approaches by using ancillary crowdsourced information, such as street-level images, to augment the scarce validation and training datasets, aiming to minimize the need for On-The-Spot Checks. In the water sector, CALLISTO can improve the ability of water industries and authorities to control their regions, without the need for physical presence and inspection. This will benefit the citizens as drinking water consumers. In the journalism sector, CALLISTO serves as a research tool for journalists, and eventually the general public, boosting awareness and usability of imagery and other datasets and supporting journalists in investigating areas and events usually hard to access. Finally, the integration of CALLISTO’s land border change detection system in SatCen´s operational workflows is expected to support the monitoring of strategic areas such as the EU pre-frontiers, thus enabling EU governments to remain aware of the situation evolution for critical infrastructures.
The specific objectives of CALLISTO are of four different categories (see Figure 2).
CALLISTO's goal is to develop Artificial Intelligence (AI) solutions that, by combining data from heterogeneous sources, aim to help monitor human and environmental activity. To this purpose, EO data from the ONDA DIAS (www.onda-dias.eu) are combined with data from other distributed sources, including crowdsourced data, videos from Unmanned Aerial Vehicles and in situ measurements, through machine learning and data fusion technologies. The outcomes are semantically-enriched and served to humans in interactive interfaces, mobile and Mixed Reality applications, creating a novel immersive environment for the Copernicus market (see Figure 1).
Four pilot cases are being developed. As concerns the EU’s Common Agricultural Policy (CAP) monitoring, CALLISTO overcomes the limitations of the existing EO-based approaches by using ancillary crowdsourced information, such as street-level images, to augment the scarce validation and training datasets, aiming to minimize the need for On-The-Spot Checks. In the water sector, CALLISTO can improve the ability of water industries and authorities to control their regions, without the need for physical presence and inspection. This will benefit the citizens as drinking water consumers. In the journalism sector, CALLISTO serves as a research tool for journalists, and eventually the general public, boosting awareness and usability of imagery and other datasets and supporting journalists in investigating areas and events usually hard to access. Finally, the integration of CALLISTO’s land border change detection system in SatCen´s operational workflows is expected to support the monitoring of strategic areas such as the EU pre-frontiers, thus enabling EU governments to remain aware of the situation evolution for critical infrastructures.
The specific objectives of CALLISTO are of four different categories (see Figure 2).
The project has achieved the milestones and deliverables foreseen in the reporting period, with no relevant deviations. Four pilot use cases have been detailed and the corresponding user requirements elicited through dedicated user workshops.
To extend the current ONDA DIAS data model, data from in situ sensors, social media and UAVs are being collected together with optical satellite data. A hyperspectral in situ sensor has been set up for monitoring surface water quality, while Sentinel-2 and Landsat-8 optical data have been processed. A social media crawler for collecting tweets about borders and air/water quality has been developed and a sample dataset has been ingested in the ONDA DIAS catalogue. All software and hardware requirements for UAV have been set.
AI architectures have been developed to address requirements in the monitoring of the land, air and water surfaces. Achievements include models for paddy rice classification in South Korea and for grassland classification in the Netherlands, no-reference image quality assessment for street-level images, and a prototype for air quality forecasting. An evaluation of existing state-of-the-art deep learning methods was performed for change detection, with the best performing model being evaluated on ONDA DIAS.
A first 3D model reconstruction has been created, as well as a multimodal event detection methodology for social media data and a UAV coverage path planner.A selected module was optimized on HPC and a multimodal hashing retrieval method has been served as a service through a user interface for searching similar UAV images.
Concepts and relationships of the CALLISTO ontology have been defined, in response to the PUCs' domain specific representation. A draft version of the ontology is uploaded to VOCOREG platform (https://www.vocoreg.com) for visualization and querying. Models for extracting locations from social media data (i.e. tweets text) are now available for five languages.
The technical requirements applying to the hardware, the software, and the data structures were specified and a platform development roadmap was delivered. The definition of the security framework has started and a first version of the Mixed Reality User Interface (UI) has been finalized. Wireframes and user stories for the mobile app for Galileo-enabled devices were designed.
Numerous dissemination activities were carried out to engage stakeholders and raise awareness about the project. As Ethics management is concerned, the specific recruiting and consent procedures required for the rightful and ethical participation of humans, were defined.
To extend the current ONDA DIAS data model, data from in situ sensors, social media and UAVs are being collected together with optical satellite data. A hyperspectral in situ sensor has been set up for monitoring surface water quality, while Sentinel-2 and Landsat-8 optical data have been processed. A social media crawler for collecting tweets about borders and air/water quality has been developed and a sample dataset has been ingested in the ONDA DIAS catalogue. All software and hardware requirements for UAV have been set.
AI architectures have been developed to address requirements in the monitoring of the land, air and water surfaces. Achievements include models for paddy rice classification in South Korea and for grassland classification in the Netherlands, no-reference image quality assessment for street-level images, and a prototype for air quality forecasting. An evaluation of existing state-of-the-art deep learning methods was performed for change detection, with the best performing model being evaluated on ONDA DIAS.
A first 3D model reconstruction has been created, as well as a multimodal event detection methodology for social media data and a UAV coverage path planner.A selected module was optimized on HPC and a multimodal hashing retrieval method has been served as a service through a user interface for searching similar UAV images.
Concepts and relationships of the CALLISTO ontology have been defined, in response to the PUCs' domain specific representation. A draft version of the ontology is uploaded to VOCOREG platform (https://www.vocoreg.com) for visualization and querying. Models for extracting locations from social media data (i.e. tweets text) are now available for five languages.
The technical requirements applying to the hardware, the software, and the data structures were specified and a platform development roadmap was delivered. The definition of the security framework has started and a first version of the Mixed Reality User Interface (UI) has been finalized. Wireframes and user stories for the mobile app for Galileo-enabled devices were designed.
Numerous dissemination activities were carried out to engage stakeholders and raise awareness about the project. As Ethics management is concerned, the specific recruiting and consent procedures required for the rightful and ethical participation of humans, were defined.
The CALLISTO-generated annotated datasets, progressively becoming available through the CALLISTO repository, are countering the shortage of labelled data on the street-level, especially for the agricultural domain. An efficient CAP monitoring tool, difficult to be achieved only using Sentinel data, will be developed by leveraging Sentinel data, VHR satellite data, UAV data and street-level images. This is expected to bring Paying Agencies a step closer to the exhaustive CAP monitoring.
A 3D reconstruction algorithm will allow multiple visualizations of the same data. The generated 3D models can be used in Mixed Reality applications, enabling virtual presence in places where it is not feasible or safe to be.
The event detection module for non-EO data, the coverage path planning solution and the multimodal retrieval method have or are expected to outperform the literature methodologies; this will unlock a wide range of applications, including search and rescue, precision agriculture and crisis management.
A CALLISTO ontology and knowledge graph is being developed for semantic representation of the project’s EO and PUCs domains, enhancing the existing state-of-the-art. More datasets will be represented in the ontology, which will be available on VOCOREG platform for querying, documentation and visualization. This will enable users to access integrated data e.g. for journalistic purposes or hazards prevention. A service will also be developed for detecting locations in an input text, in many different languages.
CALLISTO has started demonstrating the extension of the ONDA data model with unstructured data made available in the project. This will enhance the capacity of analysis of large volumes of EO and non-EO data, without the need to download them.
The CALLISTO platform architecture includes a workflow engine supporting artificial intelligence frameworks and generic infrastructure deployments. It will result in a secure platform with an easily extensible catalogue of added-value geospatial data and analytics, offering a reusable, scalable, interoperable and secure Cloud environment to run and distribute services. Time-to-value for the service developers will be reduced with the acceleration of the services industrialization and distribution.
A 3D reconstruction algorithm will allow multiple visualizations of the same data. The generated 3D models can be used in Mixed Reality applications, enabling virtual presence in places where it is not feasible or safe to be.
The event detection module for non-EO data, the coverage path planning solution and the multimodal retrieval method have or are expected to outperform the literature methodologies; this will unlock a wide range of applications, including search and rescue, precision agriculture and crisis management.
A CALLISTO ontology and knowledge graph is being developed for semantic representation of the project’s EO and PUCs domains, enhancing the existing state-of-the-art. More datasets will be represented in the ontology, which will be available on VOCOREG platform for querying, documentation and visualization. This will enable users to access integrated data e.g. for journalistic purposes or hazards prevention. A service will also be developed for detecting locations in an input text, in many different languages.
CALLISTO has started demonstrating the extension of the ONDA data model with unstructured data made available in the project. This will enhance the capacity of analysis of large volumes of EO and non-EO data, without the need to download them.
The CALLISTO platform architecture includes a workflow engine supporting artificial intelligence frameworks and generic infrastructure deployments. It will result in a secure platform with an easily extensible catalogue of added-value geospatial data and analytics, offering a reusable, scalable, interoperable and secure Cloud environment to run and distribute services. Time-to-value for the service developers will be reduced with the acceleration of the services industrialization and distribution.