Periodic Reporting for period 1 - ThinkingEarth (Copernicus Foundation Models for a Thinking Earth)
Okres sprawozdawczy: 2024-01-01 do 2025-06-30
At ThinkingEarth, we view the Earth as a complex unified and interconnected system. To harness the power of Artificial Intelligence (AI), we use cutting-edge techniques, including deep learning, causality, eXplainable AI, and physics-aware machine learning. We leverage the predictive abilities of Self-Supervised Learning and Graph Neural Networks to develop the first task-agnostic Copernicus Foundation Models and a graph representation model of the Earth.
We demonstrate the potential of these assets through small-scale downstream Spotlight Applications, as well as large-scale Use Cases (UCs) that integrate distributed industrial and user non-EO datasets. These use cases address ambitious problems with high socio-environmental impact and new business growth opportunities, such as accelerating Europe's clean energy transition and independence from volatile fossil fuels, understanding Earth's processes by modeling causal Earth system teleconnections, and assessing and modeling the impact of current and future climate emergency in biodiversity and food security.
We demonstrate the potential of these assets through small-scale downstream Spotlight Applications, as well as large-scale Use Cases (UCs) that integrate distributed industrial and user non-EO datasets. These use cases address ambitious problems with high socio-environmental impact and new business growth opportunities, such as accelerating Europe's clean energy transition and independence from volatile fossil fuels, understanding Earth's processes by modeling causal Earth system teleconnections, and assessing and modeling the impact of current and future climate emergency in biodiversity and food security.
Big data set the basis of developing EO foundation models. During the first phase of ThinkingEarth, we introduced the following datasets and benchmarks supporting our foundation models:
-SSL4EO-S12-ML, which is a global multi-label land-cover land-use classification dataset combining multispectral and Synthetic Aperture Radar (SAR) imagery with open land-cover products to create scene-level multi-label annotations;
-SSL4EO-S, which integrates ~15TB data from all Copernicus Sentinel missions and the Copernicus DEM GLO-30 product, offering a large-scale, multi-modal dataset to advance unified EO foundation models with a potential connection to weather and climate;
-Kuro Siwo, which is a global multi-temporal manually annotated SAR dataset for rapid flood mapping, consisting of 43 global flood events;
-FoMo-Bench, which is the first unified forest monitoring benchmark, spanning 15 datasets with diverse data modalities and tasks, with a total volume exceeding 10TB;
-GAIA, which is a large-scale vision-language dataset designed to bridge the gap between remote sensing imagery and natural language understanding.
Using innovative self-supervised and cross-modal learning techniques to improve performance and generalizability, we developed the first version of ThinkingEarth's Copernicus foundation models:
-SupCon and SoftCon leverage the SSL4EO-S12-ML dataset to guide self-supervised pretraining using multi-label land cover annotations;
-FoMo-Net is a pre-training paradigm for learning to process all of the most common modalities in the Remote Sensing domain with a single, sensor-agnostic foundation model. Its pre-training scheme contains rich multi-sensor information, both paired and unpaired, from most parts of the world;
-In our work on Remote Sensing Vision-Language models we focused on Contrastive Language-Image Pre-training (CLIP), an open-vocabulary foundation model, which achieves high accuracy across many image classification tasks and is often competitive with a fully supervised baseline without being explicitly trained. We introduce a novel approach to align RS imagery modalities with the visual and textual capabilities of CLIP, enhancing its zero-shot capability in EO.
Towards the creation of a graph representation of the Earth, we developed a robust framework with baseline implementations of leading models for medium-range weather forecasting, enabling optimized training workflows and efficient parallelization of data and model computations across multiple GPUs. We investigated a range of model architectures and implemented architectural enhancements to improve forecasting accuracy on ERA5 climate data. We applied causal analysis methods for capturing teleconnections. We used Direct Effect Analysis (DEA), a framework for constructing causal representations of environmental variables by isolating the direct effects of specific drivers while controlling for confounders. We demonstrated DEA's utility through a case study of ENSO’s direct effects on NDVI in Africa, highlighting its broader applicability to environmental and policy-related challenges.
In UC1 "Distributed solar energy production forecasting and demand management" we developed a Multi-Layer Cloud Motion Vector forecasting method and a Deep-Learning Emulator for radiative transfer modelling. We released the first prototypes in the form of interactive online applications for Lower Austria State and Metropolitan Athens Region.
In UC2a "Biodiversity monitoring in urban environment" we enhanced the spatial resolution of Sentinel-2 imagery using DL-based super-resolution. We developed a web-based decision support system prototype in the form of an online platform that visualizes Sentinel-2 super-resolved images and biodiversity indices, aiming to support urban planners with actionable insights on green/blue infrastructure.
In UC2b "Forest biomass monitoring" we developed a first prototype version of a cloud-based tool that enables scalable remote monitoring of remote forest carbon stocks, providing an accessible platform for stakeholders in the carbon credit industry to verify and track high-quality carbon credit at both local and regional levels.
In UC3 "Causal inference for food insecurity analysis" we compiled a multi-source dataset harmonizing climate, economic, and conflict data from the period 2025-2022 across 56 Somali districts. We estimated Average Treatment Effects using X-Learner to quantify the average impact of cash-based transfers on food insecurity.
Lastly, we prepared three spotlight applications within the first cycle of the project, targeting the convenient evaluation of the task-agnostic Copernicus foundation models using three different Sentinel modalities: flood mapping with Sentinel-1 SAR imagery, land use land cover classification with Sentinel-3 OLCI imagery, and air pollutants estimation with Sentinel-5P products about atmospheric constituents.
-SSL4EO-S12-ML, which is a global multi-label land-cover land-use classification dataset combining multispectral and Synthetic Aperture Radar (SAR) imagery with open land-cover products to create scene-level multi-label annotations;
-SSL4EO-S, which integrates ~15TB data from all Copernicus Sentinel missions and the Copernicus DEM GLO-30 product, offering a large-scale, multi-modal dataset to advance unified EO foundation models with a potential connection to weather and climate;
-Kuro Siwo, which is a global multi-temporal manually annotated SAR dataset for rapid flood mapping, consisting of 43 global flood events;
-FoMo-Bench, which is the first unified forest monitoring benchmark, spanning 15 datasets with diverse data modalities and tasks, with a total volume exceeding 10TB;
-GAIA, which is a large-scale vision-language dataset designed to bridge the gap between remote sensing imagery and natural language understanding.
Using innovative self-supervised and cross-modal learning techniques to improve performance and generalizability, we developed the first version of ThinkingEarth's Copernicus foundation models:
-SupCon and SoftCon leverage the SSL4EO-S12-ML dataset to guide self-supervised pretraining using multi-label land cover annotations;
-FoMo-Net is a pre-training paradigm for learning to process all of the most common modalities in the Remote Sensing domain with a single, sensor-agnostic foundation model. Its pre-training scheme contains rich multi-sensor information, both paired and unpaired, from most parts of the world;
-In our work on Remote Sensing Vision-Language models we focused on Contrastive Language-Image Pre-training (CLIP), an open-vocabulary foundation model, which achieves high accuracy across many image classification tasks and is often competitive with a fully supervised baseline without being explicitly trained. We introduce a novel approach to align RS imagery modalities with the visual and textual capabilities of CLIP, enhancing its zero-shot capability in EO.
Towards the creation of a graph representation of the Earth, we developed a robust framework with baseline implementations of leading models for medium-range weather forecasting, enabling optimized training workflows and efficient parallelization of data and model computations across multiple GPUs. We investigated a range of model architectures and implemented architectural enhancements to improve forecasting accuracy on ERA5 climate data. We applied causal analysis methods for capturing teleconnections. We used Direct Effect Analysis (DEA), a framework for constructing causal representations of environmental variables by isolating the direct effects of specific drivers while controlling for confounders. We demonstrated DEA's utility through a case study of ENSO’s direct effects on NDVI in Africa, highlighting its broader applicability to environmental and policy-related challenges.
In UC1 "Distributed solar energy production forecasting and demand management" we developed a Multi-Layer Cloud Motion Vector forecasting method and a Deep-Learning Emulator for radiative transfer modelling. We released the first prototypes in the form of interactive online applications for Lower Austria State and Metropolitan Athens Region.
In UC2a "Biodiversity monitoring in urban environment" we enhanced the spatial resolution of Sentinel-2 imagery using DL-based super-resolution. We developed a web-based decision support system prototype in the form of an online platform that visualizes Sentinel-2 super-resolved images and biodiversity indices, aiming to support urban planners with actionable insights on green/blue infrastructure.
In UC2b "Forest biomass monitoring" we developed a first prototype version of a cloud-based tool that enables scalable remote monitoring of remote forest carbon stocks, providing an accessible platform for stakeholders in the carbon credit industry to verify and track high-quality carbon credit at both local and regional levels.
In UC3 "Causal inference for food insecurity analysis" we compiled a multi-source dataset harmonizing climate, economic, and conflict data from the period 2025-2022 across 56 Somali districts. We estimated Average Treatment Effects using X-Learner to quantify the average impact of cash-based transfers on food insecurity.
Lastly, we prepared three spotlight applications within the first cycle of the project, targeting the convenient evaluation of the task-agnostic Copernicus foundation models using three different Sentinel modalities: flood mapping with Sentinel-1 SAR imagery, land use land cover classification with Sentinel-3 OLCI imagery, and air pollutants estimation with Sentinel-5P products about atmospheric constituents.
Results of the first reporting period include:
-Datasets SSL4EO-S12-ML, SSL4EO-S, Kuro Siwo, MUDDAT, GAIA, and benchmark FoMo-Bench
-Foundation Models SupCon, SoftCon, FoMo-Net, and a Vision-Language Model
-The first version of the Earth as a Graph interactive prototype
-An interactive online solar energy application for the Lower Austria State and the Metropolitan Athens Region
-A DL Emulator for Radiative Transfer Model that can support a variety of atmospheric physics studies and the renewables sector
-A web-based Decision Support System prototype for urban biodiversity monitoring
-A cloud-based biomass monitoring tool that could support carbon projects by providing accurate high resolution AGB estimates for monitoring forest carbon sinks
-X-Learner algorithm to estimate Average Treatment Effects (ATEs) of cash transfers
-Direct Effect Analysis to uncover the causal influence of ENSO on NDVI
-Conditional Average Treatment Effects (CATEs) of price shocks, stratified by climate regime
-Causal graph and treatment effect maps
Part of the work is already published to the scientific community for further research and exploitation. The publications of the first reporting period include 6 articles in journals, 4 publications in conference proceedings, and 6 arXiv preprints.
-Datasets SSL4EO-S12-ML, SSL4EO-S, Kuro Siwo, MUDDAT, GAIA, and benchmark FoMo-Bench
-Foundation Models SupCon, SoftCon, FoMo-Net, and a Vision-Language Model
-The first version of the Earth as a Graph interactive prototype
-An interactive online solar energy application for the Lower Austria State and the Metropolitan Athens Region
-A DL Emulator for Radiative Transfer Model that can support a variety of atmospheric physics studies and the renewables sector
-A web-based Decision Support System prototype for urban biodiversity monitoring
-A cloud-based biomass monitoring tool that could support carbon projects by providing accurate high resolution AGB estimates for monitoring forest carbon sinks
-X-Learner algorithm to estimate Average Treatment Effects (ATEs) of cash transfers
-Direct Effect Analysis to uncover the causal influence of ENSO on NDVI
-Conditional Average Treatment Effects (CATEs) of price shocks, stratified by climate regime
-Causal graph and treatment effect maps
Part of the work is already published to the scientific community for further research and exploitation. The publications of the first reporting period include 6 articles in journals, 4 publications in conference proceedings, and 6 arXiv preprints.