Periodic Reporting for period 1 - Edge SpAIce (Unleashing AI potential to foster space accessibility and novel Earth Observation services creation.)
Reporting period: 2023-12-01 to 2025-05-31
Satellites are crucial for Earth observation, attracting significant investment from both the private and public sector and potentially reshaping global economics. However, current data management infrastructure cannot sufficiently maximise the value of the increasing amount of data captured. To address this, the EU-funded Edge SpAIce project plans to develop an efficient approach to deploy deep neural networks (DNNs) at the edge for more effective data management, targeting continuous data flow between capture and processing. Despite challenges such as high computational power requirements and complex DNN architectures, Edge SpAIce will optimise AI execution, enabling its compatibility with various on-board satellite hardware. The ultimate goal is to demonstrate the potential of edge-AI technology by deploying a DNN for remote monitoring of marine plastic litter on a satellite.
The objectives of the project :
- Creation of an accurate DNN for marine plastic litter detection from satellite multispectral data, achieving F1 score 80%.
- Onboard DNN processing throughput above 180k pix/Watt/sec.
- Make 10-100M parameter AI model deployment onboard satellites feasible and efficient.
- Attrition rate on EO cases <2%, when reducing DNN to 1M parameters.
- Development of AI middleware for SoC-FPGAs, applicable for European AI accelerators.
- Evaluation of edge-AI solution on an actual satellite BALKAN-1.
- Evaluation of edge-AI solution on ground for European SoC-FPGA such as NanoXplore NG-Ultra.
The objectives of the project :
- Creation of an accurate DNN for marine plastic litter detection from satellite multispectral data, achieving F1 score 80%.
- Onboard DNN processing throughput above 180k pix/Watt/sec.
- Make 10-100M parameter AI model deployment onboard satellites feasible and efficient.
- Attrition rate on EO cases <2%, when reducing DNN to 1M parameters.
- Development of AI middleware for SoC-FPGAs, applicable for European AI accelerators.
- Evaluation of edge-AI solution on an actual satellite BALKAN-1.
- Evaluation of edge-AI solution on ground for European SoC-FPGA such as NanoXplore NG-Ultra.
In order to create a highly accurate DDN for marine litter detection NTUA firstly extended the benchmark dataset MARIDA with new observations from multispectral satellite data forming the new globally distributed MADOS dataset. Additionally, the MADOS dataset which includes ACOLITE Rayleigh reflectance values was reverted to Level-1C data in order to obtain the raw TOA reflectance values and enable model adaptation. The model was also trained by retaining only the Sentinel-2 bands that are identical to those of the upcoming Balkan-1 satellite.
To finally provide cloud and cloud shadow masking software modules, NTUA conducted a preliminary comprehensive evaluation of cloud masking algorithms on an extended version of MARIDA dataset with cloud annotations. This study revealed that there is a clear need for further refinement of these well-established algorithms when applied to marine environments. For this reason, NTUA proceeded to the extension of MADOS with the new cloud and cloud shadow classes in order to retrain the model with these classes.
The final candidate DNN has been elaborated and delivered by the NTUA
Agenium Space has focused on the following key elements for the first 18 months of the project :
1) The interoperability of our distillation software with the middleware developed by CERN has been achieved.
2) The early test and integrations of datasets and DNNs. This includes defining a roadmap for improving and defining the benchmarks with a maximum of automated processes. This also includes some complex reworking for the very dataset used in this project
We have actually performed improvements on the ODiToo tools : new quantization aware capabilities, new merit function, distillation on intermediates features have been added.
Our greatest achievement in the period was the enabling of the interoperability with Hls4ML. AGENIUM Space and CERN have agreed to use QONNX exchange format for the simplified model to be transferred from ODiToo to HLS4ML.
Finally we have also worked on the first steps to simplify the marine litter DNN. AGENIUM Space have prepared MADOS database to be supported in ODiToo. However, due to a very sparse annotation, new functionalities were needed to ensure a proper training and distillation with this database to handle the “no data” regions. In addition, the current NTUA best model on MADOS database has been upgraded and implemented in ODiToo and is currently being simplified.
CERN has successfully deploy DNN of up to 100k parameters using hls4ML.
EnduroSat has developed a full satellite engineering model, fully representative of the SW environment of the edge-AI board and the camera. This model is available remotely on demand when needed by Agenium.
Agenium is currently using an edge-AI fully representative of the SW environment implemented on Balkan-1 satellite. The development environment has been agreed between EnduroSat and Agenium in the Interface Control Document of the DNN.
The satellite Balkan-1 has been launched in January 2025. The actual plastic litter images of the satellite Balkan-01 for DNN training will be available after the satellite commissioning, in Q4 2025. These images will be used by Agenium in order to train the distilled DNN. Once the trained DNN will be available, the processing accuracy tests will be performed in early 2026 by EnduroSat satellite engineering model, with the remote support of Agenium.
To finally provide cloud and cloud shadow masking software modules, NTUA conducted a preliminary comprehensive evaluation of cloud masking algorithms on an extended version of MARIDA dataset with cloud annotations. This study revealed that there is a clear need for further refinement of these well-established algorithms when applied to marine environments. For this reason, NTUA proceeded to the extension of MADOS with the new cloud and cloud shadow classes in order to retrain the model with these classes.
The final candidate DNN has been elaborated and delivered by the NTUA
Agenium Space has focused on the following key elements for the first 18 months of the project :
1) The interoperability of our distillation software with the middleware developed by CERN has been achieved.
2) The early test and integrations of datasets and DNNs. This includes defining a roadmap for improving and defining the benchmarks with a maximum of automated processes. This also includes some complex reworking for the very dataset used in this project
We have actually performed improvements on the ODiToo tools : new quantization aware capabilities, new merit function, distillation on intermediates features have been added.
Our greatest achievement in the period was the enabling of the interoperability with Hls4ML. AGENIUM Space and CERN have agreed to use QONNX exchange format for the simplified model to be transferred from ODiToo to HLS4ML.
Finally we have also worked on the first steps to simplify the marine litter DNN. AGENIUM Space have prepared MADOS database to be supported in ODiToo. However, due to a very sparse annotation, new functionalities were needed to ensure a proper training and distillation with this database to handle the “no data” regions. In addition, the current NTUA best model on MADOS database has been upgraded and implemented in ODiToo and is currently being simplified.
CERN has successfully deploy DNN of up to 100k parameters using hls4ML.
EnduroSat has developed a full satellite engineering model, fully representative of the SW environment of the edge-AI board and the camera. This model is available remotely on demand when needed by Agenium.
Agenium is currently using an edge-AI fully representative of the SW environment implemented on Balkan-1 satellite. The development environment has been agreed between EnduroSat and Agenium in the Interface Control Document of the DNN.
The satellite Balkan-1 has been launched in January 2025. The actual plastic litter images of the satellite Balkan-01 for DNN training will be available after the satellite commissioning, in Q4 2025. These images will be used by Agenium in order to train the distilled DNN. Once the trained DNN will be available, the processing accuracy tests will be performed in early 2026 by EnduroSat satellite engineering model, with the remote support of Agenium.
NTUA managed to produce high-accuracy DNN with F1=93% for MSLD with 10 classes.
AGS has improved its Distillation & Quantisation software capabilities to deal with MLSD use case.
CERN improved HLS4ML, handling100k parameters DNN on FPGA
BALKAN-1 launched, going through calibration/validation phase
Edge-SpAIce consortium is confident that AI can deliver a massive help in scanning wide ocean areas and detecting plastics onboard satellites. This is the 1st of its kind application by analysing multi-spectral data to detect marine litter close to real time onboard BALKAN-1 satellite. Abbreviated as “SMLD app” (Space-based Maritime Litter Detection edge-AI application).
AGS has improved its Distillation & Quantisation software capabilities to deal with MLSD use case.
CERN improved HLS4ML, handling100k parameters DNN on FPGA
BALKAN-1 launched, going through calibration/validation phase
Edge-SpAIce consortium is confident that AI can deliver a massive help in scanning wide ocean areas and detecting plastics onboard satellites. This is the 1st of its kind application by analysing multi-spectral data to detect marine litter close to real time onboard BALKAN-1 satellite. Abbreviated as “SMLD app” (Space-based Maritime Litter Detection edge-AI application).