Periodic Reporting for period 3 - So2Sat (Big Data for 4D Global Urban Mapping – 10^16 Bytes from Social Media to EO Satellites)
Période du rapport: 2020-05-01 au 2021-10-31
MO1 Improving information retrieval: There are three sub-objectives in MO1: 1) robust Earth observation image denoising, 2) solve ill-conditioned and underdetermined problem, and 3) image fusion. Regarding the first one, we have developed the so called nonlocal-means filter to significantly improve the SNR of TanDEM-X SAR images, which in turn improves the 3D reconstruction accuracy when using the filtered images. This smart filtering algorithm was integrated into our radar tomography algorithm that is used for global 3D building reconstruction (Shi et al., 2019). For the second sub-objective, we have also achieved it. We developed a compressive sensing-based radar tomographic algorithm (Shi et al., 2019) that is tailored to solve ill-conditioned and heavily underdetermined problem. The last sub-objective is on going.
MO2 Mining social media data: The three sub-objectives in MO2 are 1) find feature representation of social media images, 2) find efficient method to update 3D building models, and 3) mine information from text messages. Regarding the first and the third points, we developed stable processors to crawl Flickr images and tweets from the internet. Currently, we have collected more than 25 million social media images and 1.5 billion tweets. Social media images from Flickr cover a broad variety of motifs, but only a small fraction of these images contains clear and useful information for individual buildings. Hence, for 2) we have developed an algorithms to identify images that is relevant for our building type classification task. To extract information from text messages we use Twitter data and implement natural language processing methods including word embeddings and deep learning models. By integrating the outcoming features in the spatial domain we create semantic information, e.g. building functions. The study of social media images and tweets have been done, and published in (Häberle et al. 2019, Kruspe et al. 2021), and are also being summarize in two phd dissertations.
MO3 Optimal information fusion of heterogeneous data: in this MO, we tackle two types of data fusion challenges. One is the fusion of different types of EO images, and the other is the fusion of EO and social media data. Regarding the former one, we have specifically focused on the fusion of synthetic aperture radar (SAR) data and multi-spectral imagery provided by the Sentinel-1 and Sentinel-2 missions, respectively. This lead to an improvement of urban land cover classification accuracy over the utilization of a single sensor source. The research has been done and the result was published in (Zhu et al. 2021). Regarding the latter, we finished a study of the fusion of street view and aerial view optical images for settlement type classification, and provided conclusion of the best fusion strategy for such highly heterogeneous data (Hoffmann et al., 2019). A study on the fusion of social media image and tweets was also done, and is being summarized in a phd dissertation. The study on the fusion of social media data and remote sensing data is ongoing.
MO4 Big data processing: In this MO, we have been constantly developing and improving our big data processing software on the supercomputer SuperMUC-NG of LRZ. This framework is the basis for our global 3D reconstruction and classification processing. We also achieved significant progress in developing fast optimizer for 3D reconstruction using SAR tomography, which improved the computational speed by a factor of 20 (Shi et al., 2018); regarding classification of global EO data, we have developed efficient inferencing code on CPUs which is currently running on the supercomputer.