The achievements in each methodological objective (MO) and application objective (AO) of the project are explained as follows.
MO1 Improving information retrieval: There are three sub-objectives in MO1: 1) robust Earth observation image denoising, 2) solve ill-conditioned and underdetermined problems, and 3) image fusion. Regarding the first one, we have developed the so-called nonlocal-means filter to significantly improve the SNR of TanDEM-X SAR images. This smart filtering algorithm was integrated into our radar tomography algorithm that is used for global 3D building reconstruction (Shi et al., 2020a; Zhu et al., 2018). For the second sub-objective, we have achieved by developing a compressive sensing-based radar tomographic algorithm (Shi et al., 2020a) that is tailored to solve ill-conditioned and heavily underdetermined problems. For the third point, we developed multiple algorithms beyond the original objective of fusing images with different resolutions. Description can be found below in MO3.
MO2 Mining social media data: The three sub-objectives in MO2 are 1) find feature representation of social media images, 2) find efficient methods to update 3D building models, and 3) mine information from text messages and metadata. As a common prerequisite for the objectives, we developed stable processors to crawl Flickr images and tweets from the internet. Currently, we have collected more than 25 million social media images and 1.5 billion tweets. Despite this large quantity of social media images, they cover a broad variety of motifs, but only a small fraction of these images contain clear and useful information for individual buildings. This renders the first objective finding a unique representation of individual buildings in the social media images less relevant and the second objective not achievable, as there are not sufficient amounts of useful images for individual buildings. Hence, we shifted our research objective to developing novel algorithms to classifying building functionality (Hoffmann et al., 2019). For the third point, we use Twitter data and implemented natural language processing methods including self-trained multilingual Twitter word embeddings, multilingual large language models and deep learning architectures for classification (Häberle et al., 2022, 2019).
MO3 Optimal information fusion of heterogeneous data: in this MO, we tackle two types of data fusion challenges. One is the fusion of different types of EO images, and the other is the fusion of EO and social media data. Regarding the former one, we have specifically focused on the fusion of synthetic aperture radar data and multi-spectral imagery provided by the Sentinel-1 and Sentinel-2 missions. We developed a decision level fusion algorithm of SAR and optical images for better prediction of the so-called local climate zones classification (Zhu et al., 2022). For the latter, we developed a fusion algorithm of street view and aerial view images for building type classification and compared different types of fusion algorithms (Hoffmann et al., 2019). Last but not least, we also developed a fusion algorithm for social media text and images for building type classification, and results show that linguistic features can indeed improve the classification (Häberle et al., 2022).
MO4 Big data processing: In this MO, we have developed a processing workflow control framework for our data processing on the supercomputer SuperMUC-NG of LRZ, as well as machine learning model training and inference pipeline for global local climate zones classification. This framework is the basis for our global 3D reconstruction and classification processing.
AO1 Classification of global informal settlements: Towards the application objective of providing a detailed mapping of informal settlements, we have successfully carried out the first studies about deep learning techniques dedicated to this purpose (Stark et al., 2019; Wurm et al., 2019). We further developed a transfer learned fully convolutional Xception network (XFCN), trained on a conceptually consistently generated large sample of globally distributed slums. Transfer learning was implemented that helped to improve segmentation results when learning on a variety of slum morphologies, with high F1 scores of up to 89% (Stark et al., 2020). Furthermore, we used the classification results on the urban built environment in So2Sat to relate the physical appearance of the built urban landscape with multiple, intertwined socio-economic or planning processes (Stark et al., 2020).
AO2 Estimation of global population density: currently we have curated a large earth observation dataset called So2Sat POP for population estimation using machine learning (Doda et al., 2022). A follow-on work on population estimation using the model trained on the So2Sat POP dataset has been finished and submitted to a journal. Since the doctoral research started in 10.2019 the work of building level population density estimation is still ongoing.
