Periodic Reporting for period 2 - So2Sat (Big Data for 4D Global Urban Mapping – 10^16 Bytes from Social Media to EO Satellites)
Reporting period: 2018-11-01 to 2020-04-30
MO1 Improving information retrieval: There are three sub-objectives in MO1: 1) robust Earth observation image denoising, 2) solve ill-conditioned and underdetermined problem, and 3) image fusion. Regarding the first one, we have developed the so called nonlocal-means filter to significantly improve the SNR of TanDEM-X SAR images, which in turn improves the 3D reconstruction accuracy when using the filtered images. This smart filtering algorithm was integrated into our radar tomography algorithm that is used for global 3D building reconstruction (Shi et al., 2019). For the second sub-objective, we have also achieved it. We developed a compressive sensing-based radar tomographic algorithm (Shi et al., 2019) that is tailored to solve ill-conditioned and heavily underdetermined problem. The last sub-objective is on going.
MO2 Mining social media data: The three sub-objectives in MO2 are 1) find feature representation of social media images, 2) find efficient method to update 3D building models, and 3) mine information from text messages. Regarding the first the third points, we developed stable processors to crawl Flickr images and tweets from the internet. Currently, we have collected more than 25 million social media images and 1.5 billion tweets. Social media images from Flickr cover a broad variety of motifs, but only a small fraction of these images contains clear and useful information for individual buildings. Hence, for 2) we are developing novel algorithms to identify images helpful for the given tasks. To extract information from text messages we use Twitter data and implement natural language processing methods including word embeddings and deep learning models. By integrating the outcoming features in the spatial domain we create semantic information, e.g. building functions.
MO3 Optimal information fusion of heterogeneous data: in this MO, we tackle two types of data fusion challenges. One is the fusion of different types of EO images, and the other is the fusion of EO and social media data. Regarding the former one, we have specifically focused on the fusion of synthetic aperture radar (SAR) data and multi-spectral imagery provided by the Sentinel-1 and Sentinel-2 missions, respectively. This lead to an improvement of urban land cover classification accuracy over the utilization of a single sensor source. Regarding the latter, we finished a preliminary study of the fusion of street view and aerial view optical images for settlement type classification, and provided conclusion of the best fusion strategy for such highly heterogeneous data (Hoffmann et al., 2019).
MO4 Big data processing: In this MO, we have developed a processing workflow control framework for our data processing on the supercomputer SuperMUC-NG of LRZ. This framework is the basis for our global 3D reconstruction and classification processing. We also achieved significant progress in developing fast optimizer for 3D reconstruction using SAR tomography, which improved the computational speed by a factor of 20 (Shi et al., 2018); regarding classification of global EO data, we have developed efficient inferencing code on CPUs which is currently running on the supercomputer.